Machine Methods for Generalisation

Statistical generalisation methods form the foundation of many modern machine learning techniques, especially deep learning. These methods infer a model from observational data, treating training data as examples from which a broader pattern or distribution is derived. The idea is to enable the model to apply its learnt patterns to new data that belong to the same underlying distribution. This approach is often implemented via empirical risk minimisation where the prediction error is minimized across the training data, with additional regularisation terms added to promote better generalisation. Despite their capacity to handle high data complexity and large-scale data, statistical methods sometimes struggle with out-of-distribution generalisation because they primarily rely on correlations instead of underlying causal relationships. The text clearly explains that while these methods excel at inference correctness and efficiency, they typically require vast amounts of data and tend to work best when the test data are similar to the training data^[1].

Knowledge-informed methods take a different approach by incorporating existing domain theories or explicit background knowledge into the learning process. These methods aim to find empirical evidence of a theory, resulting in a model that is not just statistically powerful but also meaningful from a semantic perspective. Examples mentioned include mechanistic models, causal models, and functional programs. By explicitly grounding the model with semantic information, these methods allow for inspectable parameters and provide explanations for how generalisation occurs. However, because these approaches require formalising a theory, they are typically more computationally demanding and are often restricted to simpler or well-defined scenarios. In summary, knowledge-informed methods support compositionality and contribute to explainable predictions, aligning the model's output more closely with human expectations^[1].

Instance-based methods, also known as lazy learning or non-parametric techniques, focus on using individual observed examples to perform local inference at the time of prediction. Techniques such as nearest-neighbour algorithms and case-based reasoning are central to this approach. These methods do not create a global model that represents the entire data distribution. Instead, they memorize individual instances and use them to handle new observations by identifying similarity to the stored cases. This allows for flexibility in adapting to new or shifting data distributions, as well as robustness to noise. The performance of these methods is strongly dependent on the quality of the chosen representation, as the similarity between cases is key to successful generalisation. Instance-based approaches are noted for their ability to learn from a few examples and for providing an interpretable link between training instances and outcomes, though their overall effectiveness is tied to the specific context and representation used^[1].

A critical point raised in the text is the alignment between machine generalisation approaches and human generalisation abilities. Humans tend to perform generalisation by forming sparse abstractions and conceptual representations that are composable and can be transferred across domains. In contrast, statistical methods often result in probabilistic patterns that may not capture the nuances of human abstraction. To better align machine outcomes with human reasoning, recent research suggests combining the strengths of different methods. Neurosymbolic approaches, for instance, integrate the data-driven power of statistical methods with the clarity and compositionality of knowledge-based models. Additionally, incorporating instance-based methods offers an alternative by focusing on individual data translations. This multifaceted approach is essential for developing systems that facilitate effective human-AI teaming by making machine outputs more understandable and aligned with human expectations^[1].

In conclusion, the main machine methods for generalisation are statistical, knowledge-informed, and instance-based techniques. Statistical methods provide robust performance with large datasets through empirical risk minimisation and extensive data modelling. Knowledge-informed methods bring explicit semantic grounding and support explainable predictions by incorporating domain knowledge. Instance-based methods leverage memorised examples for local inference, offering flexibility in dealing with novel and out-of-distribution scenarios. The integration and alignment of these diverse approaches are key to improving the performance of AI systems in ways that closely mirror human generalisation processes^[1].