Evaluating AI Generalisation in Human-AI Teams

The evaluation of AI generalisation is discussed using a variety of metrics and methods. One method involves measuring distributional shifts between training and test data. The source states that such shifts can be estimated using statistical distance measures, for example, the Kullback-Leibler divergence or the Wasserstein distance. Generative models provide an explicit likelihood estimate p(x) to indicate how typical a sample is of the training distribution. For discriminative models, proxy techniques include the calculation of cosine similarity between embedding vectors and nearest-neighbour distances using transformed feature spaces. In the case of large language models (LLMs), measuring perplexity is a common proxy to assess generalisation in terms of internal representations^[1].

A major part of evaluating AI generalisation involves the concepts of undergeneralisation and overgeneralisation. The text notes that undergeneralisation occurs when a slight change in the input—whether perceptible or not—leads to a significant alteration in model output. This may occur when an AI model fails to account for small variations such as camera or environmental perturbations, leading to a notable degradation in performance. On the other hand, overgeneralisation is described as the model making overconfident errors in its predictions, such as generating hallucinations or biased predictions where the model ignores critical differences in input features. The evaluation framework must therefore include tests for both types of errors, examining tasks with adversarial, counterfactual, or naturally shifted examples to assess robustness^[1].

A challenge in the evaluation process is to distinguish between memorisation and genuine generalisation. Memorisation refers to the model learning specific details from the training data, which may sometimes be beneficial (for instance, remembering factual knowledge) but can also lead to errors when the model is expected to generalise. For certain tasks such as factual question answering or legal reasoning, memorisation might be appropriate; however, for tasks that require adaptive reasoning, generalisation beyond the learnt examples is essential. The source suggests that evaluations should therefore include methods that separately assess the model’s ability to memorise and to generalise, ensuring the correct balance relative to the task context^[1].

In the context of human-AI teams, it is not enough for an AI to generalise well statistically; its outputs must also be aligned with human cognitive models and expectations. The text emphasizes that effective alignment requires a principled evaluation of both objective task outcomes and subjective process-related experiences. Evaluations should not only measure objective metrics such as prediction accuracy and robustness to noise and shifts, but also involve human studies that test the explainability and interpretability of the AI’s decisions. The human-centric approach recommends that benchmarks and evaluation methods should mirror the kind of real-world variations and anomalies encountered during human decision making, thereby ensuring that the model’s generalisation behavior is compatible with human reasoning^[1].

Although various strategies exist for evaluating generalisation, several challenges remain. One key issue is the contamination of test data with training examples, especially in foundation models, which can lead to overestimated performance metrics. Additionally, the standard evaluation setup typically assumes that training and test data are independent and identically distributed (IID). However, in real-world human-AI teaming scenarios, this IID assumption is often violated due to natural distributional shifts and context changes. The source therefore calls for more sophisticated benchmarks that take into account contextual dependencies, multimodal datasets, and varied real-world conditions. Another promising direction mentioned is the integration of neurosymbolic approaches, where the explainability inherent in symbolic methods is combined with the powerful approximation capabilities of statistical models. Future research will have to develop methods that not only generate guarantees and bounds regarding robustness but that also account for the nuances of human feedback during continuous interaction^[1].

For teams that combine human and AI capabilities, it is essential that both parties' strengths are utilised and that any misalignments are promptly detected and corrected. The source explains that when discrepancies occur between AI predictions and human decisions—such as differences in diagnosis or classification—mechanisms for realignment and error correction must be established. This involves designing collaborative workflows where explanations of AI decisions are accessible and comprehensible to human users. The evaluation framework, therefore, should include tests for not only the statistical performance of the model, but also its ability to provide transparent and explainable outputs that support real-time human feedback. Such systems would allow iterative improvements and adaptations, ultimately leading to more effective and trustworthy human-AI collaborations^[1].

In summary, evaluating AI generalisation in human-AI teams requires a comprehensive framework that addresses various dimensions including statistical robustness, handling of distributional shifts, and clear differentiation between memorisation and genuine adaptive generalisation. Essential metrics include statistical distance measurements, adversarial and counterfactual tests, and human-centric evaluations focusing on explainability and process alignment. The dynamic nature of human-AI interaction—as well as the challenges posed by real-world variability—necessitates advanced evaluation benchmarks that incorporate contextual and multimodal data. Future research may benefit from neurosymbolic approaches which promise to bridge the gap between data-driven inference and human-like compositional reasoning. Ultimately, a well-rounded evaluation strategy is fundamental for ensuring that AI systems generalise in a manner that is both technically robust and aligned with human decision-making processes^[1].