veryone who uses a machine carries a small theory of it: what it can do, when it fails, what it will do next. Human factors calls that theory a mental model, and much of human–AI interaction is the study of how good those models are.

the classic idea

Norman (1983), writing about calculators and other everyday devices, observed that people's models of machines are incomplete, unstable, superstitious in places and kept deliberately simple, and that this is normal, not a failure of the users [paraphrase, not a quotation]. Design, on this view, is the job of making the right model easy to form.

in human–AI teams

Bansal and colleagues (2019) asked what part of the mental model matters most for a person working with an AI advisor. Their answer was the error boundary: knowing when the AI is likely to be wrong, so you know when to override it. Boundaries that were simple and consistent were learned better, and the team did better [paraphrase, not a quotation]. This is why a slightly less accurate model with predictable mistakes can make a better partner.

Explanations are one way to build the model. Kulesza and colleagues (2013) varied how complete and how sound explanations of a recommender were, and found completeness mattered more than soundness: fuller explanations built better mental models, and people judged them worth the effort, while heavily oversimplified ones cost trust [paraphrase, not a quotation]. Amershi and colleagues (2019) distilled guidelines for human–AI interaction, several of which are about exactly this: make clear what the system can do, and how well it does it.

When the model is "there is no machine". Eslami and colleagues (2015) interviewed Facebook users about their news feed. More than half did not know that an algorithm chose what they saw, and some had read the absence of a friend's posts as the friend drifting away [paraphrase, not a quotation]. A mental model can be wrong about whether there is anything to model.

live debates

Do explanations reduce overreliance? Results disagree, and Vasconcelos et al. (2023) offer a reconciliation: people engage with an explanation when the cost of checking it is lower than the benefit of being right, so explanations reduce overreliance when they make verification easier, and not otherwise [paraphrase, not a quotation]. Buçinca et al. (2021) found that cognitive forcing designs reduced overreliance, at the cost of being liked less. Bansal et al. (2019) framed the useful mental model as knowing the system's error boundary, where it tends to fail.

methodological critiques

Can we see the model at all? Rouse & Morris (1986) reviewed the concept and found it used loosely, with no agreed way to elicit a mental model without changing it; verbal reports, drawings and predictions each capture something different [paraphrase, not a quotation]. The human–AI studies inherit the problem. Buçinca et al. (2020) showed that proxy tasks (predict what the model will say) and self-rated understanding did not predict real decision performance, and the survey by Lai et al. (2023) found the field's studies vary so much in task, participants and measures that results rarely compare. Poursabzi-Sangdeh et al. (2021) is a caution in the other direction: more information about the model did not produce better use of it.

still open

How to give people an accurate model of a system that changes with each update; whether explanations help ordinary users decide better or mainly make them feel better; and how mental models interact with anthropomorphism, which supplies a ready-made model, of a person, that is usually the wrong one.

Reading list

1. Norman (1983). Some observations on mental models. In D. Gentner & A. L. Stevens (eds), Mental Models. Hillsdale, NJ: Lawrence Erlbaum.
2. Bansal et al. (2019). Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. doi:10.1609/hcomp.v7i1.5285
3. Kulesza et al. (2013). Too much, too little, or just right? Ways explanations impact end users' mental models. 2013 IEEE Symposium on Visual Languages and Human Centric Computing. doi:10.1109/vlhcc.2013.6645235
4. Amershi et al. (2019). Guidelines for Human-AI Interaction. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. doi:10.1145/3290605.3300233
5. Eslami et al. (2015). "I always assumed that I wasn't really that close to [her]": Reasoning about invisible algorithms in news feeds. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. doi:10.1145/2702123.2702556
6. Vasconcelos et al. (2023). Explanations Can Reduce Overreliance on AI Systems During Decision-Making. Proceedings of the ACM on Human-Computer Interaction. doi:10.1145/3579605
7. Buçinca, Malaya & Gajos (2021). To Trust or to Think. Proceedings of the ACM on Human-Computer Interaction. doi:10.1145/3449287
8. Rouse & Morris (1986). On looking into the black box: Prospects and limits in the search for mental models. Psychological Bulletin. doi:10.1037/0033-2909.100.3.349
9. Buçinca et al. (2020). Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems. Proceedings of the 25th International Conference on Intelligent User Interfaces. doi:10.1145/3377325.3377498
10. Lai et al. (2023). Towards a Science of Human-AI Decision Making: An Overview of Design Space in Empirical Human-Subject Studies. 2023 ACM Conference on Fairness Accountability and Transparency. doi:10.1145/3593013.3594087
11. Poursabzi-Sangdeh et al. (2021). Manipulating and Measuring Model Interpretability. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. doi:10.1145/3411764.3445315

see also

	complementarity — what a good mental model is for
	anthropomorphism — the default model, and its problems
	interpretability — explanations, seen from the machine side
	algorithm aversion — what a bad experience does to the model