ilots, nurses and radiologists work alongside automated aids that are usually right. "Usually" is the problem. Automation bias is the tendency to let the aid's output stand in for your own vigilance and judgment.

the question

When should a person rely on an automated system, and do they rely on it when they should? Parasuraman & Riley (1997) gave the field its vocabulary: use (relying on automation), misuse (relying on it when you should not), disuse (neglecting it when it would help), and abuse (automating functions without regard for the effect on the people who remain).

the classic findings

Skitka, Mosier & Burdick (1999) had participants monitor a simulated flight task with and without an automated aid. With the aid, people made two kinds of errors: omission errors, missing events the aid did not flag, and commission errors, following the aid's advice even when other available information contradicted it. Goddard, Roudsari & Wyatt (2012) reviewed the healthcare literature and found automation bias across clinical decision support, more likely with high workload, task complexity and inexperience.

The deeper analysis is older. Bainbridge (1983) set out the "ironies of automation": the designer automates the easy parts and leaves the person the hardest ones, and asks them to monitor a system for rare failures, a task people are poor at, while their manual skills fade from disuse. Lee & See (2004) argued that the goal is not more trust or less trust but calibrated trust: trust that matches the system's actual capability, specific to the situations where it is and is not reliable.

aversion and appreciation

Reliance can fail in the other direction too. Dietvorst, Simmons & Massey (2015) found algorithm aversion: after seeing a forecasting algorithm make mistakes, people chose a human forecaster instead, even when the algorithm was more accurate overall. But Logg, Minson & Moore (2019) found the opposite default in many settings: lay people gave more weight to advice labelled as coming from an algorithm than from a person. Experienced forecasters, by contrast, discounted advice of either kind. Which one you see depends on the task, the person, and whether they have watched the system fail.

in human–AI work now

Generative models change the shape of the problem. Their output is fluent whether or not it is correct, and a fluent answer invites commission errors. Vaccaro, Almaatouq & Malone (2024) meta-analysed experiments that compared people, AI systems, and the two combined, and found that on average the combination did worse than the better of the two alone, especially in decision tasks. Automation bias is one plausible reason; see complementarity.

live debates

One phenomenon or two? Parasuraman & Manzey (2010) argue that complacency (monitoring automation too little) and automation bias (acting on its cues without checking) share an attentional basis and are best treated together [paraphrase, not a quotation]. Wickens et al. (2015) tested both with imperfect automation. Dzindolet et al. (2003) showed that trust and reliance depend on what people know about why automation errs. Lyell & Coiera (2017) reviewed clinical evidence and found automation bias greater when verifying the advice was harder.

Is human oversight a safeguard? Bainbridge (1983) noted the irony that automation leaves people the hardest tasks while taking away their practice. Green (2022) argues that policies requiring a "human in the loop" for government algorithms rely on oversight people cannot reliably provide, and can lend unearned legitimacy.

methodological critiques

To call monitoring "complacent" one needs a normative rate of checking; without it, low checking of reliable automation may be rational. Lab studies concentrate automation failures to get enough data, and the first failure after long reliability behaves differently from later ones, so rates do not transfer simply to operations.

the open problem

Telling a person "be careful" does little. What helps is still being worked out: showing confidence, requiring a first guess before revealing the aid's answer, and designing systems whose failures are visible. Each helps in some studies and not others.

In the tour there is a room of machines that answer, with a kiosk that is not a guide. This page is the long version of that wall label. The plain tour lists it.

Reading list

1. Parasuraman & Riley (1997). Humans and Automation: Use, Misuse, Disuse, Abuse. Human Factors: The Journal of the Human Factors and Ergonomics Society. doi:10.1518/001872097778543886
2. Skitka, Mosier & Burdick (1999). Does automation bias decision-making?. International Journal of Human-Computer Studies. doi:10.1006/ijhc.1999.0252
3. Goddard, Roudsari & Wyatt (2012). Automation bias: a systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association. doi:10.1136/amiajnl-2011-000089
4. Bainbridge (1983). Ironies of automation. Automatica. doi:10.1016/0005-1098(83)90046-8
5. Lee & See (2004). Trust in Automation: Designing for Appropriate Reliance. Human Factors: The Journal of the Human Factors and Ergonomics Society. doi:10.1518/hfes.46.1.50_30392
6. Dietvorst, Simmons & Massey (2015). Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General. doi:10.1037/xge0000033
7. Logg, Minson & Moore (2019). Algorithm appreciation: People prefer algorithmic to human judgment. Organizational Behavior and Human Decision Processes. doi:10.1016/j.obhdp.2018.12.005
8. Vaccaro, Almaatouq & Malone (2024). When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human Behaviour. doi:10.1038/s41562-024-02024-1
9. Parasuraman & Manzey (2010). Complacency and Bias in Human Use of Automation: An Attentional Integration. Human Factors: The Journal of the Human Factors and Ergonomics Society. doi:10.1177/0018720810376055
10. Wickens et al. (2015). Complacency and Automation Bias in the Use of Imperfect Automation. Human Factors: The Journal of the Human Factors and Ergonomics Society. doi:10.1177/0018720815581940
11. Dzindolet et al. (2003). The role of trust in automation reliance. International Journal of Human-Computer Studies. doi:10.1016/S1071-5819(03)00038-7
12. Lyell & Coiera (2017). Automation bias and verification complexity: a systematic review. Journal of the American Medical Informatics Association. doi:10.1093/jamia/ocw105
13. Green (2022). The flaws of policies requiring human oversight of government algorithms. Computer Law & Security Review. doi:10.1016/j.clsr.2022.105681

Try it yourself. [playful] Small toys on the toy shelf that show this in action:   • spot the change — change blindness, with a blink

see also

	complementarity — when the team should beat either member
	annotated: human–AI interaction — Licklider to the present
	glossary: automation bias — with algorithm aversion
	calibration — trust needs something to calibrate against
	J. C. R. Licklider — who wanted partnership, not replacement