model that labels pictures of horses correctly may be looking at the horse. Or it may be looking at a watermark that happens to sit on many horse photographs. On the test set the two look the same. Shortcut learning is the name for decision rules that work on the benchmark and fail on the world.

the question

Geirhos et al. (2020) gathered the cases and gave the problem its name: models trained to minimize error will use whatever features predict the labels, and if a simpler feature than the intended one does the job in the training data, they will find it. The question is how to tell, before deployment, which rule a system has learned.

the findings

Lapuschkin et al. (2019) used explanation methods to look inside image classifiers and found the horse case above: a well-performing model was relying in part on a source tag present on horse images in its dataset. They called such systems "Clever Hans predictors", after the horse on our oddities shelf.

Zech et al. (2018) trained models on chest radiographs from several hospital systems. The models could identify which hospital an image came from, and performance on pneumonia detection dropped on data from hospitals not seen in training, consistent with the model using site-specific cues.

In language, Gururangan et al. (2018) showed that natural-language inference datasets carried annotation artifacts: a classifier that saw only the hypothesis sentence, never the premise, did far better than chance. McCoy, Pavlick & Linzen (2019) built a test set to catch three such heuristics, such as assuming a hypothesis follows whenever its words all appear in the premise. Strong models failed badly on it. They named the set HANS, again after the horse.

the human precedent

The horse is the precedent. In 1907 Pfungst showed that Clever Hans answered arithmetic questions by reading tiny, involuntary movements of the questioner. When the questioner did not know the answer, Hans did not either. Psychology later found the same thing in people: experimenter expectancy and demand characteristics are shortcuts that participants and experimenters learn without trying.

what is not settled

Not every unexpected feature is a shortcut; sometimes the model has found a real signal people missed. And the explanation methods used to find shortcuts have their own failure modes. The cleanest test remains the one Pfungst used: change the conditions so the suspected cue is gone, and see whether performance survives.

live debates

Bugs or features? Ilyas et al. (2019) argued that adversarial examples exploit genuinely predictive but non-robust features in the data: what looks like a shortcut to us is signal to the model. Is robustness separate from accuracy? Recht et al. (2019) rebuilt an ImageNet test set and found accuracy dropped consistently; Taori et al. (2020) and Miller et al. (2021) found that, across many models, accuracy under natural shift is largely predicted by in-distribution accuracy, which leaves little "effective robustness" for most interventions to claim [paraphrase, not a quotation].

methodological critiques

Benchmarks that can be solved without the task. Poliak et al. (2018) and Gururangan et al. (2018) showed that inference datasets can be partly solved from the hypothesis alone. Gardner et al. (2020) proposed contrast sets, small expert edits that change the label, to test local decision boundaries; WILDS and Hendrycks & Dietterich (2019) built shift and corruption benchmarks. Each benchmark tests the shortcuts its builders thought of.

the open problem

For large models trained on most of the written record, it is hard to know what was in the training data, and so hard to know which test items can be solved by recall or surface pattern. Building evaluations that only the intended skill can pass is the central difficulty of measuring machine behavior.

Reading list

1. Geirhos et al. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence. doi:10.1038/s42256-020-00257-z
2. Lapuschkin et al. (2019). Unmasking Clever Hans predictors and assessing what machines really learn. Nature Communications. doi:10.1038/s41467-019-08987-4
3. Zech et al. (2018). Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study. PLOS Medicine. doi:10.1371/journal.pmed.1002683
4. Gururangan et al. (2018). Annotation Artifacts in Natural Language Inference Data. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). doi:10.18653/v1/N18-2017
5. McCoy, Pavlick & Linzen (2019). Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. doi:10.18653/v1/P19-1334
6. Pfungst (1911). Clever Hans (the Horse of Mr. von Osten), trans. C. L. Rahn. New York: Henry Holt. German original 1907. Free edition at Project Gutenberg. www.gutenberg.org/ebooks/33936
7. Ilyas et al. (2019). Adversarial Examples Are Not Bugs, They Are Features. arXiv:1905.02175. arxiv.org/abs/1905.02175
8. Recht et al. (2019). Do ImageNet Classifiers Generalize to ImageNet?. arXiv:1902.10811. arxiv.org/abs/1902.10811
9. Taori et al. (2020). Measuring Robustness to Natural Distribution Shifts in Image Classification. arXiv:2007.00644. arxiv.org/abs/2007.00644
10. Miller et al. (2021). Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization. arXiv:2107.04649. arxiv.org/abs/2107.04649
11. Poliak et al. (2018). Hypothesis Only Baselines in Natural Language Inference. Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics. doi:10.18653/v1/S18-2023
12. Gardner et al. (2020). Evaluating Models' Local Decision Boundaries via Contrast Sets. arXiv:2004.02709. arxiv.org/abs/2004.02709
13. Koh et al. (2020). WILDS: A Benchmark of in-the-Wild Distribution Shifts. arXiv:2012.07421. arxiv.org/abs/2012.07421
14. Hendrycks & Dietterich (2019). Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. arXiv:1903.12261. arxiv.org/abs/1903.12261

see also

	Clever Hans — the horse, the commission, and the student who found the cue
	archive: Pfungst, 1907 — the report itself, in a free edition
	glossary: the Clever Hans effect — with shortcut learning and demand characteristics
	replication — the human-subjects version of a shortcut