hen a network gets an answer right, what did it use? Interpretability is the research that tries to say, in terms a person can check. It matters most when the answer is right for the wrong reason, which is the subject of shortcut learning.

the question

A trained network is a very large set of numbers that nobody wrote by hand. Its designers know the architecture and the training data, but not which features of an input end up carrying a decision. Lipton (2018) pointed out that "interpretable" was being used for several different wishes at once: to see how the model works inside (transparency), or to get some account after the fact of why it gave one output (post-hoc explanation), and that these need not coincide [paraphrase, not a quotation]. Doshi-Velez & Kim (2017) argued for testing explanations the way any method is tested: with real tasks and users, with simplified tasks and people, or with proxies, and saying which one was done [paraphrase, not a quotation].

the classic findings

Early work made pictures. Zeiler & Fergus (2014) ran a convolutional network backwards to show which input patterns drove units in each layer: edges near the bottom, textures and object parts higher up. Bau and colleagues (2017) scored individual units against a labelled set of concepts and found many that lined up with a colour, texture, part or object. Ribeiro, Singh & Guestrin (2016) took the opposite route with LIME: leave the model closed, perturb one input many times, and fit a small simple model that imitates the big one near that input.

Then came the checks. Adebayo and colleagues (2018) randomized a trained model's weights and looked again at popular saliency maps. Some barely changed. A map that looks the same for a trained network and a scrambled one is showing you the image, not the model [paraphrase, not a quotation]. On the useful side, Lapuschkin and colleagues (2019) used relevance maps to catch classifiers leaning on things that were not the object: one model had learned to recognize horses partly by a source tag in the corner of the photographs. Clever Hans in a new body, found by looking.

Superposition. Elhage and colleagues (2022) built small models where the true features were known, and showed that a network can store more features than it has dimensions by giving them overlapping directions, as long as each feature is rarely active [paraphrase, not a quotation]. If that holds in large models, one unit will often mean several things, and looking for "the cat neuron" is the wrong search. See superposition.

what held up, and what did not

What held up: features are found, sometimes cleanly, and some explanation methods do catch real shortcuts. What did not: the assumption that a pleasing picture is a faithful one. Faithfulness has to be tested, by changing the thing an explanation points at and seeing whether the output changes.

the human side

People have the same problem about themselves. Nisbett & Wilson (1977) reviewed experiments in which participants gave confident reasons for choices that the experimenters knew had other causes, and argued that such reports often come from a theory of why people act, not from looking inward [paraphrase, not a quotation]. A language model's written reasoning has been put to a similar test: Turpin and colleagues (2023) biased models toward an answer with features of the prompt, and the step-by-step explanations rarely mentioned the bias that had moved them [paraphrase, not a quotation]. See hallucination and confabulation.

live debates

Faithful or merely plausible? Jacovi & Goldberg (2020) separate two properties that the field had run together. An explanation is plausible if people find it convincing, and faithful if it accurately reflects the process by which the model reached its output. Their argument is that the two must be evaluated separately, that human judgments measure only plausibility, and that faithfulness should be treated as graded rather than all-or-nothing [paraphrase, not a quotation]. The distinction now organises much of the debate: a fluent chain of thought can be highly plausible and still unfaithful, which is what Turpin et al. (2023) demonstrated by biasing the inputs and showing that the stated reasoning did not mention the bias.

Is attention an explanation? Jain & Wallace (2019) reported that attention weights in recurrent text classifiers often correlate weakly with gradient- and leave-one-out measures of feature importance, and that quite different attention distributions could be substituted without changing the prediction. Wiegreffe & Pinter (2019) replied that the conclusion depends on what "explanation" is taken to mean: an adversarial distribution found separately for each example does not show that a model trained with it would behave the same, and they proposed stricter tests, including training an adversarial model end to end and comparing against uniform weights [paraphrase, not a quotation]. The exchange is still cited on both sides, because each paper is right about the claim the other one did not make.

Superposition, and the limits of dictionary features. If a network stores more features than it has neurons, in superposition (Elhage et al., 2022), single neurons will not be the right unit of analysis. Sparse autoencoders try to recover the features: Cunningham et al. (2023) and Bricken et al. (2023) found many directions that humans could label, and Templeton et al. (2024) scaled the method to a production model. Marks et al. (2024) used such features as nodes in causal circuits. The doubts are specific. Chanin et al. (2024) describe feature absorption: a latent that seems to track a general concept fails to fire where a more specific latent has taken over, so the feature is less reliable than its label suggests. Engels et al. (2024) found features that are irreducibly multi-dimensional, such as circular representations of days and months, which a one-direction-per-feature picture cannot hold. And Kantamneni et al. (2025), asking whether SAE features make better probes, found that they did not reliably beat simple baselines on the tasks tested [paraphrase, not a quotation]. The open question is whether SAE latents are the model's units or an artefact of the sparsity objective and dictionary size.

Explain, or replace? Rudin (2019) argues that for high-stakes decisions, post-hoc explanations of black boxes are the wrong goal; where an inherently interpretable model performs comparably, it should be used instead, because an explanation that is not the model is, at best, an approximation of it [paraphrase, not a quotation]. Mechanistic interpretability takes the opposite bet: that the large models will be used anyway, so they must be opened.

methodological critiques

What a probe shows. A probe is a small classifier trained to read some property out of a model's activations. Hewitt & Liang (2019) showed that a sufficiently expressive probe can score well on a control task with random labels, so high accuracy may reflect what the probe learned rather than what the representation encodes; they propose reporting selectivity, the gap between real and control accuracy. Belinkov (2022)'s review sets out the deeper problem: decodability is correlational. That a property can be read out does not show the model uses it. Ravichander et al. (2021) found that probes recover properties the task does not need, and Elazar et al. (2021) proposed amnesic probing: remove the property from the representation and measure whether behaviour changes [paraphrase, not a quotation].

Interventions have their own illusions. Causal methods were meant to fix the probing problem, and they bring new ones. Meng et al. (2022) located factual associations by causal tracing and edited them; Hase et al. (2023) then found that where causal tracing localises a fact barely predicts which layer is best to edit. Makelov et al. (2023) showed that patching a subspace can change behaviour by activating a dormant pathway, so a subspace can look causally responsible for a concept it does not represent in normal use. Bolukbasi et al. (2021) found that what a single unit appears to encode in BERT depends on the dataset used to inspect it. For images, Adebayo et al. (2018) showed that some saliency methods give similar maps after the model's weights are randomised.

What counts as a mechanism? Circuit papers such as Wang et al. (2022) on indirect-object identification report criteria of faithfulness, completeness and minimality, and Conmy et al. (2023) automate circuit discovery; both show that conclusions depend on choices such as how ablated activations are replaced. Geiger et al. (2021) and Geiger et al. (2023) propose a common standard: a mechanistic hypothesis is a high-level causal model, and it is supported only to the degree that interchange interventions on the network produce the counterfactual behaviour the high-level model predicts [paraphrase, not a quotation]. This gives causal abstraction a precise meaning, and it also makes clear how partial most published circuits are.

Do explanations help people? The human-subjects record is sobering. Poursabzi-Sangdeh et al. (2021) found that a more transparent model did not help participants detect its mistakes and could overload them, and Buçinca et al. (2020) found that proxy tasks and subjective ratings did not predict performance on the actual decision task. An explanation method validated only on proxies has not been validated.

still open

Whether the methods that work in small models scale; whether any explanation can be both faithful and short enough for a person to use; and whether users given explanations actually decide better, which is a question for mental models and complementarity.

in the tour: the second room of machines has a placard called "More features than directions", and another on reading the stream too early ···

Reading list

1. Lipton (2018). The Mythos of Model Interpretability. Queue. doi:10.1145/3236386.3241340
2. Doshi-Velez & Kim (2017). Towards a rigorous science of interpretable machine learning. arXiv:1702.08608. arxiv.org/abs/1702.08608
3. Zeiler & Fergus (2014). Visualizing and Understanding Convolutional Networks. Computer Vision – ECCV 2014 (Lecture Notes in Computer Science). doi:10.1007/978-3-319-10590-1_53
4. Bau et al. (2017). Network Dissection: Quantifying Interpretability of Deep Visual Representations. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). doi:10.1109/cvpr.2017.354
5. Ribeiro, Singh & Guestrin (2016). "Why should I trust you?": Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. doi:10.1145/2939672.2939778
6. Adebayo et al. (2018). Sanity checks for saliency maps. arXiv:1810.03292 (NeurIPS 2018). arxiv.org/abs/1810.03292
7. Lapuschkin et al. (2019). Unmasking Clever Hans predictors and assessing what machines really learn. Nature Communications. doi:10.1038/s41467-019-08987-4
8. Elhage et al. (2022). Toy models of superposition. Transformer Circuits Thread; also arXiv:2209.10652. transformer-circuits.pub/2022/toy_model/index.html
9. Nisbett & Wilson (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review. doi:10.1037/0033-295x.84.3.231
10. Turpin, Michael, Perez & Bowman (2023). Language models don't always say what they think: unfaithful explanations in chain-of-thought prompting. arXiv:2305.04388 (NeurIPS 2023). arxiv.org/abs/2305.04388
11. Jacovi & Goldberg (2020). Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness?. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. doi:10.18653/v1/2020.acl-main.386
12. Jain & Wallace (2019). Attention is not Explanation. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). doi:10.18653/v1/N19-1357
13. Wiegreffe & Pinter (2019). Attention is not not Explanation. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). doi:10.18653/v1/D19-1002
14. Cunningham et al. (2023). Sparse Autoencoders Find Highly Interpretable Features in Language Models. arXiv:2309.08600. arxiv.org/abs/2309.08600
15. Bricken et al. (2023). Towards monosemanticity: decomposing language models with dictionary learning. Transformer Circuits Thread. transformer-circuits.pub/2023/monosemantic-features/index.html
16. Templeton et al. (2024). Scaling monosemanticity: extracting interpretable features from Claude 3 Sonnet. Transformer Circuits Thread. transformer-circuits.pub/2024/scaling-monosemanticity/index.html
17. Marks et al. (2024). Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models. arXiv:2403.19647. arxiv.org/abs/2403.19647
18. Chanin et al. (2024). A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders. arXiv:2409.14507. arxiv.org/abs/2409.14507
19. Engels et al. (2024). Not All Language Model Features Are One-Dimensionally Linear. arXiv:2405.14860. arxiv.org/abs/2405.14860
20. Kantamneni et al. (2025). Are Sparse Autoencoders Useful? A Case Study in Sparse Probing. arXiv:2502.16681. arxiv.org/abs/2502.16681
21. Rudin (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence. doi:10.1038/s42256-019-0048-x
22. Hewitt & Liang (2019). Designing and Interpreting Probes with Control Tasks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). doi:10.18653/v1/D19-1275
23. Belinkov (2022). Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguistics. doi:10.1162/coli_a_00422
24. Ravichander, Belinkov & Hovy (2021). Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. doi:10.18653/v1/2021.eacl-main.295
25. Elazar et al. (2021). Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals. Transactions of the Association for Computational Linguistics. doi:10.1162/tacl_a_00359
26. Meng et al. (2022). Locating and Editing Factual Associations in GPT. arXiv:2202.05262. arxiv.org/abs/2202.05262
27. Hase et al. (2023). Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models. arXiv:2301.04213. arxiv.org/abs/2301.04213
28. Makelov, Lange & Nanda (2023). Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching. arXiv:2311.17030. arxiv.org/abs/2311.17030
29. Bolukbasi et al. (2021). An Interpretability Illusion for BERT. arXiv:2104.07143. arxiv.org/abs/2104.07143
30. Wang et al. (2022). Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. arXiv:2211.00593. arxiv.org/abs/2211.00593
31. Conmy et al. (2023). Towards Automated Circuit Discovery for Mechanistic Interpretability. arXiv:2304.14997. arxiv.org/abs/2304.14997
32. Geiger et al. (2021). Causal Abstractions of Neural Networks. arXiv:2106.02997. arxiv.org/abs/2106.02997
33. Geiger et al. (2023). Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability. arXiv:2301.04709. arxiv.org/abs/2301.04709
34. Poursabzi-Sangdeh et al. (2021). Manipulating and Measuring Model Interpretability. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. doi:10.1145/3411764.3445315
35. Buçinca et al. (2020). Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems. Proceedings of the 25th International Conference on Intelligent User Interfaces. doi:10.1145/3377325.3377498

see also

	shortcut learning — what interpretability is often hunting
	mental models — what people believe about the machine
	annotated: machine behaviour — the shelf these papers sit on
	Clever Hans — the first case of a right answer for a wrong reason
	brains and models compared — the same question asked of brains
	connectomics — the full wiring, and whether it is enough