e learn most from what surprises us. That sentence is folk wisdom, but it is also the core of the most successful formal account of learning in psychology, and, decades later, of a famous result about dopamine neurons.

the question

Pairing a tone with food teaches an animal to expect food when it hears the tone. But pairing alone is not enough. If a light already predicts the food, adding a tone alongside it teaches almost nothing about the tone. This is blocking, and it says learning depends on whether the outcome was already predicted, not merely on what came together.

the classic finding

Rescorla & Wagner (1972) wrote the rule down. On each trial, every cue present changes its associative strength in proportion to the prediction error: the outcome that happened minus the outcome that all the present cues together predicted. When the outcome is fully predicted the error is zero and nothing changes, which is blocking falling straight out of the arithmetic.

Sutton (1988) generalized the idea from trials to moments. In temporal-difference learning, the system compares its prediction now with its prediction one step later plus any reward received in between, and learns from that difference. Then Schultz, Dayan & Montague (1997) showed that dopamine neurons in the monkey midbrain behave much like that error signal. Early in training they fire when juice arrives. After training they fire at the cue that predicts the juice, not at the juice. If the predicted juice is withheld, their firing dips below baseline at the moment it should have come.

what held up, and what did not

The core pattern, response transfer to the predictive cue and a dip on omission, has been seen many times and in several species. The broader claim, that dopamine is only a reward prediction error, has not held. Dopamine neurons also respond to novelty and salience, and different populations behave differently. The Rescorla–Wagner rule itself famously fails on some cases, such as the recovery of a blocked cue after the blocking cue is extinguished, which led to many successor models.

At a larger scale, Rao & Ballard (1999) proposed that the visual cortex runs on prediction errors too: higher areas predict lower-level activity and pass down predictions, and only the unexplained remainder is sent up. Friston (2010) pushed the idea toward a single principle for the whole brain. These are ambitious frameworks rather than settled findings, and the evidence for specific predictive-coding circuitry is still debated.

in machines

Temporal-difference learning is not only a model of animals; it is a working engineering method, the ancestor of much of modern reinforcement learning. Language models are trained to reduce a different but related error: the gap between the next word they predicted and the next word that came. The word "prediction" is shared; whether anything like dopamine's teaching signal is shared is a separate and open question.

live debates

What does dopamine signal? The reward-prediction-error account (Schultz, Dayan & Montague, 1997) has rivals. Berridge (2007) argues dopamine mediates incentive salience, the "wanting" of a reward, rather than learning from it. Mohebi et al. (2019) found dopamine release in the nucleus accumbens tracking motivational value with no matching change in dopamine cell firing, so release and firing can come apart. Jeong et al. (2022) proposed that dopamine conveys retrospective causal associations rather than a forward prediction error, and Sharpe et al. (2017) showed dopamine transients can drive learning of associations with no value at all. Defenders have extended the account: Kim et al. (2020) explained dopamine ramps as prediction errors under changing state, and Dabney et al. (2020) found neurons with different optimism, consistent with a distributional code for value [paraphrase, not a quotation].

Predictive processing as a theory of everything. Litwin & Miłkowski (2020) argue that predictive processing achieves its unity "by fiat": the framework is flexible enough to describe nearly any result, so its generality is not evidence for it. Colombo & Wright (2017) make a related case for explanatory pluralism against the free-energy programme (Friston, 2010) [paraphrase, not a quotation].

methodological critiques

Most of the human evidence is correlational: a regressor shaped like a prediction error correlates with a BOLD signal. That reverses the inference, and other quantities (surprise, salience, value) are often correlated with prediction error in the same task. The animal work above is persuasive because it manipulates dopamine directly; the general lesson is that a prediction-error account must say what result it forbids.

the open problem

Prediction error explains when learning happens. It says much less about what the learner represents, and how a predicted value gets attached to the right cue when the reward comes long after. That credit-assignment problem is shared by brains and machines, and neither has a fully satisfying answer.

Reading list

1. Rescorla & Wagner (1972). A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. In A. H. Black & W. F. Prokasy (eds), Classical Conditioning II. New York: Appleton-Century-Crofts.
2. Sutton (1988). Learning to predict by the methods of temporal differences. Machine Learning. doi:10.1007/BF00115009
3. Schultz, Dayan & Montague (1997). A Neural Substrate of Prediction and Reward. Science. doi:10.1126/science.275.5306.1593
4. Rao & Ballard (1999). Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience. doi:10.1038/4580
5. Friston (2010). The free-energy principle: a unified brain theory?. Nature Reviews Neuroscience. doi:10.1038/nrn2787
6. Berridge (2007). The debate over dopamine’s role in reward: the case for incentive salience. Psychopharmacology. doi:10.1007/s00213-006-0578-x
7. Mohebi et al. (2019). Dissociable dopamine dynamics for learning and motivation. Nature. doi:10.1038/s41586-019-1235-y
8. Jeong et al. (2022). Mesolimbic dopamine release conveys causal associations. Science. doi:10.1126/science.abq6740
9. Sharpe et al. (2017). Dopamine transients are sufficient and necessary for acquisition of model-based associations. Nature Neuroscience. doi:10.1038/nn.4538
10. Kim et al. (2020). A Unified Framework for Dopamine Signals across Timescales. Cell. doi:10.1016/j.cell.2020.11.013
11. Dabney et al. (2020). A distributional code for value in dopamine-based reinforcement learning. Nature. doi:10.1038/s41586-019-1924-6
12. Litwin & Miłkowski (2020). Unification by Fiat: Arrested Development of Predictive Processing. Cognitive Science. doi:10.1111/cogs.12867
13. Colombo & Wright (2017). Explanatory pluralism: An unrewarding prediction error for free energy theorists. Brain and Cognition. doi:10.1016/j.bandc.2016.02.003

see also

	glossary: prediction error — with temporal difference and blocking
	Bayes, 1763 — updating a belief is a prediction error too
	bounded rationality — when the learner cannot compute the ideal
	minds — the Cartesian theater, and what might replace it
	predictive processing — the same idea as a theory of the whole brain