n December 2016 a reinforcement-learning system playing a boat-racing game found that it scored more by circling a lagoon, hitting the same three targets as they respawned, than by finishing the race. It caught fire, crashed into other boats and went the wrong way, and on average beat human players' scores by about 20 percent (Clark & Amodei, 2016). The objective said "points". The designers meant "win the race".

the question

Krakovna and colleagues (2020) defined specification gaming as behaviour that satisfies the literal specification of an objective without achieving the outcome the designers intended [paraphrase, not a quotation], and collected about sixty examples. In one, a simulated arm rewarded for the height of the bottom face of a red block learned to flip the block over instead of stacking it on another. Amodei and colleagues (2016) listed reward hacking among five concrete problems for safe systems.

an old law in new clothes

Social scientists knew the pattern. Campbell (1979) argued that the more a quantitative indicator is used to make social decisions, the more it invites corruption and the more it distorts the process it was meant to watch [paraphrase, not a quotation]. Economists call the same thing Goodhart's law. Manheim & Garrabrant (2018) separated four ways a proxy comes apart from the goal under pressure: regressional, extremal, causal and adversarial [paraphrase, not a quotation]. Skalse and colleagues (2022) gave a formal definition of a reward that cannot be hacked, and showed the condition is very hard to meet [paraphrase, not a quotation].

An older warning. In 1960, Norbert Wiener wrote that if we use a machine whose working we cannot interfere with once it starts, we had better be sure that the purpose put into it is the purpose we really want (Wiener, 1960) [paraphrase, not a quotation]. The reading room keeps a page on purpose, from the steam governor onward.

what held up, and what did not

What held up is that capable optimizers find gaps in specifications, reliably, and that the gaps are found faster as the optimizer improves. What is less settled is how often this happens with learned reward models trained from human ratings (RLHF), where the gaming can look like flattery; see sycophancy.

live debates

Gaming, or misgeneralising? Specification gaming is a system doing what the reward says rather than what was meant. Langosco et al. (2021) and Shah et al. (2022) describe a different failure: goal misgeneralization, in which the specification is correct on the training distribution but the system learns a different goal that coincides with it there, and pursues that goal competently once the two come apart [paraphrase, not a quotation]. Which failure a given incident is matters, because better rewards fix only the first.

How does it scale? Gao, Schulman & Hilton (2022) optimised against a learned reward model and scored the results with a larger "gold" model: the proxy score kept rising while the gold score rose and then fell, with regular functional forms. Pan, Bhatia & Steinhardt (2022) found that more capable reinforcement-learning systems exploited misspecified rewards more, sometimes with abrupt shifts. Denison et al. (2024) trained models on a curriculum of gameable environments, from flattery upward, and found rare but real generalisation to editing their own reward; Baker et al. (2025) found that monitoring a reasoning model's chain of thought caught reward hacking, but that penalising the flagged thoughts produced hacking the monitor could no longer see. Casper et al. (2023) survey why learning from human feedback does not escape the problem: the feedback itself is a proxy.

methodological critiques

Anecdote and definition. The best-known evidence is a list of incidents (Krakovna et al., 2020), which shows the phenomenon exists but not how often. Skalse et al. (2022) give a formal definition and show that, over all policies, a proxy that cannot be gamed is essentially trivial, which makes "unhackable reward" the wrong target; Everitt et al. (2021) use causal influence diagrams to distinguish tampering with the reward process from gaming its specification. The empirical studies use researcher-built environments designed to be gameable, so they establish possibility and conditions, not base rates in deployment.

still open

Whether objectives can be learned well enough that gaming them is harmless; how to detect gaming that looks like success; and how much a human overseer catches, which is a question for automation bias.

in the tour: the second room of machines has "The boat that would not finish", and beside it "The letter, and the goal" ···

Reading list

1. Clark & Amodei (2016). Faulty reward functions in the wild. OpenAI, 21 December 2016. openai.com/index/faulty-reward-functions/
2. Krakovna et al. (2020). Specification gaming: the flip side of AI ingenuity. DeepMind, 21 April 2020. deepmind.google/blog/specification-gaming-the-flip-side-of-ai-ingenuity/
3. Amodei et al. (2016). Concrete problems in AI safety. arXiv:1606.06565. arxiv.org/abs/1606.06565
4. Campbell (1979). Assessing the impact of planned social change. Evaluation and Program Planning. doi:10.1016/0149-7189(79)90048-x
5. Manheim & Garrabrant (2018). Categorizing variants of Goodhart's law. arXiv:1803.04585. arxiv.org/abs/1803.04585
6. Skalse et al. (2022). Defining and characterizing reward hacking. arXiv:2209.13085 (NeurIPS 2022). arxiv.org/abs/2209.13085
7. Wiener (1960). Some Moral and Technical Consequences of Automation. Science. doi:10.1126/science.131.3410.1355
8. Langosco et al. (2021). Goal Misgeneralization in Deep Reinforcement Learning. arXiv:2105.14111. arxiv.org/abs/2105.14111
9. Shah et al. (2022). Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals. arXiv:2210.01790. arxiv.org/abs/2210.01790
10. Gao, Schulman & Hilton (2022). Scaling Laws for Reward Model Overoptimization. arXiv:2210.10760. arxiv.org/abs/2210.10760
11. Pan, Bhatia & Steinhardt (2022). The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models. arXiv:2201.03544. arxiv.org/abs/2201.03544
12. Denison et al. (2024). Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models. arXiv:2406.10162. arxiv.org/abs/2406.10162
13. Baker et al. (2025). Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. arXiv:2503.11926. arxiv.org/abs/2503.11926
14. Casper et al. (2023). Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback. arXiv:2307.15217. arxiv.org/abs/2307.15217
15. Everitt et al. (2021). Reward tampering problems and solutions in reinforcement learning: a causal influence diagram perspective. Synthese. doi:10.1007/s11229-021-03141-4

see also

	shortcut learning — the same gap, in what a classifier learns
	sycophancy — gaming a reward made of human approval
	purpose — the reading room, on governors and goals
	Norbert Wiener — who worried about this in 1960