hen a forecaster says "70 percent chance of rain", it should rain on about 70 percent of those days. That property is calibration. It is separate from being right often; a forecaster who always says the climatological rate is calibrated and nearly useless.

the question

How well do people, and models, know how much they know? And can a system be built to say "I am not sure" at the right times?

the classic findings

Brier (1950) gave weather forecasting a scoring rule, the Brier score: the mean squared difference between the forecast probability and what happened. It rewards both calibration and sharpness, and it cannot be gamed by saying something other than what you believe. Murphy & Winkler (1977) then showed that US weather forecasters' probability-of-precipitation forecasts were remarkably well calibrated, a rare case of a profession getting this right, helped by fast, unambiguous feedback every day.

Most people in most studies are not. Lichtenstein, Fischhoff & Phillips (1982) reviewed the early work: on general-knowledge questions, stated confidence usually exceeds accuracy, most of all on hard items, and the "hard–easy effect" flips toward underconfidence on easy ones. Tetlock (2005) tracked thousands of expert political forecasts over years and found them poorly calibrated and often hardly better than simple baselines.

in machines

Guo et al. (2017) found that modern deep networks were much more overconfident than older, smaller ones, and that a single learned temperature on the outputs fixed most of it. For language models, Kadavath et al. (2022) found that large pretrained models were fairly well calibrated on multiple-choice and true/false questions in the right format, and could be trained to predict whether they knew an answer [paraphrase, not a quotation]. Later reports describe calibration getting worse after preference tuning, which links this page to sycophancy.

The engineering response is old. Chow (1970) worked out the optimal trade-off for a classifier that may decline to decide: the reject option. Refuse when the best class is not probable enough, and you trade coverage for fewer errors along a curve you can compute.

what is not settled

For people, how much of overconfidence is a real bias and how much comes from how questions are selected and scored is still argued. For models, calibration on multiple-choice benchmarks says little about calibration in free-form answers, where there is no single probability to read off.

live debates

Is human overconfidence real? Gigerenzer, Hoffrage & Kleinbölting (1991) and Juslin (1994) argued that much of the classic overconfidence came from how general-knowledge items were selected; with representatively sampled items it shrinks. Erev, Wallsten & Budescu (1994) showed that random error alone can produce over- and underconfidence in the same data, depending on how it is analysed. Moore & Healy (2008) separate three things all called overconfidence: overestimation, overplacement and overprecision [paraphrase, not a quotation].

Machines. After Guo et al. (2017) reported that modern networks were badly calibrated, Minderer et al. (2021) found that newer, non-convolutional architectures were among the best calibrated, so the trend was not general. For language models, the GPT-4 report showed calibration of the pretrained model degraded after post-training, Tian et al. (2023) found verbalised confidence could be better calibrated than token probabilities in such models, and Xiong et al. (2023) found verbalised confidence generally overconfident.

methodological critiques

ECE is an estimator, not a fact. Nixon et al. (2019) showed that expected calibration error depends on binning choices and can rank models differently, and Kumar, Liang & Ma (2019) showed binned estimates can understate the true error. A calibration claim needs its estimator, bin scheme and test distribution stated.

the open problem

A calibrated system is only useful if the people relying on it can use its confidence. That turns a statistical property into a question about reliance and teamwork.

Reading list

1. Brier (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review. doi:10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2
2. Murphy & Winkler (1977). Reliability of Subjective Probability Forecasts of Precipitation and Temperature. Applied Statistics. doi:10.2307/2346866
3. Lichtenstein, Fischhoff & Phillips (1982). Calibration of probabilities: the state of the art to 1980. In D. Kahneman, P. Slovic & A. Tversky (eds), Judgment under Uncertainty: Heuristics and Biases. Cambridge University Press.
4. Tetlock (2005). Expert Political Judgment: How Good Is It? How Can We Know? Princeton University Press.
5. Guo, Pleiss, Sun & Weinberger (2017). On calibration of modern neural networks. arXiv:1706.04599 (ICML 2017). arxiv.org/abs/1706.04599
6. Kadavath et al. (2022). Language models (mostly) know what they know. arXiv:2207.05221. arxiv.org/abs/2207.05221
7. Chow (1970). On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory. doi:10.1109/TIT.1970.1054406
8. Gigerenzer, Hoffrage & Kleinbölting (1991). Probabilistic mental models: A Brunswikian theory of confidence. Psychological Review. doi:10.1037/0033-295X.98.4.506
9. Juslin (1994). The Overconfidence Phenomenon as a Consequence of Informal Experimenter-Guided Selection of Almanac Items. Organizational Behavior and Human Decision Processes. doi:10.1006/obhd.1994.1013
10. Erev, Wallsten & Budescu (1994). Simultaneous over- and underconfidence: The role of error in judgment processes. Psychological Review. doi:10.1037/0033-295X.101.3.519
11. Moore & Healy (2008). The trouble with overconfidence. Psychological Review. doi:10.1037/0033-295X.115.2.502
12. Minderer et al. (2021). Revisiting the Calibration of Modern Neural Networks. arXiv:2106.07998. arxiv.org/abs/2106.07998
13. OpenAI (2023). GPT-4 Technical Report. arXiv:2303.08774. arxiv.org/abs/2303.08774
14. Tian et al. (2023). Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. arXiv:2305.14975. arxiv.org/abs/2305.14975
15. Xiong et al. (2023). Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. arXiv:2306.13063. arxiv.org/abs/2306.13063
16. Nixon et al. (2019). Measuring Calibration in Deep Learning. arXiv:1904.01685. arxiv.org/abs/1904.01685
17. Kumar, Liang & Ma (2019). Verified Uncertainty Calibration. arXiv:1909.10155. arxiv.org/abs/1909.10155

see also

	essay: where the line is — ends at the reject option
	glossary: calibration — with Brier score and reject option
	signal detection — sensitivity and criterion
	Bayes, 1763 — what a calibrated belief is supposed to do