from Fechner's threshold to a machine that declines to answer · an essay from the house

here is a moment, turning a dimmer down, when the light is gone. Or is there? Ask carefully and the moment smears. At one setting you are sure; a little lower you think so; lower still you are guessing, and yet your guesses are right more often than chance. This essay is about that smear, and why the way psychology learned to handle it is now the way we ought to want machines to handle theirs.

I. a line in the mind

Fechner needed the line. His psychophysics (Fechner, 1860) was built on the threshold, the point at which a stimulus becomes a sensation, and on the just-noticeable difference as a unit of sensation. His methods for finding thresholds were superb. They also kept showing that the threshold was not where a line should be: it moved from trial to trial, and the curve of "yes" responses against intensity, the psychometric function, rose gradually instead of stepping.¹

For decades the gradual rise was blamed on noise around a true line. Thurstone (1927) made the noise the theory: every stimulus produces a value that varies from moment to moment, and judgments compare those values. That was half of the answer.

II. the line is a choice

The other half came from radar. Peterson, Birdsall & Fox (1954) asked how an ideal receiver should decide whether a target was present in a noisy trace. The answer has two parts that cannot be pulled apart by looking at hits alone: how separable signal is from noise, and where the receiver sets its criterion. Tanner & Swets (1954) showed that human observers looking for faint lights behaved like such receivers. Change the payoffs and their hits and false alarms moved together along a curve, the ROC, while their sensitivity, d′, stayed where it was.

Then Swets (1961) asked the question in his title, "Is there a sensory threshold?", and argued that the data did not need one [paraphrase, not a quotation]. What Fechner had been measuring as a line in the senses was, in large part, a line the observer drew: a decision about how much evidence is enough.²

This is not a technicality. It means a "yes" carries information about the world and about the person in one number, and that you cannot read either without the false alarms. A radiologist who finds more tumours may see better, or may simply say yes more often. A test that "detects" more cases may have moved its line. Signal detection theory is the habit of always asking which.

III. the line in a machine

Machines decide too, and for most of the history of pattern recognition they decided every time. Chow (1970) worked out what happens if a classifier is allowed a third answer: "I will not say". With a reject option, the classifier declines whenever its best guess is not probable enough, and Chow derived the optimal trade-off between how often it declines and how often it errs when it does not. Raise the line and errors fall while coverage shrinks; there is a best line for any given cost of an error versus a refusal.

The rule only works if the probabilities mean something, which is to say if the system is calibrated. Modern networks often are not: Guo et al. (2017) found them more confident than accurate. For language models, Kadavath et al. (2022) found reasons for hope: large models could estimate, for some kinds of question, whether they knew the answer.³

IV. what we should want

Put the three parts together and the argument is simple. A good detector, human or machine, has two separate properties: how well it can tell, and where it draws the line. We should measure both, and we should want the line drawn where the costs say it should be, not where it makes the system look most helpful. For people, that is the discipline signal detection theory imposed on psychophysics. For machines that talk, it is a system that says "I am not sure" when, and only when, it is not.

Fechner was right that there is a line. He was looking for it in the wrong place. It is not in the light. It is in whoever is deciding what counts as seeing it.⁴

notes

1. Fechner's three methods (limits, constant stimuli, adjustment) are still taught. The adaptive staircase is a descendant of the method of limits; there is one you can run on yourself on the staircase page, and the classic description is Cornsweet (1962).
2. The textbook statement is Green & Swets (1966). Whether some "high threshold" survives underneath the noise is still argued in memory research; see the signal detection page.
3. The same paper is careful to say this held in particular formats; free-form answers are harder to score. On how preference tuning interacts with confidence, see sycophancy.
4. If you would like to find your own line, we set some text out just about where it begins to fade. It is a toy, not a measurement: your screen, your room and your criterion all count.

Sources

1. Fechner (1860). Elemente der Psychophysik. Leipzig: Breitkopf und Härtel.
2. Thurstone (1927). A law of comparative judgment. Psychological Review. doi:10.1037/h0070288
3. Peterson, Birdsall & Fox (1954). The theory of signal detectability. Transactions of the IRE Professional Group on Information Theory. doi:10.1109/TIT.1954.1057460
4. Tanner & Swets (1954). A decision-making theory of visual detection. Psychological Review. doi:10.1037/h0058700
5. Swets (1961). Is There a Sensory Threshold?. Science. doi:10.1126/science.134.3473.168
6. Chow (1970). On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory. doi:10.1109/TIT.1970.1054406
7. Guo, Pleiss, Sun & Weinberger (2017). On calibration of modern neural networks. arXiv:1706.04599 (ICML 2017). arxiv.org/abs/1706.04599
8. Kadavath et al. (2022). Language models (mostly) know what they know. arXiv:2207.05221. arxiv.org/abs/2207.05221
9. Cornsweet (1962). The Staircase-Method in Psychophysics. The American Journal of Psychology. doi:10.2307/1419876
10. Green & Swets (1966). Signal Detection Theory and Psychophysics. New York: Wiley.

Try it yourself. [playful] Small toys on the toy shelf that show this in action:   • the line judge — find your Weber fraction for line length

see also

	signal detection — the area page
	calibration — the property the reject option needs
	Fechner
	timeline: psychophysics — the dates in one column
	the cave — another essay about what is really on the wall