receiver operating characteristic, or ROC curve, plots the hit rate of a detector against its false-alarm rate as its criterion moves from cautious to liberal. It separates two things a single accuracy score mixes up: how well the detector can tell signal from noise, and where it has chosen to draw the line.

where it came from

The curve came from radar and the theory of signal detectability (Peterson, Birdsall & Fox, 1954), and was carried into psychology by Tanner & Swets (1954), who showed that observers trade hits for false alarms when the payoffs change while their sensitivity stays fixed. See signal detection and d′.

the toy [playful]

Two bell curves: what the detector receives when there is only noise (grey) and when there is a signal (purple). Move the separation and the criterion; the red dot moves along the curve. Tick the box to give the signal a wider spread, as recognition memory seems to have.

This toy needs JavaScript. The rest of the page reads fine without it.

separation d′   criterion   unequal spread

the area under the curve

Hanley & McNeil (1982) showed that the area under the ROC curve (AUC) equals the probability that a randomly chosen signal case is ranked above a randomly chosen noise case, which made it the standard summary in medicine and later in machine learning. Swets, Dawes & Monahan (2000) argued that diagnostic decisions in medicine, law and engineering would improve if practitioners separated accuracy from the choice of threshold in just this way.

what's contested

Is AUC coherent? Hand (2009) showed that comparing classifiers by AUC implicitly weighs misclassification costs differently for each classifier, which makes it an inconsistent yardstick; he proposed an alternative, the H measure [paraphrase, not a quotation]. Lobo et al. (2008) listed ways AUC misleads in ecological prediction, including that it summarises regions of the curve no one would use and ignores how well probabilities are calibrated (see calibration).

What the shape means. In recognition memory the ROC is asymmetric. Wixted (2007) argues that an unequal-variance signal detection model explains this, against dual-process accounts that add a separate all-or-none recollection process; the argument is about which model of memory the curve supports [paraphrase, not a quotation].

methodological critiques

An empirical ROC needs several criteria from the same observer, usually from confidence ratings, and the assumption that rating categories behave like criteria on one dimension is itself testable and sometimes fails. With rare positives, a curve can look excellent while most positive calls are wrong; precision–recall curves exist for that case.

later reception

ROC analysis is standard in radiology, forensic science, credit scoring and machine learning. In eyewitness research it changed policy debates about line-up procedures, because it separates better discrimination from a mere shift toward saying "no".

References

1. Peterson, Birdsall & Fox (1954). The theory of signal detectability. Transactions of the IRE Professional Group on Information Theory. doi:10.1109/TIT.1954.1057460
2. Tanner & Swets (1954). A decision-making theory of visual detection. Psychological Review. doi:10.1037/h0058700
3. Hanley & McNeil (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. doi:10.1148/radiology.143.1.7063747
4. Swets, Dawes & Monahan (2000). Psychological Science Can Improve Diagnostic Decisions. Psychological Science in the Public Interest. doi:10.1111/1529-1006.001
5. Hand (2009). Measuring classifier performance: a coherent alternative to the area under the ROC curve. Machine Learning. doi:10.1007/s10994-009-5119-5
6. Lobo, Jiménez‐Valverde & Real (2008). AUC: a misleading measure of the performance of predictive distribution models. Global Ecology and Biogeography. doi:10.1111/j.1466-8238.2007.00358.x
7. Wixted (2007). Dual-process theory and signal-detection theory of recognition memory. Psychological Review. doi:10.1037/0033-295X.114.1.152

see also