ower is the probability that a study detects an effect of a given size if the effect is real. It depends on the true effect, the sample size and the significance threshold, and it is usually lower than researchers think.

the history

Cohen (1962) surveyed a year of a leading psychology journal and estimated that the typical study had only about an even chance of detecting a medium-sized effect. Sedlmeier & Gigerenzer (1989) repeated the survey a quarter of a century later and found no improvement. Button et al. (2013) estimated that the median power of neuroscience studies was around a fifth, and pointed out the consequence: in a low-powered field, a significant result is less likely to be true, and if it is true, its effect size is likely to be inflated [paraphrase, not a quotation].

the toy [playful]

Choose a true effect (as a standardised difference, Cohen's d), a sample size per group and a threshold. The bar shows power. Then simulate two thousand studies: grey bars are non-significant results, purple bars significant ones. Watch where the purple sits relative to the red line when power is low.

This toy needs JavaScript. The rest of the page reads fine without it.

true effect d: 0.2 (small)0.30.5 (medium)0.8 (large)1.0   per group: 1020305064100200   threshold: 0.050.010.005

simulate 2000 studies

Estimated effects from -1 to 2 will appear here.

Gelman & Carlin (2014) named what the histogram shows: a Type M (magnitude) error, the exaggeration of significant estimates, and a Type S (sign) error, a significant estimate in the wrong direction. Both grow as power falls.

what's contested

Is the threshold the problem? Benjamin et al. (2018) proposed moving the default threshold for new discoveries from .05 to .005. Lakens et al. (2018) replied that no single threshold suits every field and that researchers should justify their alpha, and McShane et al. (2019) argued for abandoning significance thresholds as a decision rule altogether [paraphrase, not a quotation]. The three positions remain in play.

Low power or bad incentives? Ioannidis (2005) modelled how low power, bias and many tested hypotheses make most published findings false in some fields. Smaldino & McElreath (2016) modelled labs as competing for publications and showed that selection for output favours low-powered methods even when no one chooses them deliberately.

methodological critiques

A power analysis needs an effect size, and the usual source, a published estimate, is inflated by exactly the filter the toy shows, so studies planned from the literature are underpowered. Hoenig & Heisey (2001) showed that "observed power", computed after the fact from the observed effect, is a direct function of the p-value and adds no information. The normal approximation in the toy is slightly optimistic for very small samples.

later reception

Power analysis is now required by many journals and funders, and large multi-lab replications were designed with high power partly because of these arguments. Benchmark comparisons between models raise the same issue: on a small test set, many reported improvements are within the noise. See replication.

References

1. Cohen (1962). The statistical power of abnormal-social psychological research: A review. The Journal of Abnormal and Social Psychology. doi:10.1037/h0045186
2. Sedlmeier & Gigerenzer (1989). Do studies of statistical power have an effect on the power of studies?. Psychological Bulletin. doi:10.1037/0033-2909.105.2.309
3. Button et al. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience. doi:10.1038/nrn3475
4. Gelman & Carlin (2014). Beyond Power Calculations. Perspectives on Psychological Science. doi:10.1177/1745691614551642
5. Benjamin et al. (2017). Redefine statistical significance. Nature Human Behaviour. doi:10.1038/s41562-017-0189-z
6. Lakens et al. (2018). Justify your alpha. Nature Human Behaviour. doi:10.1038/s41562-018-0311-x
7. McShane et al. (2019). Abandon Statistical Significance. The American Statistician. doi:10.1080/00031305.2018.1527253
8. Ioannidis (2005). Why Most Published Research Findings Are False. PLoS Medicine. doi:10.1371/journal.pmed.0020124
9. Smaldino & McElreath (2016). The natural selection of bad science. Royal Society Open Science. doi:10.1098/rsos.160384
10. Hoenig & Heisey (2001). The Abuse of Power. The American Statistician. doi:10.1198/000313001300339897

see also