hat fraction of published findings in psychology would you get again if you repeated the study carefully? For most of the field's history nobody knew, because almost nobody checked. Since about 2011 people have checked, at scale.

the question

Science assumes that a result reported once will appear again under the same conditions. But a field publishes mostly what worked. Rosenthal (1979) named the consequence: the file drawer of studies that found nothing and were never printed. Ioannidis (2005) argued from simple arithmetic that when studies are small, effects modest, and many hypotheses are tested, most "significant" results can be false. The argument depends on the inputs, and it was disputed, but it made the question impossible to ignore.

the classic findings

In 2011, two papers in respected journals made the problem concrete. Bem (2011) reported nine experiments that appeared to show precognition, using the field's standard methods. Simmons, Nelson & Simonsohn (2011) showed how: with a few undisclosed choices about when to stop collecting data, which measures to report and which covariates to include, a researcher can push the false-positive rate far above the nominal five percent. They demonstrated it by "finding" that listening to a particular song made participants younger. This is p-hacking, and it is usually done without anyone meaning to cheat.

Then came the large replications. The Open Science Collaboration (2015) repeated 100 studies from three psychology journals. Almost all the originals had reported significant results; about 36 percent of the replications did, and replication effect sizes averaged about half the originals. Camerer et al. (2018) repeated social-science experiments published in Nature and Science between 2010 and 2015 and found a similar pattern: a majority replicated, with effects around half as large. Many Labs 2 ran each of 28 findings in dozens of labs at once and found roughly half replicated, with surprisingly little variation from site to site for the effects that held.

what we learned

Low statistical power is a large part of the story. Button et al. (2013) estimated that the typical neuroscience study was badly underpowered, which both misses real effects and inflates the ones it finds. The reforms that followed are concrete: preregistration of hypotheses and analyses (Nosek et al., 2018), larger samples, shared data and materials, and journals that accept a study on its design before the results are known.

It is worth being exact about what the replication projects show. A failed replication does not prove the original effect is zero; it shows the evidence is weaker than it looked. Some effects that "failed" turned out to depend on details of procedure or population. Others have simply not come back, however they are run.

in machines

Machine-learning research has its own version: benchmark results that do not survive a different random seed, a fresh test set, or a tuned baseline. Studies of model behavior add another layer, because the same prompt can produce different answers on different days and model versions. The remedies are the same ones psychology learned: say in advance what you will measure, run it more than once, and publish the nulls.

live debates

What did the 2015 project show? Gilbert et al. (2016) argued that the Open Science Collaboration estimate was too pessimistic, because some replications departed from the originals and because sampling error alone predicts failures; Anderson et al. (2016) replied that the reanalysis made errors of its own. Van Bavel et al. (2016) reported that effects rated as context-sensitive replicated less often; Inbar (2016) found the association could be explained by subfield [paraphrase, not a quotation]. Many Labs 5 (Ebersole et al., 2020) tested the fidelity claim directly by revising protocols with expert review before data collection; the revisions made little difference to effect sizes.

What to do with p. Benjamin et al. proposed redefining significance for new discoveries at 0.005; Lakens et al. (2018) answered that the threshold should be justified for each study; McShane et al. (2019) argued for abandoning significance thresholds as a decision rule altogether.

methodological critiques

The base rate of positive results. Scheel, Schijen & Lakens (2021) compared standard papers with registered reports and found positive results in a much smaller share of the latter, which is what one would expect if the standard literature is filtered by outcome [paraphrase, not a quotation]. Beyond replication. Yarkoni (2022) argues that many psychology papers draw verbal conclusions far broader than their statistical models, which treat stimuli, settings and procedures as fixed; a result can replicate exactly and still not generalise. Flake & Fried (2020) show that measurement is often undocumented or ad hoc, so that two studies of "the same" construct may not measure the same thing.

the open problem

Replication rates tell you how often findings come back, not why the ones that failed were published. Building incentives that reward getting it right over getting it first is a problem about institutions, and it is not solved.

Reading list

1. Rosenthal (1979). The file drawer problem and tolerance for null results. Psychological Bulletin. doi:10.1037/0033-2909.86.3.638
2. Ioannidis (2005). Why Most Published Research Findings Are False. PLoS Medicine. doi:10.1371/journal.pmed.0020124
3. Bem (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology. doi:10.1037/a0021524
4. Simmons, Nelson & Simonsohn (2011). False-Positive Psychology. Psychological Science. doi:10.1177/0956797611417632
5. Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science. doi:10.1126/science.aac4716
6. Camerer et al. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour. doi:10.1038/s41562-018-0399-z
7. Klein et al. (2018). Many Labs 2: Investigating Variation in Replicability Across Samples and Settings. Advances in Methods and Practices in Psychological Science. doi:10.1177/2515245918810225
8. Button et al. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience. doi:10.1038/nrn3475
9. Nosek et al. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences. doi:10.1073/pnas.1708274114
10. Gilbert et al. (2016). Comment on “Estimating the reproducibility of psychological science”. Science. doi:10.1126/science.aad7243
11. Anderson et al. (2016). Response to Comment on “Estimating the reproducibility of psychological science”. Science. doi:10.1126/science.aad9163
12. Van Bavel et al. (2016). Contextual sensitivity in scientific reproducibility. Proceedings of the National Academy of Sciences. doi:10.1073/pnas.1521897113
13. Inbar (2016). Association between contextual dependence and replicability in psychology may be spurious. Proceedings of the National Academy of Sciences. doi:10.1073/pnas.1608676113
14. Ebersole et al. (2020). Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability. Advances in Methods and Practices in Psychological Science. doi:10.1177/2515245920958687
15. Benjamin et al. (2017). Redefine statistical significance. Nature Human Behaviour. doi:10.1038/s41562-017-0189-z
16. Lakens et al. (2018). Justify your alpha. Nature Human Behaviour. doi:10.1038/s41562-018-0311-x
17. McShane et al. (2019). Abandon Statistical Significance. The American Statistician. doi:10.1080/00031305.2018.1527253
18. Scheel, Schijen & Lakens (2021). An excess of positive results: comparing the standard psychology literature with registered reports. Advances in Methods and Practices in Psychological Science. doi:10.1177/25152459211007467
19. Yarkoni (2022). The generalizability crisis. Behavioral and Brain Sciences. doi:10.1017/S0140525X20001685
20. Flake & Fried (2020). Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them. Advances in Methods and Practices in Psychological Science. doi:10.1177/2515245920952393

Try it yourself. [playful] Small toys on the toy shelf that show this in action:   • the arrow lines — an illusion that knowing about does not switch off   • the Stroop box — name the ink, not the word

see also

	annotated: metascience — the replication papers, two or three sentences each
	glossary: p-hacking — with file drawer, preregistration, power
	measurement — what changes when somebody looks
	Clever Hans — the first famous failure to replicate under controls