reregistration means writing down, before seeing the data, what you will test and how: the hypotheses, the sample size, the outcome measures, the exclusions and the analysis, with a time stamp that others can check (Nosek et al., 2018). Its purpose is to make visible the difference between a prediction and a description of what happened.

why it is needed

Simmons, Nelson & Simonsohn (2011) showed by simulation that a few ordinary choices, such as which of two measures to report, whether to collect more data after peeking, whether to include a covariate, and which conditions to compare, could together push the false-positive rate from the nominal 5% to more than half, without anyone intending to cheat [paraphrase, not a quotation]. Gelman & Loken (2014) made the subtler point: even a researcher who runs only one analysis may have chosen it in light of the data, so the "garden of forking paths" inflates error without any fishing.

the toy [playful]

Here are two groups drawn from the same population, so any "effect" is a false positive. Each participant has two correlated outcome measures. Choose which freedoms to allow yourself, then run one study and see every analysis you could have reported, or run a thousand.

This toy needs JavaScript. The rest of the page reads fine without it.

report measure A, measure B, or their average if not significant, add ten more per group and test again also try dropping "outliers" beyond two standard deviations also test each half of the sample (a subgroup) per group: 203050

run one study   run 1000 studies

Nothing has run yet.

How it computes: normal data, a two-sample t-test with pooled variance for each allowed analysis. With all four boxes ticked there are up to thirty-six analyses per study. This is a simplified version of the Simmons et al. design, not a reproduction of it.

what's contested

Does it improve science, or only bookkeeping? Szollosi et al. (2020) argue that preregistration treats a symptom: if theories are too weak to dictate an analysis, writing down an arbitrary one in advance does not make the inference better, and the cure is better theory [paraphrase, not a quotation]. Nosek et al. (2019) reply that preregistration is meant to make the status of a claim, confirmatory or exploratory, transparent, not to replace theory.

Do people do what they registered? Claesen et al. (2021) checked the first generation of preregistered studies in one major journal and found that deviations from the registered plan were common and often not disclosed.

What changes when review comes first. In a registered report the journal reviews and accepts the introduction and method before the results exist (Chambers, 2013). Scheel, Schijen & Lakens (2021) compared these with standard articles testing hypotheses in psychology: positive results were reported in 96% of standard articles but in 44% of registered reports, which suggests how much the standard literature is filtered.

methodological critiques

Simulations like the toy show what flexibility can do, not how much of it researchers actually use; estimates of the real prevalence rely on surveys and audits, each with its own biases. And a preregistration is only as strict as it is specific: a vague plan leaves the garden open.

later reception

Preregistration is now routine in clinical trials, common in experimental psychology and spreading in other fields, including machine-learning evaluation, where choosing the benchmark and metric after seeing results is the same garden. See replication and the replication timeline.

in the tour: the forking corridor has placards on a round earth and on forking paths ···

References

1. Nosek et al. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences. doi:10.1073/pnas.1708274114
2. Simmons, Nelson & Simonsohn (2011). False-Positive Psychology. Psychological Science. doi:10.1177/0956797611417632
3. Gelman & Loken (2014). The Statistical Crisis in Science. American Scientist. doi:10.1511/2014.111.460
4. Szollosi et al. (2020). Is Preregistration Worthwhile?. Trends in Cognitive Sciences. doi:10.1016/j.tics.2019.11.009
5. Nosek et al. (2019). Preregistration Is Hard, And Worthwhile. Trends in Cognitive Sciences. doi:10.1016/j.tics.2019.07.009
6. Claesen et al. (2021). Comparing dream to reality: an assessment of adherence of the first generation of preregistered studies. Royal Society Open Science. doi:10.1098/rsos.211037
7. Chambers (2013). Registered reports: a new publishing initiative at Cortex. Cortex. doi:10.1016/j.cortex.2012.12.016
8. Scheel, Schijen & Lakens (2021). An excess of positive results: comparing the standard psychology literature with registered reports. Advances in Methods and Practices in Psychological Science. doi:10.1177/25152459211007467

see also