The preregistration revolution

Proceedings of the National Academy of Sciences of the United States of America

Volumn 115, Issue 11, 2018, Pages 2600-2606

  Nosek, Brian A ^a,b   Ebersole, Charles R ^b   DeHaven, Alexander C ^a   Mellor, David T ^a  

^a CENTER FOR OPEN SCIENCE   (United States)

^b UNIVERSITY OF VIRGINIA   (United States)

Author keywords

Confirmatory analysis; Exploratory analysis; Methodology; Open science; Preregistration

Indexed keywords

ARTICLE; DECISION TREE; HINDSIGHT BIAS; HYPOTHESIS; INFORMATION PROCESSING; MEASUREMENT ACCURACY; MEDICAL RESEARCH; MEMORY; MOTIVATION; PREDICTION; PRIORITY JOURNAL; REPRODUCIBILITY; STATISTICAL ANALYSIS; STUDY DESIGN; HUMAN; LABORATORY PERSONNEL; MANPOWER; PREDICTIVE VALUE; RESEARCH; SCIENCE; STANDARDS;

HUMANS; LABORATORY PERSONNEL; PREDICTIVE VALUE OF TESTS; RESEARCH; SCIENCE;

EID: 85043783065     PISSN: 00278424     EISSN: 10916490     Source Type: Journal    
DOI: 10.1073/pnas.1708274114     Document Type: Article

References (60)

1. Box GEP (1976) Science and statistics. J Am Stat Assoc 71:791–799.
2. Box GEP (1979) Robustness in the strategy of scientific model building. Robustness in Statistics, eds Launer RL, Wilkinson GN (Academic, New York), pp 201–236.
3. de Groot AD (2014) The meaning of “significance” for different types of research. Acta Psychol (Amst) 148:188–194.
4. Hoyningen-Huene P (1987) Context of discovery and context of justification. Stud Hist Philos Sci 18:501–515.
5. Kuhn TS (1970) Logic of discovery or psychology of research? Criticism and the Growth of Knowledge, eds Lakatos I, Musgrave A (Cambridge Univ Press, Cambridge, UK), pp 1–23.
6. Wagenmakers E-J, Wetzels R, Borsboom D, van der Maas HLJ, Kievit RA (2012) An agenda for purely confirmatory research. Perspect Psychol Sci 7:632–638.
7. Forstmeier W, Wagenmakers E-J, Parker TH (2017) Detecting and avoiding likely false-positive findings–A practical guide. Biol Rev Camb Philos Soc 92:1941–1968.
8. Munafò MR, et al. (2017) A manifesto for reproducible science. Nat Hum Behav 1: 0021.
9. Nosek BA, Spies JR, Motyl M (2012) Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspect Psychol Sci 7: 615–631.
10. Open Science Collaboration (2015) PSYCHOLOGY. Estimating the reproducibility of psychological science. Science 349:aac4716.
11. Swaen GG, Teggeler O, van Amelsvoort LG (2001) False positive outcomes and design characteristics in occupational cancer epidemiology studies. Int J Epidemiol 30: 948–954.
12. Kerr NL (1998) HARKing: Hypothesizing after the results are known. Pers Soc Psychol Rev 2:196–217.
13. Fischhoff B, Beyth R (1975) I knew it would happen: Remembered probabilities of once–future things. Organ Behav Hum Perform 13:1–16.
14. Fischhoff B (2003) Hindsight not equal to foresight: The effect of outcome knowledge on judgment under uncertainty. 1975. Qual Saf Health Care 12:304–311, discussion 311–312.
15. Lewis M (2016) The Undoing Project: A Friendship That Changed Our Minds (W. W. Norton & Company, New York).
16. Hawkins CB, Nosek BA (2012) Motivated independence? Implicit party identity predicts political judgments among self-proclaimed Independents. Pers Soc Psychol Bull 38:1437–1452.
17. Kahneman D, Slovic P, Tversky A (1982) Judgment Under Uncertainty: Heuristics and Biases (Cambridge Univ Press, Cambridge, UK).
18. Kahneman D (2011) Thinking, Fast and Slow (Farrar, Straus and Giroux, New York).
19. Bakker M, van Dijk A, Wicherts JM (2012) The rules of the game called psychological science. Perspect Psychol Sci 7:543–554.
20. Giner-Sorolla R (2012) Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspect Psychol Sci 7: 562–571.
21. Mahoney MJ (1977) Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognit Ther Res 1:161–175.
22. Christensen-Szalanski JJJ, Willham CF (1991) The hindsight bias: A meta-analysis. Organ Behav Hum Decis Process 48:147–168.
23. Kunda Z (1990) The case for motivated reasoning. Psychol Bull 108:480–498.
24. Nickerson RS (1998) Confirmation bias: A ubiquitous phenomenon in many guises. Rev Gen Psychol 2:175–220.
25. Nosek BA, Riskind RG (2012) Policy implications of implicit social cognition. Soc Issues Policy Rev 6:113–147.
26. Pronin E, Kugler MB (2007) Valuing thoughts, ignoring behavior: The introspection illusion as a source of the bias blind spot. J Exp Soc Psychol 43:565–578.
27. Sellke T, Bayarri MJ, Berger JO (2001) Calibration of ρ values for testing precise null hypotheses. Am Stat 55:62–71.
28. Hubbard R, Ryan PA (2000) Statistical significance with comments by editors of marketing journals: The historical growth of statistical significance testing in psychology—and its future prospects. Educ Psychol Meas 60:661–681.
29. Stephens PA, Buskirk SW, del Rio CM (2007) Inference in ecology and evolution. Trends Ecol Evol 22:192–197.
30. Simmons JP, Nelson LD, Simonsohn U (2011) False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci 22:1359–1366.
31. Wasserstein RL, Lazar NA (2016) The ASA’s statement on p-values: Context, process, and purpose. Am Stat 70:129–133.
32. Benjamin DJ, et al. (2017) Redefine statistical significance. Nat Hum Behav, 10.17605/OSF.IO/MKY9J.
33. Dunnett CW (1955) A multiple comparison procedure for comparing several treatments with a control. J Am Stat Assoc 50:1096–1121.
34. Tukey JW (1949) Comparing individual means in the analysis of variance. Biometrics 5: 99–114.
35. Benjamini Y (2010) Simultaneous and selective inference: Current successes and future challenges. Biom J 52:708–721.
36. Saxe R, Brett M, Kanwisher N (2006) Divide and conquer: A defense of functional localizers. Neuroimage 30:1088–1096, discussion 1097–1099.
37. Gelman A, Loken E (2014) The statistical crisis in science. Am Sci 102:460.
38. Franco A, Malhotra N, Simonovits G (2014) Social Science. Publication bias in the social sciences: Unlocking the file drawer. Science 345:1502–1505.
39. Greenwald AG (1975) Consequences of prejudice against the null hypothesis. Psychol Bull 82:1–20.
40. Rosenthal R (1979) The file drawer problem and tolerance for null results. Psychol Bull 86:638–641.
41. Ioannidis JPA (2005) Why most published research findings are false. PLoS Med 2: e124.
42. John LK, Loewenstein G, Prelec D (2012) Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol Sci 23:524–532.
43. Kaplan RM, Irvin VL (2015) Likelihood of null effects of large NHLBI clinical trials has increased over time. PLoS One 10:e0132382.
44. Cybulski L, Mayo-Wilson E, Grant S (2016) Improving transparency and reproducibility through registration: The status of intervention trials published in clinical psychology journals. J Consult Clin Psychol 84:753–767.
45. Odutayo A, et al. (2017) Association between trial registration and positive study findings: Cross sectional study (Epidemiological Study of Randomized Trials- ESORT). BMJ 356:j917.
46. Armitage P, McPherson CK, Rowe BC (1969) Repeated significance tests on accumulating data. J R Stat Soc Ser A 132:235–244.
47. Dutilh G, et al. (2017) A test of the diffusion model explanation for the worst performance rule using preregistration and blinding. Atten Percept Psychophys 79: 713–725.
48. MacCoun R, Perlmutter S (2015) Blind analysis: Hide results to seek the truth. Nature 526:187–189.
49. Lin W, Green DP (2016) Standard operating procedures: A safety net for pre-analysis plans. PS Polit Sci Polit 49:495–500.
50. Klionsky DJ, et al. (2016) Guidelines for the use and interpretation of assays for monitoring autophagy (3rd edition). Autophagy 12:1–222, and erratum (2016) 12: 443.
51. Greenwald AG, Nosek BA, Banaji MR (2003) Understanding and using the implicit association test: I. An improved scoring algorithm. J Pers Soc Psychol 85:197–216.
52. Campbell L, Loving TJ, Lebel EP (2014) Enhancing transparency of the research process to increase accuracy of findings: A guide for relationship researchers. Pers Relatsh 21: 531–545.
53. Cockburn A (2017) Long-term data as infrastructure: A comment on Ihle et al. Behav Ecol 28:357.
54. Fafchamps M, Labonne J (2016) Using split samples to improve inference about causal effects (National Bureau of Economic Research, Cambridge, MA).
55. Platt JR (1964) Strong inference: Certain systematic methods of scientific thinking May produce much more rapid progress than others. Science 146:347–353.
56. Holm S (1979) A simple sequentially rejective multiple test procedure. Scand J Stat 6: 65–70.
57. Anderson MS, Martinson BC, De Vries R (2007) Normative dissonance in science: Results from a national survey of U.S. Scientists. J Empir Res Hum Res Ethics 2:3–14.
58. Kidwell MC, et al. (2016) Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLoS Biol 14:e1002456.
59. Chambers CD, Feredoes E, Muthukumaraswamy SD, Etchells P (2014) Instead of “playing the game” it is time to change the rules: Registered reports at AIMS Neuroscience and beyond. AIMS Neurosci 1:4–17.
60. Nosek BA, Lakens D (2014) Registered reports: A method to increase the credibility of published results. Soc Psychol 45:137–141.