Pre-registration in social psychology—A discussion and suggested template

Journal of Experimental Social Psychology

Volumn 67, Issue , 2016, Pages 2-12

  van 't Veer, Anna Elisabeth ^a,b   Giner Sorolla, Roger ^c  

^a LEIDEN UNIVERSITY   (Netherlands)

^b TILBURG UNIVERSITY   (Netherlands)

^c UNIVERSITY OF KENT   (United Kingdom)

Author keywords

Pre registration; Research methods; Reviewed pre registration (RPR); Solid science; Unreviewed pre registration (UPR)

Indexed keywords

HUMAN; REGISTRATION; SCIENTIST;

EID: 84990050130     PISSN: 00221031     EISSN: 10960465     Source Type: Journal    
DOI: 10.1016/j.jesp.2016.03.004     Document Type: Article

References (72)

1. APA Publications and Communications Board Working Group on Journal Article Reporting Standards, Reporting standards for research in psychology: hy do we need them? What might they be?. American Psychologist 63:9 (2008), 839–851.
2. Badges to Acknowledge Open Practices, 2013, February 19 Retrieved from https://osf.io/tvyxz/.
3. Bakker, M., Wicherts, J.M., Outlier removal, sum scores, and the inflation of the type I error rate in independent samples t tests: The power of alternatives and recommendations. Psychological Methods 19:3 (2014), 409–427, 10.1037/met0000014.
4. Bateman, I., Kahneman, D., Munro, A., Starmer, C., Sugden, R., Testing competing models of loss aversion: An adversarial collaboration. Journal of Public Economics 89:8 (2005), 1561–1580.
5. Begg, C.B., Berlin, J.A., Publication bias: A problem in interpreting medical data. Journal of the Royal Statistical Society. Series A (Statistics in Society) 151:3 (1988), 419–463, 10.2307/2982993.
6. Begley, C.G., Ellis, L.M., Drug development: Raise standards for preclinical cancer research. Nature 483:7391 (2012), 531–533, 10.1038/483531a.
7. Bosco, F.A., Aguinis, H., Field, J.G., Pierce, C.A., Dalton, D.R., HARKing's threat to organizational research: Evidence from primary and meta-analytic sources. Personnel Psychology, 2015, 10.1111/peps.12111.
8. Bostyn, D.H., Roets, A., The morality of action: The asymmetry between judgments of praise and blame in the action–omission effect. Journal of Experimental Social Psychology 63 (2016), 19–25, 10.1016/j.jesp.2015.11.005.
9. Brandt, M.J., IJzerman, H., Dijksterhuis, A., Farach, F.J., Geller, J., Giner-Sorolla, R., van ’t Veer, A.E., The replication recipe: What makes for a convincing replication?. Journal of Experimental Social Psychology 50 (2014), 217–224, 10.1016/j.jesp.2013.10.005.
10. Carp, J., The secret lives of experiments: Methods reporting in the fMRI literature. NeuroImage 63:1 (2012), 289–300, 10.1016/j.neuroimage.2012.07.004.
11. Challenging traditions in research reporting, 2014, July Retrieved from http://newsroom.taylorandfrancisgroup.com/news/press-release/comprehensive-results-social-psychology#.VXa3YlKGNgs.
12. Chambers, C., Munafò, M., Trust in science would be improved by study pre-registration., 2013, June 5 Retrieved from http://www.theguardian.com/science/blog/2013/jun/05/trust-in-science-study-pre-registration.
13. Chambers, C.D., Feredoes, E., Muthukumaraswamy, S.D., Etchells, P.J., Instead of “playing the game” it is time to change the rules: Registered reports at AIMS Neuroscience and beyond. AIMS Neuroscience 1:1 (2014), 4–17, 10.3934/Neuroscience2014.1.4.
14. Clinical Trials Registration and Results Submission, 79 Fed. Reg. 225 (2014, November 21).
15. Clinical Trials - Regulation EU No 536/2014, 2014 Retrieved from http://ec.europa.eu/health/human-use/clinical-trials/regulation/index_en.htm.
16. Cumming, G., The new statistics: Why and how. Psychological Science 25:1 (2014), 7–29, 10.1177/0956797613504966.
17. de Groot, A.D., The meaning of “significance” for different types of research. Translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas. Acta Psychologica 148 (1956/2014), 188–194.
18. Dickerson, K., Rennie, D., Registering clinical trials. Journal of the American Medical Association 290:4 (2003), 516–523, 10.1001/jama.290.4.516.
19. Dwan, K., Altman, D.G., Arnaiz, J.A., Bloom, J., Chan, A.W., Cronin, E., Williamson, P.R., Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PloS One 3:8 (2008), 1–31, 10.1371/journal.pone.0003081.
20. Easterbrook, P.J., Berlin, J.A., Gopalan, R., Matthews, D.R., Publication bias in clinical research. The Lancet 337:8746 (1991), 867–872, 10.1016/0140-6736(91)90201-Y.
21. Ebersole, C.R., Atherton, O.E., Belanger, A.L., Skulborstad, H.M., Allen, J.M., Banks, J.B., Nosek, B.A., Many labs 3: Evaluating participant pool quality across the academic semester via replication. Journal of Experimental Social Psychology 67 (2016), 68–82 (in this issue).
22. Eich, E., Business not as usual. Psychological Science 25:1 (2014), 3–6, 10.1177/0956797613512465.
23. Fabrigar, L.R., Wegener, D.T., Conceptualizing and evaluating the replication of research results. Journal of Experimental Social Psychology 66 (2016), 68–80.
24. Fanelli, D., Do pressures to publish increase scientists' bias? An empirical support from US states data. PloS One, 5(4), 2010, e10271, 10.1371/journal.pone.0010271.
25. Ferguson, C.J., Heene, M., A vast graveyard of undead theories: Publication bias and psychological science's aversion to the null. Perspectives on Psychological Science 7:6 (2012), 555–561, 10.1177/1745691612459059.
26. Food and Drug Administration Amendments Act of, 121 U.S. Stat. 2007, 823.
27. Forrester, M.A., (eds.) Doing qualitative research in psychology: A practical guide, 2010, Sage, Londen, England.
28. Funder, D.C., Levine, J.M., Mackie, D.M., Morf, C.C., Sansone, C., Vazire, S., West, S.G., Improving the dependability of research in personality and social psychology: Recommendations for research and educational practice. Personality and Social Psychology Review 18 (2014), 3–12, 10.1177/1088868313507536.
29. Gelman, A., Loken, E., The statistical crisis in science. American Scientist 102:6 (2014), 460–465, 10.1511/2014.111.460.
30. Giner-Sorolla, R., Approaching a fair deal for significance and other concerns. Journal of Experimental Social Psychology 65 (2016), 1–6, 10.1016/j.jesp.2016.01.010.
31. Giner-Sorolla, R., Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspectives on Psychological Science 7:6 (2012), 562–571, 10.1177/1745691612457576.
32. Hung, H.M.J., O'Neill, R.T., Bauer, P., Kohne, K., The behavior of the p-value when the alternative hypothesis is true. Biometrics 53:1 (1997), 11–22, 10.2307/2533093.
33. Ioannidis, J.P.A., Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA: The Journal of the American Medical Association 279:4 (1998), 281–286, 10.1001/jama.279.4.281.
34. Jonas, K.J., Cesario, J., How can preregistration contribute to research in our field?. Comprehensive Results in Social Psychology, 2015, 1–7, 10.1080/23743603.2015.1070611.
35. Kahneman, D., Experiences of collaborative research. The American Psychologist 58:9 (2003), 723–730, 10.1037/0003-066X.58.9.723.
36. Kaplan, R.M., Irvin, V.L., Likelihood of null effects of large NHLBI clinical trials has increased over time. PloS One, 10(8), 2015, e0132382, 10.1371/journal.pone.0132382.
37. Kerr, N.L., HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review 2:3 (1998), 196–217, 10.1207/s15327957pspr0203_4.
38. Klein, R.A., Ratliff, K.A., Vianello, M., Adams, R.B., Bahník, Š., Bernstein, M.J., Nosek, B.A., Investigating variation in replicability: A “many labs” replication project. Social Psychology 45:3 (2014), 142–152, 10.1027/1864-9335/a000178.
39. Lai, C.K., Marini, M., Lehr, S.A., Cerruti, C., Shin, J.E.L., Joy-Gaba, J.A., Frazier, R.S., Reducing implicit racial preferences: I. A comparative investigation of 17 interventions. Journal of Experimental Psychology: General 143:4 (2014), 1765–1785, 10.1037/a0036260.
40. Lakens, D., Performing high-powered studies efficiently with sequential analyses. European Journal of Social Psychology 44 (2014), 701–710, 10.1002/ejsp.2023.
41. LeBel, E.P., Borsboom, D., Giner-Sorolla, R., Hasselman, F., Peters, K.R., Ratliff, K.A., Smith, C.T., PsychDisclosure.org: Grassroots support for reforming reporting standards in psychology. Perspectives on Psychological Science 8:4 (2013), 424–432, 10.1177/1745691613491437.
42. LeBel, E.P., Peters, K.R., Fearing the future of empirical psychology: Bem's (2011) evidence of psi as a case study of deficiencies in modal research practice. Review of General Psychology 15:4 (2011), 371–379, 10.1037/a0025172.
43. Lerner, J.S., Tetlock, P.E., Accounting for the effects of accountability. Psychological Bulletin 125:2 (1999), 255–275, 10.1037/0033-2909.125.2.255.
44. Leys, C., Ley, C., Klein, O., Bernard, P., Licata, L., Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median. Journal of Experimental Social Psychology 49:4 (2013), 764–766, 10.1016/j.jesp.2013.03.013.
45. Mahtani, K., How often are outcomes switched in clinical trials? And why does it matter?., 2016, February 5 Retrieved from http://compare-trials.org/blog/are-your-results-unusual-or-how-often-are-outcomes-switched/.
46. Matzke, D., Nieuwenhuis, S., van Rijn, H., Slagter, H.A., van der Molen, M.W., Wagenmakers, E.-J., The effect of horizontal eye movements on free recall: A preregistered adversarial collaboration. Journal of Experimental Psychology: General 144:1 (2015), e1–e15, 10.1037/xge0000038.
47. Meehl, P.E., Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science 34:2 (1967), 103–115.
48. Meehl, P.E., Why summaries of research on psychological theories are often uninterpretable. Psychological Reports 66:1 (1990), 195–244, 10.2466/pr0.1990.66.1.195.
49. Melander, H., Ahlqvist-Rastad, J., Meijer, G., Beermann, B., Evidence b(i)ased medicine—Selective reporting from studies sponsored by pharmaceutical industry: Review of studies in new drug applications. BMJ 326:7400 (2003), 1171–1173, 10.1136/bmj.326.7400.1171.
50. Mellers, B., Hertwig, R., Kahneman, D., Do frequency representations eliminate conjunction effects? An exercise in adversarial collaboration. Psychological Science 12:4 (2001), 269–275.
51. Mullane, K., Williams, M., Alzheimer's therapeutics: Continued clinical failures question the validity of the amyloid hypothesis — But what lies beyond?. Biochemical Pharmacology 85:3 (2013), 289–305, 10.1016/j.bcp.2012.11.014.
52. Murayama, K., Pekrun, R., Fiedler, K., Research practices that can prevent an inflation of false-positive rates. Personality and Social Psychology Review 18:2 (2013), 107–118, 10.1177/1088868313496330.
53. Nier, J.A., Campbell, S.D., Two outsiders' view on feminism and evolutionary psychology: An opportune time for adversarial collaboration. Sex Roles 69:9–10 (2013), 503–506, 10.1007/s11199-012-0154-2.
54. Nosek, B.A., Bar-Anan, Y., Scientific utopia: I. Opening scientific communication. Psychological Inquiry 23:3 (2012), 217–243, 10.1080/1047840X.2012.692215.
55. Nosek, B.A., Lakens, D., Registered reports: A method to increase the credibility of published results. Social Psychology 45:3 (2014), 137–141, 10.1027/1864-9335/a000192.
56. Open Science Collaboration. An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science 7:6 (2012), 657–660, 10.1177/1745691612462588.
57. Perugini, M., Gallucci, M., Costantini, G., Safeguard power as a protection against imprecise power estimates. Perspectives on Psychological Science 9:3 (2014), 319–332, 10.1177/1745691614528519.
58. Prinz, F., Schlange, T., Asadullah, K., Believe it or not: How much can we rely on published data on potential drug targets? Nature Reviews. Drug Discovery, 10(9), 2011, 712, 10.1038/nrd3439-c1.
59. Rakow, T., Thompson, V., Ball, L., Markovits, H., Rationale and guidelines for empirical adversarial collaboration: A thinking & reasoning initiative. Thinking & Reasoning 21:2 (2014), 167–175, 10.1080/13546783.2015.975405.
60. Rosner, B., On the detection of many outliers. Technometrics 17:2 (1975), 221–227, 10.2307/1268354.
61. Rousseeuw, P.J., van Zomeren, B.C., Unmasking multivariate outliers and leverage points. Journal of the American Statistical Association 85:411 (1990), 633–639, 10.1080/01621459.1990.10474920.
62. Simes, R.J., Publication bias: The case for an international registry of clinical trials. Journal of Clinical Oncology 4:10 (1986), 1529–1541.
63. Simmons, J.P., Nelson, L.D., Simonsohn, U., False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22:11 (2011), 1359–1366, 10.1177/0956797611417632.
64. Stern, J.M., Simes, R.J., Publication bias: Evidence of delayed publication in a cohort study of clinical research projects. BMJ 315:7109 (1997), 640–645, 10.1136/bmj.315.7109.640.
65. Tukey, J.W., We need both exploratory and confirmatory. The American Statistician 34:1 (1980), 23–25, 10.1080/00031305.1980.10482706.
66. Turner, E.H., Matthews, A.M., Linardatos, B.S., Tell, R.A., Rosenthal, R., Selective publication of antidepressant trials and its influence on apparent efficacy. The New England Journal of Medicine 358:3 (2008), 252–260.
67. van ’t Veer, A.E., Gallucci, M., Stel, M., van Beest, I., Unconscious deception detection measured by finger skin temperature and indirect veracity judgments—Results of a registered report. Frontiers in Psychology, 6(672), 2015, 10.3389/fpsyg.2015.00672.
68. Vazire, S., Editorial. Social Psychological and Personality Science 7:1 (2015), 3–7, 10.1177/1948550615603955.
69. Wagenmakers, E.-J., A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review 14:5 (2007), 779–804, 10.3758/BF03194105.
70. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H.L.J., Kievit, R.A., An agenda for purely confirmatory research. Perspectives on Psychological Science 7:6 (2012), 632–638, 10.1177/1745691612463078.
71. Wasserstein, R.L., Lazar, N.A., The ASA's statement on p-values: Context, process, and purpose. The American Statistician, 2016, 10.1080/00031305.2016.1154108 (in press).
72. World Medical Association, WMA Declaration of Helsinki — Ethical principles for medical research involving human subjects.., 2008 Retrieved July 13, 2015 from http://www.wma.net/en/30publications/10policies/b3.