Internal conceptual replications do not increase independent replication success

Psychonomic Bulletin and Review

Volumn 23, Issue 5, 2016, Pages 1631-1638

  Kunert, Richard ^a,b  

^a MAX PLANCK INSTITUTE FOR PSYCHOLINGUISTICS   (Netherlands)

^b RADBOUD UNIVERSITY   (Netherlands)

Author keywords

False positives; Publication bias; QRP; Replication; Reproducibility

Indexed keywords

BEHAVIORAL RESEARCH; FALSE POSITIVE RESULT; HUMAN; METHODOLOGY; PROCEDURES; PSYCHOLOGY; PUBLISHING; REPRODUCIBILITY;

BEHAVIORAL RESEARCH; FALSE POSITIVE REACTIONS; HUMANS; PSYCHOLOGY, EXPERIMENTAL; PUBLICATION BIAS; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN;

EID: 84962909407     PISSN: 10699384     EISSN: 15315320     Source Type: Journal    
DOI: 10.3758/s13423-016-1030-9     Document Type: Article

References (39)

1. Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R., Nosek, B. A., Perugini, M., Roberts, B. W., Schmitt, M., Vanaken, M. A. G., Weber, H., Wicherts, J. M et al. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27(2), 108–119. doi:10.1002/per.1919
2. Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7(6), 543–554. doi:10.1177/1745691612459060
3. Barrett, L. F. (2015). Psychology is not in crisis. The New York Times. Retrieved from http://www.nytimes.com/2015/09/01/opinion/psychology-is-not-in-crisis.html
4. Cesario, J. (2014). Priming, replication, and the hardest science. Perspectives on Psychological Science, 9(1), 40–48. doi:10.1177/1745691613513470
5. Chambers, C. D. (2013). Registered reports: A new publishing initiative at Cortex. Cortex, 49(3), 609–610. doi:10.1016/j.cortex.2012.12.016
6. Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155.
7. de Groot, A. D. (1956). The meaning of “significance” for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas]. Acta Psychologica, 148, 188–194. doi:10.1016/j.actpsy.2014.02.001
8. DGPS. (2015). Replikationen von Studien sichern Qualität in der Wissenschaft und bringen die Forschung voran. http://www.dgps.de/index.php?id=143&tx_ttnews[tt_news]=1630&cHash=6734f2c28f16dbab9de4871525b29a06. Accessed 28 September 2015
9. Etz, A., & Vandekerckhove, J. (2016). A Bayesian perspective on the reproducibility project: Psychology. PLoS ONE, 11(2), e0149794. doi:10.1371/journal.pone.0149794
10. Fiedler, K., & Schwarz, N. (2015). Questionable research practices revisited. Social Psychological and Personality Science, 1948550615612150. doi:10.1177/1948550615612150
11. Fleming Crim, F. (2015). Reliable science: The path to robust research results. http://www.nsf.gov/mps/perspectives/index.jsp. Accessed 28 September 2015
12. Francis, G. (2014). The frequency of excess success for articles in Psychological Science. Psychonomic Bulletin & Review, 21(5), 1180–1187. doi:10.3758/s13423-014-0601-x
13. Francis, G., Tanzman, J., & Matthews, W. J. (2014). Excess success for psychology articles in the journal Science. PLoS ONE, 9(12), e114255. doi:10.1371/journal.pone.0114255
14. Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505. doi:10.1126/science.1255484
15. Franco, A., Malhotra, N., & Simonovits, G. (2016). Underreporting in psychology experiments: Evidence from a study registry. Social Psychological and Personality Science, 7(1), 8–12. doi:10.1177/1948550615598377
16. Greve, W., Bröder, A., & Erdfelder, E. (2013). Result-blind peer reviews and editorial decisions: A missing pillar of scientific culture. European Psychologist, 18(4), 286–294. doi:10.1027/1016-9040/a000144
17. Guan, M., & Vandekerckhove, J. (2015). A Bayesian approach to mitigation of publication bias. Psychonomic Bulletin & Review. doi:10.3758/s13423-015-0868-6
18. Gunel, E., & Dickey, J. (1974). Bayes factors for independence in contingency tables. Biometrika, 61(3), 545–557. doi:10.2307/2334738
19. Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014). Publication and other reporting biases in cognitive sciences: Detection, prevalence, and prevention. Trends in Cognitive Sciences, 18(5), 235–241. doi:10.1016/j.tics.2014.02.010
20. Jeffreys, H. (1961). Theory of probability. New York: Oxford University Press.
21. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 0956797611430953.
22. Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196–217. doi:10.1207/s15327957pspr0203_4
23. Kruschke, J. K. (2013). Bayesian estimation supersedes the t test. Journal of Experimental Psychology. General, 142(2), 573–603. doi:10.1037/a0029146
24. Lakens, D. (2014). Performing high-powered studies efficiently with sequential analyses. European Journal of Social Psychology, 44(7), 701–710. doi:10.1002/ejsp.2023
25. LeBel, E. P., Borsboom, D., Giner-Sorolla, R., Hasselman, F., Peters, K. R., Ratliff, K. A., & Smith, C. T. (2013). PsychDisclosure.org. Grassroots Support for reforming reporting standards in psychology. Perspectives on Psychological Science, 8(4), 424–432. doi:10.1177/1745691613491437
26. Meredith, M., & Kruschke, J. K. (2015). Package “BEST” (Version 0.4.0). https://cran.r-project.org/web/packages/BEST/index.html
27. Morey, R. D., Rouder, J. N., & Jamil, T. (2015). Package “BayesFactor” (Version 0.9.12-2). http://bayesfactorpcl.r-forge.r-project.org/
28. Nosek, B. A., & Lakens, D. (2014). Registered reports. Social Psychology, 45(3), 137–141. doi:10.1027/1864-9335/a000192
29. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. doi:10.1126/science.aac4716
30. Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. doi:10.1037/0033-2909.86.3.638
31. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16(2), 225–237. doi:10.3758/PBR.16.2.225
32. Sanborn, A. N., & Hills, T. T. (2013). The frequentist implications of optional stopping on Bayesian hypothesis tests. Psychonomic Bulletin & Review, 21(2), 283–300. doi:10.3758/s13423-013-0518-9
33. Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple-study articles. Psychological Methods, 17(4), 551–566. doi:10.1037/a0029487
34. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 0956797611417632.
35. Simons, D. J. (2014). The value of direct replication. Perspectives on Psychological Science, 9(1), 76–80. doi:10.1177/1745691613514755
36. Smulders, Y. M. (2013). A two-step manuscript submission process can reduce publication bias. Journal of Clinical Epidemiology, 66(9), 946–947.
37. Stroebe, W., & Strack, F. (2014). The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science, 9(1), 59–71. doi:10.1177/1745691613514450
38. Wagenmakers, E.-J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14(5), 779–804. doi:10.3758/BF03194105
39. Walster, G. W., & Cleary, T. A. (1970). A proposal for a new editorial policy in the social sciences. The American Statistician, 24(2), 16–19. doi:10.2307/2681924