Recommendations for increasing replicability in psychology

European Journal of Personality

Volumn 27, Issue 2, 2013, Pages 108-119

  Asendorpf, Jens B ^a   Conner, Mark ^b   De Fruyt, Filip ^c   De Houwer, Jan ^c   Denissen, Jaap J A ^d   Fiedler, Klaus ^e   Fiedler, Susann ^f   Funder, David C ^g   Kliegl, Reinhold ^h   Nosek, Brian A ⁱ   Perugini, Marco ^j   Roberts, Brent W ^k   Schmitt, Manfred ^l   Van Aken, Marcel A G ^m   Weber, Hannelore ⁿ   Wicherts, Jelte M ^d  

^a HUMBOLDT UNIVERSITY   (Germany)

^b UNIVERSITY OF LEEDS   (United Kingdom)

^c GHENT UNIVERSITY   (Belgium)

^d TILBURG UNIVERSITY   (Netherlands)

^e UNIVERSITY OF HEIDELBERG   (Germany)

^f MAX PLANCK INSTITUTE FOR RESEARCH ON COLLECTIVE GOODS   (Germany)

^g UNIVERSITY OF CALIFORNIA   (United States)

^h UNIVERSITY OF POTSDAM   (Germany)

ⁱ UNIVERSITY OF VIRGINIA   (United States)

^j UNIVERSITY OF MILANO BICOCCA   (Italy)

^k UNIVERSITY OF ILLINOIS AT CHICAGO   (United States)

^l Rheinland Pfälzische Technische Universität Kaiserslautern Landau   (Germany)

^m UTRECHT UNIVERSITY   (Netherlands)

ⁿ UNIVERSITY OF GREIFSWALD   (Germany)

more..

Author keywords

Confirmation bias; Generalizability; Publication bias; Replicability; Research transparency

Indexed keywords

EID: 84876502847     PISSN: 08902070     EISSN: 10990984     Source Type: Journal    
DOI: 10.1002/per.1919     Document Type: Article

References (43)

1. Bakker, M., Van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543-554.
2. Bem, D. J. (2000). Writing an empirical article. In R. J. Sternberg (Ed.), Guide to publishing in psychology journals (pp. 3-16). Cambridge: Cambridge University Press.
3. Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100, 407-426.
4. Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B (Methodological) 57, 289-300.
5. Brunswik, E. (1955). Representative design and probabilistic theory in a functional psychology. Psychological Review, 62, 193-217.
6. Buckheit, J., & Donoho, D. L. (1995). Wavelab and reproducible research. In A. Antoniadis (ed.). Wavelets and statistics (pp. 55-81). New York, NY: Springer-Verlag.
7. Campbell, D. T. (1960). Blind variation and selective retention in creative thought as in other knowledge processes. Psychological Review, 67, 380-400.
8. Cohen, J. (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65, 145-153.
9. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
10. Cumming, G., & Finch, S. (2005). Inference by eye: Confidence intervals, and how to read pictures of data. The American Psychologist, 60, 170-180.
11. Fanelli, D. (2010). "Positive" results increase down the hierarchy of the sciences. PLoS One, 5, e10068.
12. Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891-904.
13. Fraley, R. C., & Marks, M. J. (2007). The null hypothesis significance-testing debate and its implications for personality research. In R. W. Robins, R. C. Fraley, & R. F. Krueger (Eds.), Handbook of research methods in personality psychology (pp. 149-169). New York: Guilford.
14. Frank, M. C., & Saxe, R. (2012). Teaching replication. Perspectives on Psychological Science, 7, 600-604.
15. Fuchs, H., Jenny, M., & Fiedler, S. (2012). Psychologists are open to change, yet wary of rules. Perspectives on Psychological Science, 7, 639-642.
16. Gigerenzer, G., Swijink, Z., Porter, T., Daston, L., Beatty, J., & Krüger, L. (1989). The empire of chance: How probability changed science and everyday life. Cambridge: Cambridge University Press.
17. Hartshorne, J. K., & Schachner, A. (2012). Tracking replicability as a method of post-publication open evaluation. Frontiers in Computational Science, 6, 1-14.
18. Hox, J. J. (2010). Multilevel analysis (2nd ed.). New York, NY: Routledge.
19. Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2, e124.
20. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological Science, 23, 524-532.
21. Judd, C. M., Westfall, J., & Kenny, D. A. (2012). Treating stimuli as a random factor in social psychology: A new and comprehensive solution to a pervasive but largely ignored problem. Journal of Personality and Social Psychology, 103, 54-69.
22. Kelley, K., & Maxwell, S. E. (2003). Sample size for multiple regression: Obtaining regression coefficients that are accurate, not simply significant. Psychological Methods, 8, 305-321.
23. Kelley, K., & Rausch, J. R. (2006). Sample size planning for the standardized mean difference: Accuracy in parameter estimation via narrow confidence intervals. Psychological Methods, 11, 363-385.
24. Kenny, D. A., Kashy, D. A., & Cook, W. L. (2006). Dyadic data analysis. New York, NY: Guilford Press.
25. King, G. (1995). Replication, replication. PS: Political Science and Politics, 28, 443-499.
26. King, G. (2006). Publication, publication. PS: Political Science and Politics, 34, 119-125.
27. Lee, J. J. (2012). Correlation and causation in the study of personality. European Journal of Personality, 26, 372-390.
28. Lehrer, J. (2010). The truth wears off: Is there something wrong with the scientific method? The New Yorker, December 13.
29. Lipsey, M. W., & Hurley, S. M. (2009). Design sensitivity: Statistical power for applied experimental research. In L. Bickman, & D. J. Rog (Eds.), The SAGE handbook of applied social research methods (pp. 44-76). Los Angeles, CA: SAGE Publications.
30. Maxwell, S. E., Kelley, K., & Rausch, J. R. (2008). Sample size planning for statistical power and accuracy in parameter estimation. Annual Review of Psychology, 59, 537-563.
31. Nakagawa, S. (2004). A farewell to Bonferroni: The problems of low statistical power and publication bias. Behavioral Ecology, 15, 1044-1045.
32. Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science 7, 615-631.
33. Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple-study articles. Psychological Methods 17, 551-566.
34. Sherman, R. A., & Funder, D. C. (2009). Evaluating correlations in studies of personality and behavior: Beyond the number of significant findings to be expected by chance. Journal of Research in Personality, 43, 1053-1061.
35. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366.
36. Simonton, D. K. (2003). Scientific creativity as constrained stochastic behavior: The integration of product, person, and process perspectives. Psychological Bulletin, 129, 475-494.
37. Tressoldi, P. E. (2012). Replication unreliability in psychology: Elusive phenomena or "elusive" statistical power? Frontiers in Psychology, 3. doi: 10.3389/fpsyg.2012.00218
38. Tversky, A., & Kahneman, D. (1971). Belief in the law of small numbers. Psychological Bulletin, 76, 105-110.
39. Valentine, J. C., Biglan, A., Boruch, R. F., Castro, F. G., Collins, L. M., Flay, B. R., ... Schinke, S. P. (2011). Replication in prevention science. Prevention Science, 12, 103-117.
40. Wachtel, P. L. (1980). Investigation and its discontents: Some constraints on progress in psychological research. The American Psychologist, 5, 399-408.
41. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology, 100, 426-432.
42. Yong, E. (2012). Bad copy: In the wake of high-profile controversies, psychologists are facing up to problems with replication. Nature, 485, 298-300.
43. Zimmer, C. (2012). Good scientist! You get a badge. Slate, August 14 (on-line). http://www.slate.com/articles/health_and_science/science/2012/08/reproducing_scientific_studies_a_good_housekeeping_seal_of_approval_.html