Conceptualizing and evaluating the replication of research results

Journal of Experimental Social Psychology

Volumn 66, Issue , 2016, Pages 68-80

  Fabrigar, Leandre R ^a   Wegener, Duane T ^b  

^a QUEEN S UNIVERSITY   (Canada)

^b OHIO STATE UNIVERSITY   (United States)

Author keywords

Meta analysis; Psychometrics; Replication

Indexed keywords

EID: 84940689002     PISSN: 00221031     EISSN: 10960465     Source Type: Journal    
DOI: 10.1016/j.jesp.2015.07.009     Document Type: Article

References (38)

1. Borenstein, M., Hedges, L.V., Higgins, J.P.T., Rothstein, H.R., Introduction to meta-analysis. 2009, John Wiley & Sons Ltd., Chichester, West Sussex, UK.
2. Braver, S.L., Thoemmes, F.J., Rosenthal, R., Continuously cumulating meta-analysis and replicability. Perspectives on Psychological Science 9 (2014), 333–342.
3. Brewer, M.B., Crano, W.D., Research design and issues of validity. Reis, H.T., Judd, C.M., (eds.) Handbook of research methods in social and personality psychology, 2nd ed., 2014, Cambridge University Press, New York, NY, 11–26.
4. Cesario, J., Priming, replication, and the hardest science. Perspectives on Psychological Science 9 (2014), 40–48.
5. Cook, T.D., Campbell, D.T., Quasi-experimentation: Design and analysis issues for field settings. 1979, Rand McNally College Publishing Company, Chicago, IL.
6. Elliot, A.J., Greitemeyer, T., Gramzow, R.H., Kayser, D.N., Lichtenfeld, S., Maier, M.A., et al. Red, rank, and romance, in women viewing men. Journal of Experimental Psychology. General 139 (2010), 399–417.
7. Francis, G., The psychology of replication and replication in psychology. Perspectives on Psychological Science 7 (2012), 585–594.
8. Francis, G., Publication bias in “Red, rank, and romance in women viewing men”, by Elliot et al. (2010). Journal of Experimental Psychology. General 142 (2013), 292–296.
9. Francis, G., Replication, statistical consistency, and publication bias. Journal of Mathematical Psychology 57 (2013), 153–169.
10. Galak, J., Meyvis, T., You could have just asked: Reply to Francis (2012). Perspectives on Psychological Science 7 (2012), 595–596.
11. Gorsuch, R.L., Factor analysis. 2nd ed., 1983, Lawrence Erlbaum Associates, Hillsdale, NJ.
12. Iyengar, S., Greenhouse, J.B., Selection models and the file drawer problem. Statistical Science 3 (1988), 109–135.
13. LeBel, E.P., Peters, K.R., Fearing the future of empirical psychology: Bem's evidence of psi as a case study of deficiencies in modal research practices. Review of General Psychology 15 (2011), 371–379.
14. Millsap, R.E., Meredith, W., Factorial invariance: Historical perspectives and new problems. Cudeck, R., MacCallum, R.C., (eds.) Factor analysis at 100: Historical developments and future directions, 2007, Lawrence Erlbaum Associates, Mahwah, NJ, 131–152.
15. Mullen, B., Muellerleile, P., Bryant, B., Cumulative meta-analysis: A consideration of indicators of sufficiency and stability. Personality and Social Psychology Bulletin 27 (2001), 1450–1462.
16. Nosek, B.A., Spies, J.R., Motyl, M., Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science 7 (2012), 615–631.
17. Open Science Collaboration. An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science 7 (2012), 657–660.
18. Pashler, H., Harris, C.R., Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science 7 (2012), 631–636.
19. Petty, R.E., Cacioppo, J.T., Issue-involvement can increase or decrease persuasion by enhancing message-relevant cognitive responses. Journal of Personality and Social Psychology 37 (1979), 1915–1926.
20. Petty, R.E., Cacioppo, J.T., Communication and persuasion: Central and peripheral routes to attitude change. 1986, Springer-Verlag, New York, NY.
21. Rosenberg, M.S., The file-drawer problem revisited: A general weighted method for calculating fail-safe numbers in meta-analysis. Evolution 59 (2005), 464–468.
22. Rosenthal, R., The file drawer problem and tolerance for null results. Psychological Bulletin 86 (1979), 638–641.
23. Rosenthal, R., Replication in behavioral research. Neulip, J.W., (eds.) Handbook of replication research in the behavioral and social sciences, 1990, Select Press, Corta Madera, CA, 1–30.
24. Rosenthal, R., Meta-analytic procedures for social research. Revised ed., 1991, Sage Publications, Newbury Park, CA.
25. Schmidt, S., Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology 13 (2009), 90–100.
26. Schimmack, U., The ironic effect of significant results on the credibility of multiple-study articles. Psychological Methods 17 (2012), 551–566.
27. Simmons, J.P., Nelson, L.D., Simonsohn, U., False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22 (2011), 1359–1366.
28. Simons, D.J., The value of direct replication. Perspectives on Psychological Science 9 (2014), 76–80.
29. Simonsohn, U., It does not follow: Evaluating the one-off publication bias critiques by Francis (2012a, 2012b, 2012c, 2012d, 2012e, in press). Perspectives on Psychological Science 7 (2012), 597–599.
30. Simonsohn, U., It really does not follow, comments on Francis (2013). Journal of Mathematical Psychology 57 (2013), 174–176.
31. Simonsohn, U., Small telescopes: Detectability and the evaluation of replication results. Psychological Science 26 (2015), 559–569.
32. Smith, E.R., (pp. 27–48) Reis, H.T., Judd, C.M., (eds.) Research design, 2nd, 2014, Cambridge University Press, New York, NY.
33. Stanley, D.J., Spence, J.R., Expectations for replications: Are yours realistic?. Perspectives on Psychological Science 9 (2014), 305–318.
34. Stroebe, W., Are most published social psychological findings false?. Journal of Experimental Social Psychology 66 (2016), 134–144 (this issue).
35. Stroebe, W., Strack, F., The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science 9 (2014), 59–71.
36. Verhagen, J., Wagenmakers, E., Bayesian tests to quantify the result of a replication attempt. Journal of Experimental Psychology. General 143 (2014), 1457–1475.
37. Widaman, K.F., Grimm, K.J., Advanced psychometrics: Confirmatory factor analysis, item response theory, and the study of measurement invariance. Reis, H.T., Judd, C.M., (eds.) Handbook of research methods in social and personality psychology, 2nd ed., 2014, Cambridge University Press, New York, NY, 534–570.
38. Widaman, K.F., Reise, S.P., Exploring the measurement invariance of psychological instruments: Applications in the substance use domain. Bryant, K.J., Windle, M., West, S.G., (eds.) The science of prevention: Methodological advances from alcohol and substance abuse research, 1997, American Psychological Association, Washington, DC, 281–324.