Addressing the “Replication Crisis”: Using Original Studies to Design Replication Studies with Appropriate Statistical Power

Multivariate Behavioral Research

Volumn 52, Issue 3, 2017, Pages 305-324

  Anderson, Samantha F ^a   Maxwell, Scott E ^a  

^a UNIVERSITY OF NOTRE DAME   (United States)

Author keywords

Effect size; power; replication; sample size planning

Indexed keywords

CALCULATION; EFFECT SIZE; HUMAN; HUMAN EXPERIMENT; PSYCHOLOGIST; PUBLISHING; REPLICATION STUDY; SAMPLE SIZE; SIMULATION; UNCERTAINTY; COMPUTER SIMULATION; METHODOLOGY; PROCEDURES; PSYCHOLOGY; REPRODUCIBILITY; SOFTWARE; STATISTICS;

COMPUTER SIMULATION; HUMANS; PSYCHOLOGY; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; SOFTWARE; STATISTICS AS TOPIC;

EID: 85014553788     PISSN: 00273171     EISSN: None     Source Type: Journal    
DOI: 10.1080/00273171.2017.1289361     Document Type: Article

References (43)

1. Alogna, V. K., Attaya, M. K., Aucoin, P., Bahnik, S., Birch, S., Birt, A. R., … Zwaan, R. A., (2014). Registered replication report:Schooler & Engstler-Schooler (1990). Perspectives on Psychological Science, 9, 556–578. doi:10.1177/1745691614545653
2. Anderson, S. F., & Maxwell, S. E., (2016). There's more than one way to conduct a replication study:Beyond statistical significance. Psychological Methods, 21, 1–12. doi:10.1037/met0000051
3. Bakker, M., vanDijk, A., & Wicherts, J. M., (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543–554. doi:10.1177/1745691612459060
4. Batterham, A. M, & Hopkins, W. G., (2013). Emergence of large treatment effects from small trials. JAMA, 309, 768–769. doi:10.1001/jama.2012.208828
5. Brand, A., Bradley, M. T., Best, L. A., & Stoica, G., (2008). Accuracy of effect size estimates from published psychological research. Perceptual and Motor Skills, 106, 645–649. doi:10.2466/pms.106.2.645-649
6. Brandt, M. J., Ijzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R, … Veer, A., (2014). The replication recipe:What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217–224. doi:10.1016/j.esp.2013.10.005
7. Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munaf, M. R., (2013). Power failure:Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 1–12. doi:10.1038/nrn3475
8. Cohen, J., (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ:Lawrence Erlbaum Associates.
9. Cook, T. D., Shadish, W. R., & Wong, V. C., (2008). Three conditions under which experiments and observational studies produce comparable causal estimates:New findings from within-study comparisons. Journal of Policy Analysis and Management, 27, 724–750. doi:10.1002/pam.20375
10. Dallow, N., & Fina, P., (2011). The perils with the misuse of predictive power. Pharmaceutical Statistics, 10, 311–317. doi:10.1002/pst.467
11. Dawes, R. M., (2004). Commentary on Meehl's theoretical risks and tabular asterisks:Sir Karl, Sir Ronald, and the slow progress of soft psychology. Applied and Preventive Psychology, 11, 23–25. doi:10.1016/j.appsy.2004.02.002
12. Fraley, R. C., & Vazire, S., (2014). The N-pact factor:Evaluating the quality of empirical journals with respect to sample size and statistical power. PLOS One, 9, e109019. doi:10.1371/journal.pone.0109019
13. Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D., (2016). Comment on “estimating the reproducibility of psychological science”. Science, 351, 1037–b. doi:10.1126/science.aad7243
14. Gelman, A., & Carlin, J., (2014). Beyond power calculations:Assessing type S (sign) and type M (magnitude) errors. Perspectives on Psychological Science, 9, 641–651. doi:10.10177/1745691614551642
15. Gelman, A., & Loken, E., (2014) The statistical crisis in science. American Scientist, 102, 460–465. doi:10.1511/2014.111.460
16. Hedges, L., (1984). Estimation of effect size under nonrandom sampling:The effects of censoring studies yielding statistically insignificant mean differences. Journal of Educational and Behavioral Statistics, 9, 61–85.
17. Ioannidis, J. P. A., (2005). Why most published research findings are false. PLoS Medicine, 2, e124. doi/10.1371/journal.pmed.0020124
18. Ioannidis, J. A, Pereira, T. V, & Horwitz, R. I., (2013). Emergence of large treatment effects from small Trials—Reply. JAMA, 309, 768–769. doi:10.1001/jama.2012.208831
19. Jaccard, J., & Wan, C. K., (1995). Measurement error in the analysis of interaction effects between continuous predictors using multiple regression:Multiple indicator and structural equation approaches. Psychological Bulletin, 117, 348– 357.
20. Jasny, B. R., Chin, G., Chong, L., & Vignieri, S., (2011). Again, and again, and again… Science, 334, 1225. doi:10.1126/science.334.6060.1225
21. Kerr, N. L., (1998). HARKing:Hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196–217. doi:10.1207/s15327957pspr0203_4
22. Lane, D. M., & Dunlap, W. P., (1978). Estimating effect size:Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical Psychology, 31, 107–112.
23. Liu, X. S., (2014). Statistical power analysis for the social and behavioral sciences:Basic and advanced techniques. New York:Taylor and Francis.
24. Maddock, J. E., & Rossi, J. S., (2001). Statistical power of articles published in three health psychology-related journals. Health Psychology, 20, 76–78. doi:10.1037/0278-6133.20.1.76
25. Maxwell, S. E., Kelley, K., & Rausch, J. R., (2008). Sample size planning for statistical power and accuracy in parameter estimation. Annual Review of Psychology, 59, 537–563. doi:10.1146/annurev.psych.59.103006.093735
26. Maxwell, S. E., Lau, M. Y., & Howard, G. S., (2015). Is psychology suffering from a replication crisis? What does failure to replicate really mean? American Psychologist, 70, 487–498. doi:10.1037/a0039400
27. McShane, B. B., & Bockenholt, U., (2014). You cannot step into the same river twice:When power analyses are optimistic. Perspectives on Psychological Science, 9, 612–625. doi:10.1177/1745691614548513
28. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716. doi:10.1126/science.aac4716
29. Open Science Framework. (2015). Reproducibility project:Researcher guide. Retrieved from https://osf.io/ru689/
30. Pereira, T. V., Horwitz, R. I., & Ioannidis, J. P. A., (2012). Empirical evaluation of very large treatment effects of medical interventions. JAMA, 308, 1676–1684. doi:10.1001/jama.2012.13444
31. Perugini, M., Gallucci, M., & Constantini, G., (2014). Safeguard power as a protection against imprecise power estimates. Perspectives on Psychological Science, 9, 319–332. doi:10.1177/1745691614528519
32. Prentice, D. A., & Miller, D. T., (1992). When small effects are impressive. Psychological Bulletin, 112, 160–164. doi:10.1037/0033-2909.112.1.160
33. Prinz, F., Schlange, T., & Asadullah, K., (2011). Believe it or not:How much can we rely on published data on potential drug targets? Nature Reviews Drug Discovery, 10, 712–713.
34. Richard, F., Bond, C., & Stokes-Zoota, J., (2003). One hundred years of social psychology quantitatively described. Review of General Psychology, 7, 331–363 doi:10.1037/1089-2680.7.4.331
35. Sedlmeier, P., & Gigerenzer, G., (1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105, 309–316.
36. Shadish, W. R., Cook, T. D., & Campbell, D. T., (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA:Houghton Mifflin.
37. Simmons, J. P., Nelson, L. D., & Simonsohn, U., (2011). False-positive psychology:Undisclosed flexibility in data-collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. doi:10.1177/0956797611417632
38. Simons, D. J., (2014). The value of direct replication. Perspectives on Psychological Science, 9, 76–80. doi:10.1177/1745691613514755
39. Simonsohn, U., (2015). Small telescopes:Detectability and the evaluation of replication results. Psychological Science, 26, 559–569. doi:10.1177/0956797614567341
40. Stanley, D. J., & Spence, J. R., (2014). Expectations for replications:Are yours realistic? Perspectives on Psychological Science, 9, 305–318. doi:10.1177/1745691614528518
41. Talboom, J. S., West, S. G., & Bimonte-Nelson, H. A., (2015). A primer of methods in biobehavioral research:Improving a study's design, analysis, and write up. In H. A., Bimonte-Nelson (Ed.), The maze book:Theories, practice, and protocols for testing rodent cognition. New York, NY:Springer.
42. Taylor, D. J., & Muller, K. E., (1996). Bias in linear model power and sample size calculation due to estimating noncentrality. Communications in Statistics:Theory and Methods, 25, 1595–1610. doi:10.1080/03610929608831787
43. West, S. G., & Thoemmes, F., (2010). Campbell's and Rubin's perspectives on causal inference. Psychological Methods, 15, 18–37. doi:10.1037/a0015917