There's more than one way to conduct a replication study: Beyond statistical significance

Psychological Methods

Volumn 21, Issue 1, 2016, Pages 1-12

  Anderson, Samantha F ^a   Maxwell, Scott E ^a  

^a UNIVERSITY OF NOTRE DAME   (United States)

Author keywords

Confidence interval; Data analysis; Effect size; Equivalence test; Replication

Indexed keywords

BEHAVIORAL RESEARCH; HUMAN; PROCEDURES; REPRODUCIBILITY; STANDARDS; STATISTICAL ANALYSIS;

BEHAVIORAL RESEARCH; DATA INTERPRETATION, STATISTICAL; HUMANS; REPRODUCIBILITY OF RESULTS;

EID: 84937792319     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/met0000051     Document Type: Article

References (69)

1. Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100, 407-425. http://dx.doi.org/10.1037/a0021524
2. Berger, J., & Delampady, M. (1987). Testing precise hypotheses. Statistical Science, 2, 317-335. http://dx.doi.org/10.1214/ss/1177013238
3. Biesanz, J. C., & Schrager, S. M. (2010). Sample size planning with effect size estimates. Manuscript under review. Vancouver, Canada: University of British Columbia School of Psychology.
4. Bond, C. F., Jr., Wiitala, W. L., & Richard, F. D. (2003). Meta-analysis of raw mean differences. Psychological Methods, 8, 406-418. http://dx.doi .org/10.1037/1082-989X.8.4.406
5. Bonett, D. G. (2008). Meta-analytic interval estimation for bivariate correlations. Psychological Methods, 13, 173-181. http://dx.doi.org/ 10.1037/a0012868
6. Bonett, D. G. (2009). Meta-analytic interval estimation for standardized and unstandardized mean differences. Psychological Methods, 14, 225-238. http://dx.doi.org/10.1037/a0016619
7. Bonett, D. G., & Price, R. M. (2014). Meta-analysis methods for risk differences. British Journal of Mathematical and Statistical Psychology, 67, 371-387. http://dx.doi.org/10.1111/bmsp.12024
8. Bonett, D. G., & Wright, T. A. (2007). Comments and recommendations regarding the hypothesis testing controversy. Journal of Organizational Behavior, 28, 647-659. http://dx.doi.org/10.1002/job.448
9. Braver, S. L., Thoemmes, F. J., & Rosenthal, R. (2014). Continuously cumulating meta-analysis and replicability. Perspectives on Psychological Science, 9, 333-342. http://dx.doi.org/10.1177/1745691614529796
10. Center for Open Science. (2012). Reproducibility project. Retrieved from https://osf.io/ezcuj/wiki/home/
11. Cochran, W. G. (1954). The combination of estimates from different experiments. Biometrics, 10, 101-129. http://dx.doi.org/10.2307/ 3001666
12. Cohn, L. D., & Becker, B. J. (2003). How meta-analysis increases statistical power. Psychological Methods, 8, 243-253. http://dx.doi.org/ 10.1037/1082-989X.8.3.243
13. Collins, H. M. (1985). Changing order. London, UK: SAGE.
14. Doyen, S., Klein, O., Pichon, C. L., & Cleeremans, A. (2012). Behavioral priming: It's all in the mind, but whose mind? PLoS ONE, 7, e29081. http://dx.doi.org/10.1371/journal.pone.0029081
15. Fisher, R. A. (1926). The arrangement of field experiments. Journal of the Ministry of Agriculture of Great Britain, 33, 503-513.
16. Galak, J., Leboeuf, R. A., Nelson, L. D., & Simmons, J. P. (2012). Correcting the past: Failures to replicate _. Journal of Personality and Social Psychology, 103, 933-948. http://dx.doi.org/10.1037/a0029709
17. Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013). Bayesian data analysis (3rd ed.). Boca Raton, FL:CRC Press.
18. Gelman, A., & Stern, H. (2006). The difference between "significant" and "not significant" is not itself statistically significant. The American Statistician, 60, 328-331. http://dx.doi.org/10.1198/ 000313006X152649
19. Goren, C. C., Sarty, M., & Wu, P. Y. K. (1975). Visual following and pattern discrimination of face-like stimuli by newborn infants. Pediatrics, 56, 544-549.
20. Hedges, L. V. (1982). Estimation of effect size from a series of independent experiments. Psychological Bulletin, 92, 490-499. http://dx.doi .org/10.1037/0033-2909.92.2.490
21. Hedges, L. V., & Vevea, J. L. (1998). Fixed-and random-effects models in meta-analysis. Psychological Methods, 3, 486-501. http://dx.doi.org/ 10.1037/1082-989X.3.4.486
22. Higgins, J. P., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. British Medical Journal, 327, 557-560. http://dx.doi.org/10.1136/bmj.327.7414.557
23. Hoijtink, H. (2011). Informative hypotheses: Theory and practice for behavioral and social scientists. Boca Raton, FL: CRC Press. http://dx .doi.org/10.1201/b11158
24. Hoijtink, H., & Klugkist, I. (2007). Comparison of hypothesis testing and Bayesian model selection. Quality & Quantity: International Journal of Methodology, 41, 73-91. http://dx.doi.org/10.1007/s11135-005-6224-6
25. Hsu, L. M. (1980). Tests of differences in p levels as tests of differences in effect sizes. Psychological Bulletin, 88, 705-708. http://dx.doi.org/ 10.1037/0033-2909.88.3.705
26. Hunter, J. E. (2001). The desperate need for replications. Journal of Consumer Research, 28, 149-158. http://dx.doi.org/10.1086/321953
27. Ioannidis, J. P., Patsopoulos, N. A., & Evangelou, E. (2007). Uncertainty in heterogeneity estimates in meta-analyses. British Medical Journal, 335, 914-916. http://dx.doi.org/10.1136/bmj.39343.408449.80
28. Jeffreys, H. (1961). Theory of probability (3rd ed.). Oxford, UK: Oxford University Press.
29. Kass, R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90, 773-795. http://dx.doi.org/ 10.1080/01621459.1995.10476572
30. Kelley, K., & Lai, K. (2012). Package "MBESS." Retrieved from http://cran.r-project.org/web/packages/MBESS/MBESS.pdf
31. Kruschke, J. K. (2011). Bayesian assessment of null values via parameter estimation and model comparison. Perspectives on Psychological Science, 6, 299-312. http://dx.doi.org/10.1177/1745691611406925
32. Kruschke, J. K. (2014). Doing Bayesian data analysis: A tutorial with R, JAGS, and Stan (2nd ed.). San Diego, CA: Academic Press.
33. Lane, D. M., & Dunlap, W. P. (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical Psychology, 31, 107-112. http://dx.doi .org/10.1111/j.2044-8317.1978.tb00578.x
34. Lau, J., Antman, E. M., Jimenez-Silva, J., Kupelnick, B., Mosteller, F., & Chalmers, T. C. (1992). Cumulative meta-analysis of therapeutic trials for myocardial infarction. The New England Journal of Medicine, 327, 248-254. http://dx.doi.org/10.1056/NEJM199207233270406
35. Maxwell, S. E. (2004). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9, 147-163. http://dx.doi.org/10.1037/1082-989X.9.2.147
36. Maxwell, S. E., Kelley, K., & Rausch, J. R. (2008). Sample size planning for statistical power and accuracy in parameter estimation. Annual Review of Psychology, 59, 537-563. http://dx.doi.org/10.1146/annurev .psych.59.103006.093735
37. Maxwell, S. E., Lau, M. Y., & Howard, G. S. (in press). Is psychology suffering from a replication crisis? American Psychologist.
38. Morey, R. D., & Rouder, J. N. (2011). Bayes factor approaches for testing interval null hypotheses. Psychological Methods, 16, 406-419. http://dx.doi.org/10.1037/a0024377
39. Morey, R. D., Rouder, J. N., Verhagen, J., & Wagenmakers, E. J. (2014). Why hypothesis tests are essential for psychological science: A comment on Cumming (2014). Psychological Science, 25, 1289-1290. http://dx.doi.org/10.1177/0956797614525969
40. Mullen, B., Muellerleile, P., & Bryant, B. (2001). Cumulative metaanalysis:A consideration of indicators of sufficiency and stability. Personality and Social Psychology Bulletin, 27, 1450-1462. http://dx .doi.org/10.1177/01461672012711006
41. Nordgren, L. F., & McDonnell, M. H. M. (2011). The scope-severity paradox: Why doing more harm is judged to be less harmful. Social Psychological & Personality Science, 2, 97-102. http://dx.doi.org/ 10.1177/1948550610382308
42. Nosek, B. A., & Lakens, D. (2014). A method to increase the credibility of published results. Social Psychology, 45, 137-141. http://dx.doi.org/ 10.1027/1864-9335/a000192
43. Pashler, H., & Wagenmakers, E.-J. (2012). Editors' introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7, 528-530. http://dx.doi.org/10.1177/1745691612465253
44. Ritchie, S. J., Wiseman, R., & French, C. C. (2012). Failing the future:Three unsuccessful attempts to replicate Bem's 'retroactive facilitation of recall' effect. PLoS ONE, 7, e33423. http://dx.doi.org/10.1371/journal .pone.0033423
45. Rosenthal, R. (1990). Replication in behavioral research. Journal of Social Behavior and Personality, 5, 1-30.
46. Rosenthal, R., & Rubin, D. B. (1979). Comparing significance levels of independent studies. Psychological Bulletin, 86, 1165-1168. http://dx .doi.org/10.1037/0033-2909.86.5.1165
47. Rosenthal, R., & Rubin, D. B. (1982). Comparing effect sizes of independent studies. Psychological Bulletin, 92, 500-504. http://dx.doi.org/ 10.1037/0033-2909.92.2.500
48. Rouder, J. N., & Morey, R. D. (2012). Default Bayes factors for model selection in regression. Multivariate Behavioral Research, 47, 877-903. http://dx.doi.org/10.1080/00273171.2012.734737
49. Rouder, J. N., Morey, R. D., Speckman, P. L., & Province, J. N. (2012). Default Bayes factors for ANOVA designs. Journal of Mathematical Psychology, 56, 356-374. http://dx.doi.org/10.1016/j.jmp.2012.08.001
50. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225-237. http://dx.doi.org/ 10.3758/PBR.16.2.225
51. Rozeboom, W. W. (1960). The fallacy of the null-hypothesis significance test. Psychological Bulletin, 57, 416-428. http://dx.doi.org/10.1037/h0042040
52. Schenker, N., & Gentleman, J. F. (2001). On judging the significance of differences by examining the overlap between confidence intervals. The American Statistician, 55, 182-186. http://dx.doi.org/10.1198/ 000313001317097960
53. Serlin, R. C., & Lapsley, D. K. (1985). Rationality in psychological research: The good-enough principle. American Psychologist, 40, 73-83. http://dx.doi.org/10.1037/0003-066X.40.1.73
54. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366. http://dx.doi.org/10.1177/0956797611417632
55. Simons, D. J. (2014). The value of direct replication. Perspectives on Psychological Science, 9, 76-80. http://dx.doi.org/10.1177/ 1745691613514755
56. Stroebe, W., & Strack, F. (2014). The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science, 9, 59-71. http://dx.doi.org/10.1177/1745691613514450
57. Stukas, A. A., & Cumming, G. (2014). Interpreting effect sizes: Toward a quantitative cumulative social psychology. European Journal of Social Psychology, 44, 711-722. http://dx.doi.org/10.1002/ejsp.2019
58. Taylor, D. J., & Muller, K. E. (1996). Bias in linear model power and sample size calculation due to estimating noncentrality. Communications in Statistics Theory and Methods, 25, 1-13.
59. Tukey, J. W. (1969). Analyzing data: Sanctification or detective work. American Psychologist, 24, 83-91. http://dx.doi.org/10.1037/h0027108
60. Tversky, A., & Kahneman, D. (1971). Belief in the law of small numbers. Psychological Bulletin, 76, 105-110. http://dx.doi.org/10.1037/h0031322
61. van de Schoot, R., Mulder, J., Hoijtink, H., Van Aken, M. A. G., Dubas, J. S., de Castro, B. O.,... Romeijn, J. W. (2011). An introduction to Bayesian model selection for evaluating informative hypotheses. European Journal of Developmental Psychology, 8, 713-729. http://dx.doi .org/10.1080/17405629.2011.621799
62. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632-638. http://dx.doi.org/ 10.1177/1745691612463078
63. Walker, E., & Nowacki, A. S. (2011). Understanding equivalence and noninferiority testing. Journal of General Internal Medicine, 26, 192-196. http://dx.doi.org/10.1007/s11606-010-1513-8
64. Wegner, D. M., Schneider, D. J., Carter, S. R., III, & White, T. L. (1987). Paradoxical effects of thought suppression. Journal of Personality and Social Psychology, 53, 5-13. http://dx.doi.org/10.1037/0022-3514.53.1.5
65. Wellek, S. (2010). Testing statistical hypotheses of equivalence and noninferiority (2nd ed.). Boca Raton, FL: Chapman & Hall/CRC Press. http://dx.doi.org/10.1201/EBK1439808184
66. Wilkinson, L. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604. http://dx .doi.org/10.1037/0003-066X.54.8.594
67. Yates, F. (1964). Sir Ronald Fisher and the design of experiments. Biometrics, 20, 307-321. http://dx.doi.org/10.2307/2528399
68. Zellner, A., & Siow, A. (1980). Posterior odds ratios for selected regression hypotheses. In J. M. Bernardo, M. H. DeGroot, D. V. Lindley, & A. F. M. Smith (Eds.), Bayesian statistics: Proceedings of the first international meeting (pp. 585-603). Valencia, Spain: University of Valencia Press.
69. Zou, G. Y. (2007). Toward using confidence intervals to compare correlations. Psychological Methods, 12, 399-413. http://dx.doi.org/10.1037/ 1082-989X.12.4.399