The replication paradox: Combining studies can decrease accuracy of effect size estimates

Review of General Psychology

Volumn 19, Issue 2, 2015, Pages 172-182

  Nuijten, Michèle B ^a   Van Assen, Marcel A L M ^a   Veldkamp, Coosje L S ^a   Wicherts, Jelte M ^a  

^a TILBURG UNIVERSITY   (Netherlands)

Author keywords

Effect size; Meta analysis; Power; Publication bias; Replication

Indexed keywords

EID: 84933182262     PISSN: 10892680     EISSN: 19391552     Source Type: Journal    
DOI: 10.1037/gpr0000034     Document Type: Article

References (91)

1. Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K.,... Wicherts, J. M. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108-119. http://dx.doi.org/10.1002/per.1919
2. Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543-554. http://dx.doi.org/10.1177/1745691612459060
3. Banks, G. C., Kepes, S., & Banks, K. P. (2012). Publication bias: The antagonist of meta-analytic reviews and effective policy making. Educational Evaluation and Policy Analysis, 34, 259-277. http://dx.doi.org/ 10.3102/0162373712446144
4. Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088-1101. http:// dx.doi.org/10.2307/2533446
5. Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Chichester, UK: Wiley, Ltd. http://dx.doi .org/10.1002/9780470743386
6. Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R.,... Van't Veer, A. (2014). The replication recipe:What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217-224. http://dx.doi.org/10.1016/j.jesp.2013 .10.005
7. Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365-376. http://dx.doi.org/10.1038/nrn3475
8. Chambers, C. D. (2013). Registered reports: A new publishing initiative at Cortex. Cortex, 49, 609-610. http://dx.doi.org/10.1016/j.cortex.2012.12 .016
9. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. New York, NY: Routledge.
10. Cooper, H., DeNeve, K., & Charlton, K. (1997). Finding the missing science: The fate of studies submitted for review by a human subjects committee. Psychological Methods, 2, 447- 452. http://dx.doi.org/ 10.1037/1082-989X.2.4.447
11. Copas, J. B. (2013). A likelihood-based sensitivity analysis for publication bias in meta-analysis. Journal of the Royal Statistical Society, 62, 47-66. http://dx.doi.org/10.1111/j.1467-9876.2012.01049.x
12. Crocker, J., & Cooper, M. L. (2011). Addressing scientific fraud. Science, 334, 1182. http://dx.doi.org/10.1126/science.1216775
13. de Groot, A. D. (2014). The meaning of "significance" for different types of research. Acta Psychologica, 148, 188 -194. http://dx.doi.org/ 10.1016/j.actpsy.2014.02.001
14. de Winter, J., & Happee, R. (2013). Why selective publication of statistically significant results can be effective. PLoS ONE, 8, e66463.
15. Diekmann, A. (2011). Are most published research findings false? Jahrbucher fur Nationalokonomie und Statistik, 231, 628-635.
16. Duval, S., & Tweedie, R. (2000a). A nonparametric "trim and fill" method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association, 95, 89-98.
17. Duval, S., & Tweedie, R. (2000b). Trim and fill: A simple funnel-plotbased method of testing and adjusting for publication bias in metaanalysis. Biometrics, 56, 455-463. http://dx.doi.org/10.1111/j.0006-341X.2000.00455.x
18. Egger, M., Davey Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629-634. http://dx.doi.org/10.1136/bmj.315.7109.629
19. Fanelli, D. (2010). "Positive" results increase down the Hierarchy of the Sciences. PLoS ONE, 5, e10068. http://dx.doi.org/10.1371/journal.pone .0010068
20. Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891-904. http://dx.doi.org/10.1007/ s11192-011-0494-7
21. Fanelli, D., & Ioannidis, J. P. A. (2013). US studies may overestimate effect sizes in softer research. Proceedings of the National Academy of Sciences of the United States of America, 110, 15031-15036. http://dx .doi.org/10.1073/pnas.1302997110
22. Ferguson, C. J., & Brannick, M. T. (2012). Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17, 120-128. http://dx.doi.org/10.1037/a0024445
23. Ferguson, C. J., & Heene, M. (2012). A vast graveyard of undead theories:Publication bias and psycholical science's aversion to the null. Perspectives on Psychological Science, 7, 555-561. http://dx.doi.org/10.1177/ 1745691612459059
24. Field, A. P., & Gillett, R. (2010). How to do a meta-analysis. British Journal of Mathematical and Statistical Psychology, 63, 665-694. http://dx.doi.org/10.1348/000711010X502733
25. Francis, G. (2012a). Publication bias and the failure of replication in experimental psychology. Psychonomic Bulletin & Review, 19, 975-991. http://dx.doi.org/10.3758/s13423-012-0322-y
26. Francis, G. (2012b). The same old new look: Publication bias in a study of wishful seeing. i-Perception, 3, 176-178. http://dx.doi.org/10.1068/ i0519ic
27. Francis, G. (2012c). Too good to be true: Publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin & Review, 19, 151-156. http://dx.doi.org/10.3758/s13423-012-0227-9
28. Francis, G. (2013). Publication bias in "Red, rank, and romance in women viewing men," by Elliot et al. Journal of Experimental Psychology:General, 142, 292-296, 2010. http://dx.doi.org/10.1037/a0027923
29. Francis, G. (2014). The frequency of excess success for articles in Psychological Science. Psychonomic Bulletin & Review, 21, 1180-1187. http://dx.doi.org/10.3758/s13423-014-0601-x
30. Francis, G., Tanzman, J., & Matthews, W. J. (2014). Excess success for psychology articles in the journal science. PLoS ONE, 9, e114255. http://dx.doi.org/10.1371/journal.pone.0114255
31. Franco, A., Malhotra, N., & Simonovits, G. (2014). Social science. Publication bias in the social sciences: Unlocking the file drawer. Science, 345, 1502-1505. http://dx.doi.org/10.1126/science.1255484
32. Frank, M. C., & Saxe, R. (2012). Teaching replication. Perspectives on Psychological Science, 7, 600 - 604. http://dx.doi.org/10.1177/ 1745691612460686
33. Gerber, A. S., Green, D. P., & Nickerson, D. (2001). Testing for publication bias in political science. Political Analysis, 9, 385-392. http://dx .doi.org/10.1093/oxfordjournals.pan.a004877
34. Giner-Sorolla, R. (2012). Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspectives on Psychological Science, 7, 562-571. http://dx.doi.org/ 10.1177/1745691612457576
35. Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1-20. http://dx.doi.org/10.1037/ h0076157
36. Hedges, L. V. (1984). Estimation of effect size under nonrandom sampling:The effects of censoring studies yielding statistically insignificant mean differences. Journal of Educational Statistics, 9, 61-85. http://dx.doi .org/10.2307/1164832
37. Hedges, L. V., & Vevea, J. L. (1996). Estimating effect size under publication bias: Small sample properties and robustness of a random effects selection model. Journal of Educational and Behavioral Statistics, 21, 299-332. http://dx.doi.org/10.3102/10769986021004299
38. Hedges, L. V., & Vevea, J. L. (2005). Selection method approaches. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 145-174). New York, NY: Wiley.
39. Ioannidis, J. P. A. (2008). Why most discovered true associations are inflated. Epidemiology, 19, 640-648. http://dx.doi.org/10.1097/EDE .0b013e31818131e7
40. Ioannidis, J. P. A. (2011). Excess significance bias in the literature on brain volume abnormalities. Archives of General Psychiatry, 68, 773-780. http://dx.doi.org/10.1001/archgenpsychiatry.2011.28
41. Ioannidis, J. P. A. (2013). Clarifications on the application and interpretation of the test for excess significance and its extensions. Journal of Mathematical Psychology, 57, 184-187. http://dx.doi.org/10.1016/j.jmp .2013.03.002
42. Ioannidis, J. P. A., & Trikalinos, T. A. (2005). Early extreme contradictory estimates may appear in published research: The Proteus phenomenon in molecular genetics research and randomized trials. Journal of Clinical Epidemiology, 58, 543-549. http://dx.doi.org/10.1016/j.jclinepi.2004.10 .019
43. Ioannidis, J. P. A., & Trikalinos, T. A. (2007). An exploratory test for an excess of significant findings. Clinical Trials, 4, 245-253. http://dx.doi .org/10.1177/1740774507079441
44. Jennions, M. D., & Moller, A. P. (2002). Publication bias in ecology and evolution: An empirical assessment using the 'trim and fill' method. Biological Reviews of the Cambridge Philosophical Society, 77, 211-222. http://dx.doi.org/10.1017/S1464793101005875
45. Jennions, M. D., & Moller, A. P. (2003). A survey of the statistical power of research in behavioral ecology and animal behavior. Behavioral Ecology, 14, 438-445. http://dx.doi.org/10.1093/beheco/14.3.438
46. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524 -532. http://dx.doi.org/10.1177/ 0956797611430953
47. Kavvoura, F. K., McQueen, M. B., Khoury, M. J., Tanzi, R. E., Bertram, L., & Ioannidis, J. P. A. (2008). Evaluation of the potential excess of statistically significant findings in published genetic association studies:Application to Alzheimer's disease. American Journal of Epidemiology, 168, 855-865. http://dx.doi.org/10.1093/aje/kwn206
48. Klein, R. A., Ratliff, K. A., Vianello, M., Adams, J., Reginald, B., Bahník, Š.,... Nosek, B. A. (2014). Investigating variation in replicability: A "many labs" replication project. Retrieved from https://osf.io/wx7ck/
49. Kraemer, H. C., Gardner, C., Brooks, J. O., & Yesavage, J. A. (1998). Advantages of excluding underpowered studies in meta-analysis: Inclusionist versus exclusionist viewpoints. Psychological Methods, 3, 23-31. http://dx.doi.org/10.1037/1082-989X.3.1.23
50. Lakens, D. (2015). What p-hacking really looks like: A comment on Masicampo & Lalande (2012). Quarterly Journal of Experimental Psychology, 68, 829-832.
51. Levine, T., Asada, K. J., & Carpenter, C. (2009). Sample sizes and effect sizes are negatively correlated in meta-analyses: Evidence and implications of a publication bias against nonsignificant findings. Communication Monographs, 76, 286 -302. http://dx.doi.org/10.1080/ 03637750903074685
52. Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in psychology research how often do they really occur? Perspectives on Psychological Science, 7, 537-542. http://dx.doi.org/10.1177/ 1745691612460688
53. Marszalek, J. M., Barber, C., Kohlhart, J., & Holmes, C. B. (2011). Sample size in psychological research over the past 30 years. Perceptual and Motor Skills, 112, 331-348. http://dx.doi.org/10.2466/03.11.PMS.112.2 .331-348
54. Moreno, S. G., Sutton, A. J., Ades, A. E., Stanley, T. D., Abrams, K. R., Peters, J. L., & Cooper, N. J. (2009). Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study. BMC Medical Research Methodology, 9. http://dx.doi.org/ 10.1186/1471-2288-9-2
55. Morey, R. D. (2013). The consistency test does not-and cannot- deliver what is advertised: A comment on Francis (2013). Journal of Mathematical Psychology, 57, 180-183. http://dx.doi.org/10.1016/j.jmp.2013 .03.004
56. Murayama, K., Pekrun, R., & Fiedler, K. (2013). Research practices that can prevent an inflation of false-positive rates. Personality and Social Psychology Review, 18, 107-118.
57. Neuliep, J. W., & Crandall, R. (1990). Editorial bias against replication research. Journal of Social Behavior and Personality, 5, 85-90.
58. Neuliep, J. W., & Crandall, R. (1993). Reviewer bias against replication research. Journal of Social Behavior and Personality, 8, 21-29.
59. Newcombe, R. G. (1987). Towards a reduction in publication bias. British Medical Journal, 295, 656 - 659. http://dx.doi.org/10.1136/bmj.295 .6599.656
60. Niemeyer, H., Musch, J., & Pietrowsky, R. (2012). Publication bias in meta-analyses of the efficacy of psychotherapeutic interventions for schizophrenia. Schizophrenia Research, 138, 103-112. http://dx.doi.org/ 10.1016/j.schres.2012.03.023
61. Niemeyer, H., Musch, J., & Pietrowsky, R. (2013). Publication bias in meta-analyses of the efficacy of psychotherapeutic interventions for depression. Journal of Consulting and Clinical Psychology, 81, 58-74. http://dx.doi.org/10.1037/a0031152
62. Nosek, B. A., Spies, J., & Motyl, M. (2012). Scientific Utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615-631. http://dx.doi.org/ 10.1177/1745691612459058
63. Nuijten, M. B., van Assen, M. A. L. M., van Aert, R. C. M., & Wicherts, J. M. (2014). Standard analyses fail to show that US studies overestimate effect sizes in softer research. Proceedings of the National Academy of Sciences of the United States of America, 111, E712-E713. http://dx.doi .org/10.1073/pnas.1322149111
64. Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7, 657-660. http://dx.doi.org/10.1177/ 1745691612462588
65. Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7(6), 531-536. http://dx.doi.org/10.1177/1745691612463401
66. Perugini, M., Galucci, M., & Constantini, G. (2014). Safeguard power as a protection against imprecise power estimates. Perspectives on Psychological Science, 9, 319 -332. http://dx.doi.org/10.1177/ 1745691614528519
67. Renkewitz, F., Fuchs, H. M., & Fiedler, S. (2011). Is there evidence of publication biases in JDM research? Judgment and Decision Making, 6, 870-881.
68. Rothstein, H. R., Sutton, A. J., & Borenstein, M. (Eds.). (2005). Publication bias in meta-analysis. Prevention, assessment, and adjustments. New York, NY: Wiley. http://dx.doi.org/10.1002/0470870168
69. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis al- lows presenting anything as significant. Psychological Science, 22, 1359-1366. http://dx.doi.org/10.1177/0956797611417632
70. Simonsohn, U. (2013). It really just does not follow, comments on Francis (2013). Journal of Mathematical Psychology, 57, 174-176. http://dx.doi .org/10.1016/j.jmp.2013.03.006
71. Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143, 534-547. http://dx.doi.org/10.1037/a0033242
72. Smulders, Y. M. (2013). A two-step manuscript submission process can reduce publication bias. Journal of Clinical Epidemiology, 66, 946-947. http://dx.doi.org/10.1016/j.jclinepi.2013.03.023
73. Stanley, T. D., Jarrell, S. B., & Doucouliagos, H. (2010). Could it be better to discard 90% of the data? A statistical paradox. The American Statistician, 64, 70-77. http://dx.doi.org/10.1198/tast.2009.08205
74. Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance-Or vice versa. Journal of the American Statistical Association, 54, 30-34.
75. Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995). Publication decisions revisited - The effect of the outcome of statistical tests on the decision to publish and vice-versa. The American Statistician, 49, 108-112.
76. Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000). Publication and related bias in meta-analysis: Power of statistical tests and prevalence in the literature. Journal of Clinical Epidemiology, 53, 1119-1129. http://dx .doi.org/10.1016/S0895-4356(00)00242-0
77. Stroebe, W., & Strack, F. (2014). The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science, 9, 59-71. http://dx.doi.org/10.1177/1745691613514450
78. Sutton, A. J., Duval, S. J., Tweedie, R. L., Abrams, K. R., & Jones, D. R. (2000). Empirical assessment of effect of publication bias on metaanalyses. British Medical Journal, 320, 1574-1577. http://dx.doi.org/ 10.1136/bmj.320.7249.1574
79. Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003). Adjusting for publication bias in the presence of heterogeneity. Statistics in Medicine, 22, 2113-2126. http://dx.doi.org/10.1002/sim.1461
80. Tsilidis, K. K., Papatheodorou, S. I., Evangelou, E., & Ioannidis, J. P. A. (2012). Evaluation of excess statistical significance in meta-analyses of 98 biomarker associations with cancer risk. Journal of the National Cancer Institute, 104, 1867-1878. http://dx.doi.org/10.1093/jnci/djs437
81. Tversky, A., & Kahneman, D. (1971). Belief in the law of small numbers. Psychological Bulletin, 76, 105-110. http://dx.doi.org/10.1037/ h0031322
82. van Assen, M. A. L. M., van Aert, R. C. M., Nuijten, M. B., & Wicherts, J. M. (2014a). Why publishing everything is more effective than selective publishing of statistically significant results. PLoS ONE, 9, e84896. http://dx.doi.org/10.1371/journal.pone.0084896
83. van Assen, M. A. L. M., van Aert, R. C. M., Nuijten, M. B., & Wicherts, J. M. (2014b). Why we need to publish all studies. Retrieved from http://www.plosone.org/annotation/listThread.action?root=78405
84. van Assen, M. A. L. M., van Aert, R. C. M., & Wicherts, J. M. (2014). Meta-analysis using effect size distributions of only statistically significant studies. Psychological Methods. Advance online publication. http://dx.doi.org/10.1037/met0000025
85. Vandekerckhove, J., Guan, M., & Styrcula, S. A. (2013). The consistency test may be too weak to be useful: Its systematic application would not improve effect size estimation in meta-analyses. Journal of Mathematical Psychology, 57, 170-173. http://dx.doi.org/10.1016/j.jmp.2013.03 .007
86. Vevea, J. L., Clements, N. C., & Hedges, L. V. (1993). Assessing the effects of selection bias on validity data for the General Aptitude-Test Battery. Journal of Applied Psychology, 78, 981-987. http://dx.doi.org/ 10.1037/0021-9010.78.6.981
87. Vevea, J. L., & Hedges, L. V. (1995). A general linear model for estimating effect size in the presence of publication bias. Psychometrika, 60, 419-435. http://dx.doi.org/10.1007/BF02294384
88. Vevea, J. L., & Woods, C. M. (2005). Publication bias in research synthesis:Sensitivity analysis using a priori weight functions. Psychological Methods, 10, 428-443. http://dx.doi.org/10.1037/1082-989X.10.4 .428
89. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632-638. http://dx.doi.org/10 .1177/1745691612463078
90. Walster, G., & Cleary, T. (1970). A proposal for a new editorial policy in the social sciences. The American Statistician, 24, 16-19.
91. Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E. J. (2011). Statistical evidence in experimental psychology:An empirical comparison using 855 t tests. Perspectives on Psychological Science, 6, 291-298. http://dx.doi.org/10.1177/ 1745691611406923