Meta-analysis using effect size distributions of only statistically significant studies

Psychological Methods

Volumn 20, Issue 3, 2015, Pages 293-309

  Van Assen, Marcel A L M ^a   Van Aert, Robbie C M ^a   Wicherts, Jelte M ^a  

^a TILBURG UNIVERSITY   (Netherlands)

Author keywords

Meta analysis; Publication bias; Sensitivity analysis; Test of excess significance; The trim and fill method

Indexed keywords

HUMAN; META ANALYSIS (TOPIC); PUBLISHING; STATISTICAL ANALYSIS; STATISTICAL DISTRIBUTION;

DATA INTERPRETATION, STATISTICAL; HUMANS; META-ANALYSIS AS TOPIC; PUBLICATION BIAS; STATISTICAL DISTRIBUTIONS;

EID: 84941211202     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/met0000025     Document Type: Article

References (71)

1. Aguinis, H., Dalton, D. R., Bosco, F. A., Pierce, C. A., & Dalton, C. M. (2011). Meta-analytic choices and judgment calls: Implications for theory building and testing, obtained effect sizes, and scholarly impact. Journal of Management, 37, 5-38. http://dx.doi.org/10.1177/0149206310377113
2. Aguinis, H., Pierce, C. A., Bosco, F. A., Dalton, D. R., & Dalton, C. M. (2011). Debunking myths and urban legends about meta-analysis. Organizational Research Methods, 14, 306-331. http://dx.doi.org/10.1177/1094428110375720
3. Anderson, T. W., & Darling, D. A. (1954). A test of goodness of fit. Journal of the American Statistical Association, 49, 765-769. http://dx .doi.org/10.1080/01621459.1954.10501232
4. Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., . . . Wicherts, J. M. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108-119. http://dx.doi.org/10.1002/per.1919
5. Aytug, Z. G., Rothstein, H. R., Zhou, W., & Kern, M. C. (2012). Revealed or concealed? Transparency of procedures, decisions, and judgment calls in meta-analyses. Organizational Research Methods, 15, 103-133. http://dx.doi.org/10.1177/1094428111403495
6. Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543-554. http://dx.doi.org/10.1177/1745691612459060
7. Banks, G. C., Kepes, S., & Banks, K. P. (2012). Publication bias: The antagonist of meta-analytic reviews and effective policymaking. Educational Evaluation and Policy Analysis, 34, 259-277. http://dx.doi.org/10.3102/0162373712446144
8. Banks, G. C., Kepes, S., & McDaniel, M. A. (2012). Publication bias: A call for improved meta-analytic practice in the organizational sciences. International Journal of Selection and Assessment, 20, 182-197. http://dx.doi.org/10.1111/j.1468-2389.2012.00591.x
9. Becker, B. J. (2005). Failsafe N or file-drawer number. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 111-125). Chichester, UK: Wiley.
10. Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088-1101. http://dx.doi.org/10.2307/2533446
11. Begley, C. G., & Ellis, L. M. (2012). Drug development: Raise standards for preclinical cancer research. Nature, 483, 531-533. http://dx.doi.org/10.1038/483531a
12. Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Chichester, UK: Wiley, Ltd. http://dx.doi .org/10.1002/9780470743386
13. Cooper, H., DeNeve, K., & Charlton, K. (1997). Finding the missing science: The fate of studies submitted for review by a human subjects committee. Psychological Methods, 2, 447-452. http://dx.doi.org/10.1037/1082-989X.2.4.447
14. Coursol, A., & Wagner, E. E. (1986). Effect of positive findings on submission and acceptance rates: A note on meta-analysis bias. Professional Psychology: Research and Practice, 17, 136-137. http://dx.doi .org/10.1037/0735-7028.17.2.136
15. Duval, S. (2005). The trim and fill method. In H. R. Rothstein, M. Borenstein, & A. J. Sutton (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 127-144). Chichester, UK: Wiley.
16. Duval, S., & Tweedie, R. (2000a). A nonparametric "trim and fill" method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association, 95, 89-98.
17. Duval, S., & Tweedie, R. (2000b). Trim and fill: A simple funnel-plotbased method of testing and adjusting for publication bias in metaanalysis. Biometrics, 56, 455-463. http://dx.doi.org/10.1111/j.0006-341X.2000.00455.x
18. Egger, M., Davey Smith, G., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629-634. http://dx.doi.org/10.1136/bmj.315.7109.629
19. Ellis, P. D. (2010). The essential guide to effect sizes: An introduction to statistical power, meta-analysis and the interpretation of research results. Cambridge, UK: Cambridge University Press. http://dx.doi.org/10.1017/CBO9780511761676
20. Fanelli, D. (2010). Do pressures to publish increase scientists' bias? An empirical support from U.S. States data. PLoS ONE, 5, e10271. http://dx.doi.org/10.1371/journal.pone.0010271
21. Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891-904. http://dx.doi.org/10.1007/s11192-011-0494-7
22. Ferguson, C. J., & Brannick, M. T. (2012). Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17, 120-128. http://dx.doi.org/10.1037/a0024445
23. Field, A. P., & Gillett, R. (2010). How to do a meta-analysis. The British Journal of Mathematical and Statistical Psychology, 63, 665-694. http://dx.doi.org/10.1348/000711010X502733
24. Fisher, R. A. (1932). Statistical methods for research workers. London, UK: Oliver & Boyd.
25. Francis, G. (2012). Publication bias and the failure of replication in experimental psychology. Psychonomic Bulletin & Review, 19, 975-991. http://dx.doi.org/10.3758/s13423-012-0322-y
26. Francis, G. (2013). Replication, statistical consistency, and publication bias. Journal of Mathematical Psychology, 57, 153-169. http://dx.doi .org/10.1016/j.jmp.2013.02.003
27. Fritz, A., Scherndl, T., & Küberger, A. (2013). Publication bias and the correlation between effect size and sample size in psychological research: Sources, consequences, and remedies. Manuscript submitted for publication.
28. Gerber, A. S., Green, D. P., & Nickerson, D. (2001). Testing for publication bias in political science. Political Analysis, 9, 385-392. http://dx .doi.org/10.1093/oxfordjournals.pan.a004877
29. Hedges, L. V. (1984). Estimation of effect size under nonrandom sampling: The effects of censoring studies yielding statistically insignificant mean differences. Journal of Educational Statistics, 9, 61-85. http://dx.doi .org/10.2307/1164832
30. Hedges, L. V., & Vevea, J. L. (1996). Estimating effect size under publication bias: Small sample properties and robustness of a random effects selection model. Journal of Educational and Behavioral Statistics, 21, 299-332. http://dx.doi.org/10.3102/10769986021004299
31. Hedges, L. V., & Vevea, J. L. (2005). Selection method approaches. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 145-175). Chichester, UK: Wiley.
32. Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Medicine, 2, e124. http://dx.doi.org/10.1371/journal.pmed .0020124
33. Ioannidis, J. P., & Trikalinos, T. A. (2007a). The appropriateness of asymmetry tests for publication bias in meta-analyses: A large survey. Canadian Medical Association Journal, 176, 1091-1096. http://dx.doi .org/10.1503/cmaj.060410
34. Ioannidis, J. P., & Trikalinos, T. A. (2007b). An exploratory test for an excess of significant findings. Clinical Trials, 4, 245-253. http://dx.doi .org/10.1177/1740774507079441
35. Kepes, S., Banks, G. C., McDaniel, M., & Whetzel, D. L. (2012). Publication bias in the organizational sciences. Organizational Research Methods, 15, 624-662. http://dx.doi.org/10.1177/1094428112452760
36. Kepes, S., Banks, G. C., & Oh, I. S. (2012). Avoiding bias in publication bias research: The value of null findings. Journal of Business and Psychology. Advance online publication.
37. Kisamore, J., & Brannick, M. (2008). An illustration of the consequences of meta-analysis model choice. Organizational Research Methods, 11, 35-53. http://dx.doi.org/10.1177/1094428106287393
38. Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Jr., Bahnik, S., Bernstein, M. J., . . . Nosek, B. A. (2014). Investigating variation in replicability: A "many labs" replication project. Social Psychology, 45, 142-152. http://dx.doi.org/10.1027/1864-9335/a000178
39. Kraemer, H. C., Gardner, C., Brooks, J., & Yesavage, J. A. (1998). Advantages of excluding underpowered studies in meta-analysis: Inclusionist versus exclusionist viewpoints. Psychological Methods, 3, 23-31. http://dx.doi.org/10.1037/1082-989X.3.1.23
40. Last, J. M. (2001). A dictionary of epidemiology. Oxford, UK: Oxford University Press.
41. Light, R. J., & Pillemer, D. B. (1984). Summing up: The science of reviewing research. Cambridge, MA: Harvard University Press.
42. Mahoney, M. J. (1977). Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 1, 161-175. http://dx.doi.org/10.1007/BF01173636
43. Massey, F. J., Jr. (1951). The Kolmogorov-Smirnov test for goodness of fit. Journal of the American Statistical Association, 46, 68-78. http://dx.doi .org/10.1080/01621459.1951.10500769
44. McCall, R. B., & Carriger, M. S. (1993). A meta-analysis of infant habituation and recognition memory performance as predictors of later IQ. Child Development, 64, 57-79. http://dx.doi.org/10.2307/1131437
45. Moreno, S. G., Sutton, A. J., Ades, A. E., Stanley, T. D., Abrams, K. R., Peters, J. L., & Cooper, N. J. (2009). Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study. BMC Medical Research Methodology, 9, 2. Retrieved from http://www.biomedcentral.com/1471-2288/9/2 http://dx.doi.org/10.1186/1471-2288-9-2
46. O'Boyle, E. H., Jr., Banks, G. C., & Gonzalez-Mulé, E. (2014). The Chrysalis effect: How ugly initial results metamorphosize into beautiful articles. Journal of Management. Advance online publication. http://dx .doi.org/10.1177/0149206314527133
47. Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2007). Performance of the trim and fill method in the presence of publication bias and between-study heterogeneity. Statistics in Medicine, 26, 4544-4562. http://dx.doi.org/10.1002/sim.2889
48. Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008). Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. Journal of Clinical Epidemiology, 61, 991-996. http://dx.doi.org/10.1016/j.jclinepi.2007.11.010
49. Prinz, F., Schlange, T., & Asadullah, K. (2011). Believe it or not: How much can we rely on published data on potential drug targets? Nature Reviews Drug Discovery, 10, 712. http://dx.doi.org/10.1038/nrd3439-c1
50. R Core Team. (2012). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.
51. Renkewitz, F., Fuchs, H. M., & Fiedler, S. (2011). Is there evidence of publication biases in JDM research? Judgment and Decision Making, 6, 870-881.
52. Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638-641. http://dx.doi.org/10.1037/0033-2909.86.3.638
53. Rothstein, H. R., & Bushman, B. J. (2012). Publication bias in psychological science: Comment on Ferguson and Brannick (2012). Psychological Methods, 17, 129-136. http://dx.doi.org/10.1037/a0027128
54. Rothstein, H. R., Sutton, A. J., & Borenstein, M. (2005). Publication bias in meta-analysis. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 1-7). Chichester, UK: Wiley. http://dx.doi.org/10.1002/0470870168
55. Sarewitz, D. (2012). Beware the creeping cracks of bias. Nature, 485, 149-150. http://dx.doi.org/10.1038/485149a
56. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366. http://dx.doi.org/10.1177/0956797611417632
57. Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2013). P-curve: A key to the file drawer. Journal of Experimental Psychology. Retrieved from http://ssrn.com/abstract_2256237
58. Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve and effect size: Correcting for publication bias using only significant results. Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id_2377290
59. Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance-Or vice versa. Journal of the American Statistical Association, 54, 30-34.
60. Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. The American Statistician, 49, 108-112.
61. Sterne, J. A. C., Becker, B. J., & Egger, M. (2005). The funnel plot. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 73-98). Chichester, UK: Wiley.
62. Sterne, J. A. C., & Egger, M. (2001). Funnel plots for detecting bias in meta-analysis: Guidelines on choice of axis. Journal of Clinical Epidemiology, 54, 1046-1055. http://dx.doi.org/10.1016/S0895-4356(01)00377-8
63. Sterne, J. A. C., & Egger, M. (2006). Regression methods to detect publication and other bias in meta-analysis. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 99-110). Chichester, UK: Wiley. http://dx.doi.org/10.1002/0470870168.ch6
64. Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000). Publication and related bias in meta-analysis: Power of statistical tests and prevalence in the literature. Journal of Clinical Epidemiology, 53, 1119-1129. http://dx .doi.org/10.1016/S0895-4356(00)00242-0
65. Sutton, A. J. (2006). Evidence concerning the consequences of publication and related biases. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 175-192). Chichester, UK: Wiley. http://dx.doi.org/10.1002/0470870168.ch10
66. Terrin, N., Schmid, C. H., & Lau, J. (2005). In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. Journal of Clinical Epidemiology, 58, 894-901. http://dx.doi.org/10.1016/j.jclinepi.2005.01.006
67. Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003). Adjusting for publication bias in the presence of heterogeneity. Statistics in Medicine, 22, 2113-2126. http://dx.doi.org/10.1002/sim.1461
68. Van Assen, M. A. L. M., Van Aert, R. C. M., Nuijten, M. B., & Wicherts, J. M. (2014). Why publishing everything is more effective than selective publishing of statistically significant results. PLoS ONE, e84896. http://dx.doi.org/10.1371/journal.pone.0084896
69. Vevea, J. L., & Woods, C. M. (2005). Publication bias in research synthesis: Sensitivity analysis using a priori weight functions. Psychological Methods, 10, 428-443. http://dx.doi.org/10.1037/1082-989X.10.4.428
70. Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36, 1-48.
71. Wetzels, R., Matzke, D., Lee, M. D., Rouder, J., Iverson, G. J., & Wagenmakers, E.-J. (2011). Statistical evidence in experimental psychology: An empirical comparison using 855 t tests. Perspectives on Psychological Science, 6, 291-298. http://dx.doi.org/10.1177/1745691611406923