The effect of publication bias on the Q test and assessment of heterogeneity

Psychological Methods

Volumn 24, Issue 1, 2019, Pages 116-134

  Augusteijn, Hilde E M ^a   van Aert, Robbie C M ^a   van Assen, Marcel A L M ^a,b  

^a TILBURG UNIVERSITY   (Netherlands)

^b UTRECHT UNIVERSITY   (Netherlands)

Author keywords

Extreme homogeneity; Heterogeneity; Meta analysis; Publication bias; Q test

Indexed keywords

ARTICLE; EFFECT SIZE; HUMAN; META ANALYSIS; MONTE CARLO METHOD; PUBLISHING; SAMPLE SIZE; SCIENTIST; META ANALYSIS (TOPIC); NORMAL DISTRIBUTION; STATISTICAL ANALYSIS;

DATA INTERPRETATION, STATISTICAL; HUMANS; META-ANALYSIS AS TOPIC; NORMAL DISTRIBUTION; PUBLICATION BIAS;

EID: 85057381025     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/met0000197     Document Type: Article

References (61)

1. Adams, R. A., & Essex, C. (2013). Calculus: A complete course (8th ed.). Toronto, Ontario, Canada: Pearson.
2. Aguinis, H., Gottfredson, R. K., & Wright, T. A. (2011). Best-practice recommendations for estimating interaction effects using meta-analysis. Journal of Organizational Behavior, 32, 1033-1043. http://dx.doi.org/ 10.1002/job.719.
3. APA Publications and Communications Board Working Group on Journal Article Reporting Standards. (2008). Reporting standards for research in psychology: Why do we need them? What might they be? American Psychologist, 63, 839-851. http://dx.doi.org/10.1037/0003-066X.63.9 .839.
4. Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543-554. http://dx.doi.org/10.1177/1745691612459060.
5. Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. West Sussex, UK: Wiley. http://dx.doi .org/10.1002/9780470743386.
6. Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2017, November 22). Correcting for bias in psychology: A comparison of meta-analytic methods. Retrieved from psyarxiv.com/9h3nu.
7. Cochran, W. G. (1954). The combination of estimates from different experiments. Biometrics, 10, 101-129. http://dx.doi.org/10.2307/ 3001666.
8. Cooper, H., DeNeve, K., & Charlton, K. (1997). Finding the missing science: The fate of studies submitted for review by a human subjects committee. Psychological Methods, 2, 447-452. http://dx.doi.org/10 .1037/1082-989X.2.4.447.
9. Der, G., Batty, G. D., & Deary, I. J. (2006). Effect of breast feeding on intelligence in children: Prospective study, sibling pairs analysis, and meta-analysis. BMJ: British Medical Journal, 333, 945. http://dx.doi .org/10.1136/bmj.38978.699583.55.
10. DerSimonian, R., & Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trials, 7, 177-188. http://dx.doi.org/10.1016/0197- 2456(86)90046-2.
11. Ellis, P. D. (2010). The essential guide to effect sizes: Statistical power, meta-analysis, and the interpretation of research results. New York, NY: Cambridge University Press. http://dx.doi.org/10.1017/CBO 9780511761676.
12. Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891-904. http://dx.doi.org/10.1007/ s11192-011-0494-7.
13. Field, A. P., & Gillett, R. (2010). How to do a meta-analysis. British Journal of Mathematical and Statistical Psychology, 63, 665-694. http://dx.doi.org/10.1348/000711010X502733.
14. Fraley, R. C., & Vazire, S. (2014). The N-pact factor: Evaluating the quality of empirical journals with respect to sample size and statistical power. PLoS ONE, 9, e109019. http://dx.doi.org/10.1371/journal.pone .0109019.
15. Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345, 1502-1505. http://dx.doi.org/10.1126/science.1255484.
16. Gronau, Q. F., Duizer, M., Bakker, M., & Wagenmakers, E. J. (2017). Bayesian mixture modeling of significant p values: A meta-analytic method to estimate the degree of contamination from H0. Journal of Experimental Psychology: General, 146, 1223-1233. http://dx.doi.org/ 10.1037/xge0000324.
17. Guan, M., & Vandekerckhove, J. (2016). A Bayesian approach to mitigation of publication bias. Psychonomic Bulletin & Review, 23, 74-86. http://dx.doi.org/10.3758/s13423-015-0868-6.
18. Hardy, R. J., & Thompson, S. G. (1998). Detecting and describing heterogeneity in meta-analysis. Statistics in Medicine, 17, 841-856. http:// dx.doi.org/10.1002/(SICI)1097-0258(19980430)17:8 841::AIDSIM781 3.0.CO;2.D.
19. Hartgerink, C. H., Wicherts, J. M., & van Assen, M. A. (2017). Too good to be false: Nonsignificant results revisited. Collabra: Psychology, 3, 9. http://dx.doi.org/10.1525/collabra.71.
20. Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The extent and consequences of p-hacking in science. PLoS Biology, 13, e1002106. http://dx.doi.org/10.1371/journal.pbio.1002106.
21. Hedges, L. V. (1992). Modeling publication selection effects in metaanalysis. Statistical Science, 7, 246-255. http://dx.doi.org/10.1214/ss/ 1177011364.
22. Hedges, L. V., & Vevea, J. (2005). Selection method approaches. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustment (pp. 145-174). West Sussex, UK: Wiley.
23. Higgins, J. P. T., & Green, S. (2011). Cochrane handbook for systematic reviews of interventions (Version 5.1.0). London, UK: Cochrane Collaboration. Retrieved from http://www.cochrane-handbook.org/.
24. Higgins, J. P., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21, 1539-1558. http://dx.doi.org/ 10.1002/sim.1186.
25. Higgins, J., Thompson, S., Deeks, J., & Altman, D. (2002). Statistical heterogeneity in systematic reviews of clinical trials: A critical appraisal of guidelines and practice. Journal of Health Services Research & Policy, 7, 51-61. http://dx.doi.org/10.1258/1355819021927674.
26. Higgins, J. P., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. BMJ: British Medical Journal, 327, 557-560. http://dx.doi.org/10.1136/bmj.327.7414.557.
27. Huedo-Medina, T. B., Sánchez-Meca, J., Marín-Martínez, F., & Botella, J. (2006). Assessing heterogeneity in meta-analysis: Q statistic or I2 index? Psychological Methods, 11, 193-206. http://dx.doi.org/10.1037/1082- 989X.11.2.193.
28. Ioannidis, J. P. A. (2008). Interpretation of tests of heterogeneity and bias in meta-analysis. Journal of Evaluation in Clinical Practice, 14, 951-957. http://dx.doi.org/10.1111/j.1365-2753.2008.00986.x.
29. Ioannidis, J. P. A., & Trikalinos, T. A. (2007). The appropriateness of asymmetry tests for publication bias in meta-analyses: A large survey. Canadian Medical Association Journal, 176, 1091-1096. http://dx.doi .org/10.1503/cmaj.060410.
30. Ioannidis, J. P. A., Trikalinos, T. A., & Zintzaras, E. (2006). Extreme between-study homogeneity in meta-analyses could offer useful insights. Journal of Clinical Epidemiology, 59, 1023-1032. http://dx.doi .org/10.1016/j.jclinepi.2006.02.013.
31. Iyengar, S., & Greenhouse, J. B. (1988). Selection models and the file drawer problem. Statistical Science, 3, 109-117. http://dx.doi.org/10 .1214/ss/1177013012.
32. Jackson, D. (2006a). The implications of publication bias for metaanalysis' other parameter. Statistics in Medicine, 25, 2911-2921. http:// dx.doi.org/10.1002/sim.2293.
33. Jackson, D. (2006b). The power of the standard test for the presence of heterogeneity in meta-analysis. Statistics in Medicine, 25, 2688-2699. http://dx.doi.org/10.1002/sim.2481.
34. Jackson, D. (2007). Assessing the implications of publication bias for two popular estimates of between-study variance in meta-analysis. Biometrics, 63, 187-193. http://dx.doi.org/10.1111/j.1541-0420.2006.00663.x.
35. Kicinski, M. (2013). Publication bias in recent meta-analyses. PLoS ONE, 8, e81823. http://dx.doi.org/10.1371/journal.pone.0081823.
36. Lane, D. M., & Dunlap, W. P. (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical Psychology, 31, 107-112. http://dx.doi .org/10.1111/j.2044-8317.1978.tb00578.x.
37. Langan, D., Higgins, J. P. T., & Simmonds, M. (2017). Comparative performance of heterogeneity variance estimators in meta-analysis: A review of simulation studies. Research Synthesis Methods, 8, 181-198. http://dx.doi.org/10.1002/jrsm.1198.
38. McShane, B. B., Böckenholt, U., & Hansen, K. T. (2016). Adjusting for publication bias in meta-analysis: An evaluation of selection methods and some cautionary notes. Perspectives on Psychological Science, 11, 730-749. http://dx.doi.org/10.1177/1745691616662243.
39. Nuijten, M. B., van Assen, M. A. L. M., Veldkamp, C. L. S., & Wicherts, J. M. (2015). The replication paradox: Combining studies can decrease accuracy of effect size estimate. Review of General Psychology, 19, 172-182. http://dx.doi.org/10.1037/gpr0000034.
40. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716. http://dx.doi.org/10.1126/ science.aac4716.
41. Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., Rushton, L., & Moreno, S. G. (2010). Assessing publication bias in meta-analyses in the presence of between-study heterogeneity. Journal of the Royal Statistical Society: Statistics in Society: Series A, 173, 575-591. http://dx.doi .org/10.1111/j.1467-985X.2009.00629.x.
42. Pietschnig, J., Penke, L., Wicherts, J. M., Zeiler, M., & Voracek, M. (2015). Meta-analysis of associations between human brain volume and intelligence differences: How strong are they and what do they mean? Neuroscience and Biobehavioral Reviews, 57, 411-432. http://dx.doi .org/10.1016/j.neubiorev.2015.09.017.
43. Rabelo, A. L., Keller, V. N., Pilati, R., & Wicherts, J. M. (2015). No effect of weight on judgments of importance in the moral domain and evidence of publication bias from a meta-analysis. PLoS ONE, 10, e0134808. http://dx.doi.org/10.1371/journal.pone.0134808.
44. Raudenbush, S. W. (2009). Analyzing effect sizes: Random-effects models. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (pp. 295-315). New York, NY: Russell Sage Foundation.
45. Rhodes, K. M., Turner, R. M., & Higgins, J. P. T. (2015). Predictive distributions were developed for the extent of heterogeneity in metaanalyses of continuous outcome data. Journal of Clinical Epidemiology, 68, 52-60. http://dx.doi.org/10.1016/j.jclinepi.2014.08.012.
46. Rücker, G., Schwarzer, G., Carpenter, J. R., & Schumacher, M. (2008). Undue reliance on I(2) in assessing heterogeneity may mislead. BMC Medical Research Methodology, 8, 79. http://dx.doi.org/10.1186/1471- 2288-8-79.
47. Schmidt, F. L., Oh, I. S., & Hayes, T. L. (2009). Fixed- versus randomeffects models in meta-analysis: Model properties and an empirical comparison of differences in results. British Journal of Mathematical and Statistical Psychology, 62, 97-128. http://dx.doi.org/10.1348/ 000711007X255327.
48. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366. http://dx.doi.org/10.1177/0956797611417632.
49. Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014a). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143, 534-547. http://dx.doi.org/10.1037/a0033242.
50. Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014b). P-curve and effect size: Correcting for publication bias using only significant results. Perspectives on Psychological Science, 9, 666-681. http://dx.doi.org/ 10.1177/1745691614553988.
51. Smaldino, P. E., & McElreath, R. (2016). The natural selection of bad science. Royal Society Open Science, 3, 160384. http://dx.doi.org/10 .1098/rsos.160384.
52. Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance- or vice versa. Journal of the American Statistical Association, 54, 30-34. http://dx.doi.org/10 .1080/01621459.1959.10501497.
53. Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. The American Statistician, 49, 108-112.
54. Sterne, J. A. C., & Egger, M. (2006). Regression methods to detect publication bias and other bias in meta-analysis. In M. Borenstein, H. Rothstein, & A. J. Sutton (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 99-110). West Sussex, UK: Wiley. http://dx.doi.org/10.1002/0470870168.ch6.
55. Thompson, S. G., & Pocock, S. J. (1991). Can meta-analyses be trusted? Lancet, 338, 1127-1130. http://dx.doi.org/10.1016/0140-6736(91) 91975.Z.
56. Thorlund, K., Imberger, G., Johnston, B. C., Walsh, M., Awad, T., Thabane, L. Wetterslev, J. (2012). Evolution of heterogeneity (I2) estimates and their 95% confidence intervals in large meta-analyses. PLoS ONE, 7, e39471. http://dx.doi.org/10.1371/journal.pone.0039471.
57. van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2016). Conducting meta-analyses based on p values: Reservations and recommendations for applying p-uniform and p-curve. Perspectives on Psychological Science, 11, 713-729. http://dx.doi.org/10.1177/ 1745691616650874.
58. van Assen, M. A. L. M., van Aert, R. C. M., & Wicherts, J. M. (2015). Meta-analysis using effect size distributions of only statistically significant studies. Psychological Methods, 20, 293-309. http://dx.doi.org/10 .1037/met0000025.
59. Veroniki, A. A., Jackson, D., Viechtbauer, W., Bender, R., Bowden, J., Knapp, G. Salanti, G. (2016). Methods to estimate the betweenstudy variance and its uncertainty in meta-analysis. Research Synthesis Methods, 7, 55-79. http://dx.doi.org/10.1002/jrsm.1164.
60. Viechtbauer, W. (2005). Bias and efficiency of meta-analytic variance estimators in the random-effects model. Journal of Educational and Behavioral Statistics, 30, 261-293. http://dx.doi.org/10.3102/10769 986030003261.
61. Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid p-hacking. Frontiers in Psychology, 7, 1832. http:// dx.doi.org/10.3389/fpsyg.2016.01832.