Interpretation of tests of heterogeneity and bias in meta-analysis

Journal of Evaluation in Clinical Practice

Volumn 14, Issue 5, 2008, Pages 951-957

  Ioannidis, John P A ^a,b  

^a UNIVERSITY OF IOANNINA   (Greece)

^b TUFTS UNIVERSITY SCHOOL OF MEDICINE   (United States)

Author keywords

Bias; Heterogeneity; Meta analysis; Publication bias; Selective reporting bias

Indexed keywords

ANALYTICAL ERROR; CORRELATION ANALYSIS; PRIORITY JOURNAL; PUBLISHING; REVIEW; STATISTICAL ANALYSIS;

BIAS (EPIDEMIOLOGY); CONFIDENCE INTERVALS; DATA INTERPRETATION, STATISTICAL; EVIDENCE-BASED MEDICINE; HUMANS; META-ANALYSIS AS TOPIC; PUBLICATION BIAS; REGRESSION ANALYSIS; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; RISK; SENSITIVITY AND SPECIFICITY;

EID: 55349132400     PISSN: 13561294     EISSN: 13652753     Source Type: Journal    
DOI: 10.1111/j.1365-2753.2008.00986.x     Document Type: Review

References (57)

1. Sackett, D. L., Rosenberg, W. M., Gray, J. A., Haynes, R. B. Richardson, W. S. (1996) Evidence based medicine: what it is and what it isn't. British Medical Journal, 312, 71 72.
2. Patsopoulos, N. A., Analatos, A. A. Ioannidis, J. P. (2005) Relative citation impact of various study designs in the health sciences. Journal of the American Medical Association, 293, 2362 2366.
3. Lau, J., Ioannidis, J. P. Schmid, C. H. (1998) Summing up evidence: one answer is not always enough. Lancet, 351, 123 127.
4. Sutton, A., Abrams, K., Jones, D., Sheldon, T. Song, F. (2000) Methods for Meta-analysis in Medical Research. Chichester : Wiley.
5. Gluud, L. L. (2006) Bias in clinical intervention research. American Journal of Epidemiology, 163, 493 501.
6. Dickersin, K. (1990) The existence of publication bias and risk factors for its occurrence. Journal of the American Medical Association, 263, 1385 1389.
7. Easterbrook, P. J., Berlin, J. A., Gopalan, R. Matthews, D. R. (1991) Publication bias in clinical research. Lancet, 337, 867 872.
8. Rothstein, H., Sutton, A. J. Borestein, M., eds. (2005) Publication Bias in Meta-analysis: Prevention, Assessment and Adjustments. Chichester : Wiley.
9. Begg, C. B. Mazumdar, M. (1994) Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088 1101.
10. Egger, M., Davey Smith, G., Schneider, M. Minder, C. (1997) Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629 634.
11. Duval, S. Tweedie, R. (2000) A nonparametric 'Trim and Fill' method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association, 95, 89 98.
12. Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R. Rushton, L. (2006) Comparison of two methods to detect publication bias in meta-analysis. Journal of the American Medical Association, 295, 676 680.
13. Harbord, R. M., Egger, M. Sterne, J. A. (2006) A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints. Statistics in Medicine, 25, 3443 3457.
14. Ioannidis, J. P. Trikalinos, T. A. (2007) The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey. Canadian Medical Association Journal, 176, 1091 1096.
15. Terrin, N., Schmid, C. H. Lau, J. (2005) In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. Journal of Clinical Epidemiology, 58, 894 901.
16. Cochran, W. G. (1954) The combination of estimates from different experiments. Biometrics, 10, 101 129.
17. Lau, J., Ioannidis, J. P. Schmid, C. H. (1997) Quantitative synthesis in systematic reviews. Annals of Internal Medicine, 127, 820 826.
18. Higgins, J. P. T. Thompson, S. G. (2002) Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21, 1539 1558.
19. Higgins, J. P. T., Thompson, S. G., Deeks, J. Altman, D. G. (2003) Measuring inconsistency in meta-analyses. British Medical Journal, 327, 557 560.
20. Mittlbock, M. Heinzl, H. (2006) A simulation study comparing properties of heterogeneity measures in meta-analyses. Statistics in Medicine, 25, 4321 4333.
21. Ioannidis, J. P., Patsopoulos, N. A. Evangelou, E. (2007) Uncertainty in heterogeneity estimates in meta-analyses. British Medical Journal, 335, 914 916.
22. Engels, E. A., Schmid, C. H., Terrin, N., Olkin, I. Lau, J. (2000) Heterogeneity and statistical significance in meta-analysis: an empirical study of 125 meta-analyses. Statistics in Medicine, 19, 1707 1728.
23. Ioannidis, J. P., Trikalinos, T. A. Zintzaras, E. (2006) Extreme between-study homogeneity in meta-analyses could offer useful insights. Journal of Clinical Epidemiology, 59, 1023 1032.
24. Kent, D. M. Hayward, R. A. (2007) Limitations of applying summary results of clinical trials to individual patients: the need for risk stratification. Journal of the American Medical Association, 298, 1209 1212.
25. Ioannidis, J. P. (2005) Why most published research findings are false. PLoS Medicine, 2, e124.
26. Huedo-Medina, T. B., Sanchez-Meca, J., Marin-Martinez, F. Botella, J. (2006) Assessing heterogeneity in meta-analysis: Q statistic or I2 index? Psychology Methods, 11, 193 206.
27. Macaskill, P., Walter, S. D. Irwig, L. (2001) A comparison of methods to detect publication bias in meta-analysis. Statistics in Medicine, 20, 641 654.
28. Lau, J., Ioannidis, J. P., Terrin, N., Schmid, C. H. Olkin, I. (2006) The case of the misleading funnel plot. British Medical Journal, 333, 597 600.
29. Schwarzer, G., Antes, G. Schumacher, M. (2002) Inflation of type I error rate in two statistical tests for the detection of publication bias in meta-analyses with binary outcomes. Statistics in Medicine, 21, 2465 2477.
30. Oxman, A. D. Guyatt, G. H. (1992) A consumer's guide to subgroup analyses. Annals of Internal Medicine, 116, 78 84.
31. Wang, R., Lagakos, S. W., Ware, J. H., Hunter, D. J. Drazen, J. M. (2007) Statistics in medicine - reporting of subgroup analyses in clinical trials. New England Journal of Medicine, 357, 2189 2194.
32. Patsopoulos, N. A., Tatsioni, A. Ioannidis, J. P. (2007) Claims of sex differences: an empirical assessment in genetic associations. Journal of the American Medical Association, 298, 880 893.
33. Simmonds, M. C., Higgins, J. P., Stewart, L. A., Tierney, J. F., Clarke, M. J. Thompson, S. G. (2005) Meta-analysis of individual patient data from randomized trials: a review of methods used in practice. Clinical Trials, 2, 209 217.
34. Ioannidis, J. P., Rosenberg, P. S., Goedert, J. J. O'Brien, T. R. (2002) Commentary: meta-analysis of individual participants' data in genetic epidemiology. American Journal of Epidemiology, 156, 204 210.
35. Ioannidis, J. P. (2005) Differentiating biases from genuine heterogeneity: distinguishing artifactual from substantive effects. In Publication Bias in Meta-analysis: Prevention, Assessment and Adjustments (eds H. Rothstein, A. J. Sutton M. Borestein), Chichester : Wiley.
36. Ioannidis, J. P., Cappelleri, J. C., Sacks, H. S. Lau, J. (1997) The relationship between study design, results, and reporting of randomized clinical trials of HIV infection. Controlled Clinical Trials, 18, 431 444.
37. Ioannidis, J. P. Lau, J. (1998) Heterogeneity of the baseline risk within patient populations of clinical trials: a proposed evaluation algorithm. American Journal of Epidemiology, 148, 1117 1126.
38. Ioannidis, J. P. Lau, J. (1997) The impact of high-risk patients on the results of clinical trials. Journal of Clinical Epidemiology, 50, 1089 1098.
39. McIntosh, M. W. (1996) The population risk as an explanatory variable in research synthesis of clinical trials. Statistics in Medicine, 15, 1713 1728.
40. Galbraith, R. F. (1988) A note on graphical presentation of estimated odds ratios from several clinical trials. Statistics in Medicine, 7, 889 894.
41. Baujat, B., Mahe, C., Pignon, J. P. Hill, C. (2002) A graphical method for exploring heterogeneity in meta-analyses: application to a meta-analysis of 65 trials. Statistics in Medicine, 21, 2641 2652.
42. Patsopoulos, N. A., Evangelou, E. Ioannidis, J. P. (2008) Sensitivity of between-study heterogeneity in meta-analysis: proposed metrics and empirical evaluation. International Journal of Epidemiology (in press) [Epub ahead of print, 18 April 2008].
43. Schmid, C. H., Stark, P. C., Berlin, J. A., Landais, P. Lau, J. (2004) Meta-regression detected associations between heterogeneous treatment effects and study-level, but not patient-level, factors. Journal of Clinical Epidemiology, 57, 683 697.
44. Lambert, P. C., Sutton, A. J., Abrams, K. R. Jones, D. R. (2002) A comparison of summary patient-level covariates in meta-regression with individual patient data meta-analysis. Journal of Clinical Epidemiology, 55, 86 94.
45. Thompson, S. G. Higgins, J. P. (2002) How should meta-regression analyses be undertaken and interpreted? Statistics in Medicine, 21, 1559 1573.
46. Higgins, J. P. Thompson, S. G. (2004) Controlling the risk of spurious findings from meta-regression. Statistics in Medicine, 23, 1663 1682.
47. Chan, A. W. Altman, D. G. (2005) Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. British Medical Journal, 330, 753.
48. Chan, A. W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C. Altman, D. G. (2004) Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. Journal of the American Medical Association, 291, 2457 2465.
49. Kavvoura, F. K., Liberopoulos, G. Ioannidis, J. P. (2007) Selection in reported epidemiological risks: an empirical assessment. PLoS Medicine, 4, e79.
50. Ioannidis, J. P. Trikalinos, T. A. (2007) An exploratory test for an excess of significant findings. Clinical Trials, 4, 245 253.
51. Laine, C., Horton, R., DeAngelis, C. D., et al. (2007) Clinical trial registration: looking back and moving ahead. Lancet, 369, 1909 1911.
52. Lau, J., Schmid, C. H. Chalmers, T. C. (1995) Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care. Journal of Clinical Epidemiology, 48, 45 57.
53. Ioannidis, J. P. Lau, J. (2001) Evolution of treatment effects over time: empirical evidence from recursive cumulative meta-analyses. Proceedings of the National Academy of Sciences, USA, 98, 831 836.
54. Ioannidis, J. P., Contopoulos-Ioannidis, D. G. Lau, J. (1999) Recursive cumulative meta-analysis: a diagnostic for the evolution of total randomized evidence from group and individual patient data. Journal of Clinical Epidemiology, 52, 281 291.
55. Sterne, J. A. Egger, M. (2001) Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis. Journal of Clinical Epidemiology, 54, 1046 1055.
56. Tang, J.-L. Liu, J. L. Y. (2000) Misleading funnel plot for detection of bias in meta-analysis. Journal of Clinical Epidemiology, 53, 477 484.
57. Djulbegovic, B. Hozo, I. (2007) When should potentially false research findings be considered acceptable? PLoS Medicine, 4, e26.