Cautionary tales in the interpretation of systematic reviews of therapy trials

Internal Medicine Journal

Volumn 36, Issue 9, 2006, Pages 587-599

  Scott, I ^a,e   Greenberg, P ^b   Poole, P ^d   Campbell, D ^c  

^a UNIVERSITY OF QUEENSLAND   (Australia)

^b UNIVERSITY OF MELBOURNE   (Australia)

^c MONASH UNIVERSITY   (Australia)

^d UNIVERSITY OF AUCKLAND   (New Zealand)

^e PRINCESS ALEXANDRA HOSPITAL   (Australia)

Author keywords

Caution; Clinical interpretation; Systematic review

Indexed keywords

AUTHOR; BIBLIOGRAPHIC DATABASE; DATA EXTRACTION; DOUBLE BLIND PROCEDURE; DRUG INDUCED DISEASE; EFFECT SIZE; HEALTH CARE POLICY; META ANALYSIS; METHODOLOGY; OUTCOME ASSESSMENT; POST HOC ANALYSIS; PRACTICE GUIDELINE; PRIORITY JOURNAL; PROSPECTIVE STUDY; PUBLISHING; QUALITY CONTROL; RANDOMIZATION; REVIEW; RISK FACTOR; RISK REDUCTION; SENSITIVITY ANALYSIS; STATISTICAL ANALYSIS; STATISTICAL MODEL; SYSTEMATIC REVIEW; THERAPY;

HUMANS; META-ANALYSIS; PEER REVIEW, RESEARCH; PRACTICE GUIDELINES; RANDOMIZED CONTROLLED TRIALS; REVIEW LITERATURE; TREATMENT OUTCOME;

EID: 33746804191     PISSN: 14440903     EISSN: 14455994     Source Type: Journal    
DOI: 10.1111/j.1445-5994.2006.01140.x     Document Type: Review

References (84)

1. Cochrane Collaboration. Glossary. In: Cochrane Collaboration. Cochrane Library. Issue 2. Oxford: Update Software; 2003.
2. Lau J, Ioannidis JPA, Schmid CH. Quantitative synthesis in systematic reviews. Ann Intern Med 1997; 127: 820-6.
3. Mulrow CD. The medical review article: state of the science. Ann Intern Med 1987; 106: 485-8.
4. Sackett DL. Applying overviews and meta-analyses at the bedside. J Clin Epidemiol 1995; 48: 61-6.
5. Cook DJ, Greengold NL, Ellrodt AG, Weingarten SR. The relation between systematic reviews and practice guidelines. Ann Intern Med 1997; 127: 210-16.
6. Bero LA, Jadad AR. How consumers and policymakers can use systematic reviews for decision-making. Ann Intern Med 1997; 127: 37-42.
7. Rennie D, Bero L. The Cochrane Collaboration: preparing, maintaining, and disseminating systematic reviews of the effects of health care. JAMA 1995; 274: 1935-8.
8. Montori VM, Wilczynski NL, Morgan D, Haynes RB. The Hedges Team. Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. BMJ 2005; 330: 68-73.
9. Chalmers I. The scandalous failure of science to cumulate evidence scientifically. Clin Trials 2005; 2: 229-31.
10. Young C, Horton R. Putting clinical trials into context. Lancet 2005; 366: 107-8.
11. LeLorier J, Gregoire G, Benhaddad A, Lapierre J, Derderian F. Discrepancies between meta-analyses and subsequent large randomised, controlled trials. N Engl J Med 1997; 337: 536-42.
12. Cook DJ, Reeve BK, Guyatt GH, Heyland DK, Griffith LE, Buckingham L et al. Stress ulcer prophylaxis in critically ill patients. Resolving discordant meta-analyses. JAMA 1996; 275: 308-14.
13. Oxman AD, Cook DJ, Guyatt GH. The Evidence-Based Medicine Working Group. Users' guides to the medical literature. VI. How to use an overview. JAMA 1994; 272: 1367-71.
14. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. The QUOROM group. Improving the quality of reporting of meta-analysis of randomised controlled trials: the QUOROM statement. Lancet 1999; 354: 1896-900.
15. Richardson WS, Wilson MC, Nishikawa J, Hayward RSA. The well-built clinical question: a key to evidence-based decisions. ACP J Club 1995; 123: A12-A13.
16. Counsell C. Formulating questions and locating primary studies for inclusion in systematic reviews. Ann Intern Med 1997; 127: 380-7.
17. Dickersin K, Scherer R, Lefebve C. Identifying relevant studies for systematic reviews. In: Chalmers I, Altman DG, eds. Systematic Reviews. London: BMJ Publishing Group; 1995; 17-36.
18. Moher D, Fortin P, Jadad AR, Juni P, Klassen T, LeLorier J et al. Assessing the completeness of reporting of trials published in languages other than English: implications for the conduct of systematic reviews. Lancet 1996; 347: 363-6.
19. Royle P, Milne R. Literature searching for randomized controlled trials used in Cochrane reviews: rapid versus exhaustive literature searches. Int J Technol Assess Health Care 2003; 19: 591-603.
20. Avenell A, Handoll HH, Grant AM. Lessons for search strategies from a systematic review, in The Cochrane Library, of nutritional supplementation trials in patients after hip fracture. Am J Clin Nutr 2001; 73: 505-10.
21. McAuley L, Pham B, Tugwell P, Moher D. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet 2000; 356: 1228-31.
22. Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A et al. Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA 1993; 269: 2749-53.
23. Meade MO, Richardson WS. Selecting and appraising studies for a systematic review. Ann Intern Med 1997; 127: 531-7.
24. Rossouw JE, Lewis B, Rifkind BM. The value of lowering cholesterol after myocardial infarction. N Engl J Med 1990; 323: 1112-19.
25. Woodhill JM, Palmer AJ, Leelarthaepin B, McGilchrist C, Blacket RB. Low fat, low cholesterol diet in secondary prevention of coronary artery disease. Adv Exp Med Biol 1978; 109: 317-30.
26. Davey Smith G, Song F, Sheldon TA. Cholesterol lowering and mortality: the importance of considering initial level of risk. BMJ 1993; 306: 1367-73.
27. Naylor CD, O'Rourke K, Detsky AS, Baker JP. Parenteral nutrition with branched-chain amino acids in hepatic encephalopathy. Ameta-analysis. Gastroenterology 1989; 97: 1033-42.
28. DerSimonian R. Parenteral nutrition with branched-chain amino acids in hepatic encephalopathy: a meta-analysis. Hepatology 1990; 11: 1083-4.
29. Moher D, Jadad AR, Klassen TP. Guides for reading and interpreting systematic reviews. III. How did the authors synthesize the data and make their conclusions? Arch Pediatr Adolesc Med 1998; 152: 915-20.
30. Laupacis A, Sackett D, Roberts R. An assessment of clinically useful measures of the consequences of treatment. N Engl J Med 1988; 318: 728-33.
31. Silberberg JS, Henry DA. The benefits of reducing cholesterol levels: the need to distinguish primary from secondary prevention. 1. A meta-analysis of cholesterol lowering trials. Med J Aust 1991; 155: 665-70.
32. McQuay HJ, Moore RA. Using numerical results from systematic reviews in clinical practice. Ann Intern Med 1997; 126: 712-20.
33. Chatellier G, Zapletal E, Lemaitre D, Menard J, Degoulet P. The number needed to treat: a clinically useful nomogramin its proper context. BMJ 1996; 312: 426-9.
34. McAlister FA, Straus SE, Guyatt GH, Haynes RB. The Evidence-Based Medicine Working Group. Users' guides to the medical literature. Integrating research evidence with the care of the individual patient. JAMA 2000; 283: 2829-36.
35. Furukawa TA, Guyatt GH, Griffith LE. Can we individualize the 'number needed to treat'? An empirical study of summary effect measures in meta-analyses. Int J Epidemiol 2002; 31: 72-6.
36. Thompson SG. Why sources of heterogeneity in meta-analysis should be investigated. BMJ 1994; 309: 1351-5.
37. Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ 2003; 327: 557-60.
38. Fleiss JL. Statistical Methods for Rates and Proportions, 2nd edn. New York: J Wiley; 1981.
39. Pagliaro L, D'Amico G, Sorensen TIA, Lebrec D, Burroughs AK, Morabito A et al. Prevention of first bleeding in cirrhosis: a meta-analysis of randomised trials of non-surgical treatment. Ann Intern Med 1992; 117: 59-70.
40. Herbert RD, Bo K. Analysis of quality of interventions in systematic reviews. BMJ 2005; 331: 507-9.
41. L'Abbe K, Detsky A, O'Rourke K. Meta-analysis in clinical research. Ann Intern Med 1987; 107: 224-33.
42. Lau J, Ioannidis JP, Schmid CH. Summing up evidence: one answer is not always enough. Lancet 1998; 351: 123-7.
43. Thompson SG. Why and how sources of heterogeneity should be investigated. In: Egger M, Davey Smith G, Altman DG, eds. Systematic Reviews in Health Care: Meta-analysis in Context. London: BMJ Publishing Group; 2001; 157-75.
44. Berlin JA, Laird NM, Sacks HS, Chalmers TC. A comparison of statistical methods for combining event rates from clinical trials. Stat Med 1989; 8: 141-51.
45. Montori V, Guyatt G, Oxman A, Cook D. Summarizing the evidence. Fixed-effects and random-effects models. In: Guyatt G, Rennie D, eds. Users' Guides to the Medical Literature. A Manual for Evidence-based Clinical Practice. Chicago: AMA Press; 2002; 539-45.
46. Khan KS, Daya S, Jadad AR. The importance of quality of primary studies in producing unbiased systematic reviews. Arch Intern Med 1996; 156: 661-6.
47. Schultz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995; 273: 408-12.
48. Montori VM, Devereaux PJ, Adhikari NKJ, Burns KEA, Eggert CH, Briel M et al. Randomized trials stopped early for benefit: a systematic review. JAMA 2005; 294: 2203-9.
49. Moja LP, Telaro E, D'Amico R, Moschetti I, Coe L, Liberati A. The Metaquality Study Group. Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study. BMJ 2003; 330: 1053-7.
50. Devereaux PJ, Beattie WS, Choi PTL, Badner NH, Guyatt GH, Villar JC et al. How strong is the evidence for the use of perioperative β-blockers in non-cardiac surgery? Systematic review and meta-analysis of randomised controlled trials. BMJ 2005; 331: 313-21.
51. Chalmers TC, Smith H, Blackburn B, Silverman B, Schroeder B, Reitman D et al. A method for assessing the quality of a randomized control trial. Control Clin Trials 1981; 2: 31-49.
52. Jadad AR, Moore RA, Carrol D, Jenkinson C, Reynolds DJ, Gavaghan DJ et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 1996; 17: 1-12.
53. Juni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 1999; 282: 1054-60.
54. Balk EM, Bonis PAL, Moskowitz H, Schmid CH, Ioannidis JPA, Wang C et al. Correlation of quality measures with estimates of treatment effect in meta-analyses of randomised controlled trials. JAMA 2002; 287: 2973-82.
55. Copas J. What works? Selectivity models and meta-analysis. J R Stat Soc A 1999; 162: 95-109.
56. Freemantle N, Cleland J, Young P, Mason J, Harrison J. Beta-blockade after myocardial infarction: systematic review and meta-regression analysis. BMJ 1999; 318: 1730-7.
57. Oxman AD, Guyatt GH. A consumer's guide to subgroup analyses. Ann Intern Med 1992; 116: 78-84.
58. Yusuf S, Peto R, Lewis J, Collins R, Sleight P. Beta blockade during and after myocardial infarction: an overview of the randomised trials. Prog Cardiovasc Dis 1985; 17: 335-71.
59. Sterne JAC, Egger M, Davey Smith G. Investigating and dealing with publication and other biases in meta-analysis. BMJ 2001; 323: 101-5.
60. Egger M, Davey Smith G, Schneider M, Minder CE. Bias in meta-analysis is detected by a simple, graphical test. BMJ 1997; 315: 629-34.
61. Tang JL, Liu JL. Misleading funnel plot for detection of bias in meta-analysis. J Clin Epidemiol 2000; 53: 477-84.
62. Pogue J, Yusuf S. Overcoming the limitations of current meta-analyses of randomised controlled trials. Lancet 1998; 351: 47-52.
63. Yusuf S, Teo K, Woods K. Intravenous magnesium in acute myocardial infarction: an effective, safe, simple and inexpensive intervention. Circulation 1993; 87: 2043-6.
64. Woods K, Fletcher S, Roffe C, Haider Y. Intravenous magnesium sulphate in suspected acute myocardial infarction: results of the second Leicester Intravenous Magnesium Intervention Trial (LIMIT-2). Lancet 1992; 339: 1553-8.
65. ISIS-4 (Fourth International Study of Infarct Survival) Collaborative Group. ISIS-4: a randomised factorial trial assessing early oral captopril, oral mononitrate, and intravenous magnesium sulphate in 58,050 patients with suspected acute myocardial infarction. Lancet 1995; 345: 669-85.
66. Pogue J, Yusuf S. Cumulating evidence from randomised trials: utilising sequential monitoring boundaries for cumulative meta-analysis. Control Clin Trials 1997; 18: 580-93.
67. Ernst E, Pittler MH. Assessment of therapeutic safety in systematic reviews: literature review. BMJ 2001; 323: 546.
68. Sink KM, Holden KF, Yaffe K. Pharmacological treatment of neuropsychiatric symptoms of dementia. A review of the evidence. JAMA 2005; 293: 596-608.
69. Schneider LS, Dagerman KS, Insel P. Risk of death with atypical antipsychotic drug treatment for dementia. Meta-analysis of randomized placebo-controlled trials. JAMA 2005; 294: 1934-43.
70. Freemantle N, Mason J, Eccles M. Deriving treatment recommendations from evidence within randomised trials. The role and limitations of meta-analysis. Int J Technol Assess Health Care 1999; 15: 304-15.
71. Yank V, Rennie D, Bero L. Are authors' financial ties with pharmaceutical companies associated with positive results or conclusions in meta-analyses on antihypertensive medications [abstract]. Proceedings of the 5th International Congress on Peer Review and Biomedical Publication; 2005 Sept; Chicago.
72. Jadad AR, Cook DJ, Jones A, Klassen TP, Tugwell P, Moher M et al. Methodology and reports of systematic reviews and meta-analyses. A comparison of Cochrane reviews with articles published in paper-based journals. JAMA 1998; 280: 278-80.
73. Wood-Baker RR, Gibson PG, Hannay M, Walters EH, Walters JAE. Systemic corticosteroids for acute exacerbations of chronic obstructive pulmonary disease. Cochrane Database Syst Rev 2005; 1: CD001288. DOI: 10.1002/14651858. CD001288.pub2.
74. Leizorovicz A, Haugh MC, Chapuis F-R, Samama MM, Boissel JP. Low molecular weight heparin in prevention of perioperative thrombosis. BMJ 1992; 305: 913-20.
75. Nurmohamed MT, Rosendaal FR, Bueller HR, Dekker E, Hommes DW, Vandenbroucke JP et al. Low-molecular-weight heparin versus standard heparin in general and orthopaedic surgery: a meta-analysis. Lancet 1992; 340: 152-6.
76. Jadad AR, Cook DJ, Browman GP. A guide to interpreting discordant systematic reviews. CMAJ 1997; 156: 1411-16.
77. Shuman RB, Schuster DP, Zuckermann GR. Prophylactic therapy for stress ulcer bleeding: a reappraisal. Ann Intern Med 1987; 106: 562-7.
78. Lacroix J, Infante-Rivarde C, Jenicek M, Gauthier M. Prophylaxis of upper gastrointestinal bleeding in intensive care units: ameta-analysis. Crit Care Med 1989; 17: 862-8.
79. Cook DJ, Witt LG, Cook RJ, Guyatt GH. Stress ulcer prophylaxis in the critically ill: a meta-analysis. Am J Med 1991; 91: 519-27.
80. Tryba M. Prophylaxis of stress ulcer bleeding: a meta-analysis. J Clin Gastroenterol 1991; 13(Suppl. 2): 544-55.
81. Ioannidis JP, Lau J. On meta-analyses of meta-analyses [letter]. Lancet 1996; 348: 756.
82. Ausperin A, Pignon JP, Poynard T. Review article: critical review of meta-analyses of randomized clinical trials in hepatogastroenterology. Aliment Pharmacol Ther 1997; 11: 215-25.
83. Bramwell VH, Williams CJ. Do authors of review articles use systematic methods to identify, assess and synthesize information? Ann Oncol 1997; 8: 1185-95.
84. Raina PS, Brehaut JC, Platt RW, Klassen TP, Moher D, St. John P et al. The influence of display and statistical factors on the interpretation of metaanalysis results by physicians. Med Care 2005; 43: 1242-9.