Can systematic reviews with sparse data be trusted?

Anaesthesia

Volumn 72, Issue 1, 2017, Pages 12-16

  Afshari, A ^a   Wetterslev, J ^a   Smith, A F ^b  

^a University of Copenhagen   (Denmark)

^b ROYAL LANCASTER INFIRMARY   (United Kingdom)

Author keywords

power; statistical analysis; systematic reviews; trial sequential analysis

Indexed keywords

EDITOR; EDITORIAL; HEALTH CARE; HUMAN; JOURNAL IMPACT FACTOR; MEDICAL LITERATURE; MEDLINE; OUTCOME ASSESSMENT; PATIENT; POLICY; PUBLICATION; SCIENTIFIC LITERATURE; SYSTEMATIC REVIEW (TOPIC); EVIDENCE BASED MEDICINE; META ANALYSIS (TOPIC); METHODOLOGY; PROCEDURES; PUBLISHING; RANDOMIZED CONTROLLED TRIAL (TOPIC); STANDARDS; STATISTICAL ANALYSIS;

DATA INTERPRETATION, STATISTICAL; EVIDENCE-BASED MEDICINE; HUMANS; META-ANALYSIS AS TOPIC; PUBLICATION BIAS; RANDOMIZED CONTROLLED TRIALS AS TOPIC; RESEARCH DESIGN; SYSTEMATIC REVIEWS AS TOPIC;

EID: 84996866165     PISSN: 00032409     EISSN: 13652044     Source Type: Journal    
DOI: 10.1111/anae.13730     Document Type: Editorial

References (47)

1. US National Library of Medicine. Key MEDLINE indicators. 2015. http://www.nlm.nih.gov/bsd/bsd_key.html (accessed 01/09/2016).
2. Smith AF, Cracknell J, Møller AM, Pace NL. The Cochrane Collaboration and its worldwide contributions to anaesthesia research and care. British Journal of Anaesthesia 2013; 111: 523–5.
3. Page MJ, Shamseer L, Altman DG, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Medicine 2016; 13: e1002028.
4. Detweiler BN, Kollmorgen LE, Umberham BA, Hedin RJ, Vassar BM. Risk of bias and methodological appraisal practices in systematic reviews published in anaesthesia journals: a meta-epidemiological study. Anaesthesia 2016; 71: 955–68.
5. Ge L, Wang JC, Li JL, et al. The assessment of the quality of reporting of systematic reviews/meta-analyses in diagnostic tests published by authors in China. PLoS ONE 2014; 21: e85908.
6. Fleming PS, Koletsi D, Seehra J, Pandis N. Systematic reviews published in higher impact clinical journals were of higher quality. Journal of Clinical Epidemiology 2014; 67: 754–9.
7. Choi SW, Lam DMH. Trials and tribulations of a meta-analyst. Anaesthesia 2016; 71: 228–31.
8. Higgins J, Green S. Cochrane Handbook for Systematic Reviews of Interventions version 5.1.0 [updated March 2011]. London: The Cochrane Collaboration, 2011. http://handbook.cochrane.org/ (accessed 30/09/2016).
9. Garattini S, Jakobsen JC, Wetterslev J, et al. Evidence-based clinical practice: overview of threats to the validity of evidence and how to minimise them. European Journal of Internal Medicine 2016; 32: 3–21.
10. Smith AF, Carlisle J. Reviews, systematic reviews and Anaesthesia. Anaesthesia 2015; 70: 637–53.
11. Hennekens CH, DeMets D. The need for large-scale randomized evidence without undue emphasis on small trials, meta-analyses, or subgroup analyses. Journal of the American Medical Association 2009; 302: 2361–2.
12. Jakobsen JC, Gluud C, Winkel P, Lange T, Wetterslev J. The thresholds for statistical and clinical significance – a five-step procedure for evaluation of intervention effects in randomised clinical trials. BMC Medical Research Methodology 2014; 14: 34.
13. Keus F, Wetterslev J, Gluud C, van Laarhoven CJ. Evidence at a glance: error matrix approach for overviewing available evidence. BMC Medical Research Methodology 2010; 10: 90.
14. Mills EJ, Thorlund K, Ioannidis JP. Demystifying trial networks and network meta-analysis. British Medical Journal 2013; 346: f2914.
15. Hutton B, Salanti G, Caldwell DM, et al. The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: checklist and explanations. Annals of Internal Medicine 2015; 162: 777–84.
16. Savovic J, Jones HE, Altman DG, et al. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Annals of Internal Medicine 2012; 157: 429–38.
17. Jakobsen JC, Wetterslev J, Winkel P, Lange T, Gluud C. Thresholds for statistical and clinical significance in systematic reviews with meta-analytic methods. BMC Medical Research Methodology 2014; 14: 120.
18. Williamson PR, Altman DG, Blazeby JM, et al. Developing core outcome sets for clinical trials: issues to consider. Trials 2012; 13: 132.
19. Jammer I, Wickboldt N, Sander M, et al. Standards for definitions and use of outcome measures for clinical effectiveness research in perioperative medicine: European Perioperative Clinical Outcome (EPCO) definitions: A statement from the ESA-ESICM joint taskforce on perioperative outcome measures. European Journal of Anaesthesiology 2015; 32: 88–105.
20. Serrano-Aguilar P, Trujillo-Martin MM, Ramos-Goni JM, Mahtani-Chugani V, Perestelo-Perez L, Posada-de la Paz M. Patient involvement in health research: a contribution to a systematic review on the effectiveness of treatments for degenerative ataxias. Social Science and Medicine 2009; 69: 920–5.
21. Choi SW, Cheung CW. The case of the misleading composite – one outcome is better than two. Anaesthesia 2016; 71: 1101–3.
22. Cordoba G, Schwartz L, Woloshin S, Bae H, Gotzsche PC. Definition, reporting, and interpretation of composite outcomes in clinical trials: systematic review. British Medical Journal 2010; 341: c3920.
23. Garattini S, Bertele V. Non-inferiority trials are unethical because they disregard patients' interests. Lancet 2007; 370: 1875–7.
24. Skoog M, Saarimäki JM, Gluud C, et al. Transaprency and registration in clinical research in the Nordic countries. Nordic Trial Alliance. Nordic Research 2015; 1–108.
25. Inthout J, Ioannidis JP, Borm GF. Obtaining evidence by a single well-powered trial or several modestly powered trials. Statistical Methods in Medical Research 2016; 25: 538–52.
26. Page MJ, McKenzie JE, Kirkham J, et al. Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions. Cochrane Database of Systematic Reviews 2014; 10: MR000035.
27. Thorlund K, Imberger G, Walsh M, et al. The number of patients and events required to limit the risk of overestimation of intervention effects in meta-analysis – a simulation study. PLoS ONE 2011; 6: e25491.
28. Guimarães MMF, El Dib R, Smith AF, Matos D. Incentive spirometry for prevention of postoperative pulmonary complications in upper abdominal surgery. Cochrane Database of Systematic Reviews 2009; 3: CD006058.
29. Turner RM, Bird SM, Higgins JP. The impact of study size on meta-analyses: examination of underpowered studies in Cochrane reviews. PLoS ONE 2013; 8: e59202.
30. Imberger G, Gluud C, Boylan J, Wetterslev J. Systematic reviews of anesthesiologic interventions reported as statistically significant: problems with power, precision, and type 1 error protection. Anesthesia and Analgesia 2015; 121: 1611–22.
31. Borm GF, Donders AR. Updating meta-analyses leads to larger type I errors than publication bias. Journal of Clinical Epidemiology 2009; 62: 825–30, e10.
32. Wetterslev J, Thorlund K, Brok J, Gluud C. Estimating required information size by quantifying diversity in random-effects model meta-analyses. BMC Medical Research Methodology 2009; 9: 86.
33. Wetterslev J, Thorlund K, Brok J, Gluud C. Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis. Journal of Clinical Epidemiology 2008; 61: 64–75.
34. Brok J, Thorlund K, Gluud C, Wetterslev J. Trial sequential analysis reveals insufficient information size and potentially false results in many meta-analyses. Journal of Clinical Epidemiology 2008; 61: 763–9.
35. van der Tweel I, Bollen C. Sequential meta-analysis: an efficient decision-making tool. Clinical Trials 2010; 7: 136–46.
36. Hu M, Cappelleri JC, Lan KK. Applying the law of iterated logarithm to control type I error in cumulative meta-analysis of binary outcomes. Clinical Trials 2007; 4: 329–40.
37. Higgins JPT, Whitehead A, Simmonds M. Sequential methods for random-effects meta-analysis. Statistics in Medicine 2011; 30: 903–21.
38. Abad-Gurumeta A, Ripollés-Melchor J, Casans-Francés R, et al. A systematic review of sugammadex vs neostigmine for reversal of neuromuscular blockade. Anaesthesia 2015; 70: 1441–52.
39. Imberger G, Thorlund K, Gluud C, Wetterslev J. False-positive findings in Cochrane meta-analyses with and without application of trial sequential analysis: an empirical review. BMJ Open 2016; 6: e011890.
40. Brok J, Thorlund K, Wetterslev J, Gluud C. Apparently conclusive meta-analyses may be inconclusive – trial sequential analysis adjustment of random error risk due to repetitive testing of accumulating data in apparently conclusive neonatal meta-analyses. International Journal of Epidemiology 2009; 38: 287–98.
41. Kulinskaya E, Wood J. Trial sequential methods for meta-analysis. Research Synthesis Methods 2014; 5: 212–20.
42. Pereira TV, Ioannidis JP. Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects. Journal of Clinical Epidemiology 2011; 64: 1060–9.
43. Afshari A, Wetterslev J. When may systematic reviews and meta-analyses be considered reliable? European Journal of Anaesthesiology 2015; 32: 85–7.
44. Roberts I, Ker K, Edwards P, Beecher D, Manno D, Sydenham E. The knowledge system underpinning healthcare is not fit for purpose and must change. British Medical Journal 2015; 350: h2463.
45. Song F, Parekh S, Hooper L, et al. Dissemination and publication of research findings: an updated review of related biases. Health Technology Assessment 2010; 14: 1–193.
46. Ioannidis JP, Greenland S, Hlatky MA, et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet 2014; 383: 166–75.
47. Ioannidis JP. Meta-analyses of hydroxyethyl starch for volume resuscitation. Journal of the American Medical Association 2013; 309: 2209.