Methods to systematically review and meta-analyse observational studies: A systematic scoping review of recommendations

BMC Medical Research Methodology

Volumn 18, Issue 1, 2018, Pages

  Mueller, Monika ^a   D'Addario, Maddalena ^a   Egger, Matthias ^a   Cevallos, Myriam ^b   Dekkers, Olaf ^c,d   Mugglin, Catrina ^a   Scott, Pippa ^e  

^a UNIVERSITY OF BERN   (Switzerland)

^b UNIVERSITY OF BERN   (Switzerland)

^c LEIDEN UNIVERSITY MEDICAL CENTER   (Netherlands)

^d AARHUS UNIVERSITY HOSPITAL   (Denmark)

^e UNIVERSITY OF OTAGO   (New Zealand)

Author keywords

Meta analysis; Methods; Observational studies; Recommendation; Systematic review

Indexed keywords

HUMAN; META ANALYSIS (TOPIC); OBSERVATIONAL STUDY; PRACTICE GUIDELINE; PUBLICATION; STANDARDS;

GUIDELINES AS TOPIC; HUMANS; META-ANALYSIS AS TOPIC; OBSERVATIONAL STUDIES AS TOPIC; PUBLICATIONS; SYSTEMATIC REVIEWS AS TOPIC;

EID: 85047738148     PISSN: None     EISSN: 14712288     Source Type: Journal    
DOI: 10.1186/s12874-018-0495-9     Document Type: Review

References (118)

1. Black N. Why we need observational studies to evaluate the effectiveness of health care. BMJ. 1996;312:1215-8.
2. Funai EF, Rosenbush EJ, Lee MJ, Del Priore G. Distribution of study designs in four major US journals of obstetrics and gynecology. Gynecol Obstet Investig. 2001;51:8-11.
3. Scales CD, Norris RD, Peterson BL, Preminger GM, Dahm P. Clinical research and statistical methods in the urology literature. J Urol. 2005;174(4 Pt 1):1374-9.
4. The Framingham Heart Study. https://www.framinghamheartstudy.org. Accessed 20 Jun 2017.
5. The National Child Development Study. http://www.cls.ioe.ac.uk. Accessed 20 Jun 2017.
6. The Dunedin Multidisciplinary Health & Development Study. http://dunedinstudy.otago.ac.nz. Accessed 20 Jun 2017.
7. Page MJ, Shamseer L, Altman DG, Tetzlaff J, Sampson M, Tricco AC, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med. 2016;13:e1002028.
8. Zwahlen M, Renehan A, Egger M. Meta-analysis in medical research: potentials and limitations. Urol Oncol. 2008;26:320-9.
9. Egger M, Schneider M, Davey SG. Spurious precision? Meta-analysis of observational studies. BMJ. 1998;316:140-4.
10. Higgins JPT, Green S. Cochrane Handbook. Version 5.1.0. 2011. http://handbook.cochrane.org. Accessed 20 Jun 2017.
11. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009;339:b2700.
12. Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis of observational studies in epidemiology (MOOSE) group. JAMA. 2000;283:2008-12.
13. Peters MDJ, Godfrey CM, Khalil H, Mcinerney P, Parker D, Soares CB. Guidance for conducting systematic scoping reviews. Int J Evid Based Heal. 2015;13:141-6.
14. Colquhoun HL, Levac D, Brien KKO, Straus S, Tricco AC, Perrier L, et al. Scoping reviews: time for clarity in definition, methods, and reporting. J Clin Epidemiol. 2014;67:1291-4.
15. Tricco AC, Lillie E, Zarin W, Brien KO, Colquhoun H, Kastner M, et al. A scoping review on the conduct and reporting of scoping reviews. BMC Med Res Methodol. 2016;16:15.
16. Lau J, Ioannidis JP, Schmid CH. Quantitative synthesis in systematic reviews. Ann Intern Med. 1997;127:820-6.
17. Lilford RJ, Richardson A, Stevens A, Fitzpatrick R, Edwards S, Rock F, et al. Issues in methodological research: perspectives from researchers and commissioners. Health Technol Assess. 2001;5:1-57.
18. Rutjes A, Reitsma J, Coomarasamy A, Khan K, Bossuyt P. Evaluation of diagnostic tests when there is no gold standard. A review of methods. Health Technol Assess. 2007;11(50).
19. Cochrane Methodological Register. https://community.cochrane.org. Accessed 20 Jun 2017.
20. The Cochrane Library. http://www.cochranelibrary.com. Accessed 20 Jun 2017.
21. The Center for Reviews and Dissemination. http://www.york.ac.uk/inst/crd/index.htm. Accessed 20 Jun 2017.
22. The Campbell Collaboration. http://www.campbellcollaboration.org. Accessed 20 Jun 2017.
23. The Scottish Intercollegiate Guidelines Network. http://www.sign.ac.uk/index.html. Accessed 20 Jun 2017.
24. The Agency for Healthcare Research and Quality. http://www.ahrq.gov. Accessed 20 Jun 2017.
25. The EQUATOR Network. http://www.equator-network.org. Accessed 20 Jun 2017.
26. National Institute for Health and Care Excellence. http://www.nice.org.uk. Accessed 20 Jun 2017.
27. The Effective Public Health Practice Project. https://merst.ca/ephpp. Accessed 20 Jun 2017.
28. The Joanna Briggs Institute. http://joannabriggs.org. Accessed 20 Jun 2017.
29. Egger M, Smith GD, Phillips AN. Meta-analysis: principles and procedures. BMJ. 1997;315:1533-7.
30. Khoshdel A, Attia J, Carney SL. Basic concepts in meta-analysis: a primer for clinicians. Int J Clin Pr. 2006;60:1287-94.
31. Friedenreich CM, Brant RF, Riboli E. Influence of methodologic factors in a pooled analysis of 13 case-control studies of colorectal cancer and dietary fiber. Epidemiology. 1994;5:66-79.
32. Chambers D, Rodgers M, Woolacott N. Not only randomized controlled trials, but also case series should be considered in systematic reviews of rapidly developing technologies. J Clin Epidemiol. 2009;62:1253-60. e4
33. Sagoo GS, Little J, Higgins JPT. Systematic reviews of genetic association studies. Human Genome Epidemiology Network PLoS Med. 2009;6:e28.
34. Blair A, Burg J, Foran J, Gibb H, Greenland S, Morris R, et al. Guidelines for application of meta-analysis in environmental epidemiology. ISLI risk science institute. Regul Toxicol Pharmacol. 1995;22:189-97.
35. Loke YK, Price D, Herxheimer A. Systematic reviews of adverse effects: framework for a structured approach. BMC Med Res Methodol. 2007;7:32.
36. Ashford D, Davids K, Bennett SJ. Difference-based meta-analytic procedures for between-participant and/or within-participant designs: a tutorial review for sports and exercise scientists. J Sport Sci. 2009;27:237-55.
37. Wells GA, Shea B, Higgins JPT, Sterne J, Tugwell P, Reeves BC. Checklists of methodological issues for review authors to consider when including non-randomized studies in systematic reviews. Res Synth Methods. 2013;4:63-77.
38. Simunovic N, Sprague S, Bhandari M. Methodological issues in systematic reviews and meta-analyses of observational studies in orthopaedic research. J Bone Jt Surg Am. 2009;91(Suppl 3):87-94.
39. Armstrong R, Waters E, editors. Systematic Reviews of Health Promotion and Public Health Interventions. Guidelines for Systematic Reviews in Health Promotion and Public Health Task Force. Version 2. Melbourne University: Australia.; 2007.
40. Schünemann HJ, Tugwell P, Reeves BC, Akl EA, Santesso N, Spencer FA, et al. Non-randomized studies as a source of complementary, sequential or replacement evidence for randomized controlled trials in systematic reviews on the effects of interventions. Res Synth Methods. 2013;4:49-62.
41. Moola S, Munn Z, Sears K, Sfetcu R, Currie M, Lisy K, et al. Conducting systematic reviews of association (etiology): the Joanna Briggs Institute's approach. Int J Evid Based Healthc. 2015;13:163-9.
42. O'Connor AM, Sargeant JM. Meta-analyses including data from observational studies. Prev Vet Med. 2014;113:313-22.
43. Ravani P, Ronksley PE, James MT, Strippoli GF. A nephrology guide to reading and using systematic reviews of observational studies. Nephrol Dial Transplant. 2015;30:1615-21.
44. Tufanaru C, Munn Z, Stephenson M, Aromataris E. Fixed or random effects meta-analysis? Common methodological issues in systematic reviews of effectiveness. Int J Evid Based Healthc. 2015;13:196-207.
45. Guyatt GH, Oxman AD, Kunz R, Atkins D, Brozek J, Vist G, et al. GRADE guidelines: 2. Framing the question and deciding on important outcomes. J Clin Epidemiol. 2011;64:395-400.
46. MacDonald-Jankowski DS, Dozier MF. Systematic review in diagnostic radiology. Dentomaxillofac Radiol. 2001;30:78-83.
47. Manchikanti L, Datta S, Smith HS, Hirsch JA. Evidence-based medicine, systematic reviews, and guidelines in interventional pain management: part 6. Systematic reviews and meta-analyses of observational studies. Pain Physician. 2009;12:819-50.
48. Normand SL. Meta-analysis: formulating, evaluating, combining, and reporting. Stat Med. 1999;18:321-59.
49. Price D, Jefferson T, Demicheli V. Methodological issues arising from systematic reviews of the evidence of safety of vaccines. Vaccine. 2004;22:2080-4.
50. Raman G, Gaylor JM, Rao M, Chan J, Earley A, Chang LKW, et al. AHRQ methods for effective health care. In: Quality of reporting in systematic reviews of implantable medical devices. Rockville (MD): Agency for Healthcare Research and Quality (US); 2012.
51. Rosenthal R, DiMatteo MR. Meta-analysis: recent developments in quantitative methods for literature reviews. Annu Rev Psychol. 2001;52:59-82.
52. Tak LM, Meijer A, Manoharan A, de Jonge P, Rosmalen JG. More than the sum of its parts: meta-analysis and its potential to discover sources of heterogeneity in psychosomatic medicine. Psychosom Med. 2010;72:253-65.
53. Thomas BH, Ciliska D, Dobbins M, Micucci S. A process for systematically reviewing the literature: providing the research evidence for public health nursing interventions. Worldviews Evid-Based Nurs. 2004;1:176-84.
54. Chaiyakunapruk N, Saokaew S, Sruamsiri R, Dilokthornsakul P. Systematic review and network meta-analysis in health technology assessment. J Med Assoc Thail. 2014;97(Suppl 5):S33-42.
55. Mahid SS, Hornung CA, Minor KS, Turina M, Galandiuk S. Systematic reviews and meta-analysis for the surgeon scientist. Br J Surg. 2006;93:1315-24.
56. Wille-Jorgensen P, Renehan AG. Systematic reviews and meta-analyses in coloproctology: interpretation and potential pitfalls. Color Dis. 2008;10:21-32.
57. Haynes RB, Kastner M, Wilczynski NL, Hedges T. Developing optimal search strategies for detecting clinically sound and relevant causation studies in EMBASE. BMC Med Inf Decis Mak. 2005;5:8.
58. Fraser C, Murray A, Burr J. Identifying observational studies of surgical interventions in MEDLINE and EMBASE. BMC Med Res Methodol. 2006;6:41.
59. Furlan AD, Irvin E, Bombardier C. Limited search strategies were effective in finding relevant nonrandomized studies. J Clin Epidemiol. 2006;59:1303-11.
60. Golder S, Loke Y, McIntosh HM. Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects. J Clin Epidemiol. 2008;61:440-8.
61. Thornton A, Lee P. Publication bias in meta-analysis: its causes and consequences. J Clin Epidemiol. 2000;53:207-16.
62. Kuper H, Nicholson A, Hemingway H. Searching for observational studies: what does citation tracking add to PubMed? A case study in depression and coronary heart disease. BMC Med Res Methodol. 2006;6:4.
63. Lemeshow AR, Blum RE, Berlin JA, Stoto MA, Colditz GA. Searching one or two databases was insufficient for meta-analysis of observational studies. J Clin Epidemiol. 2005;58:867-73.
64. Loke YK, Golder SP, Vandenbroucke JP. Comprehensive evaluations of the adverse effects of drugs: importance of appropriate study selection and data sources. Ther Adv Drug Saf. 2011;2:59-68.
65. Higgins JPT, Ramsay C, Reeves BC, Deeks JJ, Shea B, Valentine JC, et al. Issues relating to study design and risk of bias when including non-randomized studies in systematic reviews on the effects of interventions. Res Synth Methods. 2013;4:12-25.
66. Winegardner ML, Reaume KT, Dabaja GS, Kalus JS. Practical application of nonrandomized research to patient care: a case study of nesiritide. Pharmacotherapy. 2007;27:143-51.
67. Naumann. How to develop a search strategy. 2007. https://www.researchgate.net/file.PostFileLoader.html?id=5996c4eb48954ceeca4bf303&assetKey=AS%3A528656902483968%401503053035454. Accessed 13 Feb 2018.
68. Stansfield C, Dickson K, Bangpan M, Oliver S, Bangpan M, Stansfield C, et al. Exploring issues in the conduct of website searching and other online sources for systematic reviews: how can we be systematic? Syst Rev. 2016;5:191.
69. Thompson J, Davis J, Mazerolle L. A systematic method for search term selection in systematic reviews. Res Synth Methods. 2014;5:87-97.
70. Wong O, Raabe GK. Application of meta-analysis in reviewing occupational cohort studies. Occup Environ Med. 1996;53:793-800.
71. Norris SL, Moher D, Reeves BC, Shea B, Loke Y, Garner S, et al. Issues relating to selective reporting when including non-randomized studies in systematic reviews on the effects of healthcare interventions. Res Synth Methods. 2013;4:36-47.
72. Munn Z, Moola S, Lisy K, Riitano D, Tufanaru C. Methodological guidance for systematic reivews of observational epidemiological studies reporting prevalence and cumulative incidence data. Int J Evid Based Healthc. 2015;13:147-53.
73. Zingg W, Castro-Sanchez E, Secci FV, Edwards R, Drumright LN, Sevdalis N, et al. Innovative tools for quality assessment: integrated quality criteria for review of multiple study designs (ICROMS). Public Health. 2016;133:19-37.
74. Horton J, Vandermeer B, Hartling L, Tjosvold L, Klassen TP, Buscemi N. Systematic review data extraction: cross-sectional study showed that experience did not increase accuracy. J Clin Epidemiol. 2010;63:289-98.
75. Valentine JC, Thompson SG. Issues relating to confounding and meta-analysis when including non-randomized studies in systematic reviews on the effects of interventions. Res Synth Methods. 2013;4:26-35.
76. Austin H, Perkins LL, Martin DO. Estimating a relative risk across sparse case-control and follow-up studies: a method for meta-analysis. Stat Med. 1997;16:1005-15.
77. Colditz GA, Burdick E, Mosteller F. Heterogeneity in meta-analysis of data from epidemiologic studies: a commentary. Am J Epidemiol. 1995;142:371-82.
78. Sutton AJ, Cooper NJ, Lambert PC, Jones DR, Abrams KR, Sweeting MJ. Meta-analysis of rare and adverse event data. Expert Rev Pharmacoecon Outcomes Res. 2002;2:367-79.
79. Martin DO, Austin H. An exact method for meta-analysis of case-control and follow-up studies. Epidemiology. 2000;11:255-60.
80. Moreno V, Martin ML, Bosch FX, de Sanjose S, Torres F, Munoz N. Combined analysis of matched and unmatched case-control studies: comparison of risk estimates from different studies. Am J Epidemiol. 1996;143:293-300.
81. Souverein OW, Dullemeijer C, van't Veer P, van der Voet H. Transformations of summary statistics as input in meta-analysis for linear dose-response models on a logarithmic scale: a methodology developed within EURRECA. BMC Med Res Methodol. 2012;12:57.
82. Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919.
83. Verde PE, Ohmann C. Combining randomized and nonrandomized evidence in clinical research: a review of methods and applications. Res Synth Methods. 2015;6:45-62.
84. Balshem H, Helfand M, Schunemann HJ, Oxman AD, Kunz R, Brozek J, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol. 2011;64:401-6.
85. Greenland S. Invited commentary: a critical look at some popular meta-analytic methods. Am J Epidemiol. 1994;140:290-6.
86. Wong WC, Cheung CS, Hart GJ. Development of a quality assessment tool for systematic reviews of observational studies (QATSO) of HIV prevalence in men having sex with men and associated risk behaviours. Emerg Themes Epidemiol. 2008;5:23.
87. Salanti G, Sanderson S, Higgins JP. Obstacles and opportunities in meta-analysis of genetic association studies. Genet Med. 2005;7:13-20.
88. Ioannidis JP. Commentary: adjusting for bias: a user's guide to performing plastic surgery on meta-analyses of observational studies. Int J Epidemiol. 2011;40:777-9.
89. Shamliyan T, Kane RL, Jansen S. Systematic reviews synthesized evidence without consistent quality assessment of primary studies examining epidemiology of chronic diseases. J Clin Epidemiol. 2012;65:610-8.
90. Sanderson S, Tatt ID, Higgins JP. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol. 2007;36:666-76.
91. West S, King V, Carey TS, Lohr KN, McKoy N, Sutton SF, et al. Systems to rate the strength of scientific evidence. Evid Rep Technol Assess (Summ). 2002:1-11.
92. Herbison P, Hay-Smith J, Gillespie WJ. Adjustment of meta-analyses on the basis of quality scores should be abandoned. J Clin Epidemiol. 2006;59:1249-56.
93. Guyatt GH, Oxman AD, Vist G, Kunz R, Brozek J, Alonso-Coello P, et al. GRADE guidelines: 4. Rating the quality of evidence - study limitations (risk of bias). J Clin Epidemiol. 2011;64:407-15.
94. Robertson C, Ramsay C, Gurung T, Mowatt G, Pickard R, Sharma P. Practicalities of using a modified version of the Cochrane collaboration risk of bias tool for randomised and non-randomised study designs applied in a health technology assessment setting. Res Synth Methods. 2014;5:200-11.
95. Guyatt GH, Oxman AD, Montori V, Vist G, Kunz R, Brozek J, et al. GRADE guidelines: 5. Rating the quality of evidence -publication bias. J Clin Epidemiol. 2011;64:1277-82.
96. Davey Smith G, Egger M. Meta-analysis. Unresolved issues and future developments. BMJ. 1998;316:221-5.
97. Egger M, Smith GD. Bias in location and selection of studies. BMJ. 1998;316:61-6.
98. Davey Smith G, Egger M, Phillips AN. Meta-analysis. Beyond the grand mean? BMJ. 1997;315:1610-4.
99. Sutton AJ, Abrams KR, Jones DR. Generalized synthesis of evidence and the threat of dissemination bias. The example of electronic fetal heart rate monitoring (EFM). J Clin Epidemiol. 2002;55:1013-24.
100. Pladevall-Vila M, Delclos GL, Varas C, Guyer H, Brugues-Tarradellas J, Anglada-Arisa A. Controversy of oral contraceptives and risk of rheumatoid arthritis: meta-analysis of conflicting studies and review of conflicting meta-analyses with special emphasis on analysis of heterogeneity. Am J Epidemiol. 1996;144:1-14.
101. Dwyer T, Couper D, Walter SD. Sources of heterogeneity in the meta-analysis of observational studies: the example of SIDS and sleeping position. J Clin Epidemiol. 2001;54:440-7.
102. Abrams K, Jones DR. Meta-analysis and the synthesis of evidence. IMA J Math Appl Med Biol. 1995;12:297-313.
103. Doria AS. Meta-analysis and structured literature review in radiology. Acad Radiol. 2005;12:399-408.
104. Takkouche B, Cadarso-Suarez C, Spiegelman D. Evaluation of old and new tests of heterogeneity in epidemiologic meta-analysis. Am J Epidemiol. 1999;150:206-15.
105. Zeegers MP, Heisterkamp SH, Kostense PJ, van der Windt DA, Scholten RJ. Practice of systematic reviews. VII. Pooling of results from observational studies. Ned Tijdschr Geneeskd. 2000;144:1393-7.
106. Guyatt GH, Oxman AD, Kunz R, Woodcock J, Brozek J, Helfand M, et al. GRADE guidelines: 7. Rating the quality of evidence - inconsistency. J Clin Epidemiol. 2011;64:1294-302.
107. Hernandez AF, Gonzalez-Alzaga B, Lopez-Flores I, Lacasana M. Systematic reviews on neurodevelopmental and neurodegenerative disorders linked to pesticide exposure: methodological features and impact on risk assessment. Environ Int. 2016;92-93:657-79.
108. Weeks DL. The regression effect as a neglected source of bias in nonrandomized intervention trials and systematic reviews of observational studies. Eval Health Prof. 2007;30:254-65.
109. Hartemink N, Boshuizen HC, Nagelkerke NJ, Jacobs MA, van Houwelingen HC. Combining risk estimates from observational studies with different exposure cutpoints: a meta-analysis on body mass index and diabetes type 2. Am J Epidemiol. 2006;163:1042-52.
110. Salanti G, Ioannidis JP. Synthesis of observational studies should consider credibility ceilings. J Clin Epidemiol. 2009;62:115-22.
111. Smith SJ, Caudill SP, Steinberg KK, Thacker SB. On combining dose-response data from epidemiological studies by meta-analysis. Stat Med. 1995;14:531-44.
112. Thompson S, Ekelund U, Jebb S, Lindroos AK, Mander A, Sharp S, et al. A proposed method of bias adjustment for meta-analyses of published observational studies. Int J Epidemiol. 2011;40:765-77.
113. Tweedie RL, Mengersen KL. Meta-analytic approaches to dose-response relationships, with application in studies of lung cancer and exposure to environmental tobacco smoke. Stat Med. 1995;14:545-69.
114. McCarron CE, Pullenayegum EM, Thabane L, Goeree R, Tarride JE. The importance of adjusting for potential confounders in Bayesian hierarchical models synthesising evidence from randomised and non-randomised studies: an application comparing treatments for abdominal aortic aneurysms. BMC Med Res Methodol. 2010;10:64.
115. Thompson SG, Higgins JP. How should meta-regression analyses be undertaken and interpreted? Stat Med. 2002;21:1559-73.
116. Brockwell SE, Gordon IR. A comparison of statistical methods for meta-analysis. Stat Med. 2001;20:825-40.
117. Prevost TC, Abrams KR, Jones DR. Hierarchical models in generalized synthesis of evidence: an example based on studies of breast cancer screening. Stat Med. 2000;19:3359-76.
118. Shuster JJ, Jones LS, Salmon DA. Fixed vs random effects meta-analysis in rare event studies: the rosiglitazone link with myocardial infarction and cardiac death. Stat Med. 2007;26:4375-85.