A systematic review of tools used to assess the quality of observational studies that examine incidence or prevalence and risk factors for diseases

Journal of Clinical Epidemiology

Volumn 63, Issue 10, 2010, Pages 1061-1070

  Shamliyan, Tatyana ^a   Kane, Robert L ^a   Dickinson, Stacy ^a  

^a UNIVERSITY OF MINNESOTA   (United States)

Author keywords

Bias (epidemiology); Morbidity; Quality control; Reproducibility of results; Review literature as topic; Risk factors; Validation studies

Indexed keywords

CHECKLIST; CLINICAL ASSESSMENT TOOL; CLINICAL EXAMINATION; COCHRANE LIBRARY; DATA BASE; DOCUMENTATION; EPIDEMIOLOGY; EVALUATION; HUMAN; INFORMATION RETRIEVAL; MEDLINE; MORBIDITY; OBSERVATIONAL STUDY; PEER REVIEW; PREVALENCE; PRIORITY JOURNAL; PUBLISHING; QUALITY CONTROL; RANDOMIZED CONTROLLED TRIAL (TOPIC); RELIABILITY; REVIEW; RISK FACTOR; STANDARD; SYSTEMATIC REVIEW; VALIDATION PROCESS; GENERAL ASPECTS OF DISEASE; INCIDENCE; METHODOLOGY; REPRODUCIBILITY;

BIAS (EPIDEMIOLOGY); CHECKLIST; DISEASE; EPIDEMIOLOGY; HUMANS; INCIDENCE; PREVALENCE; QUALITY CONTROL; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; RISK FACTORS;

EID: 78349280348     PISSN: 08954356     EISSN: 18785921     Source Type: Journal    
DOI: 10.1016/j.jclinepi.2010.04.014     Document Type: Review

References (98)

1. Fox DM. Evidence of evidence-based health policy: the politics of systematic reviews in coverage decisions. Health Aff (Millwood) 2005;24:114-22.
2. Lavis J, Davies H, Oxman A, Denis JL, Golden-Biddle K, Ferlie E. Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Policy 2005;(10 Suppl 1): 35-48.
3. Bero LA, Jadad AR. How consumers and policymakers can use systematic reviews for decision making. Ann Intern Med 1997;127(1): 37-42.
4. Chan KS, Morton SC, Shekelle PG. Systematic reviews for evidence-based management: how to find them and what to do with them. Am J Manag Care 2004;10(11 Pt 1):806-12.
5. Briss PA, Zaza S, Pappaioanou M, Fielding J, Wright-De Aguero L, Truman BI, et al. Developing an evidence-based Guide to Community Preventive Servicesdmethods. The Task Force on Community Preventive Services. Am J Prev Med 2000;18(1 Suppl):35-43.
6. Agency for Healthcare Research and Quality. Systems to rate the strength of scientific evidence. Rockville, MD: University of California-San Francisco-Stanford Evidence-Based Practice Center; 2002.
7. Sanderson S, Tatt ID, Higgins JP. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol 2007;36: 666-76.
8. National Center for Biotechnology Information, U.S. National Library of Medicine. Bethesda, MD. Available at: http://www.ncbi. nlm.nih.gov/pubmed. Accessed June 2010.
9. The Cochrane database of systematic reviews. 2000-2010. New York: Wiley.
10. OCLC. WorldCat. Dublin, OH: OCLC; 2001.
11. Elsevier Science (Firm). Scirus for scientific information only. Amsterdam: Elsevier Science; 2001.
12. Higgins J, Green SCochrane Collaboration. Cochrane handbook for systematic reviews of interventions. Chichester,West Sussex; Hoboken NJ: John Wiley & Sons; 2008.
13. von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet 2007;370(9596):1453-7.
14. Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis of Observational Studies in Epidemiology (MOOSE) group. JAMA 2000;283:2008-12.
15. Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, et al. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol 2009;62:1013-20.
16. Altman DG, Simera I, Hoey J, Moher D, Schulz K. EQUATOR: reporting guidelines for health research. Lancet 2008;371(9619): 1149-50.
17. Higgins J, Green S. The Cochrane Collaboration. The Cochrane handbook for systematic reviews of interventions. 2005. Available at: http://www2.cochrane. org/resources/handbook/handbook.pdf. Accessed June 2010.
18. Aschengrau A, Seage GR. Essentials of Epidemiology in Public Health. Sudbury, MA: Jones and Bartlett; 2003.
19. BMJ Publishing Group. Clinical evidence. 2005. Available at: http:// www.clinicalevidence.com. Accessed June 2010.
20. Hulley SB. Designing clinical research: an epidemiologic approach. 2nd edition. Philadelphia, PA: Lippincott Williams & Wilkins; 2001.
21. Centers for Disease Control and Prevention (U.S.), Career Development Division. Principles of epidemiology in public health practice: an introduction to applied epidemiology and biostatistics. 3rd edition. Atlanta, GA: U.S. Dept. of Health and Human Services, Centers for Disease Control and Prevention (CDC), Office of Workforce and Career Development; 2006.
22. Stata Corporation. Stata base reference manual. Release 10 edition. College Station, TX: Stata Corporation; 2007.
23. SAS Institute. SAS com. Cary, NC: SAS Institute; 2001.
24. Loney PL, Chambers LW, Bennett KJ, Roberts JG, Stratford PW. Critical appraisal of the health research literature: prevalence or incidence of a health problem. Chronic Dis Can 1998;19(4):170-6.
25. DuRant RH. Checklist for the evaluation of research articles. J Adolesc Health 1994;15(1):4-8.
26. Woodbury MG, Houghton PE. Prevalence of pressure ulcers in Canadian healthcare settings. Ostomy Wound Manage 2004;50(10):22-4. 6, 8, 30, 2, 4, 6-8.
27. Macfarlane TV, Glenny AM, Worthington HV. Systematic review of population-based epidemiological studies of oro-facial pain. J Dent 2001;29(7):451-67.
28. Steinberg EP, Eknoyan G, Levin NW, Eschbach JW, Golper TA, Owen WF, et al. Methods used to evaluate the quality of evidence underlying the National Kidney Foundation-Dialysis Outcomes Quality Initiative Clinical Practice Guidelines: description, findings, and implications. Am J Kidney Dis 2000;36(1):1-11.
29. Oxman AD, Guyatt GH. Validation of an index of the quality of review articles. J Clin Epidemiol 1991;44:1271-8.
30. Downs SH, Black N. The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health 1998;52(6):377-84.
31. Slim K, Nini E, Forestier D, Kwiatkowski F, PanisY, Chipponi J. Methodological index for non-randomized studies (MINORS): development and validation of a new instrument. ANZ J Surg 2003;73(9):712-6.
32. van Rooyen S, Black N, Godlee F. Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. J Clin Epidemiol 1999;52(7):625-9.
33. Horwitz RI, Feinstein AR. Methodologic standards and contradictory results in case-control research. Am J Med 1979;66:556-64.
34. Lichtenstein MJ, Mulrow CD, Elwood PC. Guidelines for reading case-control studies. J Chronic Dis 1987;40:893-903.
35. Zaza S, Wright-De Aguero LK, Briss PA, Truman BI, Hopkins DP, Hennessy MH, et al. Data collection instrument and procedure for systematic reviews in the Guide to Community Preventive Services. Task Force on Community Preventive Services. Am J Prev Med 2000;18(1 Suppl):44-74.
36. Spitzer WO, Lawrence V, Dales R, Hill G, Archer MC, Clark P, et al. Links between passive smoking and disease: a best-evidence synthesis. A report of the Working Group on Passive Smoking. Clin Invest Med 1990;13(1):17-42. discussion 3-6.
37. Krogh CL. A checklist system for critical review of medical literature. Med Educ 1985;19(5):392-5.
38. Cho MK,Bero LA. Instruments for assessing the quality of drug studies published in the medical literature. JAMA 1994;272(2):101-4.
39. Gardner MJ, Machin D, Campbell MJ. Use of check lists in assessing the statistical content of medical studies. Br Med J (Clin Res Ed) 1986;292(6523):810-2.
40. Esdaile JM, Horwitz RI. Observational studies of cause-effect relationships: an analysis of methodologic problems as illustrated by the conflicting data for the role of oral contraceptives in the etiology of rheumatoid arthritis. J Chronic Dis 1986;39(10):841-52.
41. Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Ann Intern Med 1994;121(1):11-21.
42. Cowley DE. Prostheses for primary total hip replacement. A critical appraisal of the literature. Int J Technol Assess Health Care 1995 Fall;11:770-8.
43. Angelillo IF, Villari P. Residential exposure to electromagnetic fields and childhood leukaemia: a meta-analysis. Bull World Health Organ 1999;77(11):906-15.
44. Nguyen QV, Bezemer PD, Habets L, Prahl-Andersen B. A systematic review of the relationship between overjet size and traumatic dental injuries. Eur J Orthod 1999;21:503-15.
45. van derWindt DA, Thomas E, Pope DP, deWinter AF, Macfarlane GJ, Bouter LM, et al. Occupational risk factors for shoulder pain: a systematic review. Occup Environ Med 2000;57(7):433-42.
46. Borghouts JA, Koes BW, Bouter LM. The clinical course and prognostic factors of non-specific neck pain: a systematic review. Pain 1998;77(1):1-13.
47. Harris RP, Helfand M,Woolf SH, Lohr KN, Mulrow CD, Teutsch SM, et al. Current methods of the US Preventive Services Task Force: a review of the process. Am J Prev Med 2001;20(3 Suppl):21-35.
48. Reisch JS, Tyson JE, Mize SG. Aid to the evaluation of therapeutic studies. Pediatrics 1989;84:815-27.
49. Al-Jader LN, Newcombe RG, Hayes S, Murray A, Layzell J, Harper PS. Developing a quality scoring system for epidemiological surveys of genetic disorders. Clin Genet 2002;62(3):230-4.
50. Scholten-Peeters GGM, Verhagen AP, Bekkering GE, van der Windt DAWM, Barnsley L, Oostendorp RAB, et al. Prognostic factors of whiplash-associated disorders: a systematic review of prospective cohort studies. Pain 2003;104:303-22.
51. Rangel SJ, Kelsey J, Colby CE, Anderson J, Moss RL. Development of a quality assessment scale for retrospective clinical studies in pediatric surgery. J Pediatr Surg 2003;38:390-6. discussion 390-6.
52. Garber BG, Hebert PC, Yelle JD, Hodder RV, McGowan J. Adult respiratory distress syndrome: a systemic overview of incidence and risk factors. Crit Care Med 1996;24:687-95.
53. de Boer AG, Verbeek JH, van Dijk FJ. Adult survivors of childhood cancer and unemployment: a metaanalysis. Cancer 2006;107(1):1-11.
54. Anders JF, Jacobson RM, Poland GA, Jacobsen SJ, Wollan PC. Secondary failure rates of measles vaccines: a metaanalysis of published studies. Pediatr Infect Dis J 1996;15(1):62-6.
55. Manchikanti L, Singh V, Vilims BD, Hansen HC, Schultz DM, Kloth DS. Medial branch neurotomy in management of chronic spinal pain: systematic review of the evidence. Pain Physician 2002;5: 405-18.
56. Jabbour M, Osmond MH, Klassen TP. Life support courses: are they effective? Ann Emerg Med 1996;28:690-8.
57. Shea B, Boers M, Grimshaw JM, Hamel C, Bouter LM. Does updating improve the methodological and reporting quality of systematic reviews? BMC Med Res Methodol 2006;6:27.
58. Ariens GA, van Mechelen W, Bongers PM, Bouter LM, van der Wal G. Physical risk factors for neck pain. Scand J Work Environ Health 2000;26(1):7-19.
59. Longnecker MP, Berlin JA, Orza MJ, Chalmers TC. A meta-analysis of alcohol consumption in relation to risk of breast cancer. JAMA 1988;260:652-6.
60. Corrao G, Bagnardi V, Zambon A, Arico S. Exploring the dose-response relationship between alcohol consumption and the risk of several alcohol-related conditions: a meta-analysis. Addiction 1999;94:1551-73.
61. Stock SR.Workplace ergonomic factors and the development of musculoskeletal disorders of the neck and upper limbs: a meta-analysis. Am J Ind Med 1991;19(1):87-107.
62. Berlin JA, Colditz GA. A meta-analysis of physical activity in the prevention of coronary heart disease. Am J Epidemiol 1990;132:612-28.
63. Kreulen CM, Creugers NH, Meijering AC. Meta-analysis of anterior veneer restorations in clinical studies. J Dent 1998;26(4):345-53.
64. Littenberg B,Weinstein LP, McCarren M, Mead T, Swiontkowski MF, Rudicel SA, et al. Closed fractures of the tibial shaft.Ameta-analysis of three methods of treatment. J Bone Joint SurgAm1998;80(2):174-83.
65. Campos-Outcalt D, Senf J, Watkins AJ, Bastacky S. The effects of medical school curricula, faculty role models, and biomedical research support on choice of generalist physician careers: a review and quality assessment of the literature. Acad Med 1995;70(7):611-9.
66. Tricco AC, Tetzlaff J, Sampson M, Fergusson D, Cogo E, Horsley T, et al. Few systematic reviews exist documenting the extent of bias: a systematic review. J Clin Epidemiol 2008;61(5):422-34.
67. Spencer-Green G. Outcomes in primary Raynaud phenomenon: a meta-analysis of the frequency, rates, and predictors of transition to secondary diseases. Arch Intern Med 1998;158(6):595-600.
68. Meijer R, Ihnenfeldt DS, van Limbeek J, Vermeulen M, de Haan RJ. Prognostic factors in the subacute phase after stroke for the future residence after six months to one year. A systematic review of the literature. Clin Rehabil 2003;17:512-20.
69. Bornhoft G, Maxion-Bergemann S,Wolf U, Kienle GS, Michalsen A, Vollmar HC, et al. Checklist for the qualitative evaluation of clinical studies with particular focus on external validity and model validity. BMC Med Res Methodol 2006;6:56.
70. Cameron I, Crotty M, Currie C, Finnegan T, Gillespie L, Gillespie W, et al. Geriatric rehabilitation following fractures in older people: a systematic review. Health Technol Assess 2000;4(2). ieiv, 1-111.
71. HadornDC, BakerD,Hodges JS,Hicks N. Rating the quality of evidence for clinical practice guidelines. J Clin Epidemiol 1996;49:749-54.
72. Conde-Agudelo A, Villar J, Lindheimer M. Maternal infection and risk of preeclampsia: systematic review and metaanalysis. Am J Obstet Gynecol 2008;198(1):7-22.
73. Fowkes FG, Fulton PM. Critical appraisal of published research: introductory guidelines. BMJ (Clin Res Ed) 1991;302(6785):1136-40.
74. Scottish Intercollegiate Guidelines Network, Harbour RT, Forsyth L. Sign 50. A guideline developer's handbook. Revised edition. Ediburgh, Scotland: Scottish Intercollegiate Guidelines Network; 2008. 1 text file (102 p).
75. Pilote L, Tager IB. Outcomes research in the development and evaluation of practice guidelines. BMC Health Serv Res 2002;2(1):7.
76. Hirtz D, Thurman DJ, Gwinn-Hardy K, Mohamed M, Chaudhuri AR, Zalutsky R. How common are the "common" neurologic disorders? Neurology 2007;68(5):326-37.
77. Levine M, Walter S, Lee H, Haines T, Holbrook A, Moyer V. Users' guides to the medical literature. IV. How to use an article about harm. Evidence-Based Medicine Working Group. JAMA 1994;271(20): 1615-9.
78. Elwood M. Forward projectioneusing critical appraisal in the design of studies. Int J Epidemiol 2002;31:1071-3.
79. Pavia M, Pileggi C, Nobile CG, Angelillo IF. Association between fruit and vegetable consumption and oral cancer: a meta-analysis of observational studies. Am J Clin Nutr 2006;83:1126-34.
80. Genaidy AM, Lemasters GK, Lockey J, Succop P, Deddens J, Sobeih T, et al. An epidemiological appraisal instrumentda tool for evaluation of epidemiological studies. Ergonomics 2007;50:920-60.
81. Bhutta AT, Cleves MA, Casey PH, Cradock MM, Anand KJ. Cognitive and behavioral outcomes of school-aged children who were born preterm: a meta-analysis. JAMA 2002;288:728-37.
82. Dixon-Woods M, Sutton A, Shaw R, Miller T, Smith J, Young B, et al. Appraising qualitative research for inclusion in systematic reviews: a quantitative and qualitative comparison of three methods. J Health Serv Res Policy 2007;12(1):42-7.
83. Treadwell JR, Tregear SJ, Reston JT, Turkelson CM. A system for rating the stability and strength of medical evidence. BMC Med Res Methodol 2006;6:52.
84. Groenwold RH, Van Deursen AM, Hoes AW, Hak E. Poor quality of reporting confounding bias in observational intervention studies: a systematic review. Ann Epidemiol 2008;18(10):746-51.
85. Soares HP, Daniels S, Kumar A, ClarkeM, Scott C, Swann S, et al. Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. BMJ (Clin Res Ed) 2004;328(7430):22-4.
86. Ezzati M, Hoorn SV, Rodgers A, Lopez AD, Mathers CD, Murray CJ. Estimates of global and regional potential health gains from reducing multiple major risk factors. Lancet 2003;362(9380):271-80.
87. Arnold PD, Ickowicz A, Chen S, Schachar R. Attention-deficit hyperactivity disorder with and without obsessive-compulsive behaviours: clinical characteristics, cognitive assessment, and risk factors. Can J Psychiatry 2005;50(1):59-66.
88. Lindsberg PJ, Grau AJ. Inflammation and infections as risk factors for ischemic stroke. Stroke 2003;34:2518-32.
89. BeLue R, Taylor-Richardson KD, Lin J, Rivera AT, Grandison D. African Americans and participation in clinical trials: differences in beliefs and attitudes by gender. Contemp Clin Trials 2006;27: 498-505.
90. Braunstein JB, Sherber NS, Schulman SP, Ding EL, Powe NR. Race, medical researcher distrust, perceived harm, and willingness to participate in cardiovascular prevention trials. Medicine (Baltimore) 2008;87(1):1-9.
91. Smith YR, Johnson AM, Newman LA, Greene A, Johnson TR, Rogers JL. Perceptions of clinical research participation among African American women. J Womens Health (Larchmt) 2007;16:423-8.
92. Hussain-Gambles M, Leese B, Atkin K, Brown J, Mason S, Tovey P. Involving South Asian patients in clinical trials. Health Technol Assess (Winchester, Engl) 2004;8(42). iii, 1-109.
93. Corbie-Smith G, Thomas SB, Williams MV, Moody-Ayers S. Attitudes and beliefs of African Americans toward participation in medical research. J Gen Intern Med 1999;14(9):537-46.
94. Linden HM, Reisch LM, Hart A Jr, Harrington MA, Nakano C, Jackson JC, et al. Attitudes toward participation in breast cancer randomized clinical trials in the African American community: a focus group study. Cancer Nurs 2007;30(4):261-9.
95. Herbison P, Hay-Smith J, Gillespie WJ. Adjustment of meta-analyses on the basis of quality scores should be abandoned. J Clin Epidemiol 2006;59:1249-56.
96. Moja LP, Telaro E, D'Amico R, Moschetti I, Coe L, Liberati A. Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study. BMJ (Clin Res Ed) 2005;330(7499):1053.
97. Greenland S, O'Rourke K. On the bias produced by quality scores in meta-analysis, and a hierarchical view of proposed solutions. Biostatistics 2001;2:463-71.
98. Whiting P, Harbord R, Kleijnen J. No role for quality scores in systematic reviews of diagnostic accuracy studies. BMC Med Res Methodol 2005;5:19.