Quality assessment tools for evidence from environmental science

Environmental Evidence

Volumn 3, Issue 1, 2014, Pages

  Bilotta, Gary S ^a,b   Milner, Alice M ^b,c   Boyd, Ian L ^b,d  

^a UNIVERSITY OF BRIGHTON   (United Kingdom)

^b Food and Rural Affairs   (United Kingdom)

^c UNIVERSITY COLLEGE LONDON   (United Kingdom)

^d UNIVERSITY OF ST ANDREWS   (United Kingdom)

Author keywords

Bias; Evidence synthesis; External validity; Internal validity; Quality checklist; Quality scale; Systematic review

Indexed keywords

EID: 84930753074     PISSN: None     EISSN: 20472382     Source Type: Journal    
DOI: 10.1186/2047-2382-3-14     Document Type: Article

References (73)

1. Begley CG, Ellis LM: Drug development: raise standards for preclinical cancer research. Nature 2012,483(7391):531-533.
2. Prinz F, Schlange T, Asadullah K: Believe it or not: how much can we rely on published data on potential drug targets? Nat Rev Drug Discov 2011,10(9):712-712.
3. Bohannan J: Who's afraid of peer review? Science 2013, 342: 60-65.
4. Godlee F, Gale CR, Martyn CN: Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. Jama 1998,280(3):237-240.
5. Verhagen AP, de Vet HC, de Bie RA, Kessels AG, Boers M, Bouter LM, Knipschild PG: The Delphi list: a criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus. J Clin Epidemiol 1998,51(12):1235-1241.
6. Centre for Reviews and Dissemination: Systematic reviews: CRD's guidance for undertaking reviews on health care. 2009.
7. Higgins JP, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, Sterne JA: The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. Br Med J 2011, 343: d5928.
8. Gluud LL: Bias in clinical intervention research. Am J Epidemiol 2006,163(6):493-501.
9. Cozby PC: Methods in behavioural research. 10th edition. Boston: McGraw-Hill Higher Education; 2009.
10. Burgman MA, McBride M, Ashton R, Speirs-Bridge A, Flander L, Wintle B, Fidler F, Rumpff L, Twardy C: Expert status and performance. PLoS One 2011,6(7):e22998.
11. Cooper H, Ribble RG: Influences on the outcome of literature searches for integrative research reviews. Knowledge 1989, 10: 179-201.
12. Oxman AD, Guyatt GH: The science of reviewing research. Ann N Y Acad Sci 1993, 703: 125-133.
13. Wells K, Littell JH: Study quality assessment in systematic reviews of research on intervention effects. Res Soc Work Pract 2009,19(1):52-62.
14. Moher D, Jadad AR, Tugwell P: Assessing the quality of randomized controlled trials. Current issues and future directions. Int J Technol Assess Health Care 1996, 12: 195-208.
15. Moher D, Pham B, Jones A, Cook DJ, Jadad AR: Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998, 352: 609-613.
16. Colle F, Rannou F, Revel M, Fermanian J, Poiraudeau S: Impact of quality scales on levels of evidence inferred from a systematic review of exercise therapy and low back pain. Arch Phys Med Rehabil 2002,83(12):1745-1752.
17. Herbison P, Hay-Smith J, Gillespie WJ: Adjustment of meta-analyses on the basis of quality scores should be abandoned. J Clin Epidemiol 2006,59(12):1249-1256.
18. Juni P, Witschi A, Bloch R, Egger M: The hazards of scoring the quality of clinical trials for meta-analysis. J Am Med Assoc 1999,282(11):1054-1060. 1999
19. Deeks JJ, Dinnes J, D'amico R, Sowden AJ, Sakarovitch C, Song F, Petticrew M, Altman DJ: Evaluating non-randomised intervention studies. Health Technol Assess 2003,7(27):1-179.
20. Collaboration for Environmental Evidence: Guidelines for systematic review and evidence synthesis in environmental management. Version 4.2. 2013.
21. Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S: Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Control Clinical Trials 1995, 16: 62-73.
22. Jüni P, Altman DG, Egger M: Systematic reviews in health care: assessing the quality of controlled clinical trials. Br Med J 2001, 323: 42.
23. Sterne JAC, Jüni P, Schulz KF, Altman DG, Bartlett C, Egger M: Statistical methods for assessing the influence of study characteristics on treatment effects in 'meta-epidemiological' research. Stat Med 2002, 21: 1513-1524.
24. de Oliveira IR, Dardennes RM, Amorim ES, Diquet B, de Sena EP, Moreira EC: Is there a relationship between antipsychotic blood levels and their clinical efficacy? An analysis of studies design and methodology. Fundam Clin Pharmacol 1995, 9: 488-502.
25. Coleridge-Smith P: The management of chronic venous disorders of the leg: an evidence-based report of an International Task Force. Phlebology 1999, 14: 3-19.
26. Downs SH, Black N: The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Commun Health 1998, 52: 377-384.
27. Lundh A, Gøtzsche PC: Recommendations by Cochrane Review Groups for assessment of the risk of bias in studies. BMC Med Res Methodol 2008,8(1):22.
28. Greenland S, O'Rourke K: On the bias produced by quality scores in meta-analysis, and a hierarchical view of proposed solutions. Biostat 2001, 2: 463-471.
29. Kunz R, Vist G, Oxman AD: Randomisation to protect against selection bias in healthcare trials. In The Cochrane Library, issue 4. Chichester: John Wiley and Sons Ltd; 2002. http://www.cochrane.org/reviews/clibintro.htm
30. Ioannidis JP, Haidich AB, Pappa M, Pantazis N, Styliani I, Kokori MG, Tektonidou DG, Contopoulos-Ioannnidi DG, Lau J: Comparison of evidence of treatment effects in randomized and nonrandomized studies. JAMA 2001, 286: 821-830.
31. Als-Nielsen B, Gluud L, Gluud C: Methodological quality and treatment effects in randomised trials - a review of six empirical studies. Ottawa, Ontario, Canada: Canadian Cochrane Network and Centre, University of Ottawa; 2004. http://www.cochrane.org
32. Devereaux PJ, Manns BJ, Ghali WA, Quan H, Lacchetti C, Montori VM, Bhandari M, Guyatt GH: Physician interpretations and textbook definitions of blinding terminology in randomized controlled trials. J Am Med Assoc 2001,285(15):2000-2003.
33. Schulz KF, Chalmers I, Hayes RJ, Altman DG: Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. J Am Med Assoc 1995, 273: 408-412.
34. Kjaergard LL, Villumsen J, Gluud C: Reported methodological quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med 2001, 135: 982-989.
35. Balk EM, Bonis PA, Moskowitz H, Schmid CH, Ioannidis JPA, Wang C, Lau J: Correlation of quality measures with estimates of treatment effect in metaanalyses of randomized controlled trials. J Am Med Assoc 2002,287(22):2973-2982.
36. Als-Nielsen B, Chen W, Gluud L, Siersma V, Hilden J, Gluud C: Are trial size and reported methodological quality associated with treatment effects? Observational study of 523 randomised trials. Ottawa, Canada: 12th International Cochrane Colloquium; 2004.
37. Wood L, Egger M, Gluud LL, Schulz KF, Juni P, Altman DG, Gluud C, Martin RM, Wood AJG, Sterne JAC: Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. Br Med J 2008, 336: 601-605.
38. Easterbrook PJ, Gopalan R, Berlin JA, Matthews DR: Publication bias in clinical research. Lancet 1991,337(8746):867-872.
39. Chan AW, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG: Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. J Am Med Assoc 2004,291(20):2457-2465.
40. Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, Seida JK, Klassen TP: Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. Br Med J 2009, 339: b4012.
41. GRADE Working Group: Grading quality of evidence and strength of recommendations. Br Med J 2004, 328: 1490-1494.
42. Schünemann HJ, Jaeschke R, Cook DJ, Bria WF, El-Solh AA, Ernst A, Fahy BF, Gould MK, Horan KL, Krishnan JA, Manthous CA, Maurer JR, McNicholas WT, Oxman AD, Rubenfeld G, Turino GM, Guyatt G: An official ATS statement: grading the quality of evidence and strength of recommendations in ATS guidelines and recommendations. Am J Respir Crit Care Med 2006, 174: 605-614.
43. Guyatt G, Gutterman D, Baumann MH, Addrizzo-Harris D, Hylek EM, Phillips B, Raskob G, Lewis SZ, Schünemann HJ: Grading strength of recommendations and quality of evidence in clinical guidelines: report from an American College of Chest Physicians Task Force. Chest 2006, 129: 174-181.
44. Guyatt G, Vist G, Falck-Ytter Y, Kunz R, Magrini N, Schünemann HJ: An emerging consensus on grading recommendations? ACP J Club 2006, 144: A8-A9.
45. Schünemann HJ, Oxman AD, Vist GE, Higgins JPT, Deeks JJ, Glasziou P, Guyatt GH: Chapter 12: Interpreting results and drawing conclusions. In Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011) Edited by: Higgins JPT, Green S. 2011. http://www.cochrane-handbook.org
46. Atkins D, Best D, Briss PA, Eccles M, Falck Ytter Y, Flottorp S, Guyatt GH, Harbour RT, Haugh MC, Henry D, Hill S, Jaeschke R, Leng G, Liberati A, Magrini N, Mason J, Middleton P, Mrukowicz J, O'Connell D, Oxman AD, Phillips B, Schunemann HJ, Edejer TT, Varonen H, Vist GE, Williams JW Jr, Zaza S: Grade Working Group: Grading quality of evidence and strength of recommendations. Br Med J 2004,328(7454):1490.
47. Atkins D, Briss PA, Eccles M, Flottorp S, Guyatt GH, Harbour RT, Hill S, Jaeschke R, Liberati A, Magrini N, Mason J, O'Connell D, Oxman AD, Phillips B, Schunemann HJ, Edejer TT, Vist GE, Williams JW: GRADE Working Group: Systems for grading the quality of evidence and the strength of recommendations II: Pilot study of a new system. BMC Health Serv Res 2005, 5: 25.
48. Stewart GB, Coles CF, Pullin AS: Does burning degrade blanket bog. Systematic Review 2004, 1: 1-29.
49. Roberts PD, Pullin AS: Effectiveness of the control of ragwort (Senecio) species: Are currently recommended herbicides effective for control of ragwort (Senecio) species? CEE review 2004., 04-003: http://www.environmentalevidence.org/SR5a.html
50. Felton A, Knight E, Wood J, Zammit C, Lindenmayer D: Do tree plantations support higher species richness and abundance than pasture lands? CEE Review 2010., 09-012: http://www.environmentalevidence.org/SR73.html
51. McDonald MA, McLaren KP, Newton AC: What are the mechanisms of regeneration post-disturbance in tropical dry forest? CEE Review 2010., 07-013: http://www.environmentalevidence.org/SR37.html
52. Stacey CJ, Springer AE, Stevens LE: Have spring restoration projects in the southwestern United States been effective in restoring hydrology, geomporphology, and invertebrates and plant species composition comparable to natural springs with minimal anthropogenic disturbance? CEE Review 2011., 10-002: http://www.environmentalevidence.org/SR87.html
53. Humbert J-Y, Pellet J, Buri P, Arlettaz R: Does delaying the first mowing date increase biodiversity in European farmland meadows? CEE Review 2012., 09-011: http://www.environmentalevidence.org/SR72.html
54. Pullin AS, Knight TM: Support for decision making in conservation practice: an evidence-based approach. J Nat Conserv 2003, 11: 83-90.
55. Hannan EL: Randomized clinical trials and observational studies: Guidelines for assessing respective strengths and limitations. J Am Coll Cardiol Intv 2008, 1: 211-217.
56. Stewart GB, Coles CF, Pullin AS: Does burning of UK sub-montane, dry dwarf-shrub heath maintain vegetation diversity? CEE Review 2004., 03-002: http://www.environmentalevidence.org/SR2.html
57. Stewart GB, Pullin AS, Coles CF: Effects of wind turbines on bird abundance. CEE Review 2005., 04-002: http://www.environmentalevidence.org/SR4.html
58. Stewart GB, Tyler C, Pullin AS: Effectiveness of current methods for the control of bracken (Pteridium aquilinum). CEE Review 2005., 04-001: http://www.environmentalevidence.org/SR3.html
59. Tyler C, Pullin AS: Do commonly used interventions effectively control Rhododendron ponticum ? CEE Review 2005., 04-005: http://www.environmentalevidence.org/SR6.html
60. Tyler C, Clark E, Pullin AS: Do management interventions effectively reduce or eradicate populations of the American Mink, Mustela vison ? CEE review 2005., 04-006: http://www.environmentalevidence.org/SR7.html
61. Roberts PD, Pullin AS: Effectiveness of the control of ragwort (Senecio) species: Can biological control by the use of natural enemies effectively control Senecio jacobaea (common ragwort)? CEE Review 2005., 04-004: http://www.environmentalevidence.org/SR5b.html
62. Roberts PD, Pullin AS: The effectiveness of management options used for the control of Spartina species. CEE Review 2006., 06-001: http://www.environmentalevidence.org/SR22.html
63. Roberts PD, Pullin AS: The effectiveness of land-based schemes (incl. agri-environment) at conserving farmland bird densities within the UK. CEE Review 2007., 05-005: http://www.environmentalevidence.org/SR11.html
64. Kabat TJ, Stewart GB, Pullin AS: Are Japanese knotweed (Fallopia japonica) control and eradication interventions effective? CEE Review 2006., 05-015: http://www.environmentalevidence.org/SR21.html
65. Ramstead KM, Allen JA, Springer AE: Have wet meadow restoration projects in the Southwestern U.S. been effective in restoring hydrology, geomorphology, soils, and plant species composition to conditions comparable to wet meadows with minimal human-induced disturbance? CEE Review 2012., 09-014: http://www.environmentalevidence.org/SR75.html
66. Higgins JPT, Altman DG, Sterne JAC: Assessing risk of bias in included studies. In Chapter 8: Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). Edited by: Higgins JPT, Green S. The Cochrane Collaboration; 2011. http://www.cochrane-handbook.org
67. Chan AW, Altman DG: Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. Br Med J 2005,330(7494):753.
68. Treasure T, MacCrae KD: Minimisation: the platinum standard for trials? Randomisation doesn't guarantee similarity of groups; minimisation does. Br Med J 1998,317(7155):362-363.
69. Balshem H, Helfand M, Schünemann HJ, Oxman AD, Kunz R, Kunz R, Brozek J, Vist GE, Falck-Ytter Y, Meerpohl J, Norrisi S, Guyatt GH: GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol 2011,64(4):401-406.
70. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schünemann HJ: GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. Br Med J 2008, 336: 924-926.
71. Browne S, Vickery J, Chamberlain D: Densities and population estimates of breeding Skylarks Alauda arvensis in Britain in 1997. Bird Study 2000, 47: 52-65.
72. Peach WJ, Lovett LJ, Wotton SR, Jeffs C: Countryside stewardship delivers cirl buntings (Emberiza cirlus) in Devon, UK. Biological Conservation 2001, 101: 361-373.
73. Bradbury RB, Browne SJ, Stevens DK, Aebischer NJ: Five-year evaluation of the impact of the Arable Stewardship Pilot Scheme on birds. Ibis 2004, 146: 171-180.