Evaluating and improving the reliability of evidence syntheses in conservation and environmental science: A methodology

Biological Conservation

Volumn 176, Issue , 2014, Pages 54-62

  Woodcock, Paul ^a   Pullin, Andrew S ^a   Kaiser, Michel J ^a  

^a BANGOR UNIVERSITY   (United Kingdom)

Author keywords

Environmental policy; Evidence base; Evidence synthesis; Meta analysis; Review methodology; Systematic review

Indexed keywords

BIODIVERSITY; DECISION MAKING; ENVIRONMENTAL ASSESSMENT; ENVIRONMENTAL POLICY; GUIDELINE; LITERATURE REVIEW; META-ANALYSIS; RELIABILITY ANALYSIS; THEORETICAL STUDY;

EID: 84901975859     PISSN: 00063207     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.biocon.2014.04.020     Document Type: Article

References (42)

1. Borenstein W., Hedges L.V., Higgins J.P.T., Rothstein H.R. Introduction to Meta-analysis 2009, John Wiley & Sons, Chichester.
2. Burda B.U., Norris S.L., Holmer H.K., Ogden L.A., Smith M.E.B. Quality varies across clinical practice guidelines for mammography screening in women aged 40-49 years as assessed by AGREE and AMSTAR instruments. J. Clin. Epidemiol. 2011, 64:968-976.
3. Branton M., Richardson J.S. Assessing the value of the umbrella-species concept for conservation planning with meta-analysis. Conserv. Biol. 2011, 25:9-20.
4. CEE, 2013. Collaboration for Environmental Evidence. Guidelines for systematic review in environmental management Version 4.2. Environmental Evidence. <> (accessed 01.07.13). http://www.environmentalevidence.org/Documents/Guidelines4.2.pdf.
5. Cohen J. A coefficient of agreement for nominal scales. Edu. Psychol. Meas. 1960, 20:37-46.
6. CRD, 2009. Centre for Reviews and Dissemination. CRD's guidance for undertaking reviews in health care. CRD, University of York. Available at: <> (accessed 21.11.12). http://www.york.ac.uk/inst/crd/publications.htm.
7. Dent D.H., Wright S.J. The future of tropical species in secondary forests: a quantitative review. Biol. Conserv. 2009, 142:2833-2843.
8. Englund G., Samelle O., Cooper S.D. The importance of data-selection criteria: meta-analyses of stream predation experiments. Ecology 1999, 80:1132-1141.
9. Fazey I., Salisbury J.G., Lindenmayer D.B., Maindonald J., Douglas R. Can methods applied in medicine be used to summarise and disseminate conservation research?. Environ. Conserv. 2004, 31:190-198.
10. Gates S. Review of methodology of quantitative reviews using meta-analysis in ecology. J. Anim. Ecol. 2002, 71:547-557.
11. Halpern B.S. The impact of marine reserves: do reserves work and does reserve size matter?. Ecol. Appl. 2003, 13:117-137.
12. Harrison F. Getting started with meta-analysis. Methods Ecol. Evol. 2011, 2:1-10.
13. Huntington B.E. Confronting publication bias in marine reserve meta-analyses. Front. Ecol. Environ. 2011, 9:375-376.
14. Kalies, E., Covington, W., Chambers, W., Rosenstock, S., 2010. How do thinning and burning treatments in southwestern conifer forests in the United States affect wildlife density and population performance? CEE Review 09-005 (SR66). Collaboration for Environmental Evidence: <>. http://www.environmentalevidence.org/SR66.html.
15. Lajeunesse M.J., Forbes M.R. Variable reporting and quantitative reviews: a comparison of three meta-analytical techniques. Ecol. Lett. 2003, 6:448-454.
16. Lajeunesse M.J. Achieving synthesis with meta-analysis by combining and comparing all available studies. Ecology 2010, 91:2561-2564.
17. Landis J.R., Koch G.G. The measurement of observer agreement for categorical data. Biometrics 1977, 33:159-174.
18. Lawler J.J., Aukema J.E., Grant J.B., Halpern B.S., Kareiva P., Nelson C.R., Ohleth K., Olden J.D., Schlaepfer M.A., Silliman B., Zaradic P. Conservation science: a 20-year report card. Front. Ecol. Environ. 2006, 4:473-480.
19. Liberati A., Altman D.G., Tetzlaff J., Mulrow C., Gotzsche P.C., Ioannidis J.P.A., Clarke M., Devereux P.J., Kleijnen J., Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009, 6(7):e1000100.
20. Maliao R.J., White A.T., Maypa A.P., Turingan R.G. Trajectories and magnitude of change in coral reef fish populations in Philippine marine reserves: a meta-analysis. Coral Reefs 2009, 28:809-822.
21. Moher D., Liberati A., Tetzlaff J., Altman D.G. Preferred reporting items for systematic reviews and meta-analyses. The PRISMA Statement. PLoS Med. 2009, 6(7):e1000097. 10.1371/journal.pmed.1000097.
22. Norris R.H., Webb J.A., Nichols S.J., Stewardson M.J., Harrison E.T. Analysing cause and effect in environmental assessments: using weighted evidence from the literature. Freshw. Sci. 2012, 31:5-21.
23. Peppin, D., Fulé P., Beyers, J., Sieg, C., Hunter, M., 2011. Does seeding after severe forest fire in western USA mitigate impacts on soils and plant communities? CEE review 08-023 (SR60) Collaboration for Environmental Evidence: <>. http://www.environmentalevidence.org/SR60.html.
24. Philibert A., Loyce C., Makowski D. Assessment of the quality of meta-analysis in agronomy. Agri. Ecosyst. Environ. 2012, 148:72-82.
25. Popovich I., Windsor B., Jordan V., Showell M., Shea B., Farquahar C.M. Methodological quality of systematic reviews in subfertility: a comparison of two different approaches. PLoS ONE 2012, 7(12):e50403. 10.1371/journal.pone.0050403.
26. Pullin A.S., Knight T.M. Effectiveness in conservation practice: pointers from medicine and public health. Conserv. Biol. 2001, 15:50-54.
27. Pullin A.S., Knight T.M. Support for decision making in conservation practice: an evidence-based approach. J. Nat. Conserv. 2003, 11:83-90.
28. Pullin A.S., Knight T.M. Doing more good than harm - building an evidence-base for conservation and environmental management. Biol. Conserv. 2009, 142:931-934.
29. Pullin A.S., Knight T.M. Science informing policy - a health warning for the environment. Environ. Evid. 2012, 1:15.
30. Roberge J.-M., Angelstam P. Usefulness of the umbrella species concept as a conservation tool. Conserv. Biol. 2004, 18:76-85.
31. Roberts P.D., Stewart G.B., Pullin A.S. Are review articles a reliable source of evidence to support conservation and environmental management? A comparison with medicine. Biol. Conserv. 2006, 132:409-423.
32. Rosenberg M.S. The file-drawer problem revisited: a general weighted method for calculating fail-safe numbers in meta-analysis. Evolution 2005, 59:464-468.
33. Shea B.J., Bouter L.M., Peterson J., Boers M., Andersson B.M., Ortiz Z., Ramsay T., Bai A., Shukla V.K., Grimshaw J.M. External validation of a measurement tool to assess systematic reviews. PLoS ONE 2007, 2(12):e1350. 10.1371/journal.pone.0001350.
34. Shea B.J., Hamel C., Wells G.A., Bouter L.M., Kristjansson E., Grimshaw J., Henry D.A., Boers M. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J. Clin. Epidemiol. 2009, 62:1013-1020.
35. Stewart G.B. Meta-analysis in applied ecology. Biol. Lett. 2010, 6:78-81.
36. Sutherland W.J., Pullin A.S., Dolman P.M., Knight T.M. The need for evidence-based conservation. Trends Ecol. Evol. 2004, 19:305-308.
37. The Cochrane Collaboration, 2011. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. In: Higgins, J.P.T., Green, S. (Eds.), Available from <>. (updated March 2011). http://www.cochrane-handbook.org.
38. Trimble M.J., van Aarde R.J. Geographical and taxonomic biases in research on biodiversity in human-modified landscapes. Ecosphere 2012, 3(12):119. .
39. Viera A.J., Garrett J.M. Understanding interobserver agreement: the kappa statistic. Fam. Med. 2005, 37:360-363.
40. Veríssimo D., MacMillan D.C., Smith R.J. Toward a systematic approach for identifying conservation flagships. Conserv. Lett. 2011, 4:1-8.
41. Whittaker R.J. Meta-analysis and mega-mistakes: calling time on meta-analysis of the species richness-productivity relationship. Ecology 2010, 91:2522-2533.
42. Zumsteg J.M., Cooper J.S., Noon M.S. Systematic review checklist. A standardised technique for assessing and reporting reviews of life cycle assessment data. J. Ind. Ecol. 2012, 16:S12-S21.