The reliability of evidence review methodology in environmental science and conservation

Environmental Science and Policy

Volumn 64, Issue , 2016, Pages 75-82

  O'Leary, Bethan C ^a   Kvist, Kristian ^a   Bayliss, Helen R ^a   Derroire, Géraldine ^a   Healey, John R ^a   Hughes, Kathryn ^a   Kleinschroth, Fritz ^a   Sciberras, Marija ^a   Woodcock, Paul ^b   Pullin, Andrew S ^a  

^a BANGOR UNIVERSITY   (United Kingdom)

^b JOINT NATURE CONSERVATION COMMITTEE   (United Kingdom)

Author keywords

CEESAT; Decision making; Evidence syntheses; Evidence base; Review evaluation; Review methodology

Indexed keywords

ARTICLE; DECISION MAKING; DESCRIPTIVE RESEARCH; ENVIRONMENTAL MANAGEMENT; ENVIRONMENTAL PLANNING; ENVIRONMENTAL PROTECTION; EVIDENCE BASED PRACTICE; METHODOLOGY; PRIORITY JOURNAL; QUESTIONNAIRE; RELIABILITY; SCIENCE; SCORING SYSTEM; SYSTEMATIC REVIEW;

EID: 84976627451     PISSN: 14629011     EISSN: 18736416     Source Type: Journal    
DOI: 10.1016/j.envsci.2016.06.012     Document Type: Article

References (51)

1. Ayyub, B.M., On uncertainty in information and ignorance in knowledge. Int. J. Gen. Syst. 39 (2010), 415–435.
2. Biber, E., Which science? Whose science? How scientific disciplines can shape environmental law. University Chicago Law Rev. 79 (2012), 471–552.
3. Bilotta, G.S., Milner, A.M., Boyd, I., On the use of systematic reviews to inform environmental policies. Environ. Sci. Policy 42 (2014), 67–77.
4. Buller, H., Hinchliffe, S., Hockenhull, J., Barrett, D., Reyher, K., Butterworth, A., Heath, C., 2015. Systematic review and social research to further understanding of current practice in the context of using antimicrobials in livestock farming and to inform appropriate interventions to reduce antimicrobial resistance within the livestock sector, Department for Environment, Food & Rural Affairs, Report O00558.
5. Chalmers, I., Hedges, L.V., Cooper, H., A brief history of research synthesis. Eval. Health Professions 25 (2002), 12–37.
6. Chalmers, I., Trying to do more good than harm in policy and practice: the role of rigorous, transparent, up to date evaluations. Ann. Am. Acad. Political Soc. Sci. 589 (2003), 22–40.
7. Claudet, J., Osenberg, C.W., Benedetti-Cecchi, L., Domenici, P., Garcia-Charton, J.A., Perez-Ruzafa, A., Badalamenti, F., Bayle-Sempere, J., Brito, A., Bulleri, F., Culioli, J.-M., Dimech, M., Falcon, J.M., Guala, I., Milazzo, M., Sanchez-Meca, J., Somerfield, P.J., Stobart, B., Vandeperre, F., Valle, C., Planes, S., Marine reserves: size and age do matter. Ecol. Lett. 11 (2008), 481–489.
8. Cohen, J., A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20 (1960), 37–46.
9. Collaboration for Environmental Evidence, 2013. Guidelines for systematic review and evidence synthesis in environmental management. Version 4.2. Environmental Evidence.
10. Cook, D.J., Mulrow, C.D., Haynes, R.B., Systematic reviews: synthesis of best evidence for clinical decisions. Ann. Intern. Med. 126 (1997), 376–380.
11. Cook, C.N., Carter, R.W., Fuller, R.A., Hockings, M., Managers consider multiple lines of evidence important for biodiversity management decisions. J. Environ. Manage. 113 (2012), 341–346.
12. Dicks, L.V., Walsh, J.C., Sutherland, W.J., Organising evidence for environmental management decisions: a ‘4S’ hierarchy. Trends Ecol. Evol. 29 (2014), 607–613.
13. Gough, D., Oliver, S., Thomas, J., An Introduction to Systematic Reviews. 2012, Sage, London.
14. Guerrero, R., Misuse and abuse of journal impact factors. Eur. Sci. Ed. 27 (2001), 58–59.
15. Guyatt, G.H., Oxman, A.D., Schunermann, H.J., Tugwell, P., Knottnerus, A., GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology. J. Clin. Epidemiol. 64 (2011), 380–382.
16. Haddaway, N.R., Woodcock, P., Macura, B., Collins, A., Making literature reviews more reliable through application of lessons from systematic reviews. Conserv. Biol., 2015 (epub).
17. Huntington, B.E., Confronting publication bias in marine reserve meta-analyses. Front. Ecol. Environ. 9 (2011), 375–376.
18. Jarwal, S.D., Brion, A.M., King, M.L., Measuring research quality using the journal impact factor, citations and ‘Ranked Journals’: blunt Instruments or Inspired metrics?. J. Higher Educ. Policy Manage. 31 (2009), 289–300.
19. Kirsch, I., Deacon, B.J., Huedo-Medina, T.B., Scoboria, A., Moore, T.J., Johnson, B.T., Initial severity and antidepressant benefits: a meta-analysis of data submitted to the Food and Drug Administration. PLoS Med., 5, 2008, e45.
20. Koricheva, J., Gurevitch, J., Uses and misuses of meta-analysis in plant ecology. J. Ecol. 102 (2014), 828–844.
21. Koricheva, J., Gurevitch, J., Mengersen, K., Handbook of Meta-Analysis in Ecology and Evolution. 2013, Princeton University Press.
22. Land, M., Granéli, W., Grimvall, A., Hoffmann, C.C., Mitsch, W.J., Tonderski, K.S., Verhoeven, J.T.A., How effective are created or restored freshwater wetlands for nitrigen and phosphorus removal? A systematic review. Environ. Evid., 5, 2016, 9.
23. Landis, J.R., Koch, G.G., The measurement of observer agreement for categorical data. Biometrics 33 (1977), 159–174.
24. Larsen, P.O., von Ins, M., The rate of growth in scientific publication and the decline in coverage provided by Science Citation Index. Scientometrics 84 (2010), 575–603.
25. Li, W., Zhao, Y., Bibliometric analysis of global environmental assessment research in a 20-year period. Environ. Impact Assess. Rev. 50 (2015), 158–166.
26. Lortie, C.J., Formalized synthesis opportunities for ecology: systematic reviews and meta-analyses. Oikos 123 (2014), 897–902.
27. Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med., 6, 2009, e1000097.
28. Mulrow, C.D., The medical review article: state of the science. Ann. Int. Med. 106 (1987), 485–488.
29. Newman, J.R., Duenas-Lopez, M.A., Acreman, M.C., Palmer-Felgate, E.J., Verhoeven, J.T.A., Scholz, M., Maltby, E., 2015. Do on-farm natural, restored, managed and constructed wetlands mitigate agricultural pollution in Great Britain and Ireland? A systematic review. Department for Environment, Food & Rural Affairs, Report WT0989.
30. Pautasso, M., Publication growth in biological sub-fields: patterns, predictability and sustainability. Sustainability 4 (2012), 3234–3247.
31. Philibert, A., Loyce, C., Makowski, D., Assessment of the quality of meta-analysis in agronomy. Agric. Ecosyst. Environ. 148 (2012), 72–82.
32. Pielke, R.A., The Honest Broker: Making Sense of Science in Policy and Politics. 2007, Cambridge University Press.
33. Pope, C., Mays, N., Popay, J., Synthesizing Qualitative and Quantitative Health Evidence. A Guide to Methods. 2007, McGlaw Hill.
34. Pullin, A.S., Knight, T.M., Effectiveness in conservation practice: pointers from medicine and public health. Conserv. Biol. 15 (2001), 50–54.
35. Pullin, A.S., Knight, T.M., Science informing policy – a health warning for the environment. Environ. Evid., 1, 2012, 15.
36. Pullin, A.S., Stewart, G.B., Guidelines for systematic review in conservation and environmental management. Conserv. Biol. 20 (2006), 1647–1655.
37. Pullin, A.S., Knight, A.T., Stone, D.A., Charman, K., Do conservation managers use scientific evidence to support their decision-making?. Biol. Conserv. 119 (2004), 245–252.
38. Roberts, P.D., Stewart, G.B., Pullin, A.S., Are review articles a reliable source of evidence to support conservation and environmental management? A comparison with medicine. Biol. Conserv. 132 (2006), 409–423.
39. Rothstein, H.R., Lortie, C.J., Stewart, G.B., Koricheva, J., Gurevitch, J., Quality standards for research syntheses. Koricheva, J., Gurevitch, J., Mengersen, K., (eds.) Handbook of Meta-analyses in Ecology and Evolution, 2013, Princeton University Press, Princeton.
40. Sarewitz, D., How science makes environmental controversies worse. Environ. Sci. Policy 7 (2004), 385–403.
41. Seavy, N.E., Howell, C.A., How can we improve information delivery to support conservation and restoration decisions?. Biodivers. Conserv. 19 (2010), 1261–1267.
42. Sharman, A., Holmes, J., Evidence-based policy or policy-based evidence gathering? Biofuels, the EU and the 10% target. Environ. Policy Gov. 20 (2010), 309–321.
43. Shea, B.J., Hamel, C., Wells, G.A., Bouter, L.M., Kristjansson, E., Grimshaw, J., Henry, D.A., Boers, M., AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J. Clin. Epidemiol. 62 (2009), 1013–1020.
44. Stewart, G.B., Kaiser, M.J., Cote, I.M., Halpern, B.S., Lester, S.E., Bayliss, H.R., Pullin, A.S., Temperate marine reserves: global ecological effects and guidelines for future networks. Conserv. Lett. 2 (2009), 243–253.
45. The Cochrane Collaboration, Cochrane Handbook for Systematic Reviews of Interventions. Higgins, J.P.T., Green, S., (eds.), 2011 (Version 5.1.0).
46. Tranfield, D., Denyer, D., Smart, P., Towards a methodology for developing evidence-informed management knowledge by means of systematic review. Br. J. Manage. 14 (2003), 207–222.
47. Wade, L., Whitehead, H., Weilgart, L., Conflict of interest in research on anthropogenic noise and marine mammals: does funding bias conclusions?. Mar. Policy 34 (2010), 320–327.
48. Wesselink, A., Buchanan, K.S., Georgiadou, Y., Turnhout, E., Technical knowledge, discursive spaces and politics at the science-policy interface. Environ. Sci. Policy 30 (2013), 1–9.
49. Woodcock, P., Pullin, A.S., Kaiser, M.J., Evaluating and improving the reliability of evidence syntheses in conservation and environmental science: a methodology. Biol. Conserv. 176 (2014), 54–62.
50. vom Brocke, J., Simons, A., Niehaves, B., Niehaves, B., Reimer, K., Plattfaut, R., Cleven, A., Reconstructing the giant on the importance of rigour in documenting the literature search process. ECIS 2009 Proceedings, 2009 (Paper 161).
51. Zupanc, G.K.H., Impact beyond the impact factor. J. Comp. Physiol.-A 200 (2014), 113–116.