Toward rigorous use of expert knowledge in ecological research

Ecosphere

Volumn 4, Issue 7, 2013, Pages

  Drescher, M ^a   Perera, A H ^b   Johnson, C J ^c   Buse, L J ^b   Drew, C A ^d   Burgman, M A ^e  

^a UNIVERSITY OF WATERLOO   (Canada)

^b ONTARIO FOREST RESEARCH INSTITUTE   (Canada)

^c UNIVERSITY OF NORTHERN BRITISH COLUMBIA   (Canada)

^d NORTH CAROLINA STATE UNIVERSITY   (United States)

^e UNIVERSITY OF MELBOURNE   (Australia)

Author keywords

Conservation; Decision making; Ecology; Elicitation; Expert knowledge; Judgment; Landscape management; Methodology; Natural resources; Practitioner; Research; Rigor

Indexed keywords

EID: 84893906898     PISSN: None     EISSN: 21508925     Source Type: Journal    
DOI: 10.1890/ES12-00415.1     Document Type: Article

References (160)

1. Albert, I., S. Donnet, C. Guihenneue-Jouyaux, S. Low Choy, K. Mengersen, and J. Rousseau. 2012. Combining expert opinions in prior elicitation. Bayesian Analysis 7:503-532
2. Al-Ubaydli, O., and M. S. Lee. 2011. Can tailored communications motivate volunteers? A field experiment. Department of Economics Paper No. 11-19. George Mason University, Fairfax, Virginia, USA
3. Anadón, J. D., A. Giménez, R. Ballestar, and I. Pérez. 2009. Evaluation of local ecological knowledge as a method for collecting extensive data on animal abundance. Conservation Biology 23:617-625
4. Aspinall, W. 2010. A route to more tractable expert advice. Nature 463:294-95
5. Ayyub, B. M. 2001. Elicitation of expert opinions for uncertainty and risks. First edition. CRC Press, Boca Raton, Florida, USA
6. Bart, D. 2006. Integrating local ecological knowledge and manipulative experiments to find the causes of environmental change. Frontiers in Ecology and the Environment 4:541-546 (Pubitemid 44904693)
7. Bart, D. 2010. Using weed-control knowledge from declining agricultural communities in invasive species management. Human Ecology 38:77-85
8. Bauer, P., and K. Kohne. 1994. Evaluation of experiments with adaptive interim analyses. Biometrics 50:1029-1041
9. Benabou, R., and J. Tirole. 2006. Incentives and prosocial behavior. American Economic Review 96:1652-1678
10. Bentley, J.W. 1989. What farmers don't know can't help them-the strengths and weaknesses of indigenous technical knowledge in Honduras. Agriculture and Human Values 6:25-31
11. Bloom, B. S., editor. 1956. Taxonomy of educational objectives: the classification of educational goals-Handbook 1: cognitive domain. Longmans, Green & Co, London, UK
12. Boiral, O. 2002. Tacit knowledge and environmental management. Long Range Planning 35:291-317 (Pubitemid 34955961)
13. Bolger, F., and G. Wright. 2011. Improving the Delphi process: lessons from social psychological research. Technological Forecasting and Social Change 78:1500-1513
14. Booker, J. M., and L. A. McNamara. 2004. Solving black box computation problems using expert knowledge theory and methods. Reliability Engineering and System Safety 85:331-340
15. Brooks, R. P. 1997. Improving habitat suitability models. Wildlife Society Bulletin 25:163-67
16. Brugnach, M., A. Dewulf, C. Pahl-Wostl, and T. Taillieu. 2008. Toward a relational concept of uncertainty: about knowing too little, knowing too differently, and accepting not to know. Ecology and Society 13:30
17. Burgman, M. A. 2005. Risks and decisions for conservation and environmental management. Cambridge University Press, Cambridge, UK
18. Burgman, M. A., D. R. Breininger, B. W. Duncan, and S. Ferson. 2001. Setting reliability bounds on habitat suitability indices. Ecological Applications 11:70-78 (Pubitemid 32113853)
19. Burgman, M. A., A. Carr, G. Lee, R. Gregory, M. McBride, L. Flander, and L. Maguire. 2011a. Redefining expertise and improving ecological judgment. Conservation Letters 4:81-87
20. Burgman, M. A., M. McBride, R. Ashton, A. Speirs-Bridge, L. Flander, B. Wintle, F. Fidler, L. Rumpf, and C. Twardy. 2011b. Expert status and performance. PLoS ONE 6:e22998
21. Carey, J. M., and M. A. Burgman. 2008. Linguistic uncertainty in qualitative risk analysis and how to minimize it. Annals of the New York Academy of Science 1128:13-17 (Pubitemid 351652571)
22. Chalmers, N., and C. Fabricius. 2007. Expert and generalist local knowledge about land-cover change on South Africa's wild coast: Can local ecological knowledge add value to science? Ecology and Society 12:10
23. Chase, W. G., and H. A. Simon. 1973. Perception in chess. Cognitive Psychology 4:55-81
24. Christopherson, K. M. 2007. The positive and negative implications of anonymity in internet social interactions: On the Internet, Nobody Knows You're a Dog". Computers in Human Behavior 23:3038-3056 (Pubitemid 46880160)
25. Christopoulos, D. C. 2007. Peer esteem snowballing: A methodology for expert surveys. INSNA 29th Annual Conference, Corfu, May 2007. http://epp. eurostat.ec.europa.eu/portal/page/portal/research-methodology/documents/ POSTER-2P-PEER-ESTEEM-SNOWBALLING-CHROSTOPOULOS. pdf
26. Cleaves, D. A. 1994. Assessing uncertainty in expert judgment about natural resources. GTR SO-1 10. USDA Forest Service, Southern Forest Experiment Station, New Orleans, Louisiana, USA
27. Clemen, R. T., and R. L. Winkler. 1999. Combining probability distributions from experts in risk analysis. Risk Analysis 19:187-203 (Pubitemid 29374646)
28. Clevenger, A. P., J. Wierzchowski, B. Chruszcz, and K. Gunson. 2002. GIS-generated, expert-based models for identifying wildlife habitat linkages and planning mitigation passages. Conservation Biology 16:503-14 (Pubitemid 34315825)
29. CNSC [Canadian Nuclear Safety Commission]. 2013. Proactive Disclosure-Contracts. Canadian Nuclear Safety Commission, Ottawa, Ontario, Canada. http://nuclearsafety.gc.ca/eng/proactive-disclosure/contracts/index.cfm
30. Cook, C. N., M. Hockings, and R. W. Carter. 2010. Conservation in the dark? The information used to support management decisions. Frontiers in Ecology and the Environment 8:181-186
31. Cooke, R. M. 1991. Experts in uncertainty: opinion and subjective probability in science. Oxford University Press, New York, New York, USA
32. Cooke, R. M., and K. N. Probst. 2006. Highlights of the expert judgment policy symposium and technical workshop. AN RFF Conference Summary, Washington, D.C., USA
33. Coulson, R. N., L. J. Folse, and D. K. Loh. 1987. Artificial intelligence and natural resource management. Science 237:262-267 (Pubitemid 17622034)
34. Cowling, R. M., R. L. Pressey, R. Sims-Castley, A. Le Roux, E. Baard, C. J. Burgers, and G. Palmer. 2003. The expert or the algorithm? Comparison of priority conservation areas in the Cape Floristic Region identified by park managers and reserve selection software. Biological Conservation 112:147-67 (Pubitemid 36572849)
35. Czembor, C. A., and P. A. Vesk. 2009. Incorporation of between-expert uncertainty into state-and-transition simulation models for forest restoration. Forest Ecology and Management 259:165-75
36. Davis, A., and K. Ruddle. 2010. Constructing confidence: rational skepticism and systematic enquiry in local ecological knowledge research. Ecological Applications 20:880-894
37. Delbecq, A. L., A. H. Van de Ven, and D. H. Gustafson. 1975. Group techniques for program planning: a guide to nominal group and Delphi processes. Scott Foresman, Glenview, Illinois, USA
38. Delgado, M., F. Herrera, E. Herrera-Viedma, and L. Martínez. 1998. Combining numerical and linguistic information in group decision making. Journal of Information Science 107:177-194 (Pubitemid 128383780)
39. Dennis, B. 1996. Discussion: Should ecologists become Bayesians? Ecological Applications 6:1095-1103 (Pubitemid 126546187)
40. Doswald, N., F. Zimmerman, and U. Breitenmoser. 2007. Testing expert groups for a habitat suitability model for the lynx Lynx lynx in the Swiss Alps. Wildlife Biology 13:430-446
41. Drescher, M., and A. H. Perera. 2010a. Comparing two sets of forest cover change knowledge used in forest landscape management planning. Environmental Planning and Management 53:591-613
42. Drescher, M., and A. H. Perera. 2010b. A network approach for evaluating and communicating forest change models. Journal of Applied Ecology 47:57-66
43. Drescher, M., A. H. Perera, L. J. Buse, K. Ride, and S. Vasiliauskas. 2008. Uncertainty in expert knowledge of forest succession: A case study from boreal Ontario. Forestry Chronicle 84:194-209
44. Drew, C. A., and J. A. Collazo. 2011. Expert knowledge as a foundation for management of secretive species and their habitat. Pages 87-107 in A. H. Perera, C. A. Drew, and C. J. Johnson, editors. Expert knowledge and landscape ecological applications. Springer, New York, New York, USA
45. Drew, C. A., and A. H. Perera. 2011. Expert knowledge as a basis for landscape ecological predictive models. Pages 229-248 in C. A. Drew, Y. Wiersma, and F. H. Huettmann, editors. Predictive species and habitat modeling in landscape ecology: concepts and applications. Springer, New York, New York, USA
46. Dreyfus, H. L., and S. E. Dreyfus. 1986. Mind over Machine: the power of human intuition and expertise in the era of the computer. Free Press, New York, New York, USA
47. Dugger, K. M., G. Ballard, D. G. Ainley, and K. J. Barton. 2006. Effects of flipper bands on foraging behavior and survival of Adélie penguins (Pygoscelis adeliae). Auk 123:858-869 (Pubitemid 44106826)
48. Eastman, J. R., W. Jin, P. A. K. Kyem, and J. Toledano. 1995. Raster procedures for multi-criteria/multiobjective decisions. Photogrammetric Engineering and Remote Sensing 61:539-547
49. Elbroch, M., T. H. Mwampamba, M. J. Santos, M. Zylberberg, L. Liebenberg, J. Minye, C. Mosser, and E. Reddy. 2011. The value, limitations, and challenges of employing local experts in conservation research. Conservation Biology 25:1195-1202
50. Elith, J., M. A. Burgman, and H. M. Regan. 2002. Mapping epistemic uncertainties and vague concepts in predictions of species distribution. Ecological Modeling 157:313-329 (Pubitemid 35258968)
51. Erickson, B. H. 1979. Some problems of inference from chain data. Sociological Methodology 10:276-302
52. Ericsson, K. A., N. Charness, P. J. Feltovitch, and R. R. Hoffman, editors. 2006. The Cambridge handbook of expertise and expert performance. Cambridge University Press, New York, New York, USA
53. Evetts, J. 2003. The sociological analysis of professionalism: occupational change in the modern world. International Sociology 18:395-415 (Pubitemid 36814861)
54. Failing, L., G. Horn, and P. Higgins. 2004. Using expert judgment and stakeholder values to evaluate adaptive management options. Ecology and Society 9:13
55. Fazey, I., J. A. Fazey, and D. M. A. Fazey. 2005. Learning more effectively through experience. Ecology and Society 10:4
56. Fazey, I., J. A. Fazey, J. G. Salisbury, D. B. Lindenmayer, and S. Dovers. 2006. The nature and role of experiential knowledge for environmental conservation. Environmental Conservation 33:1-10 (Pubitemid 44034774)
57. Fischer, I., and D. V. Budescu. 2005. When do those who know more also know more about how much they know? The development of confidence and performance in categorical decision tasks. Organizational Behavior and Human Decision Processes 98:39-53 (Pubitemid 41179202)
58. Fischer, R., R. O'Leary, S. Low Choy, K. Mengersen, and M. J. Caley. 2012. A software tool for elicitation of expert knowledge about species richness or similar counts. Environmental Modelling and Software 30:1-14
59. Franklin, J., S. A. Sisson, M. A. Burgman, and J. K. Martin. 2008. Evaluating extreme risks in invasion ecology: learning from banking compliance. Diversity and Distributions 14:581-591 (Pubitemid 351813539)
60. Garthwaite, P. H., J. B. Kadane, and A. O'Hagan. 2005. Statistical methods for eliciting probability distributions. Journal of the American Statistical Association 100:680-700 (Pubitemid 40816603)
61. Gilchrist, G., M. Mallory, and F. Merkel. 2005. Can local environmental knowledge contribute to wildlife management? Case studies of migratory birds. Ecology and Society 10:20
62. Gilmour, J., R. Beilin, and T. Sysak. 2011. Biosecurity risk and peri-urban landholders-using a stakeholder consultative approach to build a risk communication strategy. Journal of Risk Research 14:281-295
63. Goodman, L. A. 1961. Snowball sampling. Annals of Mathematical Statistics 32:148-170
64. Grech, A., and H. Marsh. 2007. Rapid assessment of risks to a mobile marine mammal in an ecosystemscale marine protected area. Conservation Biology 22:711-720
65. Gregory, R., G. Long, M. Colligan, J. Geiger, and M. Laser. 2012. When experts disagree (and better science won't help much): Using structured deliberations to support endangered species recovery planning. Journal of Environmental Management 105:30-43
66. Gregory, R., J. Ohlson, and J. Arvai. 2006. Deconstructing adaptive management: criteria for applications to environmental management. Ecological Applications 16:2411-2425 (Pubitemid 44917596)
67. Guest, G., A. Bunce, and L. Johnson. 2006. How many interviews are enough? An experiment with data saturation and variability. Field Methods 18:59-82
68. Halford, G. S., N. Cowan, and G. Andrews. 2007. Separating cognitive capacity from knowledge-a new hypothesis. Trends in Cognitive Sciences 11:236-242 (Pubitemid 46809558)
69. Heckathorn, D. D. 1997. Respondent-driven sampling: A new approach to the study of hidden populations. Social Problems 44:174-199
70. Henry, G. T. 2009. Practical sampling. Pages 77-105 in L. Bickman and D. J. Rog, editors. The SAGE handbook of applied social research methods. Second edition. Sage, Thousand Oaks, California, USA
71. Holsapple, C. W., editor. 2003. Handbook on knowledge management 1: knowledge matters. Springer, New York, New York, USA
72. Hora, S. C. 1996. Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management. Reliability Engineering and System Safety 54:217-223 (Pubitemid 126365924)
73. Huntington, H. P. 2000. Using traditional ecological knowledge in science: methods and applications. Ecological Applications 10:1270-1274 (Pubitemid 32139995)
74. Hurley, M. V., E. K. Rapaport, and C. J. Johnson. 2009. Utility of expert-based knowledge for predicting wildlife-vehicle collisions. Journal of Wildlife Management 73:278-286
75. Iglecia, M. N., J. A. Collazo, and A. J. McKerrow. 2012. Use of occupancy models to evaluate expert knowledge-based species-habitat relationships. Avian Conservation and Ecology 7:5
76. Iglesias, R. M. R., and M. M. Kothmann. 1998. Evaluating expert knowledge: plant species responses to cattle grazing and fire. Journal of Range Management 51:332-344 (Pubitemid 28229130)
77. James, A., S. L. Low Choy, and K. Mengersen. 2010. Elicitator: an expert elicitation tool for regression in ecology. Environmental Modeling & Software 25:129-145
78. Janis, I. L. 1971. Groupthink. Psychology Today 5:43-46
79. Jessup, L. M., and D. A. Tansik. 1991. Decision making in an automated environment: the effects of anonymity and proximity with a group decision support system. Decision Sciences 22:266-279
80. Johnson, C. J., and M. P. Gillingham. 2004. Mapping uncertainty: sensitivity of wildlife habitat ratings to variation in expert opinion. Journal of Applied Ecology 41:1032-1041 (Pubitemid 40033191)
81. Kadane, J. B., and L. J. Wolfson. 1998. Experiences in elicitation. Statistician 47:3-19 (Pubitemid 128469059)
82. Kangas, J., and P. Leskinen. 2005. Modelling ecological expertise for forest planning calculations-rationale, examples and pitfalls. Journal of Environmental Management 76:125-133 (Pubitemid 40796622)
83. Kappel, C. V., B. C. Halpern, K. A. Selkoe, and R. M. Cooke. 2011. Eliciting expert knowledge of ecosystem vulnerability to human stressors to support comprehensive ocean management. Pages 253-277 in A. H. Perera, C. A. Drew, and C. J. Johnson, editors. Expert knowledge and landscape ecological applications. Springer, New York, New York, USA
84. Kennedy, K. M., E. R. Irwin, M. C. Freeman, and J. Peterson. 2006. Development of a decision support tool and procedures for evaluating dam operation in the southeastern United States. Science Support Partnership Program Final Report to USGS and USFWS http://www.rivermanagement.org/reports. html
85. Knol, A. B., P. Slottje, J. P. van der Sluijs, and E. Lebret. 2010. The use of expert elicitation in environmental health assessment: a seven step procedure. Environmental Health 9:19
86. Kovacs, J. M. 2000. Perceptions of environmental change in a tropical coastal wetland. Land Degradation and Development 11:209-220 (Pubitemid 30464291)
87. Krebs, C. J. 1989. Ecological methodology. First edition. Harper and Row, New York, New York, USA
88. Kuhnert, P. M. 2011. Four case studies in using expert opinion to inform priors. Environmetrics 22:662-674
89. Kuhnert, P. M., T. G. Martin, and S. P. Griffiths. 2010. A guide to eliciting and using expert knowledge in Bayesian ecological models. Ecological Letters 13:900-914
90. Lauer, M., and S. Aswani. 2010. Indigenous knowledge and long-term ecological change: Detection, interpretation, and responses to changing ecological conditions in Pacific Island communities. Environmental Management 45:985-997
91. Lewis-Beck, M. S., A. Bryman, and T. F. Lia, editors. 2004. The SAGE encyclopedia of social science research methods. Sage, Thousand Oaks, California, USA
92. Linstone, H. A., and M. Turoff, editors. 1975. The Delphi method-techniques and applications. First edition. Addison-Wesley, Boston, Massachusetts, USA
93. Low Choy, S., A. James, J. Murray, and K. Mengerson. 2011. Elicitator: a user-friendly, interactive tool to support scenario-based elicitation of expert. Pages 39-67 in A. H. Perera, C. A. Drew, and C. J. Johnson, editors. Expert knowledge and landscape ecological applications. Springer, New York, New York, USA
94. Low Choy, S., R. A. O'Leary, and K. Mengersen. 2009. Elicitation by design in ecology: using expert opinion to inform priors for Bayesian statistical models. Ecology 90:265-277
95. MacDougall, C., and F. Baum. 1997. The Devil's advocate: a strategy to avoid groupthink and stimulate discussion in focus groups. Qualitative Health Research 7:532-541 (Pubitemid 127649126)
96. Maddock, A., and M. J. Samways. 2000. Planning for biodiversity conservation based on the knowledge of biologists. Biodiversity and Conservation 9:1153-1169 (Pubitemid 30656527)
97. Martin, T. G., M. A. Burgman, F. Fidler, P. M. Kuhnert, S. Low Choy, M. McBride, and K. Mengersen. 2012. Eliciting expert knowledge in conservation science. Conservation Biology 26:29-38
98. Martin, T. G., P. M. Kuhnert, K. Mengersen, and H. P. Possingham. 2005. The power of expert opinion in ecological models using Bayesian methods: impacts of grazing on birds. Ecological Applications 15:266-280 (Pubitemid 40263584)
99. McBride, M. F., and M. A. Burgman. 2011. What is expert knowledge, how is knowledge gathered, and how do we use it to address questions in landscape ecology. Pages 11-38 in A. H. Perera, C. A. Drew, and C. J. Johnson, editors. Expert knowledge and landscape ecological applications. Springer, New York, New York, USA
100. McBride, M. F., F. Fidler, and M. A. Burgman. 2012. Evaluating the accuracy and calibration of expert predictions under uncertainty: predicting the outcomes of ecological research. Diversity and Distributions 18:782-794
101. McNay, R. S., B. G. Marcot, V. Brumovsky, and R. Ellis. 2006. A Bayesian approach to evaluating habitat for woodland caribou in north-central British Columbia. Canadian Journal of Forest Research 36:3117-3133 (Pubitemid 46866361)
102. Meyer, M., and J. Booker. 1991. Eliciting and analyzing expert judgment: a practical guide. Academic Press, New York, New York, USA
103. Mitchell, M., and J. M. Jolley. 2010. Research design explained. Seventh edition. Wadsworth, Belmont, California, USA
104. Moore, C. J., and J. C. Miles. 1991. Knowledge elicitation using more than one expert to cover the same domain. Artificial Intelligence Review 5:255-271
105. Morgan, M. G., and M. Henrion. 1992. Uncertainty: a guide to dealing with uncertainty in quantitative risk and policy analysis. Cambridge University Press, New York, New York, USA
106. Morgan, M. G., L. F. Pitelka, and E. Shevliakova. 2001. Elicitation of expert judgments of climate change impacts on forest ecosystems. Climatic Change 49:279-307 (Pubitemid 32382689)
107. Murray, J. V., A. W. Goldizen, R. A. O'Leary, C. A. McAlpine, H. P. Possingham, and S. Low Choy. 2009. How useful is expert opinion for predicting the distribution of a species within and beyond the region of expertise? A case study using brushtailed rock-wallabies Petrogale penicillata. Journal of Applied Ecology 46:842-851
108. Nakasuk, S., H. Paniaq, E. Ootoova, P. Angmarlik, J. Oosten, F. Laugrand, and A. Kublu. 1999. Interviewing Inuit elders. Volume 1: Introduction. Nunavut Arctic College, Iqaluit, Nunavut, Canada
109. Neuhauser, M. 2001. An adaptive interim analysis-a useful tool for ecological studies. Basic and Applied Ecology 2:203-207 (Pubitemid 32675648)
110. O'Hagan, A., C. E. Buck, A. Daneshkhah, J. R. Eiser, P. H. Garthwaite, D. J. Jenkinson, J. E. Oakley, and T. Rakow. 2006. Uncertain judgements: eliciting experts' probabilities. First edition. John Wiley Sons, Chichester, UK
111. O'Leary, R. A., S. Low Choy, J. Murray, M. Kynn, R. Denham, T. G. Martin, and K. Mengersen. 2009. Comparison of three expert elicitation methods for logistic regression on predicting the presence of the threatened brush-tailed rock-wallaby Petrogale penicillata. Environmetrics 20:379-398
112. Olson, P., and C. Folke. 2001. Local ecological knowledge and institutional dynamics for ecosystem management: a study of Lake Racken watershed, Sweden. Ecosystems 4:85-104 (Pubitemid 32402540)
113. O'Neill, S. J., T. J. Osborn, M. Hulme, I. Lorenzoni, and A. R. Watkinson. 2008. Using expert knowledge to assess uncertainties in future polar bear populations under climate change. Journal of Applied Ecology 45:1649-1659
114. Orsi, F., D. Geneletti, and A. C. Newton. 2011. Towards a common set of criteria and indicators to identify forest restoration priorities: An expert panel-based approach. Ecological Indicators 11:337-347
115. Osterloh, M., and B. S. Frey. 2002. Does pay for performance really motivate employees? Pages 107-122 in A. D. Neely, editor. Business performance measurement: theory and practice. Cambridge University Press, New York, New York, USA
116. Page, S. E. 2007. The difference: How the power of diversity creates better groups, firms, schools, and societies. Princeton University Press, Princeton, New Jersey, USA
117. Palys, T. 2008. Purposive sampling. Pages 698-699 in L. M. Given, editor. The SAGE encyclopedia of qualitative research methods. Volume 2. Sage, Thousand Oaks, California, USA
118. Parry, G. W. 1996. The characterization of uncertainty in probabilistic risk assessments of complex systems. Reliability Engineering and System Safety 54:119-126 (Pubitemid 126365916)
119. Perera, A. H., C. A. Drew, and C. J. Johnson. 2012. Experts, expert knowledge, and their roles in landscape ecological applications. Pages 1-10 in A. H. Perera, C. A. Drew, and C. J. Johnson, editors. Expert knowledge and its application in landscape ecology. Springer, New York, New York, USA
120. Petit, S., D. Chamberlain, K. Haysom, R. Pywell, J. Vickery, R.Warman, D. Allen, and L. Firbank. 2003. Knowledge-based models for predicting species occurrence in arable conditions. Ecography 26:626-640 (Pubitemid 37446859)
121. Pitchforth, J., and K. Mengersen. 2013. A proposed validation framework for expert elicited Bayesian networks. Expert Systems with Applications 40:162-167
122. Pollock, M. L., C. J. Legg, J. P. Holland, and C. M. Theobald. 2007. Assessment of expert opinion: seasonal sheep preference and plant response to grazing. Rangeland Ecology and Management 60:125-135 (Pubitemid 46549153)
123. Prutsch, A., T. Grothmann, I. Schauser, S. Otto, and S. McCallum. 2010. Guiding principles for adaptation to climate change in Europe. ETC/ACC Technical Paper 2010/6. European Topic Centre on Air and Climate Change, Bilthoven, Netherlands. http://acm.eionet.europa.eu/docs/ETCACC-TP-2010-6-guiding- principles-cc-adaptation.pdf
124. Pullinger, M. G., and C. J. Johnson. 2010. Maintaining or restoring connectivity of modified landscapes: evaluating the least-cost path model with multiple sources of ecological information. Landscape Ecology 25:1547-1560
125. Reed, M. S. 2008. Stakeholder participation for environmental management: a literature review. Biological Conservation 141:2417-2431
126. Regan, H. M., M. Colyvan, and M. A. Burgman. 2002. A taxonomy and treatment of uncertainty fo recology and conservation biology. Ecological Applications 12:618-628 (Pubitemid 36516680)
127. Ribisl, K. M., M. A. Walton, C. T. Mowbray, D. A. Luke, W. S. Davidson, and B. J. Bootsmiller. 1996. Minimizing participant attrition in panel studies through the use of effective retention and tracking strategies: review and recommendations. Evaluation and Programs Planning 19:1-25 (Pubitemid 126398956)
128. Robinson, K. A., C. R. Dennison, D. M. Waymanc, P. J. Pronovost, and D. M. Needham. 2007. Systematic review identifies number of strategies important for retaining study participants. Journal of Clinical Epidemiology 60:757-765
129. Rothlisberger, J. D., D. M. Lodge, R. M. Cooke, and D. C. Finnhoff. 2009. Future declines of the binational Laurentian Great Lakes fisheries: the importance of environmental and cultural change. Frontiers in Ecology and the Environment 8:239-244
130. Rowe, G. 1992. Perspectives on expertise in aggregation of judgments. Pages 155-180 in G. Wright and F. Bolger, editors. Expertise and decision support. Plenum, New York, New York, USA
131. Rykiel, E. J., Jr. 1989. Artificial intelligence and expert systems in ecology and natural resource management. Ecological Modelling 46:3-8 (Pubitemid 20422622)
132. Saaty, T. L. 1977. A scaling method for priorities in hierarchical structures. Journal of Mathematical Psychology 15:234-281
133. Sage, A. P. 1987. On the management of information imperfection in knowledge based systems. Lecture Notes in Computer Science 286:1-29
134. Salganik, M. J., and D. D. Heckathorn. 2004. Sampling and estimation of hidden populations using respondent-driven sampling. Sociological Methodology 34:193-239 (Pubitemid 40465521)
135. Santomauro, D., C. J. Johnson, and M. P. Fondahl. 2012. Historical-ecological evaluation of the longterm distribution of woodland caribou and moose in central British Columbia. Ecosphere 3:37
136. Scholz, R. W., and R. Hansmann. 2007. Combining experts' risk judgments on technology performance of phytoremediation -self-confidence ratings, averaging procedures, and formative consensus building. Risk Analysis 27:225-240 (Pubitemid 46411645)
137. Seoane, J., J. Bustamante, and R. Diaz-Delgado. 2005. Effect of expert opinion on the predictive ability of environmental models of bird distribution. Conservation Biology 19:512-522 (Pubitemid 40452152)
138. Shaw, A. D., J. J. Horton, and D. L. Chen. 2011. Designing incentives for inexpert human raters. Pages 275-284 in P. J. Hinds, J. C. Tang, J. Wang, J. E. Bardram, and N. Ducheneaut, editors. Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work, March 19-23, 2011, Hangzhou, China. Association for Computing Machinery, New York, New York, USA
139. Sherry, E. E. 2002. Constructing partnership: a Delphi study of shared resource management in the North Yukon. Dissertation. University of Northern British Columbia, Prince George, Canada
140. Singer, M. 1999. Studying hidden populations. Pages 125-192 in J. J. Schensul, M. D. LeCompte, R. T. Trotter II, E. K. Cromley, and M. Singer, editors. Mapping social networks, spatial data, and hidden populations. AltaMira, Lanham, Maryland, USA
141. Smith, C. S., A. L. Howes, B. Price, and C. A. McAlpine. 2007. Using a Bayesian belief network to predict suitable habitat of an endangered mammal: The Julia Creek dunnart (Sminthopsis douglasi ). Biological Conservation 139:333-347 (Pubitemid 47384700)
142. Smith, G. C., and D. Wilkinson. 2002. Modelling disease spread in a novel host: rabies in the European badger Meles meles. Journal of Applied Ecology 39:865-874 (Pubitemid 35456846)
143. Smithson, M. 1999. Conflict aversion: preference for ambiguity vs conflict in sources and evidence. Organizational Behavior and Human Decision Processes 79:179-198
144. Streeton, R., M. Cooke, and J. Campbell. 2004. Researching the researchers: using a snowballing technique. Nurse Researcher 12:35-46
145. Sutherland, W. J. 2006. Predicting the ecological consequences of environmental change: a review of methods. Journal of Applied Ecology 43:599-616
146. Sutherland, W. J. et al. 2009. One hundred questions of importance to the conservation of global biological diversity. Conservation Biology 23:557-567
147. Sutherland, W. J. et al. 2013. Identification of 100 fundamental ecological questions. Journal of Ecology 101:58-67
148. Tetlock, P. 2006. Expert political judgment: How good is it? How can we know? Princeton University Press, Princeton, New Jersey, USA
149. Theobald, D. M., T. Spies, J. Kline, B. Maxwell, N. T. Hobbs, and V. H. Dale. 2005. Ecological support for rural land-use planning. Ecological Applications 15:1906-1914 (Pubitemid 43112054)
150. Tversky, A., and D. Kahneman. 1974. Judgement under uncertainty: heuristics and biases. Science 185:1124-1131
151. USEPA [US Environmental Protection Agency]. 2011. Expert Elicitation Task Force White Paper. USEPA Science and Technology Policy Council, Washington, D.C., USA
152. Uusitalo, L., S. Kuikka, and A. Romakkaniemi. 2005. Estimation of Atlantic salmon smolt carrying capacity of rivers using expert knowledge. ICES Journal of Marine Science 62:708-722 (Pubitemid 40743032)
153. Valverde, L. J. 2001. Expert judgment resolution in technically-intensive policy disputes. Pages 221-238 in I. Linkov and J. Palma-Oliveira, editors. Assessment and management of environmental risks. Kluwer Academic, Dordrecht, Netherlands
154. Van der Lee, G. E. M., D. T. van der Molen, H. F. P. van den Boogaard, and H. van der Kils. 2006. Uncertainty analysis of a spatial habitat suitability model and implications for ecological management of water bodies. Landscape Ecology 21:1019-1032 (Pubitemid 44505334)
155. Whitfield, D. P., M. Ruddock, and R. Bullman. 2008. Expert opinion as a tool for quantifying bird tolerance to human disturbance. Biological Conservation 141:2708-2717
156. Wilson, S. K., et al. 2010. Crucial knowledge gaps in current understanding of climate change impacts on coral reef fishes. Journal of Experimental Biology 213:894-900
157. Wintle, B. C., and B. Cleeland. 2012. Interpreting risk in international trade. Journal of Risk Research 15:293-312
158. Yamada, K., J. Elith, M. McCarthy, and A. Zerger. 2003. Eliciting and integrating expert knowledge for wildlife habitat modelling. Ecological Modelling 165:251-264 (Pubitemid 36784858)
159. Yancey, A. K., A. N. Ortega, and S. K. Kumanyika. 2006. Effective recruitment and retention of minority research participants. Annual Review of Public Health 27:1-28
160. Zadeh, L. 1965. Fuzzy sets. Information and Control 8:338-353.