Guidance on Expert Knowledge Elicitation in Food and Feed Safety Risk Assessment

EFSA Journal

Volumn 12, Issue 6, 2014, Pages

[No Author Info available]

Author keywords

Cooke's method; Delphi; expert knowledge elicitation; feed safety; food safety; risk assessment; Sheffield method

Indexed keywords

EID: 85078378684     PISSN: None     EISSN: 18314732     Source Type: Journal    
DOI: 10.2903/j.efsa.2014.3734     Document Type: Article

References (194)

1. ACERA (Australian Centre of Excellence for Risk Analysis), 2009. Eliciting Expert Judgments, report #1: Literature Review. ACERA Project 0611, Melbourne, Australia, 71 pp. Available online: www.acera.unimelb.edu.au/materials/endorsed/0611.pdf
2. ACERA (Australian Centre of Excellence for Risk Analysis), 2010. Process manual – Elicitation Tool. ACERA, Melbourne, Australia, 40 pp. Available online: www.acera.unimelb.edu.au/materials/endorsed/0611-process-manual.pdf
3. Algers B, Anil H, Blokhuis H, Fuchs K, Hultgren J, Lambooij B, Nunes T, Paulsen P, and Smulders F, 2009. Project to develop animal aelfare risk assessment guidelines on stunning and killing. Technical report submitted to EFSA, CFP/EFSA/AHAW/2007/01, 88 pp.
4. Anderson JR, 1982. Acquisition of cognitive skill. Psychological Review, 89, 369–406.
5. Asante-Duah K, 2002. Public health risk assessment for human exposure to chemicals. Environmental Pollution Vol. 6, Kluwer Academic Publishers, Dordrecht, the Netherlands.
6. Aspinall W, 2010. A route to more tractable expert advice. Nature, 463, 294–295.
7. Aspinall WP, and Cooke RM, 2013. Quantifying scientific uncertainty from expert judgement elicitation. In: Risk and Uncertainty Assessment for Natural Hazards. Eds. J Rougier, S Sparks, and HJ Hill, University Press, Cambridge, 2013.
8. Ayyub BM, 2001. Elicitation of expert opinions for uncertainty and risks. CRC Press, Boca Raton, USA.
9. Bastian M, Heymann S and Jacomy M, 2009. Gephi: An Open Source software for exploring and manipulating networks. Proceedings of the Third International ICWSM Conference.
10. Berry DC, Raynor DK, Knapp P and Bersellini E, 2003. Patients' understanding of risk associated with medication use – impact of European Commission guidelines and other risk scales. Drug Safety, 26(1), 1–11.
11. Bolger F and Wright G, 1994. Assessing the quality of expert judgment: Issues and analysis. Decision Support Systems, 11, 1–24.
12. Bolger F and Wright G, 2011. Improving the Delphi process: lessons from social psychological research. Technological Forecasting and Social Change, 78, 1500–1513.
13. Booker JM, and Meyer MA, 1988. Sources and effects of interexpert correlation: an empirical study. IEEE Transactions on Systems, Man & Cybernetics, 18, 135–142.
14. Borgatti SP, Everett MG and Freeman LC, 2002. Ucinet for Windows: Software for Social Network Analysis. Harvard, MA: Analytic Technologies.
15. Brenner LA, 2003. A random support model of the calibration of subjective probabilities. Organizational Behavior and Human Decision Processes, 90, 87–110.
16. Brown J, Ehrhardt J, Goossens LHJ, Cooke RM, Fischer F, Hasemann I, Jones JA, Kraan BCP, and Smith JG, 2001. Probabilistic accident consequence uncertainty assessment using COSYMA: uncertainty from the food chain module. EC project report EUR-18823, 126 pp. Available online: ftp://ftp.cordis.europa.eu/pub/fp5-euratom/docs/eur18823_en.pdf.
17. Brown AJ and Aspinall WP, 2004. Use of expert opinion elicitation to quantify the internal erosion process in dams. Proceedings of the 13th Biennial British Dams Society Conference, University of Kent, Canterbury, UK, 22–26 June 2004, 16 pp.
18. Burgman MA, McBride M, Ashton R, Speirs-Bridge A, Flander L, Wintle B, Fidler F, Rumpff L, and Twardy C, 2011. Expert status and performance. PLoS ONE, 6(7), e22998. doi: 10.1371/journal.pone.0022998.
19. th edn. FAO, Rome, Italy, and WHO, Geneva, Switzerland.
20. Cheeseman P, 1985. In defense of probability. Proceedings of the Ninth International Joint Conference on Artificial Intelligence, 1002–1009.
21. Chen C, 2006. CiteSpace II: Detecting and visualizing emerging trends and transient patterns in scientific literature. Journal of the American Society for Information Science and Technology, 57(3), 359–377.
22. Clemen RT, 2008. Comment on Cooke's classical method. Reliability Engineering & System Safety, 93, 760–765.
23. Clemen RT and Winkler RL, 1999. Combining probability distributions from experts in risk analysis. Risk Analysis, 19, 187–203.
24. Cooke RM, 1991. Experts in uncertainty. Oxford University Press, Oxford, UK.
25. Cooke RM, 2008. Response to discussants. Reliability Engineering & System Safety, 93, 775–777.
26. Cooke RM, 2013. Validating expert judgment with the classical model. In: Experts and consensus in social science – critical perspectives from economics, sociology, politics, and philosophy. Eds Martini C and Boumans M. Springer, New York, USA, in preparation.
27. nd European Safety and Reliability Conference “Safety, Reliability and Risk Analysis: Beyond the Horizon”, Amsterdam, The Netherlands.
28. Cooke RM, ElSaadanay S, and Huang X, 2008. On the performance of social network and likelihood-based expert weighting schemes. Reliability Engineering & System Safety, 93, 745–756.
29. Cooke RM and Goossens LJH, 1999. Procedures guide for structured expert judgment. Project report EUR 18820EN. Luxembourg, Euratom, 50 pp.
30. Cooke RM and Goossens LJH, 2000. Procedures guide for structured expert judgment in accident consequence modelling. Radiation Protection Dosimetry 90, 303–309.
31. Cooke RM, and Goossens LJH, 2008. TU Delft Expert Judgment Data Base. Reliability Engineering & System Safety, 93, 657–674.
32. Cooke RM and Jager E, 1998. Failure frequency of underground gas pipelines: methods for assessment with structured expertjudgment. Risk Analysis, 18, 511–527.
33. Dalgleish LI, 1988. Decision making in child abuse cases: applications of social judgment theory and signal detection theory. In: Human judgment: the SJT view. Eds Brehmer B and Joyce CB. North Holland: Elsevier, Amsterdam, the Netherlands.
34. Dalkey NC, 1975. Toward a theory of group estimation. In: The Delphi method: techniques and applications. Eds Linstone HA and Turoff M. Addison-Wesley, Reading, MA, USA.
35. Daneshkhah A, and Oakley JE, 2010. Eliciting multivariate probability distributions. In: Rethinking risk measurement and reporting: Volume I. Ed Böcker K. Risk Books, London, UK.
36. DeGroot MH, 1970. Optimal statistical decisions. Reissued in Wiley Classics Library, Hoboken, USA.
37. de Nooy W, Mrvar A and & Batagelj V, 2005. Exploratory Social Network Analysis with Pajek. Cambridge University Press, Cambridge, UK.
38. Develee JLA and Knol AB, 2011. Software to support elicitation – An explonatory study of existing software packages. RIVM Letter Report 630003001/2011. National Institute for Public Health and the Environment (RIVM), Bilthoven, The Netherlands, 100 pp.
39. Edwards W, Miles RF Jr. and von Winterfeldt D, 2007. Advances in decision analysis: from foundations to applications. Cambridge University Press, Cambridge, UK, 638pp.
40. EFSA (European Food Safety Authority), 2010a. Application of systematic review methodology to food and feed safety assessments to support decision making. EFSA Journal 2010;8(6):1637, 90 pp. doi:10.2903/j.efsa.2010.1637
41. EFSA (European Food Safety Authority), 2010b. Report on data collection: future directions. EFSA Journal 2010; 8(5):1533, 35 pp. doi:10.2903/j.efsa.2010.1533
42. EFSA (European Food Safety Authority), 2012. Science strategy 2012-2016, 40 pp. doi:10.2805/96760
43. EFSA AHAW Panel (EFSA Panel on Animal Health and Welfare), 2009. Guidance on good practice in conducting scientific assessments in animal health using modelling. EFSA Journal 2009;7(12):1419, 38 pp.
44. EFSA AHAW Panel (EFSA Panel on Animal Health and Welfare), 2012. Guidance on risk assessment for animal welfare. EFSA Journal 2012;10(1):2513, 30 pp. doi:10.2903/j.efsa.2012.2513
45. th Feb. 2013.
46. EFSA MB (European Food Safety Authority—Management Board), 2005. Decision concerning implementing measures of transparency and confidentiality requirements. EFSA Document MB 10.03.2005-10.
47. EFSA PLH Panel (EFSA Panel on Plant Health), 2010a. Guidance on a harmonised framework for pest risk assessment and the identification and evaluation of pest risk management options by EFSA. EFSA Journal 2010;8(2):1495, 66 pp.
48. EFSA PLH Panel (EFSA Panel on Plant Health), 2010b. Scientific opinion on a quantitative pathway analysis of the likelihood of Tilletia indica M. introduction into EU with importation of US wheat. EFSA Journal 2010;8(6):1621, 88 pp.
49. EFSA PLH Panel (EFSA Panel on Plant Health), 2011. Guidance on the environmental risk assessment of plant pests. EFSA Journal 2011;9(12):2460, 121 pp.
50. EFSA SC (EFSA Scientific Committee), 2006a. Transparency in risk assessment carried out by EFSA: Guidance document on procedural aspects. EFSA Journal 2006, 353, 1–16.
51. EFSA SC (EFSA Scientific Committee), 2006b. Guidance of the Scientific Committee on a request from EFSA related to Uncertainties in Dietary Exposure Assessment. The EFSA Journal 2006, 438, 1–54.
52. EFSA SC (EFSA Scientific Committee), 2009. Guidance of the Scientific Committee on transparency in the scientific aspects of risk assessment carried out by EFSA. Part 2: general principles. The EFSA Journal 2009, 1051, 1–22.
53. EFSA SC (EFSA Scientific Committee), 2012. Scientific Opinion on risk assessment terminology. EFSA Journal 10(5):2664, 43 pp. doi:10.2903/j.efsa.2012.2664
54. Eggstaff JW, Mazzuchi TA and Sarkani S, 2014. The effect of the number of seed variables on the performance of Cooke's classical model. Reliability Engineering & System Safety, 121, 72–82.
55. Ferrell WR, 1985. Combining individual judgments. In: Behavioral decision making. Ed Wright G. Plenum Press, New York, USA.
56. Flandoli F, Giorgi E, Aspinall WP and Neri A, 2011. Comparison of a new expert elicitation model with the Classical Model, equal weights and single experts, using a cross-validation technique. Reliability Engineering & System Safety, 96, 1292–1310.
57. Fraidin SN, 2004. When is one head better than two? Interdependent information in group decision making. Organizational Behavior and Human Decision Processes, 93, 102–113.
58. French S, 1985. Group consensus probability distributions: a critical survey In: Bayesian Statistics. Eds Bernardo JM, De Groot MH, Lindley DV and Smith AFM. Elsevier North Holland, 182–201.
59. French, S, 2011. Aggregating expert judgements. Revista de la Real Academia de Ciencias Exactas, Fisicas y Naturales. Serie A, Matematicas, 105, 181–206.
60. French S and Rios Insua D, 2000. Statistical decision theory. Arnold, London, UK.
61. Frey HC and Patil SR, 2002. Identification and review of sensitivity analysis methods. Risk Analysis 22, 553–578.
62. Garthwaite PH, Kadane JB and O'Hagan A, 2005. Statistical methods for eliciting probability distributions. Journal of the American Statistical Association, 100, 680–701.
63. Greitemeyer T and Schulz-Hardt S, 2003. Preference-consistent evaluation of information in the hidden profile paradigm: beyond group-level explanations for the dominance of shared information in group decisions. Journal of Personality and Social Psychology, 84, 322–339.
64. Genest C and Zidek JV, 1986. Combining probability distributions: a critique and an annotated bibliography. Statistical Science 1, 114–118.
65. Genre V, Kenny G, Meyler A and Timmermann A, 2013. Combining expert forecasts: can anything beat the simple average? International Journal of Forecasting, 108–121.
66. Gerard HB, Wilhelmy RA and Conolley ES, 1968. Conformity and groupsize. Journal of Personality and Social Psychology, 8, 79–82.
67. Germain ML, 2006. Development and preliminary validation of a psychometric measure of expertise: the Generalized Expertise Measure (GEM). Unpublished doctoral dissertation. Barry University, Florida, USA.
68. Gigerenzer G, 1991. How to make cognitive illusions disappear: beyond “heuristics and biases”. European Review of Social Psychology 2, 83–115.
69. Gigerenzer G and Hoffrage U, 1994. How to improve Bayesian reasoning without instruction: Frequency formats. Psychological Review, 102, 684–704.
70. Goossens LHJ, Cooke RM, Woudenberg F and van der Torn P, 1992. Probit functions and expert judgment: Report prepared for the Ministry of Housing, Physical Planning and Environment, the Netherlands; Delft University of Technology, Safety Science Group and Department of Mathematics, and Municipal Health Service, Rotterdam, Section Environmental Health, October 1992.
71. Goossens LHJ and Kelly GN, 2000. Special: issue: expert judgement and accident consequence uncertainty analysis (COSYMA). Radiation Protection Dosimetry 90, 293–294.
72. Gore SM, 1987. Biostatistics and the Medical Research Council. Medical Research Council News.
73. Hardman D, 2009. Judgment and decision making: psychological perspectives. BPS textbooks in psychology, Blackwell, Chichester, UK, 2009.
74. Hastie R, 1986. Review essay: Experimental evidence on group accuracy. In: Information Pooling and Group Decision Making: Proceedings of the Second University of California, Irvine, Conference on Political Economy. Eds. Grofman B and Owen G. JAI Press, Greenwich, CT.
75. Hilbert M, 2012. Toward a synthesis of cognitive biases: how noisy information processing can bias human decision making. Psychological Bulletin, 138, 211–237.
76. Henry N and Fekete JD, 2006. MatrixExplorer: A dual-representation system to explore social networks. IEEE Transactions on Visualization and Computer Graphics, 12(5), 677–684.
77. Hoffmann S, Fischbeck P, Krupnick A and McWilliams M, 2006. Eliciting information on uncertainty from heterogeneous expert panels attributing U.S. foodborne pathogen illness to food consumption. Resources for the Future Discussion Paper DP 06-17-Rev. Resources for the Future, Washington, DC, USA.
78. INL (Idaho National Laboratory), 2005. Simplified expert elicitation guideline for risk assessment of operating events. Available online: www.inl.gov/technicalpublications/documents/3310952.pdf
79. IPCC (Intergovernmental Panel on Climate Change), 2007. Summary for policymakers: contribution of Working Group I to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change. Available online: www.ipcc.ch/pdf/assessment-report/ar4/wg3/ar4-wg3-spm.pdf
80. IPCS (International Programme on Chemical Safety), 2004. IPCS risk assessment terminology. Harmonization Project document No.1. FAO/WHO, Geneva, Switzerland.
81. IPCS (International Programme on Chemical Safety), 2009. Principles and methods for the risk assessment of chemicals in food. Environmental Health Criteria 240. FAO/WHO, Geneva, Switzerland.
82. IPPC (International Plant Protection Convention), 2004. Pest risk analysis for quarantine pests, including analysis of environmental risks and living modified organisms. International Standards for Phytosanitary Measures (ISPM) No 11. FAO (Food and agricultural Organization of the United Nations), Rome, Italy, 30 pp.
83. IRGC (International Risk Governance Council), 2005. White Paper 1 on risk governance: towards an integrative approach. IRGC, Geneva, Switzerland. Available online: www.irgc.org/IMG/pdf/IRGC_WP_No_1_Risk_Governance__reprinted_version_.pdf
84. Jacobs RA, 1995. Methods for combining experts' probability assessments. Neural Computing, 7, 867–888.
85. Janis IL, 1982. Groupthink: psychological studies of policy decisions and fiascoes, 2nd edn. Houghton Mifflin, Boston, MA, USA.
86. Johnson SR, Tomlinson GA, Hawker GA, Granton JT, Grosbein, HA and Feldman B, 2010. A valid and reliable belief elicitation method for Bayesian priors. Journal of Clinical Epidemiology, 63, 370–383.
87. Jones JA, Ehrhardt J, Goossens LHJ, Brown J, Cooke RM, Fischer F, Hasemann I, and Kraan BCP, 2001. Probabilistic accident consequence uncertainty assessment using COSYMA: Methodology and processing techniques. EC project report, EUR 18827, 77 pp. Available online: ftp://ftp.cordis.europa.eu/pub/fp5-euratom/docs/eur18827_en.pdf
88. Kahneman D and Tversky A, 1979. Prospect theory: an analysis of decision under risk. Econometrica, 47, 263–291.
89. Kahneman D, Slovic P and Tversky A (Eds), 1982. Judgment under uncertainty: heuristics and biases. Cambridge University Press, Cambridge, UK.
90. Kaplan S and Garrick BJ, 1981. On the quantitative definition of risk. Risk Analysis, 1, 11–27.
91. Keith DW, 1996. When is it appropriate to combine expert judgements? Climatic Change, 33, 139–143.
92. Keren G, 1991. Calibration and probability judgments: conceptual and methodological issues. Acta Psychologica, 77, 217–273.
93. Kerr NL, MacCoun RJ and Kramer GP, 1996. Bias in judgement: comparing individuals and groups. Psychological Review, 103, 687–719.
94. Kharroubi SA, Brazier JE, Roberts J and O'Hagan A, 2007. Modelling SF-6D health state preference data using a nonparametric Bayesian method. Journal of Health Economics, 26, 597–612.
95. Knol AB, Slottje P, van der Sluijs JP, and Lebret E, 2010. The use of expert elicitation in environmental health impact assessment: a seven step procedure. Environmental Health, 9, 1–16.
96. Koriat A, Lichtenstein S and Fischhoff B, 1980. Reasons for confidence. Journal of Experimental Psychology: Human Learning and Memory, 6, 107–118.
97. Kurowicka D, and Cooke RM, 2006. Uncertainty analysis with high dimensional dependence modelling. John Wiley and Sons, Chichester, UK.
98. Kynn M, 2008. The ‘heuristics and biases’ bias in expert elicitation. Journal of the Royal Statistical Society, Series A 171, 239–264.
99. Larson JR Jr, Christensen C, Franz TM and Abbott AS, 1998. Diagnosing groups: the pooling, management and impact of shared and unshared case information in team-based medical decision making. Journal of Personality and Social Psychology, 75, 93–108.
100. Lave J and Wenger E, 1991. Situated learning: legitimate peripheral participation. Cambridge: University Press, Cambridge, UK.
101. Lawrence MJ, Edmundson RH and O'Connor MJ, 1986. The accuracy of combining judgmental and statistical forecasts. Management Science, 32, 1521–1532.
102. Lin S and Cheng CH, 2009. The reliability of aggregated probability judgments obtained through Cooke's classical model. Journal of Modeling in Management, 4, 149–161.
103. Lindley DV, 1982. Scoring rules and the inevitability of probability. International Statistical Review, 50, 1–11.
104. Lindley DV, 1987. The probability approach to the treatment of uncertainty in artificial intelligence and expert systems. Statistical Science 2, 17–24.
105. Lock A, 1987. Integrating group judgments in subjective forecasts. In: Judgmental forecasting. Eds Wright G and Ayton P. John Wiley and Sons, Chichester, UK.
106. MacIntyre A, 2007. After virtue: a study in moral theory, 3rd edn. Duckworth, London, UK.
107. Malkiel BG, 2011. A random walk down Wall Street: the time-tested strategy for successful investing. W. Norton, New York, USA.
108. Meyer MA and Booker JM, 1987. Sources of Correlation between Experts: Empirical Results from two Extremes. Los Alamos National Laboratory Report LA-10918-LA: Los Alamos, New Mexico and Nuclear Regulatory Commission Report NUREG/CR-4814, Washington, DC, USA.
109. Meyer MA and Booker JM, 2001. Eliciting and analyzing expert judgment: a practical guide. American Statistical Association and the Society for Industrial and Applied Mathematics, Alexandria, Virginia, USA.
110. Moore G and Beadle R, 2006. In search of organizational virtue in business: agents, goods, practices, institutions and environments. Organization Studies, 27, 369–389.
111. Moore PG and Thomas H, 1988. The anatomy of decisions, 2nd edn. Penguin, Harmondsworth, UK.
112. Murphy AH and Brown BG, 1985. A comparative evaluation of objective and subjective weather forecasts in the United States. In: Behavioral decision making. Ed. Wright G. Plenum Press, New York, USA.
113. NRC (National Research Council, Committee on the institutional means for assessment of risks to public health), 1983. Risk assessment in the federal government: managing the process. National Academy Press, Washington, DC, USA.
114. Oakley JE and O'Hagan A, 2004. Probabilistic sensitivity analysis of complex models: a Bayesian approach. Journal of the Royal Statistical Society, B 66, 751–769.
115. Oakley JE and O'Hagan A, 2007. Uncertainty in prior elicitations: a nonparametric approach. Biometrika, 94, 427–441.
116. O'Hagan A, 1998. Eliciting expert beliefs in substantial practical applications. The Statistician, 47, 21–35 (with discussion, pp. 55–68).
117. O'Hagan A, 2012. Probabilistic uncertainty specification: overview, elaboration techniques and their application to a mechanistic model of carbon flux. Environmental Modelling and Software, 36, 35–48.
118. O'Hagan A and Wells FS, 1993. Use of prior information to estimate costs in a sewerage operation. In: Case studies in Bayesian statistics. Eds Gatsonis C, Hodges JS, Kass RE and Singpurwalla ND. Springer-Verlag, New York, USA, 118–163.
119. O'Hagan A, Buck CE, Daneshkhah A, Eiser JR, Garthwaite PH, Jenkinson D, Oakley JE and Rakow T, 2006. Uncertain judgements: eliciting experts' probabilities. John Wiley and Sons, Chichester, UK.
120. st edn. OIE, Paris, France.
121. Oskamp S, 1962. The relationship of clinical experience and training methods to several criteria of clinical prediction. Psychological Monographs, 176 (28, Whole No. 547).
122. Oskamp S, 1965. Overconfidence in case study judgments. Journal of Consulting Psychology 1965; 29 (3): 261-265.
123. Parenté FJ and Anderson-Parenté JK, 1987. Delphi inquiry systems. In: Judgmental forecasting. Eds Wright G and Ayton P. John Wiley and Sons, Chichester, UK.
124. Phillips LD, 1987. On the adequacy of judgmental forecasts. In: Judgmental forecasting. Eds Wright G and Ayton P. John Wiley and Sons, Chichester, UK.
125. Plous S, 1993. The psychology of judgement and decision making. Mc Graw-Hill Series in social psychology, New York, USA.
126. RIVM (Rijksinstituut voor Volksgezondheid en Milieu), 2008. Expert Elicitation: Methodological suggestions for its use in environmental health impact assessments. RIVM Letter report 630004001/2008. RIVM, Bilthoven, the Netherlands.
127. Roosen J and Hennessy DA, 2001. Capturing experts' uncertainty in welfare analysis: an application to Organophosphate use regulation in U.S. apple production. American Journal of Agricultural Economics 83, 166–182.
128. Rowe G, 1992. Perspectives on expertise in the aggregation of judgments. In: Expertise and decision support. Eds Wright G and Bolger F. Plenum, Press, London, UK, 155–180.
129. Rowe G, 2010. Assessment of the COT uncertainty framework from a social science perspective: a theoretical evaluation. Food Standards Agency, London, UK, 58 pp.
130. Rowe G and Wright G, 1999. The Delphi technique as a forecasting tool: issues and analysis. International Journal of Forecasting, 15, 353–375.
131. Rowe G and Wright G, 2001. Differences in expert and lay judgments of risk: myth or reality? Risk Analysis, 21 (2), 341–356.
132. Saltelli A, Chan K and Scott EM, 2000. Sensitivity Analysis. John Wiley and Sons.
133. Schweiger DM, Sandberg WR and Ragan JW, 1986. Group approaches for improving strategic decision-making: a comparative analysis of dialectical inquiry, devils advocacy, and consensus. Academy of Management Journal, 29, 51–71.
134. Schweiger DM, Sandberg WR and Rechner PL, 1989. Experiential effects of dialectical inquiry, devils advocacy, and consensus approaches to strategic decision making. Academy of Management Journal, 32, 745–772.
135. Seaver DA, 1978. Assessing probability with multiple individuals: Group interaction versus mathematical aggregation (Report No. 78-3). Social Science Research Institute, University of Southern California, Los Angeles CA, USA.
136. Shafer G, 1976. A mathematical theory of evidence. Princeton University Press, Princeton, NJ, USA.
137. Smits T, and Hoorens V, 2005. How probable is probable? It depends on whom you are talking about. Journal of Behavioral Decision Making, 18, 83–96.
138. Sniezek JA, 1992. Groups under uncertainty: an examination of confidence in group decision making. Organizational Behavior and Human Decision Processes, 52, 124–155.
139. Soll JB and Klayman J, 2004. Overconfidence in interval estimates. Journal of Experimental Psychology: Learning, Memory and Cognition, 30, 299–314.
140. Stanovich KE and West RF, 2007. Natural myside bias is independent of cognitive ability. Thinking and Reasoning, 13, 225–247.
141. Stasser G, Stewart DD, and Wittenbaum GM, 1995. Expert roles and information exchange during discussion: the importance of knowing who knows what. Journal of Experimental and Social Psychology, 31, 244–265.
142. Stasser G, and Titus W, 1985. Pooling of unshared information in group decision making: biased information processing during discussion. Journal of Personality and Social Psychology, 48, 1467–1478.
143. Swanson RA, and Holton EFIII, 2001. Foundations of human resource development. Berrett-Koehler Publishers, Inc., San Francisco, CA, USA.
144. Szwed PS, and van Dorp JR, 2002. A Bayesian model for rare event risk assessment using expert judgement about paired scenario comparisons. ASEM National Conference.
145. Torrance GW, 1986. Measurement of health state utilities for economic appraisal: a review. Journal of Health Economics 5, 1–30.
146. US-EPA, 2011 (United States – Environmental Protection Agency) Expert Elicitation Task Force White Paper. US_EPA, Washington, DC, USA (Draft version, 2009). Available online: www.epa.gov/stpc/pdfs/ee-white-paper-final.pdf, http://www.epa.gov/osa/pdfs/elicitation/Expert_Elicitation_White_Paper-January_06_2009.pdf
147. Van de Ven AH, and Delbecq AL, 1971. Nominal versus interacting group processes for committee decision making effectiveness. Academic Management Journal, 14, 203–213.
148. Walley P, 1991. Statistical reasoning with imprecise probabilities. Chapman & Hall, London, UK.
149. Wenger E, 1998. Communities of practice: learning, meaning and identity. Cambridge University Press, Cambridge, UK.
150. Wiper MW and French S, 1995. Combining experts' opinions using a Normal-Wishart model. Journal of Forecasting 14, 25–34.
151. Wright G, Bolger F and Rowe G, 2002. An empirical test of the relative validity of expert and lay judgments of risk. Risk Analysis, 22, 1107–1122.
152. Wright G, Rowe G, Bolger F and Gammack J, 1994. Coherence, calibration and expertise in judgmental probability forecasting. Organizational Behavior and Human Decision Processes, 57, 1–25.
153. Yunusov AR, Cooke RM and Krymsky VG, 1999. Rexcalibr-integrated system for processing expert judgement: In: Proceedings 9th Annual Conference Risk Analysis: Facing the New Millenium Rotterdam, The Netherlands, Oktober 10-13. Eds. Goossens LHJ. University Press, Delft, The Netherlands.
154. th edn. Springer, New York, USA.
155. EFSA (European Food Safety Authority), 2010. Application of systematic review methodology to food and feed safety assessments to support decision making. EFSA Journal 2010; 8(6):1637. [90 pp.]. doi:10.2903/j.efsa.2010.1637. Available online: www.efsa.europa.eu.
156. Andree KB and López MA, 2013. Species identification from archived snail shells via genetic analysis: a method for DNA extraction from empty shells, Molluscan Research, 33:1, 1–5
157. Barnes MA, Fordham RK and Burks RL, 2008. Fecundity of the exotic apple snail, Pomacea insularum. Journal of the North American Benthological Society 27, 738–745.
158. EFSA Panel on Plant Health (PLH), 2012a. Statement on the identity of apple snails. EFSA Journal 2012;10(4):2645. [278 pp.] doi:10.2903/j.efsa.2012.2645. Available online: www.efsa.europa.eu/efsajournal
159. EFSA Panel on Plant Health (PLH), 2012b. Scientific Opinion on the evaluation of the pest risk analysis on Pomacea insularum, the island apple snail, prepared by the Spanish Ministry of Environment and Rural and Marine Affairs. EFSA Journal 2012;10(1):2552. [278 pp.] doi:10.2903/j.efsa.2012.2552. Available online: www.efsa.europa.eu/efsajournal
160. Estebenet AL and Cazzaniga NJ, 1992. Growth and demography of Pomacea canaliculata (Gastropoda: Ampullariidae) under laboratory conditions. Malacological Review 25, 1–12.
161. Estebenet AL and Martín PR, 2002. Workshop: “Biology of Ampullariidae” – Minireview – Pomacea canaliculata (Gastropoda: Ampullariidae): life-history traits and their plasticity. Biocell 26, 83–89.
162. Hayes KA, Cowie RH, Thiengo SC and Strong EE, 2012. Comparing apples with apples: clarifying the identities of two highly invasive Neotropical Ampullariidae (Caenogastropoda). Zoological Journal of the Linnean Society, 2012, 166, 723–753
163. Hayes KA, Joshi R, Thiengo SC and Cowie RH, 2008. Out of South America: multiple origins of non native apple snails in Asia. Diversity and Distributions 14, 701–712.
164. He, M., J. Zhang, et al. (2012). “Potential spatio-temporal distribution of golden apple snail and the harm risk on paddy fields in Guangdong based on GIS technology.” Journal of South China Agricultural University 33(3): 288–295.
165. Heiler, K., P. v. Oheimb, et al. (2008). “Studies on the temperature dependence of activity and on the diurnal activity rhythm of the invasive Pomacea canaliculata (Gastropoda: Ampullariidae).” Mollusca 26(1): 73–81.
166. Howells R, Burlakova LE, Karatayev AY, Marfurt RK and Burks RL, 2006. Native and introduced Ampullaridae in North America: History, status, and ecology. In In Joshi RC & Sebastian LS: GLOBAL ADVANCES IN ECOLOGY AND MANAGEMENT OF GOLDEN APPLE SNAILS. Philippine Rice Research Institute, Science city of Munoz. 588p
167. Huang, P., Z. Lin, et al. (2010). “Biological characteristics of the egg stage of Pomacea canaliculata.” Journal of Fujian Agriculture and Forestry University (Natural Science Edition) 39(1): 25–29.
168. Lee, T.-G. and K. Oh (2006). “Golden apple snails in Korea.” In Joshi RC & Sebastian LS: GLOBAL ADVANCES IN ECOLOGY AND MANAGEMENT OF GOLDEN APPLE SNAILS. Philippine Rice Research Institute, Science city of Munoz. 588p
169. Liu, J., Y.-J. He, et al. (2012). “Characteristics of Pomacea canaliculata reproduction under natural conditions.” Ying yong sheng tai xue bao = The journal of applied ecology / Zhongguo sheng tai xue xue hui, Zhongguo ke xue yuan Shenyang ying yong sheng tai yan jiu suo zhu ban 23(2): 559–565.
170. Liu, Yan-bin; HAN, Wei; XIAN, Zhen-hua. 2011. Effect of different temperatures on growth, development and feeding of Pomacea canaliculata. Journal of Southern Agriculture
171. López, M.A., Altaba, C.R., Andree, K.B. & López, V. (2009) First invasion of the apple snail Pomacea insularum in Europe. Tentacle 18, 26–28.
172. Lv, S., Zhang, Y., Liu, H. X., Hu, L., Liu, Q., Wei, F. R., … & Utzinger, J. (2013). Phylogenetic evidence for multiple and secondary introductions of invasive snails: Pomacea species in the People's Republic of China. Diversity and Distributions, 19(2), 147–156.
173. Martín PR and Estebenet AL, 2002. Interpopulation variation in life-history traits of Pomacea canaliculata (Gastropoda: Ampullariidae) in southwestern Buenos Aires Province, Argentina. Malacologia, 44, 153–163.
174. Martín PR, Estebenet AL and Cazzaniga NJ, 2001. Factors affecting the distributionn of Pomacea canaliculata (Gastropoda: Ampullariidae) along its southernmost natural limit. Malacologia, 2001, 43(1-2):13–23.
175. Matsukura K, Okuda M, Cazzaniga NJ, Wada T, 2013. Genetic exchange between two freshwater apple snails, Pomacea canaliculata and Pomacea maculata invading East and Southeast Asia. Biol Invasions (2013) 15:2039–2048
176. Matsukura, K., H. Tsumuki, et al. (2009). “Physiological response to low temperature in the freshwater apple snail, Pomacea canaliculata (Gastropoda: Ampullariidae).” Journal of Experimental Biology 212(16): 2558–2563.
177. Mochida, O. 1991. Spread of freshwater Pomacea snails (Mollusca: Pilidae) from Argentina to Asia. Micronesia 3(supplement):51–62.
178. Oya S, Hirai Y and Miyahara Y, 1987. Overwintering of the apple snail, Pomacea canaliculata Lamarck, in north Kyushu, Japan. Japanese Journal of Applied Entomology and Zoology 31, 206–212.
179. Pizani, N. V., A. L. Estebenet, and P. R. Martín, 2005. “Effects of submersion and aerial exposure on clutches and hatchlings of Pomacea canaliculata (Gastropoda: Ampullariidae).” American Malacological Bulletin 20.1-2 (2005): 55–63.
180. Rawlings TA, Hayes KA, Cowie RH and Collins TM, 2007. The identity, distribution, and impacts of non-native apple snails in the continental United States. BMC Evolutionary Biology 7, 97.
181. Seuffert, M. E., & Martín, P. R. (2010). Dependence on aerial respiration and its influence on microdistribution in the invasive freshwater snail Pomacea canaliculata (Caenogastropoda, Ampullariidae). Biological Invasions, 12(6), 1695–1708.
182. Seuffert, M. E., & Martín, P. R. (2012). A lentic dweller in lotic habitats: the behavior of the invasive South American apple snail Pomacea canaliculata in flowing water. Aquatic Ecology, 46(1), 129–142.
183. Seuffert, M. E., & Martín, P. R. (2013). Juvenile growth and survival of the apple snail Pomacea canaliculata (Caenogastropoda: Ampullariidae) reared at different constant temperatures. SpringerPlus, 2(1), 312.)
184. Seuffert, M. E., Saveanu, L., & Martín, P. R. (2012). Threshold Temperatures and Degree-Day Estimates for Embryonic Development of the Invasive Apple Snail Pomacea canaliculata (Caenogastropoda: Ampullariidae). Malacologia, 55(2), 209–217.
185. Spanish Ministry of Environment and Rural and Marine Affairs, 2011. Pest Risk Analysis on the introduction of Pomacea insularum (d'Orbigny, 1835) into the EU. April 2011. Prepared by the Spanish Ministry of Environment and Rural and Marine Affairs.
186. Syobu, S., H. Mikuriya, et al. (2001). “Estimating the overwintering mortality of the apple snail, Pomacea canaliculata (Lamarck) (Gastropoda: Ampullariidae) in a paddy field of southern Japan using temperature data.” Japanese Journal of Applied Entomology and Zoology 45(4): 203–207.
187. Tamburi, N. E. & P. R. Martín, 2009a, Reaction norms of size and age at maturity of Pomacea canaliculata (Gastropoda: Ampullariidae) under a gradient of food deprivation. Journal of Molluscan Studies, 75: 19–26.
188. Tamburi, N. E. & P. R. Martín, 2009b, Feeding rates and food conversion efficiencies in the apple snail Pomacea canaliculata (Caenogastropoda: Ampullariidae). Malacologia, 51:221–232
189. Teo SS, 2004. Biology of the golden apple snail, Pomacea canaliculata (Lamarck, 1822) with emphasis on responses to certain environmental conditions in Sabah, Malaysia. Molluscan Research, 2004, 24, 139–148.
190. Wada and Matsukura, 2011. Linkage of cold hardiness with desiccation tolerance in the invasive freshwater apple snail, Pomacea canaliculata (Caenogastropoda: Ampullariidae). J. Mollus. Stud. (2011) 77 (2): 149–153. doi: 10.1093/mollus/eyq049 First published online: February 18, 2011.
191. Wada, T. and K. Matsukura (2007). “Seasonal changes in cold hardiness of the invasive freshwater apple snail, Pomacea canaliculata (Lamarck) (Gastropoda: Ampullariidae).” Malacologia 49(2): 383–392.
192. Yingying P; Xiaoping D and Shengzhang Y, 2008. Effects of temperature stress on development, feeding and survival of the apple snail, Pomacea canaliculata (Lamarck). Acta Phytophylacica Sinica, 2008, vol. 3, p. 009.
193. Yu XP, Li XF, Lu ZX, et al. 2002. Occurrence of golden apple snail, Pomacea canaliculata (Lamarck) in paddy fields and its management in China. // Joshi RC, Wada T. Proceeding of the special working group on the golden apple snail (Pomacea spp.) at the 7th Congress on Medical and Applied Malacology. Philippines, 2002)
194. Yusa, Y., Wada, T., & Takahashi, S. (2006). Effects of dormant duration, body size, self-burial and water condition on the long-term survival of the apple snail, Pomacea canaliculata (Gastropoda: Ampullariidae). Applied entomology and zoology, 41(4), 627–632.