Systematic review of expert elicitation methods as a tool for source attribution of enteric illness

Foodborne Pathogens and Disease

Volumn 12, Issue 5, 2015, Pages 367-382

  Butler, Ainslie J ^a   Thomas, M Kate ^a   Pintar, Katarina D M ^a  

^a PUBLIC HEALTH AGENCY OF CANADA   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

EXPERT ELICITATION; INTERVIEW; METHODOLOGY; PRIORITY JOURNAL; PROBABILITY; QUESTIONNAIRE; REVIEW; UNCERTAINTY; WORKSHOP; FACTUAL DATABASE; FOOD CONTAMINATION; FOOD CONTROL; FOODBORNE DISEASES; HUMAN; META ANALYSIS (TOPIC); MICROBIOLOGY;

DATABASES, FACTUAL; FOOD CONTAMINATION; FOOD MICROBIOLOGY; FOODBORNE DISEASES; HUMANS; META-ANALYSIS AS TOPIC; UNCERTAINTY;

EID: 84929458265     PISSN: 15353141     EISSN: 15567125     Source Type: Journal    
DOI: 10.1089/fpd.2014.1844     Document Type: Review

References (90)

1. Acosta H, Wu DR, Forrest BM. Fuzzy experts on recreational vessels, a risk modelling approach for marine invasions. Ecol Model. 2010; 221: 850-863
2. Arnell NW, Tompkins EL, Adger WN. Eliciting information from experts on the likelihood of rapid climate change. Risk Anal. 2005; 25: 1419-1431
3. Ashton RH. Combining the judgments of experts: How many and which ones? Organ Behav Hum Decis Process. 1986; 38: 405-414
4. Aspinall W. A route to more tractable expert advice. Nature. 2010; 463: 294-295
5. Ayyub BM. Elicitation of Expert Opinions for Uncertainty and Risks. Boca Raton, FL: CRC Press, 2001
6. Baker E, Keisler JM. Cellulosic biofuels: Expert views on prospects for advancement. Energy. 2011; 36: 595-605
7. Bartonova A How can scientists bring research to use: The HENVINET experience. Environ Health. 2012; 11(Suppl 1): S2-069X-11-S1-S2
8. Bouma JA, Kuik O, Dekker AG. Assessing the value of Earth Observation for managing coral reefs: An example from the Great Barrier Reef. Sci Total Environ. 2011; 409: 4497-4503
9. Brito M, Griffiths G, Ferguson J, Hopkin D, Mills R, Pederson R, Macneil E. A behavioral probabilistic risk assessment framework for managing autonomous underwater vehicle deployments. J Atmos Ocean Technol. 2012; 29: 1689-1703
10. Cagno E, Caron F, Mancini M, Ruggeri F. Using AHP in determining the prior distributions on gas pipeline failures in a robust Bayesian approach. Reliab Eng Syst Saf. 2000; 67: 275-284
11. Clemen RT, Winkler RL. Combining probability distributions from experts in risk analysis. Risk Anal. 1999; 19: 187-203
12. Cooke RM. Experts in Uncertainty: Opinion and Subjective Probability in Science. New York: Oxford University Press, 1991
13. Cooke RM, Goossens LLHJ. TU Delft expert judgment data base. Reliab Eng Syst Saf. 2008; 93: 657-674
14. Cooke RM, Wilson AM, Tuomisto JT, Morales O, Tainio M, Evans JS. A probabilistic characterization of the relationship between fine particulate matter and mortality: Elicitation of European experts. Envir Sci Tech. 2007; 41: 6598-6605
15. Coppersmith KJ, Youngs RR, Sprecher C. Methodology and main results of seismic source characterization for the PEGASOS Project, Switzerland. Swiss J Geosci. 2009; 102: 91-105
16. Cressey P, Lake R. Ranking Food Safety Risks: Development of NZFSA Policy 2004-2005. Client Report FW0563: 23, 2005
17. Cross P, Rigby D, Edwards-Jones G. Eliciting expert opinion on the effectiveness and practicality of interventions in the farm and rural environment to reduce human exposure to Escherichia coli O157. Epidemiol Infect. 2012; 140: 643-654
18. Davidson VJ, Ravel A, Nguyen TN, Fazil A, Ruzante JM. Food-specific attribution of selected gastrointestinal illnesses: Estimates from a Canadian expert elicitation survey. Foodborne Pathog Dis. 2011; 8: 983-995
19. de Jong A, Wardekker JA, van der Sluijs JP. Assumptions in quantitative analyses of health risks of overhead power lines. Environ Sci Policy. 2012; 16: 114-121
20. Delbecq AL, van de Ven AH. A group process model for problem identification and program planning. J Appl Behav Sci. 1971; 7: 466-492
21. Donlan CJ, Wingfield DK, Crowder LB, Wilcox C. Using expert opinion surveys to rank threats to endangered species: A case study with sea turtles. Conserv Biol. 2010; 24: 1586-1595
22. Doria MdF, Boyd E, Tompkins EL, Adger WN. Using expert elicitation to define successful adaptation to climate change. Environ Sci Policy. 2009; 12: 810-819
23. Elmer F, Seifert I, Kreibich H, Thieken AH. A Delphi method expert survey to derive standards for flood damage data collection. Risk Anal. 2010; 30: 107-124
24. Evans JS, Gray GM, Sielken RL Jr, Smith AE, Valdez-Flores C, Graham JD. Use of probabilistic expert judgment in uncertainty analysis of carcinogenic potency. Regul Toxicol Pharmacol. 1994; 20: 15-36
25. Fauss E, Gorman ME, Swami N. Using expert elicitation to prioritize resource allocation for risk identification for nanosilver. J Law Med Ethics. 2009; 37: 770-780
26. Fitch K, Bernstein SJ, Aguilar MS, Burnand B, LaCalle JR, Lazaro P, van het Loo M, McDonnell J, Vader J, Kahan JP. The RAND/UCLA Appropriateness Method User's Manual. Santa Monica, CA: The Rand Corporation, 2001; MR1269
27. Flander L, Dixon W, Mcbride M, Burgman M. Facilitated expert judgment of environmental risks: Acquiring and analysing imprecise data. Int J Risk Assess Manag. 2012; 16: 199-212
28. Forman EH, Gass SI. The Analytic Hierarchy Process-An exposition. Oper Res. 2001; 49: 469
29. Forsberg B, Braback L, Keune H, Kobernus M, von Krauss MK, Yang A, Bartonova A An expert assessment on climate change and health-with a European focus on lungs and allergies. Environ Health. 2012; 11(Suppl 1): S4
30. Garabed RB, Perez AM, Johnson WO, Thurmond MC. Use of expert opinion for animal disease decisions: An example of foot-And-mouth disease status designation. Prev Vet Med. 2009; 92: 20-30
31. Goossens LH, Harper FT. Joint EC/USNRC expert judgement driven radiological protection uncertainty analysis. J Radiol Prot. 1998; 18: 249-264
32. Gordon SN, Gallo K. Structuring expert input for a knowledgebased approach to watershed condition assessment for the Northwest Forest Plan, USA. Environ Monit Assess. 2011; 172: 643-661
33. Gosling JP, Hart A, Mouat DC, Sabirovic M, Scanlan S, Simmons A Quantifying experts' uncertainty about the future cost of exotic diseases. Risk Anal. 2012; 32: 881-893
34. Havelaar AH, Galindo AV, Kurowicka D, Cooke RM. Attribution of foodborne pathogens using structured expert elicitation. Foodborne Pathog Dis. 2008; 5: 649-659
35. Helmer-Hirschberg O. The Use of the Delphi Technique in Problems of Educational Innovations, P-3499. Santa Monica, CA: The RAND Corporation, 1966
36. Hetes B, Richmond H, Pekar Z, The Expert Elicitation Task Force. US EPA Expert Elicitation Task Force White Paper. Washington, DC: The US Environmental Protection Agency, 2011; 2013
37. Hirsch KG, Podur JJ, Janser RF, McAlpine RS, Martell DL. Productivity of Ontario initial-Attack fire crews: Results of an expert-judgement elicitation study. Can J Forest Res. 2004; 34: 705-715
38. Hnottavange-Telleen K, Chabora E, Finley RJ, Greenberg SE, Marsteller S. Risk management in a large-scale CO2 geosequestration pilot project, Illinois, USA. 10th International Conference on Greenhouse Gas Control Technologies. 2011; 4: 4044-4051
39. Hoek G, Boogaard H, Knol A, de Hartog J, Slottje P, Ayres JG, Borm P, Brunekreef B, Donaldson K, Forastiere F, Holgate S, Kreyling WG, Nemery B, Pekkanen J, Stone V, Wichmann HE, van der Sluijs J. Concentration response functions for ultrafine particles and all-cause mortality and hospital admissions: Results of a European expert panel elicitation. Environ Sci Technol. 2010; 44: 476-482
40. Hoelzer K, Oliver HF, Kohl LR, Hollingsworth J, Wells MT, Wiedmann M. Structured expert elicitation about Listeria monocytogenes cross-contamination in the environment of retail deli operations in the United States. Risk Anal. 2012; 32: 1139-1156
41. Hoffmann SA, Fischbeck PS, Krupnick AJ, McWilliams M. Attributing foodborne illnesses to their food sources: Using large expert panels to capture variability in expert judgment. Washington, DC: Resources for the Future, 2006; Discussion Paper 06-17-REV
42. Hoffmann S, Fischbeck P, Krupnick A, Mcwilliams M. Elicitation from large, heterogeneous expert panels: Using multiple uncertainty measures to characterize information quality for decision analysis. Decision Anal. 2007; 4: 91-109
43. Horst HS, Dijkhuizen AA, Huirne RBM, De Leeuw PW. Introduction of contagious animal diseases into The Netherlands: Elicitation of expert opinions. Livest Prod Sci. 1998; 53: 253-264
44. Kahneman D, Tversky A On the psychology of prediction. Psychol Rev. 1973; 80: 237-251
45. Kangas J, Alho JM, Kolehmainen O, Mononen A Analyzing consistency of experts' judgment-Case assessing forest biodiversity. Forest Sci. 1998; 44: 610-617
46. Knol AB, de Hartog JJ, Boogaard H, Slottje P, van der Sluijs JP, Lebret E, Cassee FR, Wardekker JA, Ayres JG, Borm PJ, Brunekreef B, Donaldson K, Forastiere F, Holgate ST, Kreyling WG, Nemery B, Pekkanen J, Stone V, Wichmann HE, Hoek G. Expert elicitation on ultrafine particles: Likelihood of health effects and causal pathways. Part Fibre Toxicol. 2009; 6: 19-8977-6-19
47. Knol AB, Slottje P, van der Sluijs JP, Lebret E. The use of expert elicitation in environmental health impact assessment: A seven step procedure. Environ Health. 2010; 9: 19-069X-9-19
48. Koehler JJ. The base rate fallacy reconsidered: Descriptive, normative, and methodological challenges. Behav Brain Sci. 1996; 19: 1-53
49. Kuhnert PM, Martin TG, Griffiths SP. A guide to eliciting and using expert knowledge in Bayesian ecological models. Ecol Lett. 2010; 13: 900-914
50. Kynn M. The 'heuristics and biases' bias in expert elicitation. J Roy Stat Soc A Sta. 2008; 171: 239-264
51. Leal J, Wordsworth S, Legood R, Blair E. Eliciting expert opinion for economic models: An applied example. Value Health. 2007; 10: 195-203
52. Leon CJ, VazquezPolo FJ, Gonzalez RL. Elicitation of expert opinion in benefit transfer of environmental goods. Environ Resour Econ. 2003; 26: 199-210
53. Lieu TA, Nguyen MD, Ball R, Martin DB. Health outcomes of interest for evaluation in the Post-Licensure Rapid Immunization Safety Monitoring Program. Vaccine. 2012; 30: 2824-2830
54. Marcot BG, Allen CS, Morey S, Shively D, White R. An expert panel approach to assessing potential effects of bull trout reintroduction on federally listed salmonids in the Clackamas River, Oregon. N Am J Fish Manage. 2012; 32: 450-465
55. Mcdaniels T, Mills T, Gregory R, Ohlson D. Using expert judgments to explore robust alternatives for forest management under climate change. Risk Anal. 2012; 32: 2098-2112
56. Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred reporting items for systematic reviews and meta-Analyses: The PRISMA statement. PLOS Med. 2009; 6: e1000097
57. Montangero A, Belevi H. Assessing nutrient flows in septic tanks by eliciting expert judgement: A promising method in the context of developing countries. Water Res. 2007; 41: 1052-1064
58. Moon JH, Kang CS. Use of fuzzy set theory in the aggregation of expert judgments. Ann Nucl Energy. 1999; 26: 461-469
59. Morgan G, Henrion M. Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis. Cambridge, UK: Cambridge University Press, 1992
60. Norrington L, Quigley J, Russell A, van der Meer R. Modelling the reliability of search and rescue operations with Bayesian Belief Networks. Reliab Eng Syst Saf. 2008; 93: 940-949
61. O'Hagan A Probabilistic uncertainty specification: Overview, elaboration techniques and their application to a mechanistic model of carbon flux. Environ Modell Softw. 2012; 36: 35-48
62. Page T, Heathwaite AL, Thompson LJ, Pope L, Willows R. Eliciting fuzzy distributions from experts for ranking conceptual risk model components. Environ Modell Softw. 2012; 36: 19-34
63. Pires SM. Assessing the applicability of currently available methods for attributing foodborne disease to sources, including food and food commodities. Foodborne Pathog Dis. 2013; 10: 206-213
64. Pires SM, Evers EG, van Pelt W, Ayers T, Scallan E, Angulo FJ, Havelaar A, Hald T, Med-Vet-Net Workpackage 28 Working Group. Attributing the human disease burden of foodborne infections to specific sources. Foodborne Pathog Dis. 2009; 6: 417-424
65. Podofillini L, Dang V, Zio E, Baraldi P, Librizzi M. Using expert models in human reliability analysis-A dependence assessment method based on fuzzy logic. Risk Anal. 2010; 30: 1277-1297
66. Ravel A, Davidson VJ, Ruzante JM, Fazil A Foodborne proportion of gastrointestinal illness: Estimates from a Canadian expert elicitation survey. Foodborne Pathog Dis. 2010; 7: 1463-1472
67. Ravnum S, Zimmer KE, Keune H, Gutleb AC, Murk AJ, Koppe JG, Magnanti B, Lyche JL, Eriksen GS, Ropstad E, Skaare JU, Kobernus M, Yang A, Bartonova A, Krayer von Krauss M. Policy relevant results from an expert elicitation on the human health risks of decabromodiphenyl ether (decaBDE) and hexabromocyclododecane (HBCD). Environ Health. 2012; 11(Suppl 1): S7-069X-11-S1-S7
68. Roman HA, Hammitt JK, Walsh TL, Stieb DM. Expert elicitation of the value per statistical life in an air pollution context. Risk Anal. 2012; 32: 2133-2151
69. Santori G, Valente R, Cambiaso F, Ghirelli R, Gianelli Castiglione A, Valente U. Preliminary results of an expertopinion elicitation process to prioritize an informative system funded by Italian ministry of health for cadaveric donor management, organ allocation, and transplantation activity. Transplant Proc. 2004; 36: 433-434
70. Saunders M, Magnanti BL, Correia Carreira S, Yang A, Alamo-Hernandez U, Riojas-Rodriguez H, Calamandrei G, Koppe JG, Krayer von Krauss M, Keune H, Bartonova A Chlorpyrifos and neurodevelopmental effects: A literature review and expert elicitation on research and policy. Environ Health. 2012 11(Suppl 1): S5-069X-11-S1-S5
71. Speirs-Bridge A, Fidler F, McBride M, Flander L, Cumming G, Burgman M. Reducing overconfidence in the interval judgments of experts. Risk Anal. 2010; 30: 512-523
72. Stiber NA, Pantazidou M, Small MJ. Expert system methodology for evaluating reductive dechlorination at TCE sites. Environ Sci Technol. 1999; 33: 3012-3020
73. Tan K, Baxter G, Newell S, Smye S, Dear P, Brownlee K, Darling J. Knowledge elicitation for validation of a neonatal ventilation expert system utilising modified Delphi and focus group techniques. Int J Hum-Comput St 2010; 68: 344-354
74. Thomas MK, Murray R, Flockhart L, Pintar K, Pollari F, Fazil A, Nesbitt A, Marshall B. Estimates of the burden of foodborne illness in Canada for 30 specified pathogens and unspecified agents, circa 2006. Foodborne Pathog Dis. 2013; 10: 639-648
75. Tuomisto JT, Wilson A, Evans JS, Tainio M. Uncertainty in mortality response to airborne fine particulate matter: Combining European air pollution experts. Reliab Eng Syst Saf. 2008; 93: 732-744
76. Tversky A, Kahneman D. Judgment under uncertainty: Heuristics and biases. Science. 1974; 185: 1124-1131
77. Tyshenko MG, ElSaadany S, Oraby T, Darshan S, Aspinall W, Catford A, Krewski D. Expert elicitation and probabilistic inversion for the judgment of prion disease risk uncertainties using the classical model, EXCALIBUR and UNIBALANCE. Prion. 2010; 4: 165-166
78. Tyshenko MG, ElSaadany S, Oraby T, Darshan S, Aspinall W, Cooke R, Catford A, Krewski D. Expert elicitation for the judgment of prion disease risk uncertainties. J Toxicol Environ Health A 2011; 74: 261-285
79. Vally H, Glass K, Ford L, Hall G, Kirk MD, Shadbolt C, Veitch M, Fullerton KE, Mustro J, Becker N. Proportion of illness acquired by foodborne transmission for nine enteric pathogens in Australia: An expert elicitation. Foodborne Pathog Dis. 2014; 11: 727-733
80. van der Fels-Klerx HJ, Horst HS, Dijkhuizen AA. Risk factors for bovine respiratory disease in dairy youngstock in The Netherlands: The perception of experts. Livest Prod Sci. 2000; 66: 35-46
81. van der Fels-Klerx HJ, Kandhai MC, Brynestad S, Dreyer M, Borjesson T, Martins HM, Uiterwijk M, Morrison E, Booij CJH. Development of a European system for identification of emerging mycotoxins in wheat supply chains. World Mycotoxin J 2009; 2: 119-127
82. van der Fels-Klerx IH, Goossens LH, Saatkamp HW, Horst SH. Elicitation of quantitative data from a heterogeneous expert panel: Formal process and application in animal health. Risk Anal. 2002; 22: 67-81
83. van Lenthe J. ELI: An interactive elicitation technique for subjective probability distributions. Organ Behav Hum Decis Process. 1993; 55: 379-413
84. Walker KD, Catalano P, Hammitt JK, Evans JS. Use of expert judgment in exposure assessment: Part 2. Calibration of expert judgments about personal exposures to benzene. J Expo Anal Environ Epidemiol. 2003; 13: 1-16
85. Walker KD, Evans JS, MacIntosh D. Use of expert judgment in exposure assessment part I Characterization of personal exposure to benzene. J Expo Anal Environ Epidemiol. 2001; 11: 308-322
86. Wardekker JA, De Jong A, Van Bree L, Turkenburg WC, Van Der Sluijs JP. Health risks of climate change: An assessment of uncertainties and its implications for adaptation policies. Environ Health. 2012; 11: 67
87. WHO] World Health Organization. The Global Burden of Disease 2004 Update. Geneva, Switzerland: The World Health Organization 2008
88. Winkler RL, Clemen RT. Multiple experts vs. multiple methods: Combining correlation assessments. Decision Anal. 2004; 1: 167-176
89. Wreathall J, Bley D, Roth E, Multer J, Raslear T. Using an integrated process of data and modeling in HRA. Reliab Eng Syst Safe. 2004; 83: 221-228
90. Zimmer KE, Gutleb AC, Ravnum S, Krayer von Krauss M,Murk AJ, Ropstad E, Skaare JU, Eriksen GS, Lyche JL, Koppe JG, Magnanti BL, Yang A, Bartonova A, Keune H. Policy relevant results from an expert elicitation on the health risks of phthalates. Environ Health. 2012; 11(Suppl 1): S6-069X-11-S1-S6