Combining Experts' Judgments: Comparison of Algorithmic Methods Using Synthetic Data

Risk Analysis

Volumn 33, Issue 1, 2013, Pages 109-120

  Hammitt, James K ^a,b   Zhang, Yifan ^c  

^a HARVARD CENTER FOR RISK ANALYSIS   (United States)

^b TOULOUSE SCHOOL OF ECONOMICS   (France)

^c HARVARD UNIVERSITY   (United States)

Author keywords

Bayes; Expert elicitation; Expert judgment; Subjective probability; Uncertainty analysis

Indexed keywords

ALGORITHMIC APPROACH; ALGORITHMIC METHODS; ALTERNATIVE APPROACH; BAYES; COMBINATION METHOD; EXPERT ELICITATION; EXPERT JUDGMENT; FREQUENTIST; SUBJECTIVE PROBABILITY; SYNTHETIC DATA;

PROBABILITY DISTRIBUTIONS; UNCERTAINTY ANALYSIS;

ALGORITHMS;

ALGORITHM; COMPARATIVE STUDY; DATA SET; DECISION MAKING; NUMERICAL MODEL; UNCERTAINTY ANALYSIS; WEIGHT;

ALGORITHM; ARTICLE; COMPARATIVE STUDY; COMPUTER SIMULATION; DECISION MAKING; EVALUATION; EXPERT WITNESS; HUMAN; METHODOLOGY;

ALGORITHMS; COMPUTER SIMULATION; DECISION MAKING; EXPERT TESTIMONY; HUMANS;

EID: 84872418234     PISSN: 02724332     EISSN: 15396924     Source Type: Journal    
DOI: 10.1111/j.1539-6924.2012.01833.x     Document Type: Article

References (30)

1. Cooke RM, Wilson AM, Toumisto JT, Morales O, Tanio M, Evans JS. A probabilistic characterization of the relationship between fine particulate matter and mortality: Elicitation of European experts. Environmental Science and Toxicology, 2007; 41:6598-6605.
2. Roman HA, Walker KD, Walsh TL, Conner L, Richmond HM, Hubbell BJ, Kinney PL. Expert judgment assessment of the mortality impact of changes in ambient fine particulate matter in the U.S. Environmental Science and Technology, 2008; 42:2268-2274.
3. Morgan MG, Keith DW. Subjective judgments by climate experts. Environmental Science and Technology, 1995; 29:468A-476A.
4. Morgan MG, Pitelka LF, Shevliakova E. Elicitation of expert judgments of climate change impacts on forest ecosystems. Climatic Change, 2001; 49:279-307.
5. Morgan MG, Adams P, Keith DW. Elicitation of expert judgments of aerosol forcing. Climatic Change, 2006; 75:195-214.
6. Zickfield K, Levermann A, Kuhlbrodt T, Rahmstorf S, Morgan MG, Keith D. Expert judgments on the response of the Atlantic meriodional overturning circulation response to climate change. Climatic Change, 2007; 82:235-265.
7. Curtwright A, Morgan MG, Keith D. Expert assessment of future photovoltaic technology. Environmental Science and Technology, 2008; 42:9031-9038.
8. EPA, Science Policy Council. Expert Elicitation Task Force White Paper. External Review Draft and Addendum: Selected Recent (2006-2008) Citations, January 6, 2009.
9. Keith DW. When is it appropriate to combine expert judgments? Climatic Change, 1996; 33:139-143.
10. Cooke RM. Experts in Uncertainty: Opinion and Subjective Probability in Science. New York: Oxford University Press, 1991.
11. Clemen RT, Winkler RL. Combining probability distributions from experts in risk analysis. Risk Analysis, 1999; 19:187-203.
12. Clemen RT, Winkler RL. Aggregating probability distributions. Pp. 154-176 in Edwards W, Miles RF, von Winterfeldt D (eds). Advances in Decision Analysis: From Foundations to Applications. Cambridge: Cambridge University Press; 2007.
13. Jouini MN, Clemen RT. Copula methods for aggregating expert opinions. Operations Research, 1996; 44:444-457.
14. Kallen MJ, Cooke RM. Expert aggregation with dependence. Pp. 1287-1294 in Bonano EJ, Camp AL, Majors MJ, Thompson RA (eds). Probabilistic Safety Assessment and Management. Amsterdam: Elsevier; 2002.
15. Cooke RM. Response to discussants. Reliability Engineering and System Safety, 2008; 93:775-777.
16. Cooke RM, Goossens LHJ. TU Delft expert judgment data base. Reliability Engineering and System Safety, 2008; 93:657-674.
17. Lin, S-W, Cheng C-H. The reliability of aggregated probability judgments obtained through Cooke's classical model. Journal of Modelling in Management, 2009; 4:149-161.
18. Flandoli F, Giorgi E, Aspinall WP, Neri A. Comparison of a new expert elicitation model with the classical model, equal weights and single experts, using a cross-validation technique. Reliability Engineering and System Safety, 2011; 96:1292-1301.
19. Hora SC. An analytic method for evaluating the performance of aggregation rules for probability densities. Operations Research, 2010; 58:1440-1449.
20. Clemen RT, Fischer GW, Winkler RL. Assessing dependence: Some experimental results. Management Science, 2000; 46:1100-1115.
21. Morgan MG, Henrion M. Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis. Cambridge: Cambridge University Press, 1990.
22. Winkler RL, Poses RM. Evaluating and combining physicians' probabilities of survival in an intensive care unit. Management Science, 1993; 39:1526-1543.
23. Walker KD, Catalano P, Hammitt JK, Evans JS. Use of expert judgment in exposure assessment: Part 2. Calibration of expert judgments about personal exposures to benzene. Journal of Exposure Analysis and Environmental Epidemiology, 2003; 13:1-16.
24. Lin S-W, Bier VM. A study of expert overconfidence. Reliability Engineering and System Safety, 2008; 93:711-721.
25. Tversky A, Kahneman D. Judgment under uncertainty: Heuristics and biases. Science, 1974; 185:1124-1131.
26. Hammitt JK, Shlyakhter AI. The expected value of information and the probability of surprise. Risk Analysis, 1999; 19:135-152.
27. Clemen RT, Winkler RL. Limits for the precision and value of information from dependent sources. Operations Research, 1985; 33:427-442.
28. Hora SC. Probability judgments for continuous quantities: Linear combinations and calibration. Management Science, 2004; 50:567-604.
29. Clemen RT. Comment on Cooke's classical method. Reliability Engineering and System Safety, 2008; 93:760-765.
30. Clemen RT, Reilly T. Correlations and copulas for decision and risk analysis. Management Science, 1999; 45:208-224.