Probabilistic coherence weighting for optimizing expert forecasts

Decision Analysis

Volumn 10, Issue 4, 2013, Pages 305-326

  Karvetski, Christopher W ^a   Olson, Kenneth C ^a   Mandel, David R ^b   Twardy, C R ^c  

^a Krasnow Institute for Advanced Study   (United States)

^b YORK UNIVERSITY   (Canada)

^c GEORGE MASON UNIVERSITY   (United States)

Author keywords

Crowdsourcing; Fifty fifty blip; Forecast aggregation; Linear opinion pool; Practice; Probabilistic coherence

Indexed keywords

EID: 84884928653     PISSN: 15458490     EISSN: 15458504     Source Type: Journal    
DOI: 10.1287/deca.2013.0279     Document Type: Article

References (44)

1. Brier GW (1950) Verification of forecasts expressed in terms of probability. Monthly Weather Rev. 78(1):1-3.
2. Bruine de Bruin W, Fischbeck PS, Stiber NA, Fischhoff B (2002) What number is "fifty-fifty"?: Redistributing excessive 50% responses in elicited probabilities. Risk Anal. 22(4):713-723.
3. Burgman MA, McBride M, Ashton R, Speirs-Bridge A, Flander L, Wintle B, Fidler F, Rumpff L, Twardy C (2011) Expert status and performance. PLoS One 6(7):1-7.
4. Clemen RT (2008) Comment on Cooke's classical method. Reliability Engrg. System Safety 93:760-765.
5. Clemen RT, Winkler RL (1999) Combining probability distributions from experts in risk analysis. Risk Anal. 19(2):187-203.
6. Cooke RM (1991) Experts in Uncertainty (Oxford University Press, Oxford, UK).
7. Cooke RM, Goossens LLHJ (2008) TU Delft expert judgment data base. Reliability Engrg. System Safety 93(5):657-674.
8. De Finetti B (1990) Theory of Probability: A Critical Introductory Treatment (John Wiley & Sons, New York).
9. Genest C, McConway KJ (1990) Allocating the weights in the linear opinion pool. J. Forecasting 9(1):53-73.
10. Gneiting T (2011) Making and evaluating point forecasts. J. Amer. Statist. Assoc. 106(494):746-762.
11. Herzog SM, Hertwig R (2009) The wisdom of many in one mind: Improving individual judgments with dialectical bootstrapping. Psych. Sci. 20(2):231-237.
12. Hirt ER, Markman KD (1995) Multiple explanation: A consider-analternative strategy for debiasing judgments. J. Personality Soc. Psych. 69(6):1069-1086.
13. Hsee C (1996) The evaluability hypothesis: An explanation for preference reversals between joint and separate evaluationsof alternatives. Organ. Behav. Human Decision Processes 67(1): 247-257.
14. Karvetski CW, Olson KC, Gantz DT, Cross GA (2013) Structuring and analyzing competing hypotheses with Bayesian networks for intelligence analysis. EURO J. Decision Processes, ePub ahead of print April 30, http://link.springer. com/article/10.1007/s40070-013-0001-x.
15. Kolmogorov A (1956) Foundations of the Theory of Probability (Chelsea Publishing Company, New York).
16. Larrick RP, Soll JB (2006) Intuitions about combining opinions: Misappreciation of the averaging principle. Management Sci. 52(1):111-127.
17. Lindley DV, Tversky A, Brown RV (1979) On the reconciliation of probability assessments. J. Roy. Statist. Soc. 142(2):146-180.
18. Lord CG, Lepper MR, Preston E (1984) Considering the opposite: A corrective strategy for social judgment. J. Personality Soc. Psych. 47(6):1231-1243.
19. Macchi L, Osherson D, Krantz DH (1999) A note on superadditive probability judgment. Psych. Rev. 106(1):210-214.
20. Mandel DR (2005) Are risk assessments of a terrorist attack coherent? J. Experiment. Psych.: Appl. 11(4):277-288.
21. Mandel DR (2008) Violations of coherence in subjective probability: A representational and assessment process account. Cognition 106(1):130-156.
22. Merrick JRW (2008) Getting the right mix of experts. Decision Anal. 5(1):43-52.
23. Murphy AH (1973) A new vector partition of the probability score. J. Appl. Meteorology 12(4):595-600.
24. Oppenheimer DM, Meyvis T, Davidenko N (2009) Instructional manipulation checks: Detecting satisficing to increase statistical power. J. Experiment. Soc. Psych. 45(4):867-872.
25. Osherson D, Vardi MY (2006) Aggregating disparate estimates of chance. Games Econom. Behav. 56(1):148-173.
26. Pate-Cornell ME (1996) Uncertainties in risk analysis: Six levels of treatment. Reliability Engrg. Systems Safety 54(2-3):95-111.
27. Peters E, Dieckmann NF, Dixon A, Hibbard JH, Mertz CK (2007) Less is more in presenting quality information to consumers. Medical Care Res. Rev. 64(2):169-190.
28. Predd JB, Osherson DN, Kulkarni SR, Poor HV (2008) Aggregating probabilistic forecasts from incoherent and abstaining experts. Decision Anal. 5(4):177-189.
29. Rohrer D, Pashler H (2007) Increasing retention without increasing study time. Current Directions Psych. Sci. 16(4):183-186.
30. Sharp GL, Cutler BL, Penrod SD (1988) Performance feedback improves the resolution of confidence judgments. Organ. Behav. Human Decision Processes 42(3):271-283.
31. Sieck WR, Merkle EC, Van Zandt T (2007) Option fixation: A cognitive contributor to overconfidence. Organ. Behav. Human Decision Processes 103(1):68-83.
32. Sprenger AM, Dougherty MR, Atkins SM, Franco-Watkins AM, Thomas RP, Lange N, Abbs B (2011) Implications of cognitive load for hypothesis generation and probability judgment. Frontiers Psych. 2(129):1-15.
33. Surowiecki J (2005) The Wisdom of Crowds (Double Day, New York).
34. Tetlock PE (2005) Expert Political Judgment: How Good Is It? How Can We Know? (Princeton University Press, Princeton, NJ)
35. Tsai J, Kirlik A (2012) Coherence and correspondence competence: Implications for elicitation and aggregation of probabilistic forecasts of world events. Proc. Human Factors and Ergonomics Soc. 56th Annual Meeting (Sage, Thousand Oaks, CA), 313-317.
36. Tversky A, Kahneman D (1974) Judgment under uncertainty: Heuristics and biases. Science 185(4157):1124-1131.
37. Tversky A, Koehler DJ (1994) Support theory: A nonextensional representation of subjective probability. Psych. Rev. 101(4):547-567.
38. Vul E, Pashler H (2008) Measuring the crowd within: Probabilistic representations within individuals. Psych. Sci. 19(7):645-647.
39. Wang G, Kulkarni SR, Poor HV, Osherson DN (2011) Aggregating large sets of probabilistic forecasts by weighted coherent adjustment. Decision Anal. 8(2):128-144.
40. Weller JA, Dieckmann NF, Tusler M, Mertz CK, Burns WJ, Peters E (2012) Development and testing of an abbreviated numeracy scale: A Rasch analysis approach. J. Behavioral Decision Making 26(2):198-212.
41. Williams JJ, Mandel DR (2007) Do evaluation frames improve the quality of conditional probability judgment? McNamara DS, Trafton JG, eds. Proc. 29th Annual Meeting of the Cognitive Sci. Soc. (Lawrence Erlbaum Associates, Inc., Mahwah, NJ), 1653-1658.
42. Winkler RL, Clemen RT (2004) Multiple experts vs. multiple methods: Combining correlation assessments. Decision Anal. 1(3):167-176.
43. Wright G, Rowe G, Bolger F, Gammack J (1994) Coherence, calibration, and expertise in judgmental probability forecasting. Organ. Behav. Human Decision Processes 57(1):1-25.
44. Yaniv I, Yates JF, Smith JEK (1991) Measures of discrimination skill in probabilistic judgment. Psych. Bull. 110(3):611-617.