An analytic method for evaluating the performance of aggregation rules for probability densities

Operations Research

Volumn 58, Issue 5, 2010, Pages 1440-1449

  Hora, Stephen C ^a  

^a University of Southern California ^*  (United States)

Author keywords

Brier score; Calibration; Consensus probability; Dependence; Elicitation; Expert combination; Expert judgment; Geometric average; Linear opinion pools; Overconfidence; Scoring rules; Subjective probability

Indexed keywords

BRIER SCORE; CONSENSUS PROBABILITY; DEPENDENCE; ELICITATION; EXPERT COMBINATION; EXPERT JUDGMENT; GEOMETRIC AVERAGE; LINEAR OPINION POOLS; OVERCONFIDENCE; SCORING RULES; SUBJECTIVE PROBABILITY;

GEOMETRY; LAKES; PROBABILITY;

CALIBRATION;

EID: 77958475014     PISSN: 0030364X     EISSN: 15265463     Source Type: Journal    
DOI: 10.1287/opre.1100.0789     Document Type: Article

References (34)

1. Anderson, T. W., D. A. Darling. 1952. The asymptotic theory of certain "goodness of fit criteria." Ann. Math. Statist. 23(2) 193-212.
2. Ashton, R. H., A. H. Ashton, eds. 1995. Judgment and Decision-Making Research in Accounting and Auditing. Cambridge University Press, Cambridge, UK.
3. Bordley, R. F., R. W. Wolff. 1981. On the aggregation of individual probability estimates. Management Sci. 27(8) 959-964.
4. Box, G. E. P., G. C. Tiao. 1992. Bayesian Inference in Statistical Analysis. Wiley Classics Library. Wiley-Interscience, New York.
5. Brier, G. W. 1950. Verification of weather forecasts expressed in terms of probabilities. Monthly Weather Rev. 78(1) 1-3.
6. Clemen, R. T., T. Reilly. 1999. Correlations and copulas for decision and risk analysis. Management Sci. 45(2) 208-224.
7. Clemen, R. T., R. L. Winkler. 1985. Limits for the precision and value of information from dependent sources. Oper. Res. 33(2) 427-442.
8. Clemen, R. T., R. L. Winkler. 2007. Aggregating probability distributions. W. Edwards, R. Miles, D. von Winterfeldt, eds. Advances in Decision Analysis. Cambridge University Press, Cambridge, UK.
9. Clemen, R. T., S. K. Jones, R. L. Winkler. 1996. Aggregating forecasts: An empirical evaluation of some Bayesian methods. D. A. Berry, K. M. Chalone, J. K. Geweke, eds. Bayesian Analysis in Statistics and Econometrics. John Wiley & Sons, New York.
10. Committee on Methodological Improvements to the Department of Homeland Security's Biological Agent Risk Analysis. 2008. Department of Homeland Security Bioterrorism Risk Assessment: A Call for Change. National Research Council, Washington, DC.
11. Conover, W. J. 1999. Practical Nonparametric Statistics, 3rd ed. John Wiley & Sons, New York.
12. Cooke, R. M. 1991. Experts in Uncertainty. Oxford University Press, Oxford, UK.
13. Croushore, D. 1993. Introducing: The survey of professional forecasters. Business Review. Federal Reserve Bank of Philadelphia, 3-13.
14. Dalkey, N. 1972. An Impossibility Theorem for Group Probability Functions. The RAND Corporation, Santa Monica, CA.
15. Feller, W. 1971. An Introduction to Probability Theory and Its Applications, 2nd ed. Wiley, New York.
16. Genest, C. 1984. A conflict between two axioms for combining subjective distributions. J. Roy. Statist. Soc., Ser. B 46(3) 403-405.
17. Genest, C., K. J. McConway. 1990. Allocating weights in the linear opinion pool. J. Forecasting 9(1) 53-73.
18. Genest, C., J. V. Zidek. 1986. Combining probability distributions: A critique and annotated bibliography. Statist. Sci. 1(1) 114-135.
19. Hora, S. C. 2004. Probability judgments for continuous quantities: Linear combinations and calibration. Management Sci. 50(5) 597-604.
20. Hora, S. C. 2007. Expert judgment. W. Edwards, D. von Winterfeldt, R. Miles, eds. Advances in Decision Analysis: From Foundations to Applications. Cambridge University Press, Cambridge, UK.
21. Kahneman, D., P. Slovic, A. Tversky, eds. 1982. Judgment Under Uncertainty: Heuristics and Biases. Cambridge University Press, Cambridge, UK.
22. Keeney, R. L., D. von Winterfeldt. 1991. Eliciting probabilities from experts in complex technical problems. IEEE Trans. Engrg. Management 38(3) 191-201.
23. Laddaga, R. 1977. Lehrer and the consensus proposal. Synthese 36(4) 473-477.
24. Matheson, J. E., R. L. Winkler. 1976. Scoring rules for continuous probability distributions. Management Sci. 22(10) 1087-1096.
25. McConway, J. 1981. Marginalization and linear opinion pools. J. Amer. Statist. Assoc. 76(374) 410-414.
26. McKay, M. D., R. J. Beckman, W. J. Conover. 1979. A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21(2) 239-245.
27. Morgan, M. G., M. Henrion 1990. Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis. Cambridge University Press, Cambridge, UK.
28. Murphy, A. H., R. L. Winkler. 1977. Reliability of subjective probability forecasts of precipitation and temperature. J. Roy. Statist. Soc., Ser. C (Appl. Statist.) 26(1) 41-47.
29. Press, W. H., S. A. Teukolshy, W. T. Vetterling, B. P. Flannery. 1992. Numerical Recipes in Fortran, 2nd ed. Cambridge University Press, Cambridge, UK.
30. Vick, S. G. 2002. Degrees ofBelief: Subjective Probability and Engineering Judgment. ASCE Press, Reston, VA.
31. Wagner, C. G. 1982. Allocation, Lehrer models, and the consensus of probabilities. Theory and Decision 14(2) 207-220.
32. Winkler, R. L. 1969. Scoring rules and the evaluation of probability assessors. J. Amer. Statist. Assoc. 64(3) 1073-1078.
33. Winkler, R. L. 1971. Probabilistic prediction: Some experimental results. J. Amer. Statist. Assoc. 66(336) 675-685.
34. Winkler, R. L. 1981. Combining probability distributions from dependent information sources. Management Sci. 27(4) 479-488.