Combining probability distributions from experts in risk analysis

Risk Analysis

Volumn 19, Issue 2, 1999, Pages 187-203

  Clemen, Robert T ^a   Winkler, Robert L ^a  

^a DUKE UNIVERSITY   (United States)

Author keywords

Combining probabilities; Expert judgment; Probability assessment

Indexed keywords

DECISION MAKING; PROBABILITY DISTRIBUTIONS;

EXPERT JUDGMENT;

RISK ASSESSMENT;

ARTICLE; BEHAVIOR; DECISION MAKING; HUMAN; MATHEMATICAL ANALYSIS; PROBABILITY; RISK ASSESSMENT; RISK MANAGEMENT; SOCIAL PSYCHOLOGY;

EID: 0032839653     PISSN: 02724332     EISSN: None     Source Type: Journal    
DOI: 10.1023/A:1006917509560     Document Type: Article

References (117)

1. Argote, L., Seabright, M. A., & Dyer, L. (1986). Individual versus group use of base-rate and individuating information. Organizational Behavior and Human Decision Processes, 38, 65-75.
2. Armstrong, J. S., Denniston, W. B., & Gordon, M. M. (1975). The use of the decomposition principle in making judgments. Organizational Behavior and Human Performance, 14, 257-263.
3. Arrow, K. J. (1951). Social Choice and Individual Values. New York: Wiley.
4. Aumann, R. J. (1976). Agreeing to Disagree. Annals of Statistics, 4, 1236-1239.
5. Bacharach, M. (1979). Normal Bayesian dialogues. Journal of the American Statistical Association, 74, 837-846.
6. Bates, J. M., & Granger, C. W. J. (1969). The combination of forecasts. Operational Research Quarterly, 20, 451-468.
7. Bonano, E. J., Hora, S. C., Keeney, R. L., & von Winterfeldt, D. (1990). Elicitation and Use of Expert Judgment in Performance Assessment for High-Level Radioactive Waste Repositories. NUREG/CR-5411. Washington, DC: Nuclear Regulatory Commission.
8. Bonduelle, Y. (1987). Aggregating Expert Opinions by Resolving Sources of Disagreement. PhD Dissertation, Stanford University.
9. Brockhoff, K. (1975). The performance of forecasting groups in computer dialogue and face-to face discussion. In H. Linstone & M. Turoff (Eds.), The Delphi Method: Techniques and Applications. Reading, MA: Addison-Wesley.
10. Bunn, D. W. (1975). A Bayesian approach to the linear combination of forecasts. Operational Research Quarterly, 26, 325-329.
11. Bunn, D. W. (1988). Combining forecasts. European Journal of Operational Research, 33, 223-229.
12. Bunn, D., & Wright, G. (1991). Interaction of judgmental and statistical forecasting methods: Issues and Analysis. Management Science, 37, 501-518.
13. Chandrasekharan, R., Moriarty, M. M., & Wright, G. P. (1994). Testing for unreliable estimators and insignificant forecasts in combined forecasts. Journal of Forecasting, 13, 611-624.
14. Chhibber, S., & Apostolakis, G. (1993). Some approximations useful to the use of dependent information sources. Reliability Engineering and System Safety, 42, 67-86.
15. Chhibber, S., Apostolakis, G., & Okrent, D. (1992). A taxonomy of the use of expert judgments in safety studies. Reliability Engineering & System Safety, 38, 27-45.
16. Clemen, R. T. (1985). Extraneous expert information. Journal of Forecasting, 4, 329-348.
17. Clemen, R. T. (1986). Calibration and the aggregation of probabilities. Management Science, 32, 312-314.
18. Clemen, R. T. (1989). Combining forecasts: A review of annotated bibliography. International Journal of Forecasting, 5, 559-583.
19. Clemen, R. T., Jones, S. K., & Winkler, R. L. (1996). Aggregating forecasts: An empirical evaluation of some Bayesian methods. In D. Berry, K. Chaloner, & J. Geweke (Eds.), Bayesian Statistics and Econometrics: Essays in Honor of Arnold Zellner (pp 3-13). New York: Wiley.
20. Clemen, R. T., & Murphy, A. H. (1986). Objective and subjective precipitation probability forecasts: Statistical analysis of some interrelationships. Weather and Forecasting, 1, 56-65.
21. Clemen, R. T., & Reilly, T. (1999). Correlations and copulas for decision and risk analysis. Management Science, 45, 208-224.
22. Clemen, R. T., & Winkler, R. L. (1985). Limits for the precision and value of information from dependent sources. Operations Research, 33, 427-442.
23. Clemen, R. T., & Winkler, R. L. (1987). Calibrating and combining precipitation probability forecasts. In R. Viertl (Ed.), Probability and Bayesian Statistics (pp. 97-110). New York: Plenum.
24. Clemen, R. T., & Winkler, R. L. (1990). Unanimity and compromise among probability forecasters. Management Science, 36, 767-779.
25. Clemen, R. T., & Winkler, R. L. (1993). Aggregating point estimates: A flexible modeling approach. Management Science, 39, 501-515.
26. Cooke, R. M. (1991). Experts in Uncertainty: Opinion and Subjective Probability in Science. New York: Oxford University Press.
27. Dalkey, N. C. (1969). The Delphi method: An experimental study of group opinions. Report No. RM-5888-PR. The Rand Corporation.
28. Dalkey, N. C., & Brown, B. (1971). Comparison of group judgment techniques with short-range predictions and almanac questions. Report No. R-678-ARPA. The RAND Corporation.
29. Dall'Aglio, G., Kotz, S., & Salinetti, G. (1991). Advances in Probability Distributions with Given Marginals: Beyond the Copulas. Dordrecht, Netherlands: Kluwer.
30. Davis, J. (1992). Some compelling intuitions about group consensus decisions, theoretical and empirical research, and interpersonal aggregation phenomena: Selected examples, 1950-1990. Organizational Behavior and Human Decision Processes, 52, 3-38.
31. Dawes, R. M., Faust, D., & Meehl, P. A. (1989). Clinical versus actuarial judgment. Science, 243, 1668-1673.
32. de Finetti, B. (1937). La Prévision: Ses Lois Logiques, Ses Sources Subjectives. Annales De L'Institut Henri Poincaré, 7.
33. Delbecq, A. L., Van de Ven, A. H., & Gustafson, D. H. (1975). Group Techniques for Program Planning. Glenview, IL: Scott Foresman.
34. Einhorn, H. J., Hogarth, R. M., & Klempner, E. (1977). Quality of group judgment. Psychological Bulletin, 84, 158-172.
35. EPRI (1986). Seismic Hazard Methodology for the Central and Eastern United States. Vol. 1: Methodology. NP-4/26. Palo Alto, CA: Electric Power Research Institute.
36. Ferrell, W. R. (1985) Combining individual judgments. In G. Wright (Ed.), Behavioral Decision Making (pp. 111-145). New York: Plenum.
37. Fischer, G. (1975). An experimental study of four procedures for aggregating subjective probability assessments. Technical report 75-7. Decisions and Designs, Inc.
38. Flores, B. E., & White, E. M. (1989). Subjective vs. objective combining of forecasts: An experiment. Journal of Forecasting, 8, 331-341.
39. French, S. (1981). Consensus of opinion. European Journal of Operational Research, 7, 332-340.
40. French, S. (1985). Group consensus probability distributions: A critical survey. In J. M. Bernardo, M. H. DeGroot, D. V. Lindley, & A. F. M. Smith (Eds.), Bayesian Statistics 2 (pp. 183-197). Amsterdam: North-Holland.
41. Gelfand, A. E., Mallick, B. K., & Dey, D. K. (1995). Modeling expert opinion rising as a partial probabilistic specification. Journal of the Ameican Statistical Association, 90, 598-604.
42. Genest, C. (1984). Pooling operators with the marginalization property. Canadian Journal of Statistics, 12, 153-163.
43. Genest. C., & McConway, K. J. (1990). Allocating the weights in the linear opinion pool. Journal of Forecasting, 9, 53-73.
44. Genest, C., & Schervish, M. J. (1985). Modeling expert judgments for Bayesian updating. Annals of Statistics, 13, 1198-1212.
45. Genest, C., & Zidek, J. V. (1986). Combining probability distributions. A Critique and annotated bibliography. Statistical Science, 1, 114-148.
46. Gigone, D., & Hastie, R. (1997). Proper analysis of the accuracy of group judgments. Psychological Bulletin, 121, 149-167.
47. Gokhale, D. V., & Press, S. J. (1982). Assessment of a prior distribution for the correlation coefficient in a bivariate normal distribution. Journal of the Royal Statistical Society, Series A, 145, 237-249.
48. Goodman, B. (1972). Action selection and likelihood estimation by individuals and groups. Organizational Behavior and Human Performance, 7, 121-141.
49. Gough, R. (1975). The effects of group format on aggregate subjective probability distributions. In Utility, Probability, and Human Decision Making. Dordrecht, Netherlands: Reidel.
50. Gustafson, D. H., Shukla, R. U., Delbecq, A., & Walster, G. W. (1973). A comparative study of differences in subjective likelihood estimates made by individuals, interacting groups, Delphi groups, and nominal groups. Organizational Behavior and Human Performance, 9, 280-291.
51. Hastie, R. (1986). Review essay: Experimental evidence on group accuracy. In B. Grofman & G. Owen (Eds.), Information Pooling and Group Decision Making: Proceedings of the Second University of California, Irvine, Conference on Political Economy. Greenwich, CT: JAI Press.
52. Hammitt, J. K., & Shlyakhter, A. I. (1999). The expected value of information and the probability of surprise. Risk Analysis, 19, 135-152.
53. Hill, G. W. (1982). Group vs. individual performance: Are N + 1 heads better than one? Psychological Bulletin, 91, 517-539.
54. Hogarth, R. M. (1977). Methods for aggregating opinions. In H. Jungermann & G. DeZeeuw (Eds.), Decision Making and Change in Human Affairs (pp. 231-255). Dordrecht, Netherlands: Reidel.
55. Hogarth, R. M. (1987). Judgment and Choice: 2nd Ed. Chichester, England: Wiley.
56. Hora, S. C. (1992). Acquisition of expert judgment: Examples from risk assessment. Journal of Energy Engineering, 118, 136-148.
57. Hora, S. C., Dodd, N. G., & Hora, J. A. (1993). The use of decomposition in probability assessments on continuous variables. Journal of Behavioral Decision Making, 6, 133-147.
58. Hora, S. C., & Iman, R. L. (1989). Expert opinion in risk analysis: The NUREG-1150 methodology. Nuclear Science and Engineering, 102, 323-331.
59. Innami, I. (1994). The quality of group decisions, group verbal behavior, and intervention. Organizational Behavior and Human Decision Processes, 60, 409-430.
60. Janis, I. L. (1982). Groupthink: Psychological Studies of Policy Decisions and Fiascoes, 2nd Ed. Boston: Houghton Mifflin.
61. Janis, I. L., & Mann, L. (1977). Decision Making. New York: Free Press.
62. Jouini, M. N., & Clemen, R. T. (1996). Copula models for aggregating expert opinions. Operations Research, 44, 444-457.
63. Kahneman, D., & Tversky, A. (1972). Subjective probability: A judgment of representativeness. Cognitive Psychology, 3, 430-454.
64. Kaplan, S. (1990). 'Expert information' vs 'expert opinions': Another approach to the problem of eliciting/combining/using expert knowledge in PRA. Journal of Reliability Engineering and System Safety, 39.
65. Keeney, R. L., & von Winterfeldt, D. (1989). On the uses of expert judgment on complex technical problems. IEEE Transactions on Engineering Management, 36, 83-86.
66. Keeney, R. L., & von Winterfeldt, D. (1991). Eliciting probabilities from experts in complex technical problems. IEEE Transactions on Engineering Management, 38, 191-201.
67. Lawrence, M. J., Edmundson, R. H., & O'Connor, M. J. (1986). The accuracy of combining judgmental and statistical forecasts. Management Science, 32, 1521-1532.
68. Lindley, D. V. (1983). Reconciliation of probability distributions. Operations Research, 31, 866-880.
69. Lindley, D. V. (1985). Reconciliation of discrete probability distributions. In J. M. Bernardo, M. H. DeGroot, D. V. Lindley, & A. F. M. Smith (Eds.), Bayesian Statistics 2 (pp. 375-390). Amsterdam: North-Holland.
70. Linstone, H. A., & Turoff, M. (1975). The Delphi Method: Techniques and Applications. Reading, MA: Addison-Wesley.
71. Lipscomb, J., Parmigiani, G., & Hasselblad, V. (1998). Combining expert judgment by hierarchical modeling: An application to physician staffing. Management Science, 44, 149-161.
72. Lock, A. (1987). Integrating group judgments in subjective forecasts. In G. Wright & P. Ayton (Eds.), Judgmental Forecasting (pp. 109-127). Chichester, England: Wiley.
73. MacGregor, D., Lichtenstein, S., & Slovic, P. (1988). Structuring knowledge retrieval: An analysis of decomposing quantitative judgments. Organizational Behavior and Human Decision Processes, 42, 303-323.
74. Makridakis, S., and Winkler, R. L. (1983). Averages of forecasts: Some empirical results. Management Science, 29, 987-996.
75. Mendel, M. B., & Sheridan, T. B. (1989). Filtering information from human experts. IEEE Transactions on Systems, Man, and Cybernetics, 36, 6-16.
76. Merkhofer, M. W. (1987). Quantifying judgmental uncertainty: Methodology, experience, and insights. IEEE Transactions on Systems, Man, and Cybernetics, 17, 741-752.
77. Morgan, M. G., & Henrion, M. (1990). Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis. Cambridge, MA: Cambridge University Press.
78. Morgan, M. G., & Keith, D. W. (1995). Subjective judgments by climate experts. Environmental Science and Technology, 29, 468-476.
79. Morris, P. A. (1974). Decision analysis expert use. Management Science, 20, 1233-1241.
80. Morris, P. A. (1977). Combining expert judgments: A Bayesian approach. Management Science, 23, 679-693.
81. Morris, P. A. (1983). An axiomatic approach to expert resolution. Management Science, 29, 24-32.
82. Mosleh, A., Bier, V. M., & Apostolakis, G. (1987). A critique of current practice for the use of expert opinions in probabilistic risk assessment. Reliability Engineering and System Safety, 20, 63-85.
83. Myers, D. G., & Lamm, H. (1975). The polarizing effect of group discussion. American Scientist, 63, 297-303.
84. Newbold, P., & Granger, C. W. J. (1974). Experience with forecasting univariate time series and the combination of forecasts. Journal of the Royal Staistical Society, Series A, 137, 131-149.
85. Otway, H., & von Winterfeldt, D. (1992). Expert judgment in risk analysis and management: Process, context, and pitfalls. Risk Analysis, 12, 83-93.
86. Parenté, F. J., & Anderson-Parenté, J. K. (1987). Delphi inquiry systems. In G. Wright & P. Ayton (Eds.), Judgmental Forecasting (pp. 129-156). Chichester, England: Wiley.
87. Park, W. (1990). A review of research on groupthink. Journal of Behavioral Decision Making, 3, 229-245.
88. Phillips, L. D. (1984). A theory of requisite decision models. Acta Psychologica, 56, 29-48.
89. Phillips, L. D. (1987). On the adequacy of judgmental forecasts. In G. Wright & P. Ayton (Eds.), Judgmental Forecasting (pp. 11-30). Chichester, England: Wiley.
90. Phillips, L. D., & Phillips, M. C. (1990). Facilitated work groups: Theory and practice. Unpublished manuscript, London School of Economics and Political Science.
91. Pious, S. (1993). The Psychology of Judgment and Decision Making. New York: McGraw-Hill.
92. Ravinder, H. V., Kleinmuntz, D. N., & Dyer, J. S. (1988). The reliability of subjective probabilities obtained through decomposition. Management Science, 34, 186-199.
93. Reagan-Cirincione, P. (1994). Improving the accuracy of group judgment; A process intervention combining group facilitation, social judgment analysis, and information technology. Organizational Behavior and Human Decision Processes, 58, 246-270.
94. Rohrbaugh, J. (1979). Improving the quality of group judgment: Social judgment analysis and the Delphi technique. Organizational Behavior and Human Performance, 24, 73-92.
95. Savage, L. J. (1954). The Foundations of Statistics. New York: Wiley.
96. Schmittlein, D. C., Kim, J., & Morrison, D. G. (1990). Combining forecasts: Operational adjustments to theoretically optimal rules. Management Science, 36, 1044-1056.
97. Seaver, D. A. (1978). Assessing probability with multiple individuals: Group interaction versus mathematical aggregation (Report No. 78-3). Social Science Research Institute, University of Southern California.
98. Shlyakhter, A. I. (1994). Improved framework for uncertainty analysis: Accounting for unsuspected errors. Risk Analysis, 14, 441-447.
99. Shlyakhter, A. I., Kammen, D. M., Brodio, C. L., & Wilson, R. (1994). Quantifying the credibility of energy projections from trends in past data: The U.S. energy sector. Energy Policy, 22, 119-130.
100. Sniezek, J. A. (1989). An examination of group process in judgmental forecasting. International Journal of Forecasting, 5, 171-178.
101. Sniezek, J. (1992). Groups under uncertainty: An examination of confidence in group decision making. Organizational Behavior and Human Decision Processes, 52, 124-155.
102. Sniezek, J. A., & Henry, R. A. (1989). Accuracy and confidence in group judgment. Organizational Behavior and Human Decision Processes, 43, 1-28.
103. Sniezek, J. A., & Henry, R. A. (1990). Revision, weighting, and commitment in consensus group judgment. Organizational Behavior and Human Decision Processes, 45, 66-84.
104. Staël von Holstein, C.-A. S. (1972). Probabilistic forecasting: An experiment related to the stock market. Organizational Behavior and Human Performance, 8, 139-158.
105. Stone, M. (1961). The opinion pool. Annals of Mathematical Statistics, 32, 1339-1342.
106. Tindale, R. S., Sheffey, S., & Filkins, J. (1990). Conjunction errors by individuals and groups. Paper presented at the annual meeting of the Society for Judgment and Decision Making, New Orleans, LA.
107. Uecker, W. C. (1982). The quality of group performance in simplified information evaluation. Journal of Accounting Research, 20, 388-402.
108. West, M. (1992). Modelling agent forecast distributions. Journal of the Royal Statistical Society B, 54, 553-567.
109. West, M., & Crosse, J. (1992). Modelling probabilistic agent opinion. Journal of the Royal Statistical Society B, 54, 285-299.
110. Winkler, R. L. (1968). The consensus of subjective probability distributions. Management Science, 15, 361-375.
111. Winkler, R. L. (1981). Combining probability distributions from dependent information sources. Management Science, 27, 479-488.
112. Winkler, R. L., & Clemen, R. T. (1992). Sensitivity of weights in combining forecasts. Operations Research, 40, 609-614.
113. Winkler, R. L., & Makridakis, S. (1983). The combination of forecasts. Journal of the Royal Statistical Society, Series A, 146, 150-157.
114. Winkler, R. L., & Poses, R. M. (1993). Evaluating and combining physicians' probabilities of survival in an intensive care unit. Management Science, 39, 1526-1543.
115. Winkler, R. L., Wallsten, T. S., Whitfield, R. G., Richmond, H. M., Hayes, S. R., & Rosenbaum, A. S. (1995). An assessment of the risk of chronic lung injury attributable to longterm ozone exposure. Operations Research, 43, 19-28.
116. Wright, G., Saunders, C., & Ayton, P. (1988). The consistency, coherence and calibration of holistic, decomposed, and recomposed judgmental probability forecasts. Journal of Forecasting, 7, 185-199.
117. Wu, J. S., Apostolakis, G., & Okrent, D. (1990). Uncertainties in system analysis: Probabilistic versus nonprobabilistic theories. Reliability Engineering & System Safety, 30, 163-181.