Quantifying scientific uncertainty from expert judgement elicitation

Risk and Uncertainty Assessment for Natural Hazards

Volumn 9781107006195, Issue , 2011, Pages 64-99

  Aspinall, W P ^a   Cooke, R M ^b  

^a UNIVERSITY OF BRISTOL   (United Kingdom)

^b RESOURCES FOR THE FUTURE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DISASTERS; VOLCANOES;

EXPERT JUDGEMENT; EXPERT OPINION; GOVERNMENT DECISIONS; HAZARD AND RISK ASSESSMENTS; NATURAL DISASTERS; SCIENTIFIC ADVISORY COMMITTEES; SCIENTIFIC UNCERTAINTY; VOLCANIC ERUPTIONS;

RISK ASSESSMENT;

EID: 84924141507     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1017/CBO9781139047562.005     Document Type: Chapter

References (93)

1. Abrahamson, N. A., Coppersmith, K. J., Koller, M., et al. (2004) Probabilistic seismic hazard analysis for Swiss nuclear power plant sites, PEGASOS Project, Vols 1–6, NAGRA.
2. ACERA (2007) Eliciting expert judgments: literature review. http://www.acera.unimelb. edu.au/materials/endorsed/0611.pdf(accessed 28 December 2011).
3. ACERA (2010) Elicitation tool: process manual. http://www.acera.unimelb.edu.au/materials/software.html(accessed 28 December 2011)
4. Aspinall, W. P. (2006) Structured elicitation of expert judgment for probabilistic hazard and risk assessment in volcanic eruptions. In Statistics in Volcanology, ed. H. M. Mader, S. G. Coles, C. B. Connor and L. J. Connor, London: The Geological Society for IAVCEI, pp. 15–30
5. Aspinall, W. (2010) Opinion: a route to more tractable expert advice. Nature 463: 294–295
6. Aspinall, W. and Cooke, R. M. (1998) Expert judgment and the Montserrat Volcano eruption. In Proceedings, 4th International Conference on Probabilistic Safety Assessment and Management PSAM4, ed. Ali Mosleh and Robert A. Bari, vol. 3, New York, NY: Springer, pp. 2113–2118
7. Aspinall, W. P., Loughlin, S. C., Michael, A. D., et al. (2002) The Montserrat Volcano Observatory; its evolution, organization, role and activities. In The Eruption of Soufrière Hills Volcano, Montserrat from 1995 to 1999, ed. T. H. Druitt and B. P. Kokelaar, London: Geological Society of London, pp. 71–91
8. Aspinall, W. P., Woo, G., Voight, B., et al. (2003) Evidence-based volcanology; application to eruption crises. Journal of Volcanology and Geothermal Research: Putting Volcano Seismology in a Physical Context; In Memory of Bruno Martinelli 128: 273–285
9. Brockhoff, K. (1975) The performance of forecasting groups in computer dialogue and face to face discussions. In The Delphi Method, Techniques and Applications, ed. H. A. Linstone and M. Turoff, Reading, MA: Addison Wesley, pp. 291–321
10. Brown, A. J. and Aspinall, W. P. (2004) Use of expert elicitation to quantify the internal erosion processes in dams. In Proceedings of the British Dam Society Conference: Long-term Benefits and performance of Dams, Canterbury: Thomas Telford, pp. 282–297
11. Budnitz, R., Apostolakis, G., Boore, D. M., et al. (1998) Use of technical expert panels: applications to probabilistic seismic hazard analysis. Risk Analysis 18: 463–469
12. Budnitz, R. J., C. A. Cornell and Morris, P. A. (2005) Comment on J.U. Klugel’s ‘Problems in the application of the SSHAC probability method for assessing earthquake hazards at Swiss nuclear power plants,’ in Engineering Geology, vol. 78: 285–307 Engineering Geology 82: 76–78
13. Clemen, R. T. and Reilly, T. (1999) Correlations and copulas for decision and risk analysis. Management Science 45: 208–224
14. Clemen, R. T. and Winkler, R. L. (1986) Combining economic forecasts. Journal of Business & Economic Statistics 4 (1): 39–46
15. Clemen, R. T. and Winkler, R. L. (1987) Calibrating and combining precipitation probability forecasts. In Probability & Bayesian Statistics, ed. Viertl, R., New York, NY: Plenum Press, pp. 97–110
16. Clemen, R. T. and Winkler, R. L. (1999) Combining probability distributions from experts in risk analysis. Risk Analysis 19: 187–203
17. Cooke, R. (1987) A theory of weights for combining expert opinions. Delft University of Technology, Department of Mathematics and Informatics, Report 87–25
18. Cooke, R. M. (1991) Experts in Uncertainty:Opinion and Subjective Probability in Science. Oxford: Oxford University Press.
19. Cooke, R. M. (2008) Special issue on expert judgment. Reliability Engineering & System Safety 93: 655–656
20. Cooke, R. M. and Goossens, L. L. H. J. (2008) TU Delft expert judgment data base. Reliability Engineering & System Safety 93: 657–674
21. Cooke, R. M., Mendel, M. and Thijs, W. (1988) Calibration and information in expert resolution. Automatica 24: 87–94
22. Cooke, R. M., ElSaadany, S. and Huang, X. (2008) On the performance of social network and likelihood-based expert weighting schemes. Reliability Engineering & System Safety 93: 745–756
23. Coppersmith, K. J., Youngs, R. R. and Sprecher, C. (2009) Methodology and main results of seismic source characterization for the PEGASOS Project, Switzerland. Swiss Journal of Geosciences 102: 91–105
24. De Groot, M. (1974) Reaching a consensus. Journal of the American Statistical Association 69: 118–121
25. Delbecq, A. L., Van de Ven, A. and Gusstafson, D. (1975) Group Techniques for Program Planning. Glenview, IL: Scott, Foresman.
26. EPA (2009) Expert elicitation white paper: external review draft, January. http://www.epa. gov/spc/expertelicitation/index.htm (accessed 6 November 2009)
27. Fischer, G. (1975) An experimental study of four procedures for aggregating subjective probability assessments. Technical Report 75–7, Decisions and Designs Incl., McLean, Virginia.
28. Flandoli, F., Giorgi, E., Aspinall, W. P., et al. (2011) Comparison of a new expert elicitation model with the Classical Model, equal weights and single experts, using a crossvalidation technique. Reliability Engineering & System Safety 96: 1292–1310
29. French, S. (2008) Comments by Prof. French. Reliability Engineering & System Safety Expert Judgment 93: 766–768
30. French, S. (2011) Aggregating expert judgment. Revista de la Real Academia de Ciencias Exactas, Fisicas y Naturales. Serie A. Matematicas (RACSAM) 105: 181–206
31. Gigerenzer, G., Hertwig, R., van den Broek, E., et al. (2005) ‘A 30% chance of rain tomorrow’: how does the public understand probabilistic weather forecasts? Risk Analysis 25: 623–629
32. Goossens, L. H. J. and Harper, F. T. (1998) Joint EC/USNRC expert judgment driven radiological protection uncertainty analysis. Journal of Radiological Protection 18: 249– 264
33. Gough, R. (1975) The effect of group format on aggregate subjective probability distributions. In Utility, Probability and Human Decision Making, ed. D. Wendt and C. Vlek, Dordrecht, Reidel.
34. Hanks, T. C., Abrahamson, N. A., Boore, D. M., et al. (2009) Implementation of the SSHAC Guidelines for Level 3 and 4 PSHAs: experience gained from actual applications. US Geological Survey Open-File Report 2009–1093. http://pubs.er.usgs.gov/publication/ofr20091093 (accessed 27 July 2012).
35. Helmer, O. (1966) Social Technology, New York, NY: Basic Books.
36. Helmer, O. (1968) Analysis of the future: the Delphi method, and The Delphi Method–an illustration. In Technological Forecasting for Industry and Government, ed. J. Bright, Englewood Cliffs, NJ: Prentice Hall.
37. Hora, S. C. (2004) Probability judgments for continuous quantities: linear combinations and calibration. Management Science 50: 597–604
38. Hora, S. C. and Iman, R. L. (1989) Expert opinion in risk analysis: the NUREG-1150 experience. Nuclear Science and Engineering 102: 323–331
39. IAEA (2010) Seismic Hazards in Site Evaluation for Nuclear Installations. Vienna: International Atomic Energy Agency.
40. Kahneman, D. (2011) Thinking, Fast and Slow, London: Allen Lane.
41. Kahneman, D., Slovic, P. and Tversky, A. (eds) (1982) Judgment Under Uncertainty: Heuristics and Biases, Cambridge: Cambridge University Press.
42. Kallen, M. J. and Cooke, R. M. (2002) Expert aggregation with dependence. In Probabilistic Safety Assessment and Management, ed. E. J. Bonano, A. L. Camp, M. J. Majors, R. A. Thompson, London: Elsevier, pp. 1287–1294
43. Kelly G. B., Kenneally R. M. and Chokshi N. C. (1991) Consideration of seismic events in severe accidents. In Proceedings of Probabilistic Safety Assessment and Management 2, pp. 871–876
44. Klügel, J.-U. (2005a) Problems in the application of the SSHAC probability method for assessing earthquake hazards at Swiss nuclear power plants. Engineering Geology 78: 285–307
45. Klügel, J.-U. (2005b) Reply to the comment of Krinitzsky on J.U. Klugel’s ‘Problems in the application of the SSHAC probability method for assessing earthquake hazards at Swiss nuclear power plants’, in Engineering Geology, vol. 78: 285–307 Engineering Geology 82: 69–70
46. Klügel, J.-U. (2005c) Reply to the comment on J.U. Klugel’s ‘Problems in the application of the SSHAC probability method for assessing earthquake hazards at Swiss nuclear power plants’, in Engineering Geology, Vol. 78: 285–307, by Budnitz, by, J.U, Klugel. Engineering Geology 82: 79–85
47. Klügel, J.-U. (2005d) Reply to the comment on J.U. Klugel’s: ‘Problems in the application of the SSHAC probability method for assessing earthquake hazards at Swiss nuclear power plants,’ Eng. Geol. Vol. 78: 285–307, by Musson et al. Engineering Geology 82: 56–65
48. Klügel, J.-U. (2005e) Reply to the comment on J. U. Klugel’s: ‘Problems in the application of the SSHAC probability method for assessing earthquake hazards at Swiss nuclear power plants,’ in Engineering Geology, vol. 78: 285–307, by Lomnitz, by J.U. Klugel. Engineering Geology 82: 74–75
49. Klügel, J.-U. (2005f) Reply to the comment on J. U. Klugel’s: Problems in the application of the SSHAC probability method for assessing earthquake hazards at Swiss nuclear power plants, in Engineering Geology, Vol. 78: 285–307, by Wang, by J.U, Klugel. Engineering Geology 82: 89–90
50. Klügel, J.-U. (2007) Error inflation in probabilistic seismic hazard analysis. Engineering Geology 90: 186–192
51. Klügel, J.-U. (2009) Probabilistic seismic hazard analysis for nuclear power plants: current practice from a European perspective. Nuclear Engineering and Technology 41: 1–12
52. Klügel, J. U., Mualchin, L. and Panza, G. F. (2006) A scenario-based procedure for seismic risk analysis. Engineering Geology 88: 22
53. Kotra, J. P., Lee, M. P., Eisenberg, N. A., et al. (1996) Branch Technical Position on the Use of Expert Elicitation in the High-Level Radioactive Waste Program. NUREG-1563Washington, DC: US Nuclear Regulatory Commission.
54. Kraan, B. C. P. and Bedford, T. J. (2005) Probabilistic inversion of expert judgments in the quantification of model uncertainty. Management Science 51 (6): 995–1006
55. Krinitzsky, E. L. (2005) Comment on J.U. Klugel’s ‘Problems in the application of the SSHAC probability method for assessing earthquake hazards at Swiss nuclear power plants’, in Engineering Geology, vol. 78: 285–307 Engineering Geology 82: 66–68
56. Kurowicka, D. and Cooke, R. (2006) Uncertainty Analysis with High Dimensional Dependence Modelling. Chichester: John Wiley and Sons.
57. Kynn, M. (2008) The ‘heuristics and biases’ in expert elicitation. Journal of the Royal Statistical Society: Series A 171: 239–264
58. Lehrer, K. and Wagner, C. G. (1981) Rational Consensus in Science and Society. Dordrecht: Reidel.
59. Lin, S.-W. and Bier, V.M. (2008) A study of expert overconfidence. Reliability Engineering & System Safety 93: 775–777
60. Lin, S.-W. and Cheng, C.-H. (2008) Can Cooke’s Model sift out better experts and produce well-calibrated aggregated probabilities? 2008 IEEE International Conference on Industrial Engineering and Engineering Management, Singapore: IEEE Singapore Section, pp. 425–429
61. Lin, S.-W. and Cheng, C.-H. (2009) The reliability of aggregated probability judgments obtained through Cooke’s classical model. Journal of Modelling in Management 4: 149–161
62. Lindley, D.V. (1971) Making Decisions, London: John Wiley and Sons. Lindley, D. (1983) Reconciliation of probability distributions. Operations Research 31: 866–880
63. Lomnitz, C. (2005) Comment on J.U. Klugel’s ‘Problems in the application of the SSHAC probability method for assessing earthquake hazards at Swiss nuclear power plants’, in Engineering Geology, vol. 78: 285–307 Engineering Geology 82: 71–73
64. Mendel, M. and Sheridan, T. (1989) Filtering information from human experts. IEEE Transactions on Systems, Man and Cybernetics 19 (1): 6–16
65. Meyer, M. A. and Booker, J.M. (2001) Eliciting and Analyzing Expert Judgment: A Practical Guide. Philadelphia, PA: ASA-SIAM.
66. Morgan, M. G. and Henrion, M. (1990) Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis. New York, NY: Cambridge University Press.
67. Morris, P. (1974) Decision analysis expert use. Management Science 20: 1233–1241
68. Morris, P. (1977) Combining expert judgments: a Bayesian approach. Management Science 23: 679–693
69. Mosleh, A. and Apostolakis, G. (1982) Models for the use of expert opinions.Workshop on low probability high consequence risk analysis, Society for Risk Analysis, Arlington, VA, June.
70. Musson, R. M.W., Toro, G. R., Coppersmith, K. J., et al. (2005) Evaluating hazard results for Switzerland and how not to do it: A discussion of ‘Problems in the application of the SSHAC probability method for assessing earthquake hazards at Swiss nuclear power plants’ by J-U Klugel. Engineering Geology 82: 43–55
71. Neslo, R., Micheli, F., Kappel, C.V., et al. (2009) Modeling stakeholder preferences with probabilistic inversion: application to prioritizing marine ecosystem vulnerabilities. In Real Time and Deliberative Decision Making: Application to Risk Assessment for Nonchemical Stressors, ed. I. Linkov, E. A. Ferguson and V. S. Magar, Amsterdam: Springer: pp. 265–284
72. O’Hagan, A., Buck, C. E., Daneshkhah, A., et al. (2006) Uncertain Judgments: Eliciting Experts’ Probabilities. Chichester: John Wiley and Sons.
73. Parascandola, M. (2010) Epistemic risk: empirical science and the fear of being wrong. Law, Probability and Risk 9: 201–214
74. Rothlisberger, J. D., Lodge, D. M., Cooke, R. M., et al. (2009) Frontiers in ecology and the environment. The Ecological Society of America. www.frontiersinecology.org (accessed 27 July 2012).
75. Sackman, H. (1975) Delphi Critique, Expert Opinion, Forecasting and Group Processes, Lexington, MA: Lexington Books,.
76. Seaver, D. (1977) How groups can assess uncertainty. Proceedings of the International Conference on Cybernetics and Society, New York: Institute of Electrical and Electronics Engineers, pp. 185–190
77. Seaver, D. A. (1978) Assessing probabilities with multiple individuals: group interaction versus mathematical aggregation. Technical Report SSRI-78–3, Social Science Research Institute, University of Southern California, Los Angeles.
78. Skipp, B. O. and Woo, G. (1993). A question of judgement: expert or engineering? In Risk and Reliability in Ground Engineering, London: Thomas Telford, pp. 29–39
79. SSHAC (1995) Recommendations for Probabilistic Seismic Hazard Analysis: Guidance on Uncertainty and Use of Experts; prepared by Senior Seismic Hazard Analysis Committee (SSHAC), Livermore, CA: Lawrence Livermore National Laboratory.
80. SSHAC (1997) [Senior Seismic Hazard Analysis Committee, R. J. Budnitz, Chairman, G. Apostolakis, D. M. Boore, L. S. Cluff, K. J. Coppersmith, C. A. Cornell and P. A. Morris] Recommendations for Probabilistic Seismic Hazard Analysis: Guidance on Uncertainty and Use of Experts, NUREG/CR-6372, Washington, DC: US Nuclear Regulatory Commission.
81. Stone, J., Aspinall W. P. and Sparks, R. S. J. (2011) Poster presented at Soufriere Hills Volcano: 15 Years on Conference, Montserrat, West Indies, April.
82. Tuomisto, J. T., Wilson, A., Evans, J. S., et al. (2008) Uncertainty in mortality response to airborne fine particulate matter: combining European air pollution experts. Reliability Engineering & System Safety Expert Judgment 93: 732–744
83. Tversky, A. and Kahneman, D. (2005) Judgment under uncertainty: heuristics and biases. In Social Cognition Key Readings, ed. D. L. Hamilton, New York, NY: Psychology Press, chapter 10
84. Tyshenko, G. M., ElSaadany, S., Oraby, T., et al. (2011) Expert elicitation for the judgment of Prion disease risk uncertainties. Journal of Toxicology and Environmental Health, Part A: Current Issues 74: 261–285
85. Vick, S. G. (2002) Degrees of Belief: Subjective Probability and Engineering Judgment, Reston, VA: ASCE Press.
86. Wang, Z. (2005) Comment on J.U. Klugel’s: Problems in the application of the SSHAC probability method for assessing earthquake hazards at Swiss nuclear power plants, in Engineering Geology, vol. 78: 285–307 Engineering Geology 82: 86–88
87. Winkler, R. L. (1981) Combining probability distributions from dependent information sources. Management Science 27: 479–488
88. Wisse, B., Bedford, T. and Quigley, J. (2008) Expert judgement combination using moment methods. Reliability Engineering & System Safety Expert Judgement 93: 675–686
89. Woo, G. (1999) The Mathematics of Natural Catastrophes, River Edge, NJ: Imperial College Press.
90. Woo, G. (2011) Calculating Catastrophe, River Edge, NJ: Imperial College Press.
91. Woudenberg, F. (1991) An evaluation of Delphi. Technological Forecasting and Social Change 40: 131–150
92. Yunusov, A. R., Cooke, R. M. and Krymsky, V. G. (1999) Rexcalibr-integrated system for processing expert judgment. In Proceedings 9th Annual Conference: Risk Analysis: Facing the New Millenium. Rotterdam, 10–13 October, ed. L. H. J. Goossens, Netherlands: Delft University, pp. 587–589
93. Zuidema, P. (2006) PEGASOS: A PSHA for Swiss nuclear power plants–some comments from a managerial point of view. OECD-IAEA Specialist meeting on the Seismic Probabilistic Safety Assessment of Nuclear Facilities, Seogwipo, Jeju Island, Korea.