Testing for ontological errors in probabilistic forecasting models of natural systems

Proceedings of the National Academy of Sciences of the United States of America

Volumn 111, Issue 33, 2014, Pages 11973-11978

  Marzocchi, Warner ^a   Jordan, Thomas H ^b  

^a ISTITUTO NAZIONALE DI GEOFISICA E VULCANOLOGIA   (Italy)

^b UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

Author keywords

Bayesian statistics; Expert opinion; Significance testing; Subjective probability; System science

Indexed keywords

AMBIGUITY; ARTICLE; BAYES THEOREM; BINOMIAL DISTRIBUTION; DECISION MAKING; EARTHQUAKE; EXPLANATORY VARIABLE; FORECASTING; HAZARD ASSESSMENT; LANDSLIDE; NATURAL SCIENCE; NULL HYPOTHESIS; ONTOLOGICAL ERROR; ONTOLOGY; PREDICTION; PRIORITY JOURNAL; PROBABILISTIC SEISMIC HAZARD ANALYSIS; PROBABILITY; STATISTICAL MODEL; STATISTICAL SIGNIFICANCE; SYSTEMATIC ERROR; TSUNAMI; UNCERTAINTY;

EID: 84906309154     PISSN: 00278424     EISSN: 10916490     Source Type: Journal    
DOI: 10.1073/pnas.1410183111     Document Type: Article

References (85)

1. American Association for the Advancement of Science (1989) Science for All Americans: A Project 2061 Report on Literacy Goals in Science, Mathematics and Technology (American Association for the Advancement of Science, Washington, DC).
2. Oreskes N, Shrader-Frechette K, Belitz K (1994) Verification, validation, and confirmation of numerical models in the Earth sciences. Science 263(5147):641-646.
3. Pilkey OH, Pilkey-Jarvis L (2007) Useless Arithmetic: Why Environmental Scientists Can't Predict the Future (Columbia Univ Press, New York).
4. Hacking I (1975) The Emergence of Probability (Cambridge Univ Press, Cambridge, UK).
5. Senior Seismic Hazard Analysis Committee (1997) Recommendations for Probabilistic Seismic Hazard Analysis: Guidance on Uncertainty and Use of Experts (U.S. Nuclear Regulatory Commission, U.S. Dept. of Energy, Electric Power Research Institute), NUREG/CR-6372, UCRL-ID-122160.
6. Goldstein M (2013) Observables and models: Exchangeability and the inductive argument. Bayesian Theory and Its Applications, eds Damien P, Dellaportas P, Olson NG, Stephens DA (Oxford Univ Press, Oxford, UK), pp 3-18.
7. Murphy AH, Winkler RL (1987) A general framework for forecast verification. Mon Weather Rev 115:1330-1338.
8. Schorlemmer D, Gerstenberger MC, Wiemer S, Jackson DD, Rhoades DA (2007) Earthquake likelihood model testing. Seismol Res Lett 78:17-29.
9. Mayo DG (1996) Error and the Growth of Experimental Knowledge (Univ of Chicago Press, Chicago).
10. Gelman A, Shalizi CR (2013) Philosophy and the practice of Bayesian statistics. Br J Math Stat Psychol 66(1):8-38.
11. Cox DR, Hinkley DV (1974) Theoretical Statistics (Chapman & Hall, London).
12. Musson R (2012) On the nature of logic trees in probabilistic seismic hazard assessment. Earthquake Spectra 28(3):1291-1296.
13. Dequech D (2004) Uncertainty: Individuals, institutions and technology. Camb J Econ 28(3):365-378.
14. Lindley DV (2000) The philosophy of statistics. Statistician 49(3):293-337.
15. Mayo DG, Spanos A (2006) Severe testing as a basic concept in a Neyman-Pearson philosophy of induction. Br J Philos Sci 57:323-357. (Pubitemid 43959230)
16. Bayarri MJ, Berger J (2000) P-values for composite null models. J Am Stat Assoc 95(452):1127-1142.
17. Box GEP (1980) Sampling and Bayes inference in scientific modelling and robustness. Roy Statist Soc Ser A 143(4):383-430.
18. Gelman A, Ming XL, Stern HS (1996) Posterior predictive assessment of model fitness via realized discrepancies (with discussion). Stat Sin 6:733-807. (Pubitemid 126706389)
19. Sellke T, Bayarri MJ, Berger JO (2001) Calibration of p values for testing precise null hypotheses. Am Stat 55(1):62-71.
20. Gelman A, Carlin JB, Stern HS, Rubin DB (2003) Bayesian Data Analysis (Chapman & Hall/CRC, London), 2nd Ed.
21. Mayo D (2011) Statistical science and philosophy of science: Where do/should they meet in 2011 (and beyond)? Rationality. Mark Morals 2:79-102.
22. Morey RD, Romeijn J-W, Rouder JN (2013) The humble Bayesian: Model checking from a fully Bayesian perspective. Br J Math Stat Psychol 66(1):68-75.
23. Jaffe AH (2003) A polemic on probability. Science 301(5638):1329-1330.
24. Jaynes ET (2003) Probability Theory: The Logic of Science (Cambridge Univ Press, New York).
25. Cornell CA (1968) Engineering seismic risk analysis. Bull Seismol Soc Am 58(5):1583-1606.
26. McGuire RK (2004) Seismic Hazard and Risk Analysis (Earthquake Engineering Research Institute, Oakland, CA).
27. Working Group on California Earthquake Probabilities (1995) Seismic hazards in southern California: Probable earthquakes, 1994-2024. Bull Seismol Soc Am 85(2):379-439.
28. Headquarters for Earthquake Research Promotion (2005) National Seismic Hazard Maps for Japan (Report issued on 23 March 2005, available in English translation at http://www.jishin.go.jp/main/index-e.html).
29. Frankel AD, et al. (1996) National Seismic Hazard Maps; Documentation June 1996. (US Geological Survey Open-file Report1996-532).
30. MPS Working Group (2004) Redazione della mappa di pericolosita sismica prevista dall'Ordinanza PCM del 20 marzo 2003, rapporto conclusivo per il Dipartimento della Protezione Civile (Istituto Nazionale di Geofisica e Vulcanologia, Milano-Roma, http://zonesismiche.mi.ingv.it).
31. Petersen MD, et al. (2008) Documentation for the 2008 Update of the United States National Seismic Hazard Maps. (US Geological Survey Open-file Report 2008-1128).
32. Bedford T, Cooke R (2001) Probabilistic Risk Analysis: Foundations and Methods (Cambridge Univ Press, Cambridge).
33. Baker JW, Cornell CA (2008) Uncertainty propagation in probabilistic seismic loss estimation. Struct Saf 30:236-252. (Pubitemid 351267472)
34. Stein S, Geller R, Liu M (2011) Bad assumptions or bad luck: Why earthquake hazard maps need objective testing. Seismol Res Lett 82(5):623-626.
35. Stein S, Geller R, Liu M (2012) Why earthquake hazard maps often fail and what to do about it. Tectonophysics 562-563:1-25.
36. Stirling M (2012) Earthquake hazard maps and objective testing: The hazard mapper's point of view. Seismol Res Lett 83(2):231-232.
37. Hanks TC, Beroza GC, Toda S (2012) Have recent earthquakes exposed flaws in or misunderstandings of probabilistic seismic hazard analysis? Seismol Res Lett 83(5):759-764.
38. Wyss MA, Nekrasova A, Kossobokov V (2012) Errors in expected human losses due to incorrect seismic hazard estimates. Nat Hazards 62(3):927-935.
39. Frankel A (2013) Comment on "Why earthquake hazard maps often fail and what to do about it" by S. Stein, R. Geller, and M. Liu. Tectonophys 592:200-206.
40. Savage JC (1991) Criticism of some forecasts of the National Earthquake Prediction Evaluation Council. Bull Seismol Soc Am 81(3):862-881.
41. Field EH, et al. (2009) Uniform California Earthquake Rupture Forecast, Version 2 (UCERF 2). Bull Seismol Soc Am 99(4):2053-2107.
42. Geller RJ (2011) Shake-up time for Japanese seismology. Nature 472(7344):407-409.
43. Krinitzsky EL (1995) Problems with logic trees in earthquake hazard evaluation. Eng Geol 39:1-3.
44. Castaños H, Lomnitz C (2002) PSHA: Is it science? Eng Geol 66(3-4):315-317.
45. Klügel J-U (2009) Probabilistic seismic hazard analysis for nuclear power plants - Current practice from a European perspective. Nucl Eng Technol 41(10):1243-1254.
46. Cox LA, Jr (2012) Confronting deep uncertainties in risk analysis. Risk Anal 32(10):1607-1629.
47. Marzocchi W, Zechar JD, Jordan TH (2012) Bayesian forecast evaluation and ensemble earthquake forecasting. Bull Seismol Soc Am 102(6):2574-2584.
48. Kulkarni RB, Youngs RR, Coppersmith KJ (1984) Assessment of confidence intervals for results of seismic hazard analysis. Proceedings of the Eighth World Conference on Earthquake Engineering, vol 1, 263-270, International Association for Earthquake Engineering, San Francisco.
49. Runge AK, Scherbaum F, Curtis A, Riggelsen C (2013) An interactive tool for the elicitation of subjective probabilities in probabilistic seismic-hazard analysis. Bull Seismol Soc Am 103(5):2862-2874.
50. Bommer JJ, Scherbaum F (2008) The use and misuse of logic trees in probabilistic seismic hazard analysis. Earthquake Spectra 24(4):997-1009.
51. Hoeting JA, Madigan D, Raftery AE, Volinsky CT (1999) Bayesian model averaging: A tutorial. Stat Sci 14(4):382-417.
52. Scherbaum F, Kuhen NM (2011) Logic tree branch weights and probabilities: Summing up to one is not enough. Earthquake Spectra 27(4):1237-1251.
53. Working Group on California Earthquake Probabilities (2003) Earthquake Probabilities in the San Francisco Bay Region: 2002-2031 (USGS Open-File Report 2003-214).
54. National Research Council Panel on Seismic Hazard Evaluation (1997) Review of Recommendations for Probabilistic Seismic Hazard Analysis: Guidance on Uncertainty and Use of Experts (National Academy of Sciences, Washington, DC).
55. Musson RMW (2005) Against fractiles. Earthquake Spectra 21(3):887-891.
56. Lindley DV, Novick MR (1981) The role of exchangeability in inference. Ann Stat 9(1):45-58.
57. Draper D, Hodges JS, Mallows CL, Pregibon D (1993) Exchangeability and data analysis. J R Stat Soc Ser A Stat Soc 156(1):9-37.
58. Bernardo JM (1996) The concept of exchangeability and its applications. Far East J Math Sci 4:111-122.
59. McGuire RK, Barnhard TP (1981) Effects of temporal variations in seismicity on seismic hazard. Bull Seismol Soc Am 71(1):321-334.
60. Stirling M, Petersen M (2006) Comparison of the historical record of earthquake hazard with seismic hazard models for New Zealand and the continental United States. Bull Seismol Soc Am 96(6):1978-1994. (Pubitemid 46070691)
61. Albarello D, D'Amico V (2008) Testing probabilistic seismic hazard estimates by comparison with observations: An example in Italy. Geophys J Int 175:1088-1094.
62. Stirling M, Gerstenberger M (2010) Ground motion-based testing of seismic hazard models in New Zealand. Bull Seismol Soc Am 100(4):1407-1414.
63. McGuire RK, Cornell CA, Toro GR (2005) The case for using mean seismic hazard. Earthquake Spectra 21(3):879-886. (Pubitemid 41226651)
64. Cox LA, Brown GC, Pollock SM (2008) When is uncertainty about uncertainty worth characterizing? Interfaces 38(6):465-468.
65. de Finetti B (1974) Theory of Probability: A Critical Introductory Treatment (John Wiley and Sons, London).
66. Feller W (1966) An Introduction to Probability Theory and its Applications (John Wiley and Sons, New York), 3rd Ed, Vol II.
67. Box GEP (1976) Science and statistics. J Am Stat Assoc 71(356):791-799.
68. Popper KR (1983) Realism and the Aim of Science (Hutchinson, London).
69. de Finetti B (1989) Probabilism. Erkenntnis 31:169-223.
70. Vick SG (2002) Degrees of Belief: Subjective Probability and Engineering Judgment (ASCE Press, Reston, VA).
71. Galton F (1907) Vox populi. Nature 75(1949):450-451.
72. Cooke RM (1991) Experts in Uncertainty: Opinion and Subjective Probability in Science (Oxford Univ Press, New York).
73. U.S. Environmental Protection Agency (2011) Expert Elicitation Task Force White Paper (Science and Technology Policy Council, U.S. Environmental Protection Agency, Washington, DC).
74. O'Neill B (2009) Exchangeability, correlation, and Bayes' effect. Int Stat Rev 77(2):241-250.
75. Gelman A (2011) Induction and deduction in Bayesian data analysis. Rationality. Mark Morals 2:67-78.
76. Rubin DB (1984) Bayesian justifiable and relevant frequency calculations for the applied statistician. Ann Stat 12(4):1151-1172.
77. Bayarri MJ, Castellanos ME (2007) Bayesian checking of the second levels of hierarchical models. Stat Sci 22(3):322-343.
78. Gelman A (2007) Comment: Bayesian checking of the second levels of hierarchical models. Stat Sci 22(3):349-352.
79. Jordan TH, et al. (2011) Operational earthquake forecasting: State of knowledge and guidelines for implementation. Ann Geophys 54(4):315-391.
80. Inness P, Dorling S (2013) Operational Weather Forecasting (John Wiley and Sons, New York).
81. Marzocchi W, Lombardi AM, Casarotti E (2014) The establishment of an operational earthquake forecasting system in Italy. Seismol Res Lett, in press.
82. Zechar JD, et al. (2010) The Collaboratory for the Study of Earthquake Predictability perspectives on computational earthquake science. Concurr Comput 22:1836-1847.
83. Davidson P (1996) Reality and economic theory. J Post Keynes Econ 18(4):479-508.
84. Silver N (2012) The Signal and the Noise: Why So Many Predictions Fail But Some Don't (Penguin Press, New York).
85. Bayarri MJ Berger JO (2013) Hypothesis testing and model uncertainty. Bayesian Theory and Its Applications, ed Damien P, Dellaportas P, Olson NG, and Stephens DA (Oxford Univ Press, Oxford, UK), pp. 361-194.