Validation of imprecise probability models

International Journal of Reliability and Safety

Volumn 3, Issue 1-3, 2009, Pages 3-22

  Ferson, Scott ^a   Oberkampf, William L ^b  

^a APPLIED BIOMATHEMATICS   (United States)

^b NONE   (United States)

Author keywords

Agreement between theory and data; Area metric; Distribution; Interval; L1 metric; Observation; P box; Prediction; Probability box; Validation

Indexed keywords

FORECASTING;

AREA METRIC; DISTRIBUTION; INTERVAL; L1 METRIC; OBSERVATION; PROBABILITY BOX; VALIDATION;

PROBABILITY DISTRIBUTIONS;

EID: 78651554322     PISSN: 1479389X     EISSN: 14793903     Source Type: Journal    
DOI: 10.1504/IJRS.2009.026832     Document Type: Article

References (69)

1. AIAA (1998) Guide for the Verification and Validation of Computational Fluid Dynamics Simulations, AIAA G-077-1998, American Institute of Aeronautics and Astronautics, Reston, Virginia.
2. Angus, J.E. (1994) 'The probability integral transform and related results', SIAM Review, Vol. 36, no. 4, pp.652-654.
3. ASME (2006) Guide for Verification and Validation in Computational Solid Mechanics, ASME V&V 10-2006, American Society of Mechanical Engineers. Available online at: http://catalog.asme.org/Codes/PrintBook/ VV_10_2006_Guide_Verification.cfm
4. Berger, J.O. (1985) Statistical Decision Theory and Bayesian Analysis Springer-Verlag, New York.
5. Brier, G.W. (1950) 'Verification of forecasts expressed in terms of probability', Monthly Weather Review, Vol. 78, pp.1-3.
6. Chen, W., Baghdasaryan, L., Buranathiti, T. and Cao, J. (2004) 'Model validation via uncertainty propagation', AIAA Journal, Vol. 42, pp.1406-1415.
7. Chen, W., Xiong, Y., Tsui, K-L. and Wang, S. (2006) 'Some metrics and a Bayesian procedure for validating predictive models in engineering design', Proceedings of ASME 2006 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Philadelphia, American Society of Mechanical Engineers. Available online at: http://ideal.mech.northwestern.edu/pdf/ DAC_validation06.pdf
8. Chen, W., Xiong, Y., Tsui, K-L. and Wang, S. (2008) 'A design-driven validation approach using Bayesian prediction models', Journal of Mechanical Design, Vol. 130, pp.021101 (12 pages).
9. Colyvan, M. (2004) 'The philosophical significance of Cox's theorem', International Journal of Approximate Reasoning, Vol. 37, No. 1, pp.71-85. Available online at: http://homepage.mac.com/mcolyvan/papers/ cox.pdf
10. Cox, R.T. (1946) 'Probability, frequency and reasonable expectation', American Journal of Physics, Vol. 14, pp.1-13.
11. Devore, J.L. (2000) Probability and Statistics for Engineers and Scientists, Duxbury, Pacific Grove, California.
12. Dobrushin, R.L. (1970) 'Prescribing a system of random variables by conditional distributions', Theory of Probability and its Applications, Vol. 15, pp.458-486.
13. Dowding, K.J., Pilch, M. and Hills, R.G. (2008) 'Formulation of the thermal problem', Computer Methods in Applied Mechanics and Engineering, Vol. 197, pp.2385-2389.
14. Dowding, K.J., Hills, R.G., Leslie, I., Pilch, M., Rutherford, B.M. and Hobbs, M.L. (2004) Case Study for Model Validation: Assessing a Model for Thermal Decomposition of Polyurethane Foam, SAND2004-3632, Sandia National Laboratories, Albuquerque, NM.
15. Draper, D. (1995) 'Assessment and propagation of model uncertainty', Journal of the Royal Statistical Society Series, Vol. B, No. 57, pp.45-97.
16. Feller, W. (1948) 'On the Kolmogorov-Smirnov limit theorems for empirical distributions', Annals of Mathematical Statistics, Vol. 19, pp.177-189.
17. Ferson, S. (2002) RAMAS Risk Calc 4.0 Software: Risk Assessment with Uncertain Numbers, Lewis Publishers, Boca Raton, Florida.
18. Ferson, S., Kreinovich, V., Ginzburg, L., Myers, D.S. and Sentz, K. (2003) Constructing Probability Boxes and Dempster-Shafer Structures SAND2002-4015, Sandia National Laboratories, Albuquerque, New Mexico. Available online at: http://www.ramas.com/unabridged.zip
19. Ferson, S., Joslyn, C.A., Helton, J.C., Oberkampf, W.L. and Sentz, K. (2004) 'Summary from the epistemic uncertainty workshop: Consensus amid diversity', Reliability Engineering and System Safety, Vol. 85, pp. 355-370.
20. Ferson, S., Kreinovich, V., Hajagos, J., Oberkampf, W.L. and Ginzburg, L. (2007) Experimental Uncertainty Estimation and Statistics for Data Having Interval Uncertainty, SAND2007-0939, Sandia National Laboratories, Albuquerque, NM. Available online at: http://www.ramas.com/ intstats.pdf
21. Ferson, S., Oberkampf, W.L. and Ginzburg, L. (2008) 'Model validation and predictive capability for the thermal challenge problem', Computer Methods in Applied Mechanics and Engineering, Vol. 197, pp.2408-2430.
22. de Finetti, B. (1962) 'Does it make sense to speak of "good probability appraisers"?', in Good, I.J. (Ed.): The Scientist Speculates: An Anthology of Partly-Baked Ideas, Wiley, New York, pp.357-363
23. Frank, M.J., Nelsen, R.B. and Schweizer, B. (1987) 'Best-possible bounds for the distribution of a sum - a problem of Kolmogorov', Probability Theory and Related Fields, Vol. 74, pp.199-211.
24. Fréchet, M. (1906) 'Sur quelques points du calcul fonctionnel (Thèse)', Rendiconti Circolo Mathematico di Palermo, Vol. 22, pp.1-74.
25. Gioia, F. and Lauro. C.N. (2005) 'Basic statistical methods for interval data', Statistica Applicata [Italian Journal of Applied Statistics, Vol. 17, No. 1, pp.75-104.
26. Hansen, K.M. (1999) A framework for assessing uncertainties in simulation predictions', Physica, Vol. D, No. 133, pp.179-188.
27. Hazelrigg, G. (2003) 'Thoughts on model validation for engineering design', Proceedings of ASME 2003 Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Chicago.
28. Helton, J.C. and Oberkampf, W.L. (Eds) (2004) Reliability and Engineering System Safety, Vol. 85, Nos. 1-3.
29. Hills, R.G. (2006) 'Model validation: Model parameter and measurement uncertainty', Journal of Heat Transfer, Vol. 128, pp.339-351.
30. Hills, R.G. and Leslie, I. (2003) Statistical Validation of Engineering and Scientific Models: Validation Experiments to Application SAND2003-0706, Sandia National Laboratories, Albuquerque, NM.
31. Hills, R.G. and Truncano, T.G. (2002) Statistical Validation of Engineering and Scientific Models: A Maximum Likelihood Based Metric, SAND2002-1783, Sandia National Laboratories, Albuquerque, NM.
32. Hills, R.G., Pilch, M., Dowding, K.J., Red-Horse, J., Paez, T.L., Babuska, I. and Tempone, R. (2008) 'Validation challenge workshop', Computer Methods in Applied Mechanics and Engineering, Vol. 197, pp.2375-2380.
33. ISO [International Organization for Standardization] (1993) Guide to the Expression of Uncertainty in Measurement, International Organization for Standardization, Geneva, Switzerland.
34. JCGM [Joint Committee for Guides in Metrology] (2006) 'Evaluation of measurement data - Supplement 1 to the "Guide to the expression of uncertainty in measurement" - Propagation of distributions using a Monte Carlo method', Available online at: http://www.intermet.jp/JCGM/ 0610news/Supplement_to_GUM.pdf
35. Kennedy, M. and O'Hagan, A. (2001) 'Bayesian calibration of computer models (with discussion)', Journal of the Royal Statistical Society, Series B, Vol. 63, pp.425-464.
36. Klir, G. and Wierman, M.J. (1999) Uncertainy-based Information: Elements of Generalized Information Theory, Physica-Verlag, Heidelberg.
37. Kolmogorov [Kolmogoroff], A. (1941) 'Confidence limits for an unknown distribution function', Annals of Mathematical Statistics, Vol. 12, pp.461-463.
38. Kullback, S. (1959) Information Theory and Statistics, Wiley, New York.
39. Kullback, S. and Leibler, R.A. (1951) 'On information and sufficiency', Annals of Mathematical Statistics, Vol. 22, pp.79-86.
40. Levi, I. (1980) The Enterprise of Knowledge, MIT Press, Cambridge, Massachusetts.
41. Lindley, D.V., Tversky, A. and Brown, R.V. (1979) 'On the reconciliation of probability assessments', Journal of the Royal Statistical Society Vol. A 142 (Part 2), pp.146-180.
42. Manski, C.F. (2003) Partial Identification of Probability Distributions, Springer Series in Statistics, Springer, New York.
43. Matheron, G. (1975) Random Sets and Integral Geometry, J. Wiley, New York.
44. Mathiassen, J.R., Skavhaug, A. and Bø, K. (2002) 'Texture similarity measure using Kullback-Leibler divergence between gamma distributions', Computer Vision - ECCV 2002: 7th European Conference on Computer Vision, Copenhagen, Denmark, May 28-31, 2002. Proceedings, Part III, Lecture Notes in Computer Science, Vol. 2352, Springer, Berlin, pp.19-49
45. Menger, K. (1942) 'Statistical metrics', Proceedings of the National Academy of Science U.S.A., Vol. 28, pp.535-537.
46. Molchanov, I. (2005) Theory of Random Sets, Springer, London.
47. Neapolitan, R.E. (1992) 'A survey of uncertain and approximate inference', in Zadeh, L. and Kacprzyk, J. (Eds): Fuzzy Logic for the Management of Uncertainty, John Wiley & Sons, New York, pp.55-82
48. Nikolaidis, E. and Haftka, R. (2001) 'Theories of uncertainty for risk assessment when data is scarce', International Journal of Advanced Manufacturing Systems, Vol. 4, No. 1, pp.49-56.
49. Oberkampf, W.L. and Barone, M.F. (2006) 'Measures of agreement between computation and experiment: Validation metrics', Journal of Computational Physics, Vol. 217, pp.5-36.
50. Oberkampf, W.L. and Helton, J.C. (2005) 'Evidence theory for engineering applications', in Nikolaidis, E., Ghiocel, D.M. and Singhal, S. (Eds): Engineering Design Reliability Handbook, CRC Press, Boca Raton, Florida, pp.10-1-10-30.
51. Oberkampf, W.L. and Trucano, T.G. (2007) Verification and Validation Benchmarks, SAND2007-0853, Sandia National Laboratories, Albuquerque, NM.
52. Oberkampf, W.L., Helton, J.C. and Sentz, K. (2001) 'Mathematical representation of uncertainty', American Institute of Aeronautics and Astronautics Non-Deterministic Approaches Forum, Seattle, WA, Paper No. 2001-1645, April, 2001.
53. Oberkampf, W.L., Trucano, T.G. and Hirsch, C. (2004) 'Verification, validation, and predictive capability in computational engineering and physics', Applied Mechanics Reviews, Vol. 57, No. 5, pp.345-384.
54. Pompeiu, D. (1905) Sur la continuité des fonctions de variables complexes (Thèse)', Gauthier-Villars, Paris, Ann. Fac. Sci. de Toulouse, Vol. 7, pp.264-315.
55. Rabinovich, S. (1993) Measurement Errors: Theory and Practice, American Institute of Physics, New York.
56. Romero, V.J. (2007) 'Validated model? Not so fast. The need for model "conditioning" as an essential addendum to model validation', AIAA-2007-1953 in Proceedings of the 2007 AIAA Non-Deterministic Approaches Conference, Honolulu, American Institute of Aeronautics and Astronautics.
57. Rutherford, B.M. and Dowding, K.J. (2003) An Approach to Model Validation and Model-based Prediction - Polyurethane Foam Case Study, SAND2003-2336, Sandia National Laboratories, Albuquerque, NM.
58. Shafer, G. (1976) A Mathematical Theory of Evidence, Princeton University Press, Princeton, New Jersey.
59. Song, K-S. (2002) 'Goodness-of-fit tests based on Kullback-Leibler discrimination information', IEEE Transactions on Information Theory Vol. 48, pp.1103-1117
60. Stephens, M.A. (1974) 'EDF statistics for goodness of fit and some comparisons', Journal of the American Statistical Association, Vol. 69, pp.730-737.
61. Smirnov [Smirnoff], N. (1939) 'On the estimation of the discrepancy between empirical curves of distribution for two independent samples', Bulletin de l'Université de Moscou, Série internationale (Mathématiques), Vol. 2: (fasc. 2).
62. Taylor, B.N. and Kuyatt, C.E. (1994) Guidelines for Evaluating and Expressing the Uncertainty of NIST Measurement Results, NIST Technical Note 1297, National Institute of Standards and Technology, Washington, DC. Available online at: http://physics.nist.gov/Pubs/ guidelines/contents.html. See also web guidance at http:// physics.nist.gov/cuu/Uncertainty/index.html.
63. Trucano, T.G., Swiler, L.P., Igusa, T., Oberkampf, W.L. and Pilch, M. (2006) 'Calibration, validation and sensitivity analysis: What's what', Reliability Engineering and System Safety, Vol. 91, pp.1331-1357.
64. Williamson, R.C. and Downs, T. (1990) 'Probabilistic arithmetic I: numerical methods for calculating convolutions and dependency bounds', International Journal of Approximate Reasoning, Vol. 4, pp.89-158.
65. Winkler, R.L. (1996) 'Scoring rules and the evaluation of probabilities', Test 5, pp.1-60.
66. 2 test', Journal of the Royal Statistical Society (Supplement), Vol. 1, pp.217-235.
67. Vallender, S.S. (1973) 'Calculation of the Wasserstein distance between probability distributions on the line', Theory of Probability and its Applications, Vol. 18, pp.784-786.
68. Walley, P. (1991) Statistical Reasoning with Imprecise Probabilities, Chapman and Hall, London.
69. Zhang, R. and Mahadevan, S. (2003) 'Bayesian methodology for reliability model acceptance', Reliability Engineering and System Safety, Vol. 80, pp.95-103.