Calibration, validation, and sensitivity analysis: What's what

Reliability Engineering and System Safety

Volumn 91, Issue 10-11, 2006, Pages 1331-1357

  Trucano, T G ^a   Swiler, L P ^a   Igusa, T ^b   Oberkampf, W L ^a   Pilch, M ^a  

^a SANDIA NATIONAL LABORATORIES   (United States)

^b JOHNS HOPKINS UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ADAPTIVE SYSTEMS; CALIBRATION; CODES (SYMBOLS); COMPUTER PROGRAMMING LANGUAGES; MATHEMATICAL TECHNIQUES; SENSITIVITY ANALYSIS; SIGNAL FILTERING AND PREDICTION;

COMPUTATIONAL SCIENCE; INTERPRETATION; TECHNICAL CHALLENGE; VALIDATION;

COMPUTER SCIENCE;

EID: 33745699543     PISSN: 09518320     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ress.2005.11.031     Document Type: Article

References (86)

1. United States Department of Energy. Advanced Simulation and Computing Program Plan. Sandia National Laboratories Fiscal Year 2005; Report SAND 2004-4607PP. Issued by Sandia National Laboratories for NNSA's Office of Advanced Simulation & Computing, NA-114, 2004.
2. United States Department of Defense. DoD Directive No. 5000.59: Modeling and Simulation (M&S) Management, Defense Modeling and Simulation Office, Office of the Director of Defense Research and Engineering, 1994.
3. United States Department of Defense. Verification, Validation, and Accreditation (VV&A) Recommended Practices Guide, Defense Modeling and Simulation Office, Office of the Director of Defense Research and Engineering, 1996.
4. AIAA. Guide for the verification and validation of computational fluid dynamics simulations. Reston, VA: American Institute of Aeronautics and Astronautics, AIAA-G-077-1998; 1998.
5. Trucano TG, Pilch M, Oberkampf WL. On the role of code comparisons in verification and validation. Report SAND2003-2752, Sandia National Laboratories 2003 (Available at http://www.sandia.gov).
6. Oberkampf W.L., and Trucano T.G. Verification and validation in computational fluid dynamics. Prog Aerospace Sci 38 (2002) 209-272
7. Roache P.J. Verification and validation in computational science and engineering (1998), Hermosa Publishers, Albuquerque, NM
8. Konikov L.F., and Bredehoeft J.D. Groundwater models cannot be validated. Adv Water Resour 15 (1992) 75-83
9. Oreskes N., Shrader-Frechette K., and Belitz K. Verification, validation, and confirmation of numerical models in the earth sciences. Science 263 (1994) 641-646
10. Oberkampf W.L., Trucano T.G., and Hirsch C. Verification, validation, and predictive capability in computational engineering and physics. Appl Mech Rev 57 5 (2004) 345-384
11. Beizer B. Software testing techniques (1990), Van Nostrand Reinhold, New York
12. Kaner C., Falk J., and Nguyen H.Q. Testing computer software. 2nd ed (1999), Wiley, New York
13. Trucano TG, Pilch M, Oberkampf WL. General concepts for experimental validation of ASCI code applications. Report SAND2002-0341, Sandia National Laboratories 2002 (Available at http://www.sandia.gov).
14. Chen MI, Trucano TG. ALEGRA validation studies for regular, Mach, and double Mach shock reflection in gas dynamics. Report SAND2002-2240, Sandia National Laboratories 2002 (Available at http://www.sandia.gov).
15. Sandoval DL. CAVEAT calculations of shock interactions. Report LA-11001-MS, Los Alamos National Laboratory 1987.
16. Trucano TG, et al. Description of the Sandia validation metrics project. Report SAND2001-1339, Sandia National Laboratories 2001 (Available at http://www.sandia.gov).
17. Stewart T.R., and Lusk C.M. Seven components of judgmental forecasting skill: implications for research and the improvement of forecasts. J Forecasting 13 (1994) 579-599
18. Knupp P., and Salari K. Verification of computer codes in computational science and engineering (2003), Chapman & Hall/CRC, Boca Raton, FL
19. Parry G.W., and Winter P.W. Characterization and evaluation of uncertainty in probabilistic risk analysis. Nucl Saf 22 1 (1981) 28-42
20. Paté-Cornell M.E. Probability and uncertainty in nuclear safety decisions. Nucl Eng Des 92 2-3 (1986) 319-327
21. Parry G.W. On the meaning of probability in probabilistic safety assessment. Reliab Eng Sys Saf 23 4 (1988) 309-314
22. Apostolakis G.E. Uncertainty in probabilistic risk assessment. Nucl Eng Des 115 (1989) 173-179
23. Apostolakis G.E. The concept of probability in safety assessments of technological systems. Science 250 4986 (1990) 1359-1364
24. Helton J.C. Treatment of uncertainty in performance assessments of complex systems. Risk Anal 14 4 (1994) 483-511
25. Hoffman F.O., and Hammonds J.S. Propagation of uncertainty in risk assessments: the need to distinguish between uncertainty due to lack of knowledge and uncertainty due to variability. Risk Anal 14 5 (1994) 707-712
26. Helton J.C., and Burmaster D.E. Guest editorial: treatment of aleatory and epistemic uncertainty in performance assessments for complex systems. Reliab Eng Syst Saf 54 2-3 (1996) 91-94
27. Paté-Cornell M.E. Uncertainties in risk analysis: six levels of treatment. Reliab Eng Syst Saf 54 2-3 (1996) 95-111
28. Helton J.C. Uncertainty and sensitivity analysis in the presence of stochastic and subjective uncertainty. J Stat Comput Simul 57 1-4 (1997) 3-76
29. Helton JC, Oberkampf WL editor. Special issue: alternative representations of epistemic uncertainty. Reliab Eng Syst Saf 2004;85(1-3):1-376.
30. Oberkampf W.L., DeLand S.M., Rutherford B.M., Diegert K.V., and Alvin K.F. Error and uncertainty in modeling and simulation. Reliab Eng Syst Saf 75 3 (2002) 333-357
31. Hills RG, Trucano TG. Statistical validation of engineering and scientific models: background. Report SAND99-1256, Sandia National Laboratories 1999 (Available at http://www.sandia.gov).
32. Zhang R., and Mahadevan S. Bayesian methodology for reliability model acceptance. Reliab Eng Syst Saf 80 1 (2003) 95-103
33. Mahadevan S., and Rebba R. Validation of reliability computational models using Bayesian networks. Reliab Eng Syst Saf 87 (2005) 223-232
34. Singpurwalla N.D., and Wilson S.P. Statistical methods in software engineering (1999), Springer, New York
35. Campbell K. A brief survey of statistical model calibration ideas. Los Alamos Technical Report LA-UR-02-3157, 2002.
36. Nichols A.L., and Zeckhauser R.J. The perils of prudence: how conservative risk assessments distort regulation. Regul Toxicol Pharmacol 8 (1988) 61-75
37. Sielken Jr. R.L., Bretzlaff R.S., and Stevenson D.E. Challenges to default assumptions stimulate comprehensive realism as a new tier in quantitative cancer risk assessment. Regul Toxicol Pharmacol 21 (1995) 270-280
38. Caruso M.A., et al. An approach for using risk assessment in risk-informed decisions on plant-specific changes to the licensing basis. Reliab Eng Syst Saf 63 (1999) 231-242
39. Paté-Cornell M.E. Risk and uncertainty analysis in government safety decisions. Risk Anal 22 (2002) 633-646
40. Oxford English Dictionary. Concise. 11th ed (2004), Oxford University Press, Oxford
41. Red-Horse J, et al. Nondeterministic analysis of mechanical systems. Report SAND2000-0890, Sandia National Laboratories 2000 (Available at http://www.sandia.gov).
42. Earman J. Bayes or bust? (1992), MIT Press, Cambridge, MA
43. Woodward P., and Colella P. The numerical simulation of two-dimensional fluid flow with strong shocks. J Comput Phys 54 1 (1984) 115-173
44. NIST website on statistical process models, regression, parameter estimation, etc.: http://www.itl.nist.gov/div898/handbook/pmd/section1/pmd141.htm
45. Neter J., Wasserman W., and Kuter M.H. Applied linear statistical models: regression, analysis of variance, and experimental designs (1985), Irwin, Homewood, IL
46. Bates D.M., and Watts D.G. Nonlinear regression analysis and its applications (1988), Wiley, New York
47. Kennedy M.C., and O'Hagan A. Bayesian calibration of computer models. J R Stat Soc 63 (2001) 425-464
48. Higdon D, Williams B, Moore L, McKay M, Keller-McNulty S. Uncertainty quantification for combining experimental data and computer simulations. Los Alamos Technical Report, LA-UR 04-6562, 2004.
49. Bayarri MJ, Berger JO, Higdon D, Kennedy MC, Kotas RP, Sacks J, et al. A framework for validation of computer models. V&V Foundations Conference, 2002.
50. Cressie N.A.C. Statistics for spatial data (1993), Wiley, New York
51. Williams C. Gaussian processes. In: Arbib M. (Ed). The handbook of brain theory and neural networks (2002), MIT Press, Cambridge, MA
52. Gelman A.B., Carlin J.S., Stern H.S., and Rubin D.B. Bayesian data analysis (1995), Chapman & Hall/CRC, Boca Raton, FL
53. Press S.J. Subjective and objective Bayesian statistics: principles, models, and applications. 2nd Ed (2003), Wiley, New York
54. Gilks W.R., Richardson S., and Spiegelhalter D.J. Markov Chain Monte Carlo in practice (1996), Chapman and Hall/CRC, Boca Raton, FL
55. Gamerman D. Markov Chain Monte Carlo: stochastic simulation for Bayesian inference (1997), Chapman & Hall/CRC, Boca Raton, FL
56. Swiler LP, Trucano TG. Treatment of model uncertainty under calibration. In: Proceedings of the nineth ASCE specialty conference on probabilistic mechanics and structural reliability, 2004.
57. Campbell K. Exploring Bayesian model calibration: a guide to intuition. Los Alamos Technical Report LA-UR-02-7175, 2002.
58. Cox D.D., Park J.-S., and Singer C.E. A statistical method for tuning a computer code to a data base. Comput Stat Data Anal 37 (2001) 77-92
59. Wolpert D.H. The status of supervised learning science circa 1994: the search for a consensus. In: Wolpert D.H. (Ed). The mathematics of generalization (1995), Addison-Wesley, Reading, MA
60. Cucker F., and Smale S. On the mathematical foundations of learning. Bull Am Math Soc 39 (2001) 1-49
61. Poggio T., and Smale S. The mathematics of learning: dealing with data. Notices Am Math Soc May (2003) 537-544
62. Igusa T, Trucano TG. Role of computational learning theory in calibration and prediction. In: Proceedings of the nineth ASCE specialty conference on probabilistic mechanics and structural reliability, 2004.
63. Hastie T., Tibshirani R., and Friedman J. The elements of statistical learning (2001), Springer, New York
64. Cooke R.M. Experts in uncertainty: opinion and subjective probability in science (1991), Oxford University Press, New York
65. Meyer M.A., and Booker J.M. Eliciting and analyzing expert judgment: a practical guide (1991), Academic Press, New York
66. Ayyub B.M. Elicitation of expert opinions for uncertainty and risks (2001), CRC Press, Boca Raton, FL
67. Johnson R.A., and Wichern D.W. Applied multivariate statistical analysis (1998), Prentice-Hall, Englewood Cliffs, NJ
68. Hardle W. Applied nonparametric regression (1990), Cambridge University Press, New York
69. Beck M.B. Water quality modeling: a review of the analysis of uncertainty. Water Resour Res 23 (1987) 1393-1442
70. Box G.E.P., and Draper N.R. Empirical model-building and response surfaces (1987), Wiley, New York
71. Iman RL. Uncertainty and sensitivity analysis for computer modeling applications. In: Cruse TA, editor. Reliability technology-1992, Winter annual meeting of the American Society of Mechanical Engineers, Anaheim, CA, November 8-13, 1992. New York: American Society of Mechanical Engineers, Aerospace Division, 1992; (28). p. 153-68.
72. Helton J.C. Uncertainty and sensitivity analysis techniques for use in performance safety assessment for radioactive waste disposal. Reliab Eng Syst Saf 42 2-3 (1993) 327-367
73. Blower S.M., and Dowlatabadi H. Sensitivity and uncertainty analysis of complex models of disease transmission: an HIV model, as an Example. Int Stat Rev 62 2 (1994) 229-243
74. Hamby D.M. A review of techniques for parameter sensitivity analysis of environmental models. Environ Monit Assess 32 2 (1994) 135-154
75. Saltelli A., and Scott M. Guest editorial: the role of sensitivity analysis in the corroboration of models and its link to model structural and parametric uncertainty. Reliab Eng Syst Saf 57 (1997) 1-4
76. Kleijnen J.P.C., and Helton J.C. Statistical analyses of scatterplots to identify important factors in large-scale simulations, 1: review and comparisons of techniques. Reliab Eng Syst Saf 65 2 (1999) 147-185
77. Saltelli A., Chan K., and Scott E.M. Sensitivity analysis (2000), Wiley, New York
78. Frey H.C., and Patil S.R. Identification and review of sensitivity analysis methods. Risk Anal 22 3 (2002) 553-578
79. Helton J.C., and Davis F.J. Illustration of sampling-based methods for uncertainty and uncertainty analysis. Risk Anal 22 3 (2002) 591-622
80. Ionescu-Bujor M., and Cacuci D.G. A comparative review of sensitivity and uncertainty analysis of large-scale systems-I: deterministic methods. Nucl Sci Eng 147 3 (2004) 189-203
81. Cacuci D.G., and Ionescu-Bujor M.A. Comparative review of sensitivity and uncertainty analysis of large-scale systems-II: statistical methods. Nucl Sci Eng 147 3 (2004) 204-217
82. Pilch M, et al. Guidelines for Sandia ASCI verification and validation plans-content and format: version 2.0. Report SAND2000-3101, Sandia National Laboratories 2001 (Available at http://www.sandia.gov).
83. Boyack B.E. Quantifying reactor safety margins. Part 1: an overview of the code scaling, applicability, and uncertainty evaluation methodology. Nucl Eng Des 119 (1990) 1-15
84. Breeding R.J., Helton J.C., Gorham E.D., and Harper F.T. Summary description of the methods used in the probabilistic risk assessments for NUREG-1150. Nucl Eng Des 135 1 (1992) 1-27
85. Helton J.C., Anderson D.R., Basabilvazo G., Jow H.-N., and Marietta M.G. Conceptual structure of the 1996 performance assessment for the waste isolation pilot plant. Reliab Eng Syst Saf 69 1-3 (2000) 151-165
86. Wilson G.E., and Boyack B.E. The role of the PIRT process in experiments, code development and code applications associated with reactor safety analysis. Nucl Eng Des 186 (1998) 23-37