A framework for validation of computer models

Technometrics

Volumn 49, Issue 2, 2007, Pages 138-154

  Bayarri, Maria J ^a   Paulo, Rui ^b   Berger, James O ^c   Sacks, Jerry ^d   Cafeo, John A ^e   Cavendish, James ^e   Lin, Chin Hsu ^e   Tu, Jian ^e  

^a UNIVERSITY OF VALENCIA   (Spain)

^b UNIVERSITY OF LISBON   (Portugal)

^c DUKE UNIVERSITY   (United States)

^d National Institute of Statistical Sciences   (United States)

^e GENERAL MOTORS CORPORATION   (United States)

Author keywords

Bayesian analysis; Identifiability; Model discrepancy; Prediction

Indexed keywords

BAYESIAN ANALYSIS; IDENTIFIABILITY; LIKELIHOOD METHODOLOGY; MODEL DISCREPANCY;

BAYESIAN NETWORKS; ERROR ANALYSIS; INFORMATION RETRIEVAL; MATHEMATICAL MODELS; MAXIMUM LIKELIHOOD; UNCERTAINTY ANALYSIS;

COMPUTER SYSTEMS;

EID: 34249069162     PISSN: 00401706     EISSN: None     Source Type: Journal    
DOI: 10.1198/004017007000000092     Document Type: Article

References (46)

1. Aslett, R., Buck, R. J., Duvall, S. G., Sacks, J., and Welch. W. J. (1998), "Circuit Optimization via Sequential Computer Experiments: Design of an Output Buffer," Applied Statistics, 47, 31-48.
2. Bates, R. A., Buck, R. J., Riccomagno, E., and Wynn, H. P. (1996), "Experimental Design and Observation for Large Systems," Journal of the Royal Statistical Society, Ser. B, 58, 77-94.
3. Bayarri, M., Berger, J., Cafeo, J., Garcia-Donato, G., Liu, F., Palomo, J., Parthasarathy, R., Paulo, R., Sacks, J., and Walsh, D. (2006a), "Computer Model Validation With Functional Output," Technical Report 165, National Institute of Statistical Sciences.
4. Bayarri, M. J., Berger, J. O., Higdon, D., Kennedy, M. C., Kottas, A., Paulo, R., Sacks, J., Cafeo, J. A., Cavendish, J., Lin, C. H., and Tu, J. (2002), "A Frame-work for Validation of Computer Models," Technical Report 128, National Institute of Statistical Sciences.
5. Bayarri, M. J., Berger, J. O., Kennedy, M. C., Kottas, A., Paulo, R., Sacks, J., Cafeo, J. A., Lin, C. H., and Tu, J. (2006b), "Bayesian Validation of a Computer Model for Vehicle Crashworthiness," Technical Report 163, National Institute of Statistical Sciences.
6. Berk, R., Bickel, P., Campbell, K., Fovell, R., Keller-McNulty, S., Kelly, E., Linn, R., Park, B., Perelson, A., Rouphail, N., Sacks, J., and Schoenberg, F. (2002), "Workshop on Statistical Approaches for the Evaluation of Complex Computer Models," Statistical Science, 17, 173-192.
7. Cafeo, J., and Cavendish, J. (2001), "A Framework for Verification and Validation of Computer Models and Simulations," internal document, General Motors, Research & Development Center.
8. Caswell, H. (1976), "The Validation Problem," in Systems Analysis and Simulation in Ecology, Vol. IV, ed. B. Patten, New York: Academic Press, pp. 313-325.
9. Chapman, W., Welch, W., Bowman, K., Sacks, J., and Walsh, J. (1994), "Arctic Sea Ice Variability: Model Sensitivities and a Multidecadal Simulation," Journal of Geophysical Research, 99, 919-935.
10. Conti, S., Anderson, C., O'Hagan, A., and Kennedy, M. (2005), "Bayesian Analysis of Complex Dynamic Computer Models," in Sensitivity Analysis of Model Output, eds. K. Hanson and F. Hemez, Los Alamos National Laboratory, pp. 147-156; available at http://library.lanl. gov/ccw/samo2004/.
11. Craig, P. S., Goldstein, M., Rougier, J. C., and Seheult, A. H. (2001), "Bayesian Forecasting for Complex Systems Using Computer Simulators," Journal of the American Statistical Association, 96, 717-729.
12. Craig, P. S., Goldstein, M., Seheult, A. H., and Smith, J. A. (1997), "Pressure Matching for Hydrocarbon Reservoirs: A Case Study in the Use of Bayes Linear Strategies for Large Computer Experiments," in Case Studies in Bayesian Statistics, Vol. III, eds. C. Gatsonis, J. S. Hodges, R. E. Kass, R. McCulloch, P. Rossi, and N. D. Singpurwalla, New York: Springer, pp. 36-93.
13. Currin, C., Mitchell, T., Morris, M., and Ylvisaker, D. (1991), "Bayesian Prediction of Deterministic Functions, With Applications to the Design and Analysis of Computer Experiments," Journal of the American Statistical Association, 86, 953-963.
14. Easterling, R. G. (2001), "Measuring the Predictive Capability of Computational Models: Principles and Methods, Issues and Illustrations," Technical Report SAND2001-0243, Sandia National Laboratories.
15. Goldstein, M., and Rougier, J. C. (2003), "Calibrated Bayesian Forecasting Using Large Computer Simulators," technical report, University of Durham, Statistics and Probability Group.
16. _ (2004), "Probabilistic Formulations for Transferring Inferences From Mathematical Models to Physical Systems," technical report, University of Durham, Statistics and Probability Group.
17. Gough, W., and Welch, W. (1994), "Parameter Space Exploration of an Ocean General Circulation Model Using an Isopycnal Mixing Parametrization," Journal of Marine Research, 52, 773-796.
18. Gramacy, R., Lee, H., and Macready, W. (2004), "Parameter Space Exploration With Gaussian Process Trees," in Proceedings of the International Conference on Machine Learning, pp. 353-360.
19. Gustafson, P. (2005), "On Model Expansion, Model Contraction, Identifiability and Prior Information: Two Illustrative Scenarios Involving Mismeasured Variables," Statistical Science, 20, 111-140.
20. _ (2006), "Sample Size Implications When Biases Are Modeled Rather Than Ignored," Journal of the Royal Statistical Society, Ser. A, 169, 1-17.
21. Higdon, D. (2002), "Space and Space-Time Modeling Using Process Convolutions," in Quantitative Methods for Current Environmental Issues, eds. C. Anderson, V. Barnett, P. Chatwin, and A. El-Shaarawi, London: Springer-Verlag, pp. 37-56.
22. Higdon, D., Gattiker, J., and Williams, B. (2005), "Computer Model Calibration Using High Dimensional Output," Technical Report LAUR-05-6410, Los Alamos National Laboratories.
23. Higdon, D., Kennedy, M. C., Cavendish, J., Cafeo, J., and Ryne, R. D. (2004), "Combining Field Data and Computer Simulations for Calibration and Prediction," SIAM Journal on Scientific Computing, 26, 448-466.
24. Kennedy, M. C., and O'Hagan, A. (2001), "Bayesian Calibration of Computer Models" (with discussion), Journal of the Royal Statistical Society, Ser. B., 63, 425-464.
25. Kennedy, M. C., O'Hagan, A., and Higgins, N. (2002), "Bayesian Analysis of Computer Code Outputs," in Quantitative Methods for Current Environmental Issues, eds. C. W. Anderson, V. Barnett, P. C. Chatwin, and A. H. El-Shaarawi, London: Springer-Verlag, pp. 227-243.
26. Lee, H., Higdon, D., Bi, Z., Ferreira, M., and West, M. (2002), "Markov Random Field Models for High-Dimensional Parameters in Simulations of Fluid Flow in Porous Media," Technometrics, 44, 230-241.
27. Lee, H., Higdon, D., Calder, C., and Holloman, C. (2005), "Efficient Models for Correlated Data via Convolutions of Intrinsic Processes," Statistical Modelling, 5, 53-74.
28. Morris, M. D., Mitchell, T. J., and Ylvisaker, D. (1993), "Bayesian Design and Analysis of Computer Experiments: Use of Derivatives in Surface Prediction," Technometrics, 35, 243-255.
29. Oakley, J. (2004), "Estimating Percentiles of Computer Code Outputs," Applied Statistics, 53, 83-93.
30. Oakley, J., and O'Hagan, A. (2002), "Bayesian Inference for the Uncertainty Distribution of Computer Model Outputs," Biometrika, 89, 769-784.
31. _ (2004), "Probabilistic Sensitivity Analysis of Complex Models: A Bayesian Approach," Journal of the Royal Statistical Society, Ser. B, 66, 751-769.
32. Oberkampf, W., and Trucano, T. (2000), "Validation Methodology in Computational Fluid Dynamics," Technical Report 2000-2549, American Institute of Aeronautics and Astronautics.
33. Oreskes, N., Shrader-Frechette, K., and Belitz, K. (1994), "Verification, Validation and Confirmation of Numerical Models in the Earth Sciences," Science, 263, 641-646.
34. Paulo, R. (2005), "Default Priors for Gaussian Processes," The Annals of Statistics, 33, 556-582.
35. Pilch, M., Trucano, T., Moya, J. L., Froehlich, G., Hodges, A., and Peercy, D. (2001), "Guidelines for Sandia ASCI Verification and Validation Plans: Content and Format, Version 2.0," Technical Report SAND 2001-3101, Sandia National Laboratories.
36. Roache, P. (1998), Verification and Validation in Computational Science and Engineering, Albuquerquenm; Hermosa Publishers.
37. Robert, C., and Casella, G. (1999), Monte Carlo Statistical Methods, New York: Springer-Verlag.
38. Sacks, J., Welch, W. J., Mitchell, T. J., and Wynn, H. P. (1989), "Design and Analysis of Computer Experiments," Statistical Science, 4, 409-423.
39. Saltelli, A., Chan, K., and Scott, M. (eds.) (2000), Sensitivity Analysis, Chichester, U.K.: Wiley.
40. Santner, T., Williams, B., and Notz, W. (2003), The Design and Analysis of Computer Experiments, New York: Springer-Verlag.
41. Schmidt, A., and O'Hagan, A. (2003), "Bayesian Inference for Nonstationary Spatial Covariance Structure via Spatial Deformatations," Journal of the Royal Statistical Society, Ser. B, 65, 745-758.
42. Sellke, T., Bayarri, M., and Berger, J. (2001), "Calibration of p Values for Testing Precise Null Hypotheses," The American Statistician, 55, 62-71.
43. Stainforth, D. A., Aina, T., Christensen, C., Collins, M., Faull, N., Frame, D. J., Kettleborough, J. A., Knight, S., Martin, A., Murphy, J. M., Piani, C., Sexton, D., Smith, L A., Spicer, R. A., Thorpe, A. J., and Allen, M. R. (2005), "Uncertainty in Predictions of the Climate Response to Rising Levels of Greenhouse Gases," Nature, 433, 403-406.
44. Trucano, T., Pilch, M., and Oberkampf, W. O. (2002), "General Concepts for Experimental Validation of ASCII Code Applications," Technical Report SAND 2002-0341, Sandia National Laboratories.
45. Wang, P., and Hayden, D. ( 1999), "Computational Modeling of Resistance Spot Welding of Aluminum," Research Report R&D-9152, GM Research & Development Center.
46. Welch, W. J., Buck, R. J., Sacks, J., Wynn, H. P., Mitchell, T. J., and Morris, M. D. (1992), "Screening, Predicting, and Computer Experiments," Technometrics, 34, 15-25.