Some metrics and a Bayesian procedure for validating predictive models in engineering design

Proceedings of the ASME Design Engineering Technical Conference

Volumn 2006, Issue , 2006, Pages

  Chen, Wei ^a   Xiong, Ying ^a   Tsui, Kwok Leung ^b   Wang, Shuchun ^b  

^a Northwestern University   (United States)

^b Geogia Institute of Technology   (United States)

Author keywords

Bayesian approach; Design; Model validation; Predictive modeling; Uncertainty quantification; Validation metrics

Indexed keywords

COMPUTER SIMULATION; MATHEMATICAL MODELS; RISK ASSESSMENT;

BAYESIAN APPROACH; MODEL VALIDATION; UNCERTAINTY QUANTIFICATION; VALIDATION METRICS;

PRODUCT DESIGN;

EID: 33751320936     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (37)

1. Ang, J.A., Trucano, T.G., and Luginbuhl, D.R., "Confidence in ASCI Scientific Simulations", Ninth Nuclear Explosives Code Developers' Conference, San Diego, CA, Oct. 22-25, 1996.
2. Apley, D. W., Liu, J., Chen, W., "Understanding the Effects of Model Uncertainty in Robust Design with Computer Experiments," accepted by ASME Journal of Mechanical Design, 2005.
3. Apostolakis, G., "A Commentary on Model Uncertainty", Model Uncertainty: Its Characterization and Quantification, editors: Mosleh, A, Siu, N, Smidts, C. and Lui, C, NUREG/CP-0138, U.S. Nuclear Regulatory Commission, 1994.
4. Bayarri, M.J.., Berger, J.O., Higdon, D., Kennedy, M.C., Kottas, A., Paulo, R., Sacks, J., Cafeo, J.A., Cavendish, J., Lin, C.H., and Tu, J., "A Framework for Validation of Computer Models", Foundations for Verification and Validation in 21st Century Workshop, Johns Hopkins University, October 22-23, 2002.
5. Bayarri, M.J.., Berger, J.O., Higdon, D., Kennedy, M.C., Kottas, A., Paulo, R., Sacks, J., Cafeo, J.A., Cavendish, J., Lin, C.H., and Tu, J., "A Framework for Validation of Computer Models", Foundations for Verification and Validation in 21st Century Workshop, Johns Hopkins University, October 22-23, 2002.
6. Buranathiti, T., Cao, J., Chen, W., Baghdasaryan, L., and Xia, Z.C., "Approaches for Model Validation: Methodology and Illustration on a Sheet Metal Flanging Process", ASME Journal of Manufacturing Science and Engineering, in press, 126, November, 2004.
7. Buslik, A., "A Bayesian Approach to Model Uncertainty", Model Uncertainty: Its Characterization and Quantification, editors: Mosleh, A, Siu, N, Smidts, C. and Lui, C, U.S. Nuclear Regulatory Commission, NUREG/CP-0138, 1994.
8. Cafeo, J.A., and Thacker, B.H., "Concepts and Terminology of Validation for Computational Solid Mechanics Models", SAE 2004 World Congress & Exhibition, Detroit, MI, March, 2004.
9. Chen, W., Baghdasaryan, L., Buranathiti, T., and Cao, J., "Model Validation via Uncertainty Propagation and Data Transformations", AIAA Journal, 42(7), 1406-1415, 2004.
10. DoD, DoD Directive No. 5000.61, "Modeling and Simulation (M&S) Verification, Validation, and Accreditation (VV&A)", Defense Modeling and Simulation Office, www.dmso.mil/docslib.
11. Doebling, S.W., Hemex, F.M., Schultz, J.F., Girrens, S.P., "Overview of Structural Dynamics Model Validation Activities at Los Alamos National Laboratory", Proc AIAA/ASME/ASCE/AHS/ASC 43rd Structures, Structural Dynamics, and Materials Conf., AIAA 2002-1643, Denver, CO, April 22-25, 2002.
12. Easterling, R. G. & Berger, J. O., Statistical Foundations for The Validation of Computer Models, presented at Computer Model Verification and Validation in the 21st Century Workshop, Johns Hopkins University, 2002.
13. Freese, F., Testing Accuracy, Forest Science, 6(2), 139-145, 1960.
14. Geyer, C. J., "Practical Markov Chain Monte Carlo," Statistical Science, 7(4):473-511, 1992.
15. Gregoire, T., G., and Reynolds, M., R., Accuracy Testing and Estimation Alternatives, Forest Science, 34(2), 302-320, 1988.
16. Gu, L., Yang, R.J., "Recent Applications on Reliability-based Optimization of Automotive Structures," SAE Technical Paper Series: 2003-01-0152, 2003.
17. Hanson, K.M., "A Framework for Assessing Uncertainties in Simulation Predictions", Physica D. 133, 179-188, 1999.
18. Hastie, T., Tibshirani, R., Friedman, J., The Elements of Statistical Learning, Springer-Verleg, 2000.
19. Hazelrigg, G.A., "Thoughts on Model Validation for Engineering Design", ASME Design Technical Conference, DETC2003/DTM-48632, Chicago, IL, Sept. 2-6, 2003.
20. Hazelrigg, G.A., "On the Role and Use of Mathematical Models in Engineering Design", Transactions of ASME, Journal of Mechanical Design, 121(3), 336-341, 1999.
21. Hills, G. R., and Trucano, T. G., "Statistical Validation of Engineering and Scientific Models: Background", SAND99-1256, 1999.
22. Jin, R., Chen, W., Sudjianto, A., "Analytical metamodel-based global sensitivity analysis and uncertainty propagation for robust design," to be published in Journal of Quality Technology, 2005
23. Kennedy, M. C. and O'Hagan, A., "Bayesian Calibration of Computer Experiments", Journal of the Royal Statistical Society, B. 63, 425-464, 2001.
24. Mahadevan, S., and Rebba, R., "Validation of Reliability Computational Models using Bayes Networks," Journal of Reliability Engineering and System Safety, 87, pp. 223-232, 2005.
25. Malak, R.J. and Paredis, "On Characterizing and Assessing the Validity of Behavioral Models and Their Predictions", 2004 ASME Design Technical Conferences, DETC2004-57452, Salt Lake City, Utah, Sept. 28-Oct. 2, 2004.
26. Marczyk, J., Holzner, M., et.al., "Stochastic Automotive Crash Simulation; A new Frontier in Virtual prototyping," Proceedings of the Pam 97 user Conference, Pragu, Czech Republic, October 16-17, 1997.
27. McAdams, D.A. and Dym, C.L., "Modeling and Information in the Design Process", 2004 ASM Design Technical Conferences, DETC2004-57101, Salt Lake City, Utah, Sept. 28-Oct. 2, 2004.
28. Oberkampf, W. and Barone, M., "Measures of Agreement Between Computation and Experiment: Validation Metrics", 34th AIAA Fluid Dynamics Conference and Exhibit, AIAA-2004-2626, Portland, Oregon, June 28-1, 2004.
29. Oberkampf, W.L., Deland S.M., Rutherford, B.M, Diegert, K.V. and Alvin, K.F., "A New Methodology for the Estimation of Total Uncertainty in Computational Simulation", AIAA System Dynamics, Material, Conference, AIAA-99-1612, 3061-3083, 1999.
30. Oberkampf, W.L. and Trucano, T.G., "Validation Methodology in Computational Fluid Dynamics", AIAA 2000-2549, Fluids 2000, Denver, CO, 2000.
31. Oberkampf, W.L., Trucano, T.G., and Hirsch, C., "Verification, Validation, and Predictive Capability in Computational Engineering and Physics", SAND2003-3769, February, 2003.
32. Reynolds, M.R., Jr., "Estimating the Error in Model Predictions", Forest Science, 30(2), 454-469, 1984.
33. Sacks, J., Welch, W. J., Mitchell, T. J. and Wynn, H. P., "Design and analysis of computer experiments," Statistical Science, 4(4), pp. 409-435, 1989
34. Santner, T.J., B.J. Williams, and W.I. Notz (2003), "The Design and Analysis of Computer Experiments," New York: Springer-Verlag.
35. Trucano, T. G., "Prediction and Uncertainty in Computational Modeling of Complex Phenomena, A Whitepaper", Sandia report, SAND98-2776, 1998.
36. Wang, N., Ge, P., "Study of Metamodeling Techniques and Their Applications in Engineering Design," ASME-MED Manufacturing Science and Engineering, Vol. 10, pp. 89-95, 1999.
37. Wang, S., Chen, W., and Tsui, K, "Bayesian Validation of Computer Models," working paper, 2006.