Towards a better understanding of model validation metrics

13th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference 2010

Volumn , Issue , 2010, Pages

  Liu, Yu ^a   Chen, Wei ^b   Arendt, Paul ^b   Huang, Hong Zhong ^a  

^a UNIVERSITY OF ELECTRONIC SCIENCE AND TECHNOLOGY OF CHINA   (China)

^b Northwestern University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ACCURACY ASSESSMENT; EXPERIMENTAL DATUM; KEY CHARACTERISTICS; PHYSICAL EXPERIMENTS; QUANTITATIVE MEASURES; SOURCES OF UNCERTAINTY; STOCHASTIC CHARACTERISTIC; STOCHASTIC VALIDATION;

EXPERIMENTS; FORECASTING; OPTIMIZATION; STATISTICAL TESTS; STOCHASTIC MODELS; STOCHASTIC SYSTEMS;

COMPUTER SIMULATION;

EID: 84880827535     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.2514/6.2010-9240     Document Type: Conference Paper

References (53)

1. Oberkampf, W.L., Trucano, T.G., and Hirsch, C., "Verification, Validation, and Predictive Capability in Computational Engineering and Physics, " Applied Mechanics Reviews, Vol. 57, No. 5, 2004, pp. 345-384.
2. Sornette, D., Davis, A.B., Ide, K., Vixie, K.R., Pisarenko V., and Kamm, J.R., "Algorithm for Model Validation: Theory and Applications, " Proceedings of the National Academy of Sciences, Vol. 104, No. 16, 2007, pp. 6562-6567.
3. Ang, J.A., Trucano, T.G., and Luginbuhl, D.R., "Confidence in ASCI Scientific Simulations, " Sandia National Laboratories, SAND98-1525C, 1998.
4. Defense Modeling and Simulation Office., "Dod Verification, Validation, and Accreditation (VV&A) Recommended Practices Guide, " Office of the Director of Defense Research and Engineering, 1996.
5. AIAA., "Guide for The Verification and Validation of Computational Fluid Dynamics Simulations, " AIAA, G-077, 1998,.
6. ASME., "Guide for Verification and Validation in Computational Solid Mechanics, " ASME, PTC 60/V&V 10, 2006.
7. Chen, W., Xiong, Y., Tsui, K.L., and Wang, S., "A Design-Driven Validation Approach Using Bayesian Prediction Models, " ASME Journal of Mechanical Design, Vol. 130, No. 2, 2008, pp. 021101.
8. Xiong, Y., Chen, W., Tsui, K.L., and Apley, D.W., "A Better Understanding of Model Updating Strategies in Validating Engineering Models, " Computer Methods in Applied Mechanics and Engineering, Vol. 198, No. 15-16, 2009, pp. 1327-1337.
9. Trucano, T.G., Swiler, L.P., Igusa, T., Oberkampf, W.L., and Pilch, M., "Calibration, Validation, and Sensitivity Analysis: What's What, " Reliability Engineering & System Safety, Vol. 91, No. 10-11, 2006, pp. 1331-1357.
10. Mayer, D.G., and Butler, D.G., "Statistical Validation, " Ecological Modelling, Vol. 68, No. 1-2, 1993, pp. 21-32.
11. Hills, R.G., and Trucano T.G., "Statistical Validation of Engineering and Scientific Models: Background, " Sandia National Laboratories, SAND99-1256, 1999.
12. Sugawara, Y., Shinohara, K., and Kobayashi, N., "Quantitative Validation of Dynamic Stiffening Represented By Absolute Nodal Coordinate Formulation, " Proceedings of the ASME 2009 International Design Engineering Technical Conferences & Computers and Information in Engineering Conference, 2009, DETC2009-86955.
13. Oberkampf, W.L., and Trucano, T.G., "Verification and Validation Benchmarks, " Nuclear Engineering and Design, Vol. 238, No. 3, 2008, pp. 716-743.
14. Kennedy, M.C., and O'Hagan, A., "Bayesian Calibration of Computer Models, " Journal of the Royal Statistical Society: Series B, Vol. 63, No. 3, 2001, pp. 425-464.
15. Oberkampf, W.L., and Barone, M.F., "Measures of Agreement Between Computation and Experiment: Validation Metrics, " Journal of Computational Physics, Vol. 217, No. 1, 2006, pp. 5-36.
16. Romero, V.J., "Validated Model? Not So Fast. The Need for Model 'Conditioning' as An Essential Addendum to Model Validation, " Proceedings of the 48th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, 2007, AIAA2007-1953.
17. Sprague, M.A., and Geers, T.L., "Spectral Elements and Field Separation for An Acoustic Fluid Subject to Cavitation, " Journal of Computational Physics, Vol. 184, No. 1, 2003, pp. 149-162.
18. Russell, D.M., "Error Measures for Comparing Transient Data: Part I, Development of A Comprehensive Error Measure, " Proceedings of the 68th Shock and Vibration Symposium, 1997, pp. 175-184.
19. Russell, D.M., "Error Measures for Comparing Transient Data: Part II, Error Measures Case Study, " Proceedings of the 68th Shock and Vibration Symposium, 1997, pp. 185-198.
20. Schwer, L.E., "Validation Metrics for Response Histories: Perspectives and Case Studies, " Engineering with Computers, Vol. 23, No. 4, 2007, pp. 295-309.
21. Romero, V.J., Luketa, A., and Sherman, M., "Application of A Pragmatic Interval Based 'Real Space' Approach to Fire CFD Model Validation Involving Aleatory and Epistemic Uncertainty, " Proceedings of the 50th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, 2009, AIAA-2009-2279.
22. Ferson, S., and Oberkampf, W.L., "Validation of Imprecise Probability Models, " International Journal of Reliability and Safety, Vol. 3, No. 1-3, 2009, pp. 3-22.
23. Chen, W., Baghdasaryan, L., Buranathiti, T., and Cao, J., "Model Validation via Uncertainty Propagation and Data Transformations, " AIAA Journal, Vol. 42, No. 7, 2004, pp. 1406-1415.
24. Buranathiti, T., Cao, J., Chen, W., Baghdasaryan, L., and Xia, Z.C., "Approaches for Model Validation: Methodology and Illustration on A Sheet Metal Flanging Process, " ASME Journal of Manufacturing Science and Engineering, Vol. 128, No. 2, 2006, pp. 588-597.
25. Mahadevan, S., and Rebba, R., "Validation of Reliability Computational Models Using Bayes Networks, " Reliability Engineering & System Safety, Vol. 87, No. 1, 2005, pp. 223-232.
26. Lew, J.S., "Model Validation Using Coordinate Distance With Performance Sensitivity, " Mathematical Problems in Engineering, 2008, doi:10.1155/2008/298146.
27. Zhang, R., and Mahadevan, S., "Bayesian Methodology for Reliability Model Acceptance, " Reliability Engineering & System Safety, Vol. 80, No. 1, 2003, pp. 95-103.
28. Hills, R.G., "Model Validation: Model Parameter and Measurement Uncertainty, " ASME Journal of Heat Transfer, Vol. 128, No. 4, 2006, pp. 339-351.
29. Yang, R.J., Barbat, S., and Weerappuli, P., "Bayesian Probabilistic PCA Approach for Model Validation of Dynamic System, " Proceedings of the SAE World Congress, 2009, 2009-01-1404.
30. Dowding, K.J., Pilch, M., and Hills, R.G., "Formulation of The Thermal Problem, " Computer Methods in Applied Mechanics and Engineering, Vol. 197, No. 29-32, 2008, pp. 2385-2389.
31. Sarin, H., Kokkolaras, M., Hulbert, G., Papalambros, P., Barbat, S., and Yang, R.J., "A Comprehensive Metric for Comparing Time Histories in Validation of Simulation Models With Emphasis on Vehicle Safety Applications, " Proceedings of the ASME 2008 International Design Engineering Technical Conferences & Computers and Information in Engineering Conference, 2009, DETC2008-49669.
32. Red-Horse, J.R., and Paez, T.L., "Sandia National Laboratories Validation Workshop: Structural Dynamics Application, " Computer Methods in Applied Mechanics and Engineering, Vol. 197, No. 29-32, 2008, pp. 2578-2584.
33. Yang, R.J., Li, G., and Fu, Y., "Development of Validation Metrics for Vehicle Frontal Impact Simulation, " Proceedings of ASME 2007 International Design Engineering Technical Conferences & Computers and Information in Engineering Conference, 2007, DETC2007-34124.
34. Rebba, R., and Mahadevan, S., "Validation of Models With Multivariate Output, " Reliability Engineering & System Safety, Vol. 91, No. 8, 2006, pp. 861-871.
35. Jiang, X., and Mahadevan, S., "Bayesian Validation Assessment of Multivariate Computational Models, " Journal of Applied Statistics, Vol. 35, No. 1, 2008, pp. 49-65.
36. Ferson, S., Oberkampf, W.L., and Ginzburg, L., "Model Validation and Predictive Capability for The Thermal Challenge Problem, " Computer Methods in Applied Mechanics and Engineering, Vol. 197, No. 29-32, 2008, pp. 2408-2430.
37. Rebbaa, R., Mahadevan, S., and Huang, S., "Validation and Error Estimation of Computational Models, " Reliability Engineering & System Safety, Vol. 91, No. 10-11, 2006, pp. 1390-1397.
38. Rebba, R., Huang, S., Liu, Y., Mahadevan, S., "Statistical Validation of Simulation Models, " International Journal of Materials and Product Technology, Vol. 25, No. 1-3, 2006, pp. 164-181.
39. Hills, R.G., and Trucano, T.G., "Statistical Validation of Engineering and Scientific Models: A Maximum Likelihood Based Metric, " Sandia National Laboratories, 2002, SAND2001-1783.
40. Rosenblatt, M., "Remarks on Multivariate Transformation, " The Annals of Mathematical Statistics, Vol. 23, No. 3, 1952, pp. 470-472.
41. Nataf, A., "Determination Des Distributions De Probabilités Dont Des Marges Sont Données, " C.R. Academy of Sciences, Vol. 255, 1962, pp. 42-43.
42. Box, G.E.P., and Cox, D.R., "An Analysis of Transformations, " Journal of the Royal Statistical Society Series B, Vol. 26, No. 2, 1964, pp. 211-252.
43. Pericchi, L.R., "A Bayesian Approach to Transformations to Normality, " Biometrika, Vol. 68, No. 1, 1981, pp. 35-43.
44. Rebba, R., and Mahadevan, S., "Computational Methods for Model Reliability Assessment, " Reliability Engineering & System Safety, Vol. 93, No. 8, 2008, pp. 1197-1207.
45. Haldar, A., and Mahadevan, S., Probability, Reliability, and Statistical Methods in Engineering Design, Wiley, New York, 2000.
46. Ghanem, R.G., Doostan, A., and Red-Horse, J., "A Probabilistic Construction of Model Validation, " Computer Methods in Applied Mechanics and Engineering, Vol. 197, No. 29-32, 2008, pp. 2585-2595.
47. Berger, J.O., and Pericchi, L.R., "The Intrinsic Bayes Factor for Model Selection and Prediction, " Journal of the American Statistical Association, Vol. 91, No. 433, 1996, pp. 109-122.
48. Kass, R.E., and Rafter, A., "Bayesian Factors, " Journal of the American Statistical Association, Vol. 90, No. 430, 1995, pp. 773-795.
49. Oberkampf, W.L., and Trucano, T.G., "Verification and Validation in Computational Fluid Dynamics, " Progress in Aerospace Sciences, Vol. 38, No. 2, 2002, pp. 209-272.
50. Loehle, C., "A Hypothesis Testing Framework for Evaluating Ecosystem Model Performance, " Ecological Modelling, Vol. 97, No. 3, 1997, pp. 153-165.
51. Robinson, A.P., and Froese, R.E., "Model Validation Using Equivalence Tests, " Ecological Modelling, Vol. 176, No. 3-4, 2004, pp. 349-358.
52. Wellek, S. Testing Statistical Hypotheses of Equivalence, Chapman & Hall, London, 2003.
53. Blaci, O., and Sargent, R.G., "A Methodology for Cost-Risk Analysis in The Statistical Validation of Simulation Models, " Communications of the ACM, Vol. 24, No. 4, 1981, pp. 190-197.