Toward a better understanding of model validation metrics

Journal of Mechanical Design

Volumn 133, Issue 7, 2011, Pages

  Liu, Yu ^a   Chen, Wei ^b   Arendt, Paul ^b   Huang, Hong Zhong ^a  

^a NORTHWEST UNIVERSITY   (China)

^b Northwestern University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

STOCHASTIC MODELS; UNCERTAINTY ANALYSIS;

COMPUTATIONAL MODELLING; ENGINEERING ANALYSIS; ENGINEERING DESIGN; MODEL SELECTION; MODEL VALIDATION; PHYSICAL EXPERIMENTS; PREDICTIVE CAPABILITIES; QUANTITATIVE MEASURES; UNCERTAINTY; VALIDATION METRIC;

STOCHASTIC SYSTEMS;

EID: 79960691853     PISSN: 10500472     EISSN: None     Source Type: Journal    
DOI: 10.1115/1.4004223     Document Type: Article

References (56)

1. Oberkampf, W. L., Trucano, T. G., and Hirsch, C., 2004, "Verification, Validation, and Predictive Capability in Computational Engineering and Physics," Appl. Mech. Rev., 57(3), pp. 345-384. (Pubitemid 40748670)
2. Sornette, D., Davis, A. B., Ide, K., Vixie, K. R., Pisarenko, V., and Kamm, J. R., 2007, "Algorithm for Model Validation: Theory and Applications," Proc. Natl. Acad. Sci. U.S.A., 104(16), pp. 6562-6567. (Pubitemid 47175540)
3. Ang, J. A., Trucano, T. G., and Luginbuhl, D. R., 1998, "Confidence in ASCI Scientific Simulations," Sandia National Laboratories, Report No. SAND98-1525C.
4. Defense Modeling and Simulation Office, 1996, DoD Verification, Validation, and Accreditation (VV&A) Recommended Practices Guide, Office of the Director of Defense Research and Engineering.
5. AIAA, 1998, "Guide for the Verification and Validation of Computational Fluid Dynamics Simulations," AIAA Paper No. G-077-1998.
6. ASME, 2006, "Guide for Verification and Validation in Computational Solid Mechanics" ASME, PTC 60/V&V 10.
7. Chen, W., Xiong, Y., Tsui, K. L., and Wang, S. A., 2008, "Design-Driven Validation Approach Using Bayesian Prediction Models," ASME J. Mech. Des., 130(2), p. 021101.
8. Xiong, Y., Chen, W., Tsui, K. L., and Apley, D. W., 2009, "A Better Understanding of Model Updating Strategies in Validating Engineering Models," Comput. Methods Appl. Mech. Eng., 198(15-16), pp. 1327-1337.
9. Messer, M., Panchal, J. H., Krishnamurthy, V., Klein, B., Yoder, P. D., Allen, J. K., and Mistree, F., 2010, "Model Selection Under Limited Information Using a Value-of-Information-Based Indicator," ASME J. Mech. Des., 132(12), p. 121008.
10. Malak, R. J., and Paredis, C. J. J., 2010, "Using Support Vector Machines to Formalize the Valid Input Domain of Predictive Models in Systems Design Problems," ASME J. Mech. Des., 132(10), p. 101001.
11. Shan, S., and Wang, G. G., 2010, "Metamodeling for High Dimensional Simulation-Based Design Problem," ASME J. Mech. Des., 132(5), p. 051009.
12. Apley, D. W., Liu, J., and Chen, W., 2006, "Understanding the Effect of Model Uncertainty in Robust Design with Computer Experiments," ASME J. Mech. Des., 128(4), pp. 945-958. (Pubitemid 44123184)
13. Shao, T., and Krishnamurty, S., 2008, "A Clustering-Based Surrogate Model Updating Approach to Simulation-Based Engineering Design," ASME J. Mech. Des., 130(4), p. 041101.
14. Mayer, D. G., and Butler, D. G., 1993 "Statistical Validation," Ecol. Modell., 68(1-2), pp. 21-32.
15. Hills, R. G., and Trucano, T. G., 1999, "Statistical Validation of Engineering and Scientific Models: Background," Sandia National Laboratories, Report No. SAND99-1256.
16. Sugawara, Y., Shinohara, K., and Kobayashi, N., 2009, "Quantitative Validation of Dynamic Stiffening Represented by Absolute Nodal Coordinate Formulation," Proceedings of the ASME 2009 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, DETC2009-86955.
17. Oberkampf, W. L., and Trucano, T. G., 2008, "Verification and Validation Benchmarks," Nucl. Eng. Des., 238(3), pp. 716-743. (Pubitemid 351253682)
18. Kennedy, M. C., and O'Hagan, A., 2001, "Bayesian Calibration of Computer Models," J. R. Stat. Soc. Ser.B, 63(3), pp. 425-464.
19. Helton, J. C., 1994, "Treatment of Uncertainty in Performance Assessments for Complex Systems," Risk Anal., 14(4), pp. 483-511. (Pubitemid 24300014)
20. Helton, J. C., Johnson, J. D., and Oberkampf, W. L., 2004, "An Exploration of Alternative Approaches to the Representation of Uncertainty in Model Predictions," Reliab. Eng. Syst. Saf., 85(1-3), pp. 39-71.
21. Rebba, R., and Mahadevan, S., 2006, "Model Predictive Capability Assessment Under Uncertainty," AIAA J., 44(10), pp. 2376-2384. (Pubitemid 44672253)
22. Oberkampf,W. L., and Barone,M. F., 2006, "Measures of Agreement Between Computation and Experiment: Validation Metrics," J. Comput. Phys., 217(1), pp. 5-36. (Pubitemid 44111681)
23. Romero, V. J., Luketa, A., and Sherman, M., 2010, "Application of a Versatile 'Real Space' Validation Methodology to a Fire Model," AIAA J. Thermophys. Heat Transfer, 24(4), pp. 730-744.
24. Sprague, M. A., and Geers, T. L., 2003, "Spectral Elements and Field Separation for an Acoustic Fluid Subject to Cavitation," J. Comput. Phys., 184(1), pp. 149-162. (Pubitemid 36233163)
25. Russell, D. M., 1997, "Error Measures for Comparing Transient Data: Part I, Development of a Comprehensive Error Measure," Proceedings of the 68th Shock and Vibration Symposium, pp. 175-184.
26. Russell, D. M., 1997, "Error Measures for Comparing Transient Data: Part II, Error Measures Case Study," Proceedings of the 68th Shock and Vibration Symposium, pp. 185-198.
27. Schwer, L. E., 2007, "Validation Metrics for Response Histories: Perspectives and Case Studies," Eng. Comput., 23(4), pp. 295-309. (Pubitemid 47611104)
28. Romero, V. J., Luketa, A., and Sherman, M., 2009, "Application of a Pragmatic Interval Based 'Real Space' Approach to Fire CFD Model Validation Involving Aleatory and Epistemic Uncertainty," Proceedings of the 50th AIAA/ASME/ ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, AIAA Paper No. 2009-2279.
29. Ferson, S., and Oberkampf, W. L., 2009, "Validation of Imprecise Probability Models," Int. J. Reliab. Saf., 3(1-3), pp. 3-22.
30. Chen, W., Baghdasaryan, L., Buranathiti, T., and Cao, J., 2004, "Model Validation via Uncertainty Propagation and Data Transformations, " AIAA J., 42(7), pp. 1406-1415.
31. Buranathiti, T., Cao, J., Chen, W., Baghdasaryan, L., and Xia, Z. C., 2006, "Approaches for Model Validation: Methodology and Illustration on a Sheet Metal Flanging Process," ASME J. Manuf. Sci. Eng., 128(2), pp. 588-597. (Pubitemid 43952213)
32. Mahadevan, S., and Rebba, R., 2005, "Validation of Reliability Computational Models Using Bayes Networks," Reliab. Eng. Syst. Saf., 87(1), pp. 223-232.
33. Lew, J. S., 2008, "Model Validation Using Coordinate Distance with Performance Sensitivity," Mathematical Problems in Engineering, vol. 2008, 298146, p. 8
34. Zhang, R., and Mahadevan, S., 2003 "Bayesian Methodology for Reliability Model Acceptance," Reliab. Eng. Syst. Saf., 80(1), pp. 95-103.
35. Hills, R. G., 2006, "Model Validation: Model Parameter and Measurement Uncertainty," ASME J. Heat Transfer, 128(4), pp. 339-351.
36. Yang, R. J., Barbat, S., and Weerappuli, P., 2009, "Bayesian Probabilistic PCA Approach for Model Validation of Dynamic Systems," Proceedings of the SAE World Congress, 2009-01-1404.
37. Dowding, K. J., Pilch, M., and Hills, R. G., 2008, "Formulation of the Thermal Problem," Comput. Methods Appl. Mech. Eng., 197(29-32), pp. 2385-2389.
38. Sarin, H., Kokkolaras, M., Hulbert, G., Papalambros, P., Barbat, S., and Yang, R. J., 2009, "A Comprehensive Metric for Comparing Time Histories in Validation of Simulation Models with Emphasis on Vehicle Safety Applications," Proceedings of the ASME 2008 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, DETC2008-49669.
39. Red-Horse, J. R., and Paez, T. L., 2008, "Sandia National Laboratories Validation Workshop: Structural Dynamics Application," Comput. Methods Appl. Mech. Eng., 197(29-32), pp. 2578-2584.
40. Yang, R. J., Li, G., and Fu, Y., 2007, "Development of Validation Metrics for Vehicle Frontal Impact Simulation," Proceedings of ASME 2007 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, DETC2007-34124.
41. Rebba, R., and Mahadevan, S., 2006, "Validation of Models with Multivariate Output," Reliab. Eng. Syst. Saf., 91(8), pp. 861-871.
42. Jiang, X., and Mahadevan, S., 2008, "Bayesian Validation Assessment of Multivariate Computational Models," J. Appl. Stat., 35(1), pp. 49-65.
43. Ferson, S., Oberkampf, W. L., and Ginzburg, L., 2008, "Model Validation and Predictive Capability for the Thermal Challenge Problem," Comput. Methods Appl. Mech. Eng., 197(29-32), pp. 2408-2430.
44. Rebba, R., Huang, S., Liu, Y., and Mahadevan, S., 2006, "Statistical Validation of Simulation Models," Int. J. Mater. Prod. Technol., 25(1-3), pp. 164-181. (Pubitemid 43128358)
45. Hills, R. G., and Trucano, T. G., 2002, "Statistical Validation of Engineering and Scientific Models: A Maximum Likelihood Based Metric," Sandia National Laboratories, Report No. SAND2001-1783.
46. Rebba, R., Mahadevan, S., and Huang, S., 2006, "Validation and Error Estimation of Computational Models," Reliab. Eng. Syst. Saf., 91(10-11), pp. 1390-1397.
47. Rebba, R., and Mahadevan, S., 2008, "Computational Methods for Model Reliability Assessment," Reliab. Eng. Syst. Saf., 93(8), pp. 1197-1207. (Pubitemid 351418141)
48. Haldar, A., and Mahadevan, S., 2000, Probability, Reliability, and Statistical Methods in Engineering Design, Wiley, New York.
49. Ghanem, R. G., Doostan, A., and Red-Horse, J., 2008, "A Probabilistic Construction of Model Validation," Comput. Methods Appl. Mech. Eng., 197(29-32), pp. 2585-2595.
50. Kass, R. E., and Raftery, A., 1995, "Bayesian Factors," J. Am. Stat. Assoc., 90(430), pp. 773-795.
51. Oberkampf, W. L., and Trucano, T. G., 2002, "Verification and Validation in Computational Fluid Dynamics," Prog. Aerosp. Sci., 38(2), pp. 209-272. (Pubitemid 34863048)
52. Loehle, C., 1997, "A Hypothesis Testing Framework for Evaluating Ecosystem Model Performance," Ecol. Modell., 97(3), pp. 153-165.
53. Robinson, A. P., and Froese, R. E., 2004, "Model Validation Using Equivalence Tests," Ecol. Modell., 176(3-4),pp. 349-358. (Pubitemid 38791508)
54. Wellek, S., 2003, Testing Statistical Hypotheses of Equivalence, Chapman and Hall, London.
55. Balci, O., and Sargent, R. G., 1981, "A Methodology for Cost-Risk Analysis in the Statistical Validation of Simulation Models," Commun. ACM, 24(4), pp. 190-197.
56. Trucano, T. G., Swiler, L. P., Igusa, T., Oberkampf, W. L., and Pilch, M., 2006, "Calibration, Validation, and Sensitivity Analysis: What's What," Reliab. Eng. Syst. Saf., 91(10-11), pp. 1331-1357.