Comparison of several model validation conceptions against a "real space" end-to-end approach

SAE Technical Papers

Volumn , Issue , 2011, Pages

  Romero, Vicente ^a  

^a SANDIA NATIONAL LABORATORIES   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

FORECASTING; HEAT TRANSFER;

HIERARCHICAL MODEL; INDUSTRIAL SCALE; MODEL UNCERTAINTIES; MODEL VALIDATION; MODELING AND PREDICTIONS; NONLINEAR BEHAVIOR; STRUCTURAL MECHANICS; TIME VARYING;

UNCERTAINTY ANALYSIS;

EID: 84872462543     PISSN: 01487191     EISSN: 26883627     Source Type: Journal    
DOI: None     Document Type: Conference Paper

References (87)

1. Department of Defense Instruction 5000.61: Modeling and Simulation Verification, Validation, & Accreditation (VV&A), Defense Modeling and Simulation Office, Office of the Director of Defense Research and Engineering, dated April 29, 1996, www.dmso.mil/docslib.
2. American Institute of Aeronautics and Astonautics, Guide for the Verification and Validation of Computational Fluid Dynamics Simulations, AIAA G-077-1998, AIAA, Reston, Va.
3. Department of Energy-Defense Programs, Accelerated Strategic Computing Initiative Program Plan, 2000, DOE/DP-99-000010592.
4. American Society of Mechanical Engineers, V&V 10-2006 Guide for Verification and Validation in Computational Solid Mechanics, available from ASME Codes & Standards website.
5. NASA, Technical Standard for Models and Simulations, report NASA-STD-7009, released July 11, 2008.
6. American Society of Mechanical Engineers, V&V 20-2009 Standard for Verification and Validation in Computational Fluid Dynamics and Heat Transfer, available from ASME Codes & Standards website.
7. American Society of Mechanical Engineers, V&V 10-draft document in preparation illustrating the concepts of V&V in an end-to-end computational solid mechanics example
8. Babuska, I., Nobile, F., Tempone, R., "Reliability of Computational Science," Num. Methods in Partial Differential Equations, (2007) Vol. 23, pp. 753-784.
9. Babuska, I., Oden, J.T., Papadrakakis, M., "Verification and Validation in Computational Engineering and Science: Basic Concepts," Comput. Methods in Applied Mechanics and Engrng., Vol. 193, 2004, pp. 4057-4066.
10. Balci, O., "Verification, Validation and Testing," Handbook of Simulation, ed. Banks, J., Engineering and Management Press (1998), pp. 335-393.
11. Bayarri, M., Berger, J., Rui, P., Sacks, J., Cafeo, J., Cavendish, J., Lin, C-H., Tu, J., "A Framework for Validation of Computer Models," Technometrics, May 2007, Vol. 49, no. 2, pp. 138-154.
12. th AIAA Computational Fluid Dynamics Conference, Miami, FL, June 25-28, paper AIAA-2007-4204.
13. Blottner, F.G., "Accurate Navier-Stokes Results for the Hypersonic Flow over a Spherical Nosetip," AIAA J. Spacecraft and Rockets, Vol. 27, No. 2, March-April 1990, pp. 113-122.
14. Campbell, K., "A Brief Survey of Statistical Model Calibration Ideas," International Conference on Sensitivity Analysis of Model Output, March 8-11, 2004, Santa Fe, NM.
15. Calder, A., Fryxell, B., Plewa, T., Rosner, R., Dursi, L., Weirs, V., Dupont, T., Robey, H., Kane, J., Remington, B., Drake, R., Dimonte, G., Zingale, M., Timmes, F., Olson, K., Ricker, P., MacNeice, P., Tufo, H., "On Validating an Astrophysical Simulation Code," submitted to J. Comp. Phys.
16. Chen, W., Xiong, Y., Tsui, K-L., Wang, S., "Some Metrics and a Bayesian Procedure for Validating Predictive Models in Engineering Design," ASME Design Technical Conference, Design Automation Conf., Philadelphia, PA, Sept. 10-13, 2006.
17. Coleman, H.W., and Stern, F., "Uncertainties in CFD Code Validation," Journal of Fluids Engineering, Dec. 1997, vol. 119, pp. 795-803.
18. Dempsey, F., Wellman, G., Scherzinger, W., Romero, V., Antoun, B., "Development and Validation of a High-Temperature Constitutive Model of Stainless Steel Yield to Failure," Sandia National Laboratories report in preparation.
19. Dowding, K.J., Hills, R.G., Leslie, I.H., Pilch, M., Rutherford, B.M., Hobbs, M.L., "Case Study for Model Validation: Assessing a Model for Thermal Decomposition of Polyurethane Foam," Sandia National Laboratories report SAND2004-3632, printed September 2004.
20. Easterling, R. G., "Measuring the Predictive Capability of Computational Models: Principles and Methods, Issues and Illustrations," Sandia National Laboratores report, SAND2001-0243, Unlimited Release, February 2001.
21. Ferson, S., Oberkampf, W.L., Ginzburg, L., "Model Validation and Predictive Capability for the Thermal Challenge Problem," Comput. Methods in Applied Mechanics and Engrng., Vol. 197, 2009, pp. 2408-2430.
22. Hanson, K.M., and Hemez, F.M., 2001, "A Framework for Assessing Confidence in Computational Predictions," Experimental Techniques, Vol. 25.
23. th International Conference on Structural Safety and Reliability (ICOSSAR), Rome, Italy, June 19-23, 2005.
24. Hasselman, T., Lloyd, G., Hinnerichs, T., O'Gorman, C., Urbina, A., "Validation of Uncertainty Quantification For Structures with Encapsulated Epoxy Foam," 8th AIAA Non-Deterministic Analysis Conference, Newport, RI, May 1-4, 2006.
25. Hazelrigg, G. A., "Thoughts on Model Validation for Engineering Design," Proceedings DETC'03, ASME 2003 Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Chicago, Illinois, September 2-6, 2003.
26. Hills, R.H., and Trucano, T.G., "Statistical Validation of Engineering and Scientific Models: Background," Sandia National Laboratories Report SAND99-1256, printed May 1999.
27. Hills, R.G., and Dowding, K.J., "Statistical Validation of Engineering and Scientific Models: Bounds, Calibration, and Extrapolation," Sandia National Laboratories document SAND2005-1826, April 2005.
28. Hills, R.G., "Model Validation: Model Parameter and Measurement Uncertainty," ASME J. Heat Transfer, April 2006, Vol. 128, pp. 339-351.
29. Hobbs, M.L., and Romero, V.J., "Uncertainty Analysis of Decomposing Polyurethane Foam," Thermochemica Acta, 384 (2002), 393-401
30. Horta, L.G., Kenny, S.P., Crespo, L.G., Elliott, K.B., "NASA Langley's approach to the Sandia's structural dynamics challenge problem," Comput. Methods in Applied Mechanics and Engrng., Vol. 197, 2009, pp. 2607-2620.
31. Kleindorfer, G.B., O'Niell, L., Ganeshan, R., "Validation in Simulation: Various Positions in the Philosophy of Science," Management Science, Vol. 44, No. 8, August 1998, pp. 1087-1099.
32. th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, 23-26 April, 2007, Honolulu, HI.
33. Logan, R.W., Nitta, C.K., and Chidester, S. K., "Uncertainty Quantification During Integral Validation: Estimating Parametric, Model Form, and Solution Contributions," 8th AIAA Non-Deterministic Analysis Conference, May 1-4, 2006, Newport, RI.
34. McFarland, J., Mahadevan, S., Romero, V., Swiler, L., "Calibration and Uncertainty Analysis for Expensive Computer Simulations with Multivariate Output," AIAA Journal, vol. 46, no. 5, May 2008, pp. 1253-65.
35. Miser, H.J., and Quade, E.S., "Validation," Chapter 13 in Handbook of Systems Analysis: Craft Issues and Procedural Choices, North-Holland, New York, 1988, pp. 527-565.
36. Oberkampf, W.L., and Trucano, T.G., 2002, "Verification and Validation in Computational Fluid Dynamics," Progress in Aerospace Sciences, 38 (3), pp. 209-272.
37. Oberkampf, W.L., Trucano, T.G., Hirsch, C., "Verification, Validation, and Predictive Capability in Computational Engineering and Physics," Sandia National Laboratories Report SAND2003-3769, printed Feb. 2003.
38. Oberkampf, W.L., and Barone, M.F., 2004, "Measures of Agreement between Computation and Experiment: Validation Metrics," AIAA 34th Fluid Dynamics Conference, Portland, OR, June 2004.
39. Oberkampf, W.L., and Roy, C.J., Verification and Validation in Scientific Computing, Cambridge University Press, 2010.
40. Popper, K., Conjectures and Refutations: The Growth of Scientific Knowledge, Basic Books, New York, 1963.
41. Ricks, A., Nicolette, V., Romero, V., Erickson, W., Hewson, J., "Fuego Solid-Propellant Fire Model Verification and Validation," Sandia National Laboratories report in preparation.
42. Roache, P.J., Fundamentals of Verification and Validation, Hermosa Publishers, Socorro, NM, 2009.
43. Romero, V.J., "On Model Validation and Extrapolation for Best-Estimate-Plus-Uncertainty Predictions," Sandia National Laboratories document SAND2005-7678C, November 2005.
44. th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, April 18-21, 2005, Austin, TX.
45. Romero, V.J., "Issues and Needs in Quantification of Margins and Uncertainty (QMU) for Phenomenologically Complex Coupled Systems," paper AIAA-2006-1989, 8th AIAA Non-Deterministic Analysis Conference, Newport, RI, May 1-4, 2006.
46. Romero, V.J., "A Paradigm of Model Validation and Validated Models for Best-Estimate-Plus-Uncertainty Predictions in Systems Engineering," SAE Technical Paper 2007-01-1746, 2007, doi: 10.4271/2007-01-1746.
47. Romero, V.J., "Validated Model? Not So Fast. The Need for Model 'Conditioning' as an Essential Addendum to Model Validation," paper AIAA-2007-1953, 9th Non-Deterministic Approaches Conference, Honolulu, HI, April 23-26, 2007.
48. Romero, V.J., "Type X and Y Errors and Data & Model Conditioning for Systematic Uncertainty in Model Calibration, Validation, and Extrapolation," SAE Technical Paper 2008-01-1368, 2008, doi: 10.4271/2008-01-1368.
49. Romero, V.J., "Data & Model Conditioning for Multivariate Systematic Uncertainty in Model Calibration, Validation, and Extrapolation," paper AIAA-2010-2511, 12th AIAA Non-Deterministic Approaches Conference, April 12-15, 2010, Orlando, FL.
50. Romero, V.J., Luketa, A., Sherman, M., "Application of a Versatile "Real Space" Validation Methodology to a Fire Model" AIAA J. of Thermophysics and Heat Transfer, Vol. 24, No. 4, Oct.-Dec. 2010, pp. 730-744.
51. Romero, V. J., "A Practical and Versatile Framework for Characterizing and Propagating Uncertainty in Hierarchical Modeling: Data and Model Conditioning; Model Verification and Validation; Extrapolative Prediction with Uncertainty," Sandia National Laboratories report in preparation. Summary presentation available: "A New Interval-Based "Real Space" Approach to Model Validation Involving Aleatory and Epistemic Uncertainty," Sandia National Laboratories document SAND2009-3160P, April 30, 2009.
52. Romero, V.J., Shelton, J.W., and Sherman, M.P., "Modeling Boundary Conditions and Thermocouple Response in a Thermal Experiment," 2006 ASME Int'l. Mechanical Engineering Congress and Exposition, Nov. 5-10, 2006, Chicago, IL.
53. th AIAA Aerospace Sciences Meeting, 4-7 Jan. 2010, Orlando, FL.
54. Rutherford, B.M., and Dowding, K.J., "An Approach to Model Validation and Model-Based Prediction: Polyurethane Foam Case Study," Sandia National Laboratories report SAND2003-2336, printed July 2003.
55. Rutherford, R., Romero, V., Castro, J., Hoekstra, R., "Methods Used in Uncertainty Quantification, Calibration, Validation, and Prediction for the QASPR Silicon Device Prototype Demonstration," Sandia National Laboratories report in post-review final revision.
56. Schlesinger, S., Crosbie, R., Gagne, R., Innis, G., Lalwani, C., Loch, J., Sylvester, R., Wright, R., Kheir, N., Bartos, D., "Terminology for Model Credibility," report of the Technical Committee on Model Credibility to the general membership of the Society for Modeling and Simulation International (SCS), Simulation, 1979, Vol. 32, pp. 103-104.
57. rd ASME/JSME Joint Fluids Engineering Conference, July 18-23, 1999, San Francisco, CA.
58. Trucano, T.G., Pilch, M., Oberkampf, W.L., "General Concepts for Experimental Validation of ASCI Code Applications," Sandia National Laboratories Report SAND2002-0341, printed March 2002.
59. Trucano, T.G., Swiler, L.P., Igusa, T., Oberkampf, W.L., Pilch, M., "Calibration, validation, and sensitivity analysis: What's what," Reliability Engrng. and System Safety, Vol. 91 (2006), No. 11, pp. 1331-1357.
60. Urbina, A., Paez, T.L., Smallwood, D.O., "A Hierarchy of Validation Measures for Structural Dynamics," paper #181 of the 2006 International Modal Analysis Conference (IMAC XXIV), January 30-Feb. 2, 2006, St. Louis, MO.
61. Swiler, L.P., Adams, B.M., Eldred, M.S., "Model Calibration under Uncertainty: Matching Distribution Information," paper AIAA-2009-5944, 11th AIAA Non-Deterministic Approaches Conference, 4-7 May 2009, Palm Springs, CA.
62. Xiong, W., Chen, W., Tsui, K-L., Apley, D., "A better understanding of model updating strategies in validating engineering models," CMAME 198 (2009) 1327-1337.
63. Konikow, L.F., and Bredehoeft, J.D., "Groundwater models cannot be validated," Advances in Water Resources, Vol. 15, 1992, pp. 75-83.
64. De Marsily, G., Combes, P., Goblet, P., "Comment on 'Ground-water models cannot be validated' by L.F. Konikow & J.D. Bredehoeft (Advances in Water Resources, Vol. 15, 1992, pp. 75-83)," Advances in Water Resources, Vol. 15, 1992, pp. 367-369.
65. Leijnse, A., and Hassanizadeh, S.M., "Model definition and model validation," Short Communication in Advances in Water Resources, Vol. 17, 1994, pp. 197-200.
66. Tsang, Chin-Fu, "The Modeling Process and Model Validation," Ground Water, Vol. 29, No. 6, Nov.-Dec. 1991, pp. 825-831.
67. Oreskes, N., Frechette, K. Shrader, Belitz, K., "Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences," Science, Vol. 263, Feb. 1994, pp. 641-646.
68. Refsgaard, J.C., and Henriksen, H.J., "Modeling Guidelines- Terminology and Guiding Principles," Advances in Water Resources, Vol. 27, 2004, pp. 4057-4066.
69. Sterman, J. D., Letter to the Editor, in Science Vol. 264, April 1994,re: "Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences," by, Oreskes, et al., Science, Vol. 263, Feb. 1994, pp. 641-646.
70. Anderson, M.G., and Bates, P.D., editors, Model Validation-Perspectives in Hydrological Science, Wiley, 2001.
71. Romero, V.J., "Advanced Concepts of a Workable End-to-End Framework for Experiments through Prediction involving Bias and Uncertainty: •Experiment Design and Data Conditioning; •Real-Space Model Conditioning and Validation; •Hierarchical Modeling and Extrapolative Prediction," Sandia National Laboratories report in post-review final revision,expected to be available by current paper's publication date for SAE 2011 Congress, April 11-15, 2011.
72. Zeigler, B.P., Theory of Modeling and Simulation, John Wiley & Sons, 1976.
73. Mehta, U.B., "Guide to Credible Computer Simulations of Fluid Flows," Journal of Propulsion and Power, vol. 12, no. 5, Sept.-Oct. 1996, pp. 940-948.
74. Rebba, R., Mahadevan, S., "Computational methods for model reliability assessment," Reliability Engineering and System Safety, Vol. 93 (2008), pp. 1197-1207.
75. Sargent, R.G., "Validation of simulation models," Proc. Winter Simulation Conf., San Diego, CA, December 1979, 497-503.
76. Sargent, R.G., "Verification and validation of simulation models," Chapt. IX in Progress in Modelling and Simulation, ed. Cellier, F.E., Academic Press, London, 1982.
77. Balci, O., and Sargent, R.G., "A methodology for cost-risk analysis in the statistical validation of simulation models," Commun. of the ACM, 24 (6) April 1981, 190-197.
78. Balci, O., and Sargent, R.G., "Bibliography on validation of simulation models," Newsletter -TIMS College on Simulation and Gaming, 4 (2), Spring 1980, 11-15.
79. Balci, O., "Verification, Validation, and Accreditation of Simulation Models," Proc. 1997 Winter Simulation Conf., ed. Andradottir, S., Healy, K.J., Withers, D.H., Nelson, B.L., 135-141.
80. Balci, O., "Verification, Validation, and Testing," Chapt. 10 in The Handbook of Simulation, ed. Banks, J., 1997, John Wiley & Sons.
81. Balci, O., "Quality assessment, verification, and validation of modeling and simulation applications," Proc. 2004 Winter Simulation Conf., ed. Ingalls, R.G., Rossetti, M.D., Smith, J.S., Peters, H.A., IEEE Piscataway, NJ, 122-129.
82. Sargent, R.G., "Verification and Validation of Simulation Models," Proc. 2005 Winter Simulation Conf., ed. Kuhl, M.E., Steiger, N.M., Armstrong, F.B., Joines, J.A., 130-143.
83. Boehm, B.W., Software Engineering Economics, Prentice-Hall, 1981.
84. Ferson, S., and Ginzburg, L., (1996) "Different Methods are Needed to Propagate Ignorance and Variability," Reliability Engineering and System Safety, Vol. 54, No. 11, pp. 133-144.
85. Liu, Y., Chen, W., Arendt, P., Huang, H-Z, "Towards a Better Understanding of Model Validation Metrics," paper AIAA-2010-9240, 13th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, Sept. 12-15, 2010, Fort Worth, TX.
86. Ben-Haim, Y., Hemez, F.M., "Robustness, Fidelity and Prediction-Looseness of Models," paper ESDA 2004-58008, Proc. of ESDA04, 7th Biennial Conf. on Engr. Sys. Design and Analysis, July 19-22, 2004, Manchester, United Kingdom. Revised and extended version submitted Jan. 2011 to Royal Society Proceedings A, as Los Alamos National Laboratories document LA-UR-11-0497.
87. Ben-Haim, Y., Information Gap Decision Theory- Decisions under severe uncertainty, Academic Press, London, 2001.