Modeling and simulation verification and validation challenges

Johns Hopkins APL Technical Digest (Applied Physics Laboratory)

Volumn 25, Issue 2, 2004, Pages 163-172

  Pace, Dale K ^a  

^a APL ^*  (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER MODELING; SIMULATION VERIFICATION; UNIFIED MODELING LANGUAGES (UML); VALIDATION;

COMPUTER PROGRAMMING LANGUAGES; COMPUTER SIMULATION; DISTRIBUTED COMPUTER SYSTEMS; MATHEMATICAL MODELS; SOCIETIES AND INSTITUTIONS; SOFTWARE ENGINEERING;

DECISION SUPPORT SYSTEMS;

EID: 16244422023     PISSN: 02705214     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (45)

1. Nance, R. E., and Sargent, R. O., "Perspectives on the Evolution of Simulation," Oper. Res. 50(1), 161-174 CMFeb 2002).
2. Balci, O., and Sargent, R. G., "Bibliography on Validation of Simulation Models," Newsletter of TIMS College on Simulation and Gaming 4(2), 11-15 (1980).
3. Topcu, O., Review of Verification and Validation Methods in Simulation, Technical Memorandum 2003-055, Defense R&D Canada-Atlantic, Dartmouth, NS, Canada (Apr 2003).
4. For example, DOD Simulations: Improved Assessment Procedures Would Increase the Credibility of Results, GAO/PEMD-88-3, Government Accounting Office (1987).
5. For example, Roth, P. F., Gass, S. I., and Lemoine, A. J., "Some Considerations for Improving Federal Modeling," in Proc. 10th Winter Simulation Conf. 1, IEEE Press, Piscataway, NJ, pp. 213-218 (1978).
6. Software Engineering Institute (SEI) Capability Maturity Models, available at http://www.sei.cmu.edu/cmm/cmms/cmms.html; and Capability Maturity Modeling Integration, available at http://www.sei.cmu.edu/cmmi/.
7. Unified Modeling Language, available at http://www.uml.org/.
8. Formal Methods Repositories, available at http://vl.fmnet.info/reposi- tories/.
9. IEEE 1278.1-4 for Distributed Interactive Simulation (1995-1998, with 2002 revisions).
10. IEEE 1516/1516.1-3 for Modeling and Simulation (M&S) High Level Architecture (HLA) (2000 to 2003).
11. No comprehensive listing of such V&V/IV&V/VV&A policy/guidance is available. The briefings and papers of the T3 Session of the Foundations '02 V&V Workshop (proceedings available from the SCS or linked to the DMSO W&A Web page) identify most of DoD's model and simulation V&V/W&A policy and guidance as well as those from several professional societies (AIAA, ASME, IEEE, etc.). Not included in these are V&V guidance from NASA (which established a software IV&V center in 1993), V&V guidance from the National Institute of Standards and Technology (NIST) from the 1990s, the substantive V&V efforts begun in the late 1990s as part of the Department of Energy's Accelerated Strategic Computing Initiative (ASCI) program, software validation guidance from the Federal Drug Administration (PDA), or V&V guidance from a number of other government, industrial, and professional organizations.
12. For example, the description of software validation from Soft Solutions International: "Software validation is essentially a design verification function and includes all of the verification and testing activities conducted throughout the software life cycle," available at http://www.ssi-ltd.wjm/ services/software-validation-ethos-definitions.asp.
13. Knepell, P. L., and Arangno, D. C., Simulation Validation: A Confidence Assessment Methodology, IEEE Computer Society Press, Los Alarmtos, CA (1993).
14. Roache, P. J., Verification and Validation in Computational Science and Engineering, Hermosa Publishers, Albuquerque, NM (1998).
15. Pace, D. K. (ed.), V&V State of the Art: Proc. Foundations '02, a Workshop on Model and Simulation Verification and Validation for the 21st Century, Laurel, MD, CD-ROM, published by The Society for Modeling and Simulation (2002); see the W&A page of the DMSO Web site, http://www.dmso.mil/public/transition/wa/foundations.
16. DoD Modeling and Simulation Master Plan 5000.59-P, Under Secretary of Defense for Acquisition and Technology, Washington, DC (Oct 1995); DoD Modeling and Simulation (M&S) Verification, Validation, and Accreditation (W&A), Instruction No. 5000.61, Office of the Director of Defense Research and Engineering, Defense Modeling and Simulation Office, Washington, DC (23 May 2003).
17. For example, Software Verification and Validation Plan, IEEE Standard 1012-1998 (1998), or AIAA Guide for the Verification and Validation of Computational Fluid Dynamics Simulations, AIAA-G-077-1998, Reston, VA (1998).
18. The Sargent Circle was promulgated by the Society for Computer Simulation in Schlesinger, S., 'Terminology for Model Credibility," Simulation 32(3), 103-104 (1979).
19. The evolved Sargent Circle was first published in Pace, D. K., "Simulation Conceptual Model Role in Determining Compatibility of Candidate Simulations for a HLA Federation," in Proc. Spring 2001 Simulation Interoperability Workshop (SIW), paper 01S-SIW-024, available at http://www.sisostds.org/doclib/doclib.cfm?SISO_FID_5912.
20. The Referent Study Final Report (May 2004), available at http://www.dmso.mil/public/transition/wa; Pace, D. K., "The Referent in Simulation Development and Assessment," paper E04-SIW-024, in Proc. 2004 European Simulation Interoperability Workshop (EuroSIW), available at http://www.sisostds.org.
21. The Referent Study Final Report (May 2004), available at http://www.dmso.mil/public/transition/wa; Pace, D. K., "The Referent in Simulation Development and Assessment," paper E04-SIW-024, in Proc. 2004 European Simulation Interoperability Workshop (EuroSIW), available at http://www.sisostds.org.
22. Balci, O., Nance, R. E., Arthur, J. D., and Ormsby, W. F., "Expanding Our Horizons in Verification, and Accreditation Research and Practice," in Proc. 2002 Winter Simulation Conf., available at http://manta.cs.vt.edu/balci/papers/VVAHorizons.pdf.
23. Pace, D. K., and Sheehan, J., "Subject Matter Expert (SME)/Peer Use in M&S V&V," an A4 paper in Proc. Workshop on Foundations for Modeling and Simulation (M&S) Verification and Validation (V&V) in the 21st Century, SCS, San Diego, CA, CD-ROM (2002).
24. Metz, M., and Harmon, S., "Using Subject Matter Experts for Results Validation: A Progress Report," paper 02S-SIW-095, in Proc. Spring 2002 Simulation Interoperability Workshop (SIW), available at http://www.sisostds. org/conference/View_Public_Number.cfm?Phase_ID-2.
25. CJCSI 3170.01D, Joint Capabilities Integration and Development System (JCIDS) (12 Mar 2004), available at http://www.dtic.mil/cjcs_directives/cdata/ unlimit/3170_01.pdf.
26. Heitmeyer, C., "On the Need for Practical Formal Methods," invited paper, 5th Int. School and Symp.: Formal Techniques in Real Time and Fault Tolerant Systems (FTRTFT), pp. 18-26 (1998).
27. Kuhn, D. R., Chandramouli, R., and Butler, R. W., "Cost Effective Use of Formal Methods in Verification and Validation," an A5 paper in Proc. Workshop on Foundations for Modeling and Simulation (M&S) Verification and Validation (V&V) in the 21st Century, SCS, San Diego, CA, CD-ROM (2002).
28. Worley, D. R., Simpson, H. K., Moses, F. L., Aylward, M., Bailey, M., and Fish, D., Utility of Modeling and Simulation in the Department of Defense Initial Data Collection, paper D-1825, Institute of Defense Analyses (IDA) (May 1996), available at http://www.msiac.dmso.mil/ia/generaldocs.asp.
29. Costs are not reported for the majority of the 359 models and simulations in 22 DoD acquisition programs. See Hillegas, A., Backschies, J., Donley, M., Duncan, R. C., and Edgar, W., The Use of Modeling and Simulation (M&S) Tools in Acquisition Program Offices: Results of a Survey, Hicks & Associates (31 Jan 2001).
30. Kilikauskas, K. L., Brade, D., Gravitz, R. M., Hall, D. H., Hoppus, M. L., et al., "Estimating V&V Resource Requirements and Schedule Impact," a B4 paper in Proc. Workshop on Foundations for Modeling and Simulation (M&S) Verification and Validation (V&V) m the 21st Century, SCS, San Diego, CA, CD-ROM (2002).
31. Examples of efforts to address uncertainties with more rigor and to facilitate scientifically defensible predictions from models and simulations include Ghanem, R., and Wojtkiewicz, S., Uncertainty Quantification Short Course at the Soc. for Industrial and Applied Mathematics (SIAM) Conf. on Computational Science and Engineering (9 Feb 2003), available at http://www.siam.org/meetings/cse03/UQShortcourse.htm; Oberkampf, W. L., DeLand, S. M., Rutherford, B. M., Diegert, K. V., and Alvin, K. F., Estimation of Total Uncertainty in Modeling and Simulation, Report SAND2000-0824, Sandia National Laboratories (Apr 2000); and 4th Int. Conf. on Sensitivity Analysis of Model Output (SAMO) 2004 (8-11 Mar 2004), available at http://www.floodrisknet.org.uk/ events/SAMO%202004.
32. Examples of efforts to address uncertainties with more rigor and to facilitate scientifically defensible predictions from models and simulations include Ghanem, R., and Wojtkiewicz, S., Uncertainty Quantification Short Course at the Soc. for Industrial and Applied Mathematics (SIAM) Conf. on Computational Science and Engineering (9 Feb 2003), available at http://www.siam.org/meetings/cse03/UQShortcourse.htm; Oberkampf, W. L., DeLand, S. M., Rutherford, B. M., Diegert, K. V., and Alvin, K. F., Estimation of Total Uncertainty in Modeling and Simulation, Report SAND2000-0824, Sandia National Laboratories (Apr 2000); and 4th Int. Conf. on Sensitivity Analysis of Model Output (SAMO) 2004 (8-11 Mar 2004), available at http://www.floodrisknet.org.uk/ events/SAMO%202004.
33. Examples of efforts to address uncertainties with more rigor and to facilitate scientifically defensible predictions from models and simulations include Ghanem, R., and Wojtkiewicz, S., Uncertainty Quantification Short Course at the Soc. for Industrial and Applied Mathematics (SIAM) Conf. on Computational Science and Engineering (9 Feb 2003), available at http://www.siam.org/meetings/cse03/UQShortcourse.htm; Oberkampf, W. L., DeLand, S. M., Rutherford, B. M., Diegert, K. V., and Alvin, K. F., Estimation of Total Uncertainty in Modeling and Simulation, Report SAND2000-0824, Sandia National Laboratories (Apr 2000); and 4th Int. Conf. on Sensitivity Analysis of Model Output (SAMO) 2004 (8-11 Mar 2004), available at http://www.floodrisknet.org.uk/ events/SAMO%202004.
34. Oberkampf, W. L., Trucano, T. G., and Hersch, C., "Verification, Validation, and Predictive Capability in Computational Engineering and Physics," a B1 paper in Proc. Workshop on Foundations for Modeling and Simulation (M&S) Verification and Validation (V&V) in the 21st Century, SCS, San Diego, CA, CD-ROM (2002). Subsequently published as Sandia Report SAND2003-3769 (Feb 2003).
35. Menzies, T., "Verification and Validation and Artificial Intelligence," a B5 paper in Proc. Workshop on Foundations for Modeling and Simulation (M&S) Verification and Validation (V&V) in the 21st Century, SCS, San Diego, CA, CD-ROM (2002).
36. Bigelow, J. H., and Davis, P. K., Implications for Model Validation of Multiresolution, Multiperspective Modeling (MRMPM) and Exploratory Analysis, Report MR-1750-AF, RAND Corp. (2003).
37. For example, Lee, J. K., Lee, M. M., and Chi, S-D., "DEVS/HLA-Based Modeling and Simulation for Intelligent Transportation Systems," SIMUL-T Soc. Mod. Sim. 79(8), 423-439 (1 Aug 2003); or Bunus, P., and Fritzson, P., "DEVS-Based Multi-Formalism Modeling and Simulation in Modelica," in Proc. Summer Computer Simulation Conf., Vancouver, B.C., Canada, CD-ROM (2000).
38. For example, Lee, J. K., Lee, M. M., and Chi, S-D., "DEVS/HLA-Based Modeling and Simulation for Intelligent Transportation Systems," SIMUL-T Soc. Mod. Sim. 79(8), 423-439 (1 Aug 2003); or Bunus, P., and Fritzson, P., "DEVS-Based Multi-Formalism Modeling and Simulation in Modelica," in Proc. Summer Computer Simulation Conf., Vancouver, B.C., Canada, CD-ROM (2000).
39. Giordano, J. C., abstract for a colloquium at the University of Virginia, A Master's Project Presentation: Exploring the Constraints of Human Behavior Representation (28 Apr 2004), available at http://www.cs.virginia.edu/colloquia/ event391.html.
40. Hughes, T., and Rolek, E., "Fidelity and Validity: Issues of Human Behavioral Representation Requirements Development," in Proc. Winter Simulation Conf., New Orleans, LA, pp. 976-982 (2003).
41. Harmon, S. Y., Hoffman, C. W. D., Gonzalez, A. J., Knauf, R., and Barr, V. B., "Validation of Human Behavior Representations," a B3 paper in Proc. Workshop on Foundations for Modeling and Simulation (M&S) Verification and Validation (V&V) in the 21st Century, SCS, San Diego, CA, CD-ROM (2002).
42. MacKenzie, G. R., Schulmeyer, G. G., and Yilmaz, L., "Verification Technology Potential with Different Modeling and Simulation Development and Implementation Paradigms," an Al paper in Proc. Workshop on Foundations for Modeling and Simulation (M&S) Verification and Validation (V&V) m the 21st Century, SCS, San Diego, CA, CD-ROM (2002).
43. Theodorakis, M., Analyti, M. A., Constantopoulos, P., and Spyratos, N., "Contextualization as an Abstraction Mechanism for Conceptual Modeling," in Proc. 18th Int. Conf. on Conceptual Modeling (ER99), available at http://www.ics.forth.gr/isl/publications/paperlink/ER99_paper.pdf.
44. Others besides this author have drawn similar conclusions, though they have often been stated less bluntly. For example, Modeling and Simulation m Defense Acquisition: Pathways to Success, National Research Council (NRC) Board on Manufacturing and Engineering Design Committee on Modeling and Simulation Enhancements for 21st Century Manufacturing and Acquisition, available at http://www.dtic.mil/ndia/2002sba/castro.pdf.
45. Available at http://www.cs.virginia.edu/~MaSTRi/. While the Modeling and Simulation Technology Research initiative (MaSTRi) vision shows appreciation for the reality that these areas are interrelated and MaSTRi plans to collaborate with the Illinois Institute of Technology Research Institute (IITRI) Modeling and Simulation Technology Research Institute, substantive endeavors have yet to be accomplished.