Concepts and Criteria to Assess Acceptability of Simulation Studies: A Frame of Reference

Communications of the ACM

Volumn 24, Issue 4, 1981, Pages 180-189

  Ören, Tuncer I ^a  

^a UNIVERSITY OF OTTAWA   (Canada)

Author keywords

assessment of simulation studies; code verification; figure of merit; model calibration; model certification; model fitting; model robustness; model validation; program efficiency; program referability; software certification; software reliability; software robustness

Indexed keywords

COMPUTER SIMULATION;

EID: 0018999461     PISSN: 00010782     EISSN: 15577317     Source Type: Journal    
DOI: 10.1145/358598.358605     Document Type: Article

References (85)

1. Allen, B. and Clema, J.K. Independent Verification/Validation Support Software. Proc. Nat'l Aerospace and Electron. Conf., Dayton, Ohio, March pp. 276-281.
2. Annino, J.S. and Russell, E.C. The ten most frequent causes of simulation analysis failure-and how to avoid them. Simulation Today, 60, (June 1979), pp. 137-140.
3. Bahr, D. A note on figures of merit, A CM Performance Evaluations Rev., 4, 3, (July 1975), pp. 1-3.
4. Balci, O. and Sargent, R.G. Bibliography on Validation of Simulation Models. Newsletter-TI M S College on Simulation and Gaming, Spring 1980, pp. 11-15.
5. Beer, S. Platform for Change. Wiley, New York, 1975.
6. Belady, L.A. and Leavenworth, B. Program Modiftability. IBM, Thomas J. Watson Research Center, Res, Rept. RC 8147, 1980.
7. Boulton, J. and Gibson, A.R. First Ann. Rept. Canadian Nuclear Fuel Waste Management Program, Rept AECL-6443, Atomic Energy of Canada Ltd., Pinawa, Manitoba, 1979.
8. Brilling, K.R. Backward integration of system dynamics modelsA useful validation test? In World Modeling: A Dialogue. C.W. Churchman and R.O. Mason (Eds.) Elsevier, New York, 1976, pp. 141-149.
9. Cellier, F.E. Combined continuous/discrete system simulation by use of digital computers: Techniques and tools, Ph.D. Th.,Diss ETH No. 6483, Zurich, Switzerland. 1979,
10. Cellier, F. E. How to enhance the robustness of simulation software. In Progress in Modelling and Simulation. F.E. Cellier (Ed.) Academic (London), England, (In press).
11. Cellier, F.E. and Moebius, P.J. Towards robust general purpose simulation software. ACM SIGNUM 14, 2, (June 1979).
12. Cutler, M. M. Proving properties of simulation programs for system verification and validation. Proc., Summer Computer Simulation Conf. Toronto, Canada, 1979, pp. 610-616.
13. Cutler, M.M., A formal program model for discrete event simulations and its use in the mathematical verification and validation of system models and implementations. Ph.D. Th., UCLA, Calif., 1979,
14. Deeter, C.R. and Hoffman, A.A.J. An Annotated Bibliography of Mathematical Models Classified by Validation and Error Analysis Methods. Texas Christian University, Fort Worth, TX 1978.
15. De Millo, R.A., Lipton, R.J., Perlis, A.J. Social processes and proofs of theorems and programs. Comm. ACM, 22, 5, (May, 1979), pp. 271-280.
16. Dickhoven, S. Software requirements for an improvement of transfer and adaptability of models. In Utility and Use of Large-Scale Mathematical Models, S. Gass (Ed.). Nat'l Bur Stand, Special Publication No. 534, U.S. Dept. of Commerce, Washington, D.C., May 1979, pp. 195-205.
17. Lizas, M.S. Whither simulation? Proc., UKSC Conf. on Computer Simulation, IPC Science and Technology Press, Chester, England, April 1978, pp. 494-501.
18. Elzas, M.S. Results of a simulation: Can they be trusted? Proc. Seminar on Modelling in Business of IFIP Applied Information Processing Group, Amsterdam, The Netherlands, Jan. 1978, pp. 170-183.
19. Elzas, M.S. What is needed for robust simulation? In Methodology in Systems Modelling and Simulation, B.P. Zeigler et al. (Eds.) North-Holland, Amsterdam, 1979, pp. 57-91.
20. Lmshoff, J. R. and Sisson, R.L. Design and Use of Computer Simulation Models, Collier-MacMillan, New York, 1970.
21. Gass, S.L. Evaluation of complex models. Computers and Oper. Res. 1 (March, 1977), pp. 27-35.
22. Gass, S.I. A procedure for the evaluation of complex models. Proc First Int. Conf. in Mathematical Models, Univ. of Missouri, 1977, pp. 247-258,
23. Gass, S.I. Computer model documentation: A review and an approach. Nat. Bur. Stand. Special Publication No. 500-39, U.S. Dept. of Commerce, Washington, D.C. Feb. 1979.
24. Gass, S.L. (Ed.). Utility and use of large-scale mathematical models. Proc. of a Workshop held at the Nat'l. Bur. Stand., Gaithersburg, Maryland, Apr, 28-29, 1977, National Bureau of Standards, Special Publication 534, Washington, D.C., U.S. Dept. of Commerce, May 1979.
25. Gass, S.I. BibliographyModel evaluation and assessment. In Utility and Use of Large-Scale Mathematical Models, S.I. Gass (Ed.). Nat'l. Bur. Stand. Special Publication No. 534, U.S. Dept. of Commerce, Washington, D.C., May 1979, pp. 215-217.
26. Gass, S.I. Validation and Assessment Issues of Energy Models, Prepared for the NATO Advanced Research Institute, The Application of Systems Science to Energy Policy Planning. Nov. 1216, Brookhaven National Laboratory, Upton, New York, 1979.
27. Greenberger, M. and Richels, R. Assessing energy policy models: Current state and future directions. Ann. Rev. Energy, 4, (1979).
28. Lenire, J. Evaluating policy models. Proc. Summer Computer Simulation Conf., Toronto, Canada, 1979, pp. 550-556.
29. Holling, C.S. Adaptive Environmental Assessment and Management, John Wiley, New Yok, 1978.
30. House, P.W. and Mcleod, J. Large-Scale Models for Policy Evaluation, Wiley-Interscience, New York, 1977.
31. Howden, W.E. Applicability of software validation techniques to scientific programs. A CM Trans. Programming Languages and Systems. 2, 3, (July 1980), pp. 307-320.
32. 32, Hunt, R. Large-scale systems theories: Some paradigms from software systems engineering. In Cybernetics and Modelling and Simulation of Large-Scale Systems. T.I. ren (Ed.) Int. Assoc. for Cybernetics, Namur, Belgium, 1978, pp. 141-150,
33. Kahne, S. Model credibility for large-scale systems. IEEE Trans. SMC, (Aug. 1976), pp. 586-590.
34. Karplus, W.J. The spectrum of mathematical modelling and systems simulation In Simulation of Systems, L. Dekker (Ed.) North-Holland, Amsterdam, 1976, pp. 5-13.
35. Kheir, N.A. and W.M. Homes. On validating simulation models of missile systems. Simulation, 30, 4, (Apr. 1978), pp. 117-128.
36. Kleijnen, J.P.C. Statistical design and analysis of simulation experiments. Proc. Simulation '75, Ziirich, Switzerland Acta Press, 1975.
37. Kleijnen, J.P.C. The role of statistical methodology in simulation. In Methodology in Systems Modelling and Simulation, B. P. Zeigler, M.S. El?.as, G.J. Klir, and T.I. ren (Eds.) North-Holland, Amsterdam, 1979, pp. 425-445.
38. Lee, D.B. Requiem for large-scale models. J. Am. Inst, of Planners, 39, 3 (May 1973), pp. 163-178.
39. Marmor-Squires, A.B. Software validation: An annotated bibliography. In Current Trends in Programming Methodology. Vol. II. Program Validation, R.T. Yeh (Ed.) Prentice-Hall, Englewood Cliffs, N.J., 1977, pp. 269-319.
40. Mcleod, J. Terms and definitions-'Play it Again, Stew'. Simulation Today, 29, 4 (Oct. 1977) pp. vii-ix.
41. Mihram, G.A. Simulation Statistical Foundations and Methodology. Academic Press, New York, 1972.
42. Mihram, G.A. Some practical aspects of the verification and validation of simulation models. Operational Res. Quart., 23, (1972) pp. 17-29.
43. Mihram, G.A. Four questions regarding the credibility of simulation. Proc, 7th Ann. Pittsburgh Conf., Part 2-Modeling and Simulation, Pittsburgh, PA, Instrum. Soc. Am. Apr 1976, pp. 1225-1230.
44. Mihram, G.A., G. Innis, E.J. Mastascusa, S. Schlesinger, and T.J. Teorey. Bibliography on simular credibility. Simulation, 23, 5, (Dec. 1974) p. vi.
45. Miller, E., and Howden, W.E. Tutorial: Software Testing and Validation Techniques. IEEE Comptr Soc., New York, NY, 1978.
46. Myers, G.J. The Art of Software Testing. Wiley, New York, 1979.
47. Nance, R.E. The feasibility of and methodology for developing federal documentation standards for simulation models. Rept. Dept. of Comptr Sci. VP1 and State University, Blacksburg, VA, 1977.
48. Newton, O.L. and Weatherbee, J.E. Guidelines for documenting computer simulation models. Project Memorandum MV-068-060-USAF, Federal Computer Performance Evaluation and Simulation Center, Washington, DC, 1979.
49. Newton, O.L. and Whetherbee, J.E. Guidelines for documenting simulation models: A review and procedures. In Simulation with Discrete Models; A State-of-the-Art View, T.L ren, C.M. Shub, and P.F. Roth (Eds.). IEEE, New York, 1980 pp. 243-258.
50. Oren, T.I. Deductive general systems theories and simulation of large-scale systems. Proc., Summer Comptr Simulation Conf., Houston, Texas, July 1974 pp. 13-16
51. Oren, T.I. Simulation is it has been, is, and should be. Simulation, 29, 5 (Nov. 1977) pp. 182-183.
52. Oren, T.I. Modelling, model manipulation and programming concepts in simulation. In Modelling, Identification, and Control in Environmental Systems, G.C. van Steenkiste (Ed.) North-Holland, Amsterdam, 1978 pp. 833-842.
53. Oren, T.L. Rationale for large-scale systems simulation software based on cybernetics and general system theories. In Cybernetics and Modelling of Large-Scale Systems, T.I. Oren (Ed.) Int. Assoc. for Cybernetics, Namur, Belgium, 1978 pp. 151-179.
54. Oren, T.I. Reference manual of GEST 78-Level 1. (A modelling and simulation language for combined systems). Tech. Rept. 78-02, Univ. of Ottawa, Comptr Sci. Dept., Ottawa, Ontario, Canada, 1978.
55. Oren, T.I. Epilogue. In Cybernetics and Modelling of Large-Scale Systems. T.I. Oren (Ed.) Int. Assoc. for Cybernetics, Namur, Belgium, 1978, pp. 181-189.
56. Oren, T.I. Concepts for advanced computer-assisted modelling. In Methodology in Systems Modelling and Simulation, B.P. Zeigler, M.S. Elzas, G.J, Klir, and T.I. ren (Eds.). North-Holland, Amsterdam, 1979, pp. 29-55.
57. Oren, T.I. Computer-aided modelling systems. In Progress in Modelling and Simulation F.E. Cellier (Ed.), Academic (London), England (In press).
58. Oren, T.I. and Collie, B. Design of SEMA: A software system for computer-aided modelling and simulation of SEquential MAchines. In Proc, 1980 Winter Simulation Conf., T.L ren, C.M. Shub, and P.F. Roth (Eds.), 1980, pp 113-123.
59. Oren, T.I. and Den Dulk, J. Ecological models expressed in GEST 78, Rept prepared for the Dept. of Theoretical Production Ecology, Dutch Agricultural Univ., Wageningen, the Netherlands, 1978.
60. Oren, T.L. and Zeigler, B. P. Concepts for advanced simulation methodologies. Simulation, 32, 3 (March 1979) pp. 69-82.
61. O'Shaugnessy, J. inquiry and Decisions. Allen Unwin, London, 1972.
62. Pugh, R.E. Evaluation of Policy Simulation Models. Information Resources Press, Washington, DC, 1977.
63. Roth, P. F., Gass, S.I., and Lemoine, A.J. Some considerations for improving federal modelling. Proc. 1978 Winter Simulation Conf. Dec 1978 pp. 213-217.
64. Salaway, G. and Siler, K.F. Validation considerations during parameter development for simulation models. Proc., Summer Comptr Simulation Conf., Toronto, Canada, July 1979 pp. 624-628.
65. Sargent, R.G. Validation of discrete-event simulation models. Proc. NSF Workshop on Validation of Mathematical Models in Energy Related Research and Development. Forth Worth, Texas, 1978.
66. Sargent, R.G. Validation of simulation models. Proc., Winter Simulation Conf., Dec. 1979 pp. 497-503.
67. Sargent, R.G. Verification and validation of simulation models. In Progress in Modelling and Simulation. F.E. Cellier (Ed.), Academic (London), England, (In Press).
68. Saridis, G.N. Comparison of six on-line identification algorithms. Automatica, 10, 1 (Jan. 1974) pp. 69-79.
69. Schellenberger, R.E. Criteria for assessing model validity for managerial purposes. Decision Science, 5, 4 (Oct. 1974) pp, 644-653.
70. Schlesinger, S., J R. Buyan, E.D. Collender, W.K. Clarkson, and F.M. Perkins, Developing standard procedures for simulation validation and verification. Proc. Summer Comptr Simulation Conf., July 1974, pp. 927-933.
71. Schlesinger, S. et al. Terminology for model credibility. Simulation, 32 3, (1979) pp. 103-104.
72. Schruben, L.W. Establishing the credibility of simulations. Simulation, 3 (March 1980) pp. 101-105.
73. Teorey, T.J. Validation criteria for computer systems simulation. Proc. Symp. Simulation of Computer Systems, Boulder, Colorado, Nat'l Bur Stand, Aug. 1975, pp. 161-173.
74. Thissen, W. Investigations into the Club of Rome's World 3 Model - Lessons for understanding complicated models. Ph.D. Th., Dept. of Technical Physics, Eindhoven University of Technology, The Netherlands, 1978,
75. Thissen, W Investigations into the World 3 Model: Lessons for understanding complicated models. IEEE Trans, on Systems, Man, and Cybernetics, SMC-8, 3 ( March 1978) pp. 183-193.
76. Toffler, A. The Ecu-spasm Report. Bantam Books, New York, 1975.
77. Waller, J.D., Scanion, J.W., Kemp, D.M., and Nalley, P.G. Developing useful evaluation capability: Lessons from the model evaluation program, Stock Number 27-000-00798-8, U.S. Govt Printing Office, Washington, DC, 1979.
78. Watt, K.F. Why won't anyone believe us? Simulation, 28 (Jan. 1977), pp. 1-3.
79. Wigatt, M.R. The fitting, calibration, and validation of simulation models. Simulation, 18, 5, (May 1972) pp. 188-192.
80. Wong. C.S.Y. Criteria sensitivity analysis: A new approach to parameter sensitivity. Applied Mathematical Modelling. 4, (Feb. 1980) pp. 7-15.
81. Wright, R.D. Retrodictive tests of dynamic models. Proc. Summer Computer Simulation Conf., Montreal, Canada, July 1973.
82. Wymore, A.W. Systems Engineering Methodology for Interdisciplinary Teams. Wiley, New York, 1974.
83. Yeh, R.T. (Ed.) Current Trends in Programming Methodoloy, Vol. II Program Validation. Prentice-Hall, Englewood Cliffs, NJ, 1977.
84. Zeigler, B.P. Theory of Modelling and Simulation. Wiley-Interscience, New York, 1976.
85. Zeigler, B.P. Structuring the Organization of Partial Models. In Cybernetics and Modelling of Large-Scale Systems. T.I. Oren (Ed.). Int. Assoc. for Cybernetics, Namur, Belgium, 1978 pp, 127-139.