Methods for the external validation of continuous system simulation models: A review

Mathematical and Computer Modelling of Dynamical Systems

Volumn 4, Issue 1, 1998, Pages 5-31

  Murray Smith, D J ^a  

^a UNIVERSITY OF GLASGOW   (United Kingdom)

Author keywords

External validation; Model; Simulation

Indexed keywords

EID: 3643128589     PISSN: 13873954     EISSN: None     Source Type: Journal    
DOI: 10.1080/13873959808837066     Document Type: Review

References (88)

1. Pescatore, C.: Validation: an overview of definitions, in GEOVAL '94 Validation Through Model Testing, Proceedings of an NEA/SKI Symposium, Paris, France, 11-14 October 1994, 15-25, Nuclear Energy Agency, OECD, Paris, 1995.
2. S.C.S. Technical Committee on Model Credibility: Terminology for model credibility, Simulation, 32 (1979) 103-4.
3. Murray-Smith, D.J. and Carson, E.R.: The modelling process in respiratory medicine, in Cramp D.G. and Carson, E.R. (eds.) The Respiratory System, 296-333, Croom Helm, London, 1988.
4. Murray-Smith, D.J.: A review of methods for the validation of continuous system simulation models, in Nock, K.G. (ed.) UKSC'90: Proceedings of the 1990 UKSC Conference on Computer Simulation, 108-111, United Kingdom Simulation Council, Burgess Hill, 1990.
5. Isermann, R.: Process fault detection based on modeling and estimation methods: a survey, Automatica, 20, (1984), 4, 387-404.
6. Theil, H.: Economic Forecasting and Policy, North Holland, Amsterdam, 1970.
7. Kheir, N.A. and Holmes, W.: On validating simulation models of missile systems, Simulation, 30 (1978), 117-128.
8. Zhou Xianmin: A new method with high confidence for validation of computer simulation models for flight system, Chinese J. Syst. Eng. & Electronics, 4 (1993), 43-52.
9. Tan, K.C., Li, Y., Murray-Smith, D.J. and Sharman, K.C.: System identification and linearisation using genetic algorithms with simulated annealing, Proceedings International Conference on Genetic Algorithms in Engineering Systems: Innovations and Applications, 12-14 September 1995, Sheffield, U.K. (IEE Conference Publication No. 414) 164-169, IEE, London, 1985.
10. Kleijnan, J.P.C.: Verification and validation of simulation models, European J. Operational Research, 82, (1995), 1, 145-162.
11. Box, G.E.P. and Jenkins, G.M.: Time Series Analysis, Holden Day, San Francisco, 1976.
12. Kortmann, M., Janiszowski, K. and Unbehauen, H.: Application and comparison of different identification schemes under industrial conditions, Int. J. Control, 48, (1988), 6, 2275-2296.
13. Draper, N.R. and Smith, H.: Applied Regression Analysis, Wiley, New York, 1966.
14. Goodwin, G.C. and Payne, R.L.: Dynamic System Identification: Experiment and Design and Data Analysis, Academic Press, New York, 1977.
15. Barlas, Y.: An autocorrelation function test for output validation, Simulation, 55, (1990), 1, 7-16.
16. Butterfield, M.H. and Thomas, P.J.: Methods of quantitative validation for dynamic simulation models, Part 1, Trans. Inst. Measurement and Control, 8, (1986), 182-200.
17. Butterfield, M.H. and Thomas, P.J.: Methods of quantitative validation of dynamic simulation models, Part 2, Trans. Inst. Measurement and Control, 8, (1986), 201-219.
18. Butterfield, M.H.: A method of quantitative validation based on model distortion, Trans. Inst. Measurement and Control, 12, (1990), 167-173.
19. Cameron, R.G.: Model validation by the distortion method: linear state space systems IEE Proc.-D, 139, (1992), 296-300.
20. Cameron, R., Marcos, R.L. and de Prada, C.: Model validation of discrete transfer functions using the distortion method, Mathematical Modelling of Systems, this issue.
21. Bellman, R. and Åström, K.J.: On structural identifiability, Math. Biosci., 7, (1970), 329-339.
22. Pohjaanpalo, H.: System identifiability based on the power series of the solution, Math. Biosci., 41, (1978), 21-33.
23. Norton, J.P., Brown, R.F. and Godfrey, K.R.: Modal analysis of identifiability of linear compartmental models, IEE Proc.- Part D, 127, (1980), 83-92.
24. Walter, E.: Identifiability of state space models. Springer Lecture Notes in Biomathematics No. 46, Springer Verlag, Berlin, 1982.
25. Godfrey, K.R. and DiStefano III, J.J.: Identifiability of model parameters, in Barker, H.A. and Young, P.C. (eds.), Identification and System Parameter Estimation 1985 (Proc. 7th IFAC/IFORS Symposium, York, 3-7 July 1985), Pergamon Press, Oxford, 1985.
26. Beck, J.V. and Arnold, K.J.: Parameter Estimation in Engineering and Science, Wiley, New York, 1977.
27. Bradley, R., Padfield, G.D., Murray-Smith, D.J. and Thomson, D.G.: Validation of helicopter mathematical models, Trans. Inst. Measurement and Control, 12, (1990), 186-196.
28. Ding, Z. and Edmunds, J.M.: A statistical way for dynamic model validation, Preprints Inst. of Measurement and Control Symposium on Model Validation, 16th May 1989, London.
29. Billings, S.A. and Voon, W.S.F.: Structural detection and model validity tests in the identification of nonlinear systems, IEE Proc. - Pt. D., 130, (1983), 193-199.
30. Thomson, M., Schooling, S.P. and Soufian, M.: The practical application of a nonlinear identification methodology, Control Eng. Practice, 4, (1996), 3, 295-306.
31. Johansen, T.A.: Operating Regime Based Process Modeling and Identification, Dr. Ing. Thesis, Department of Engineering Cybernetics, The Norwegian Institute of Technology, University of Trondheim, 1994 (Report 94-109-W).
32. Johansen, T.A.: Identification of nonlinear systems using empirical data and prior knowledge - an optimization approach, Automatica, 32, (1996), 3, 337-356.
33. Murray-Smith, R.: A Local Model Network Approach to Nonlinear Modelling, Ph.D. Thesis, Department of Computer Science, University of Strathclyde, 1994.
34. Stone, M.: Cross-validatory choice and assessment of statistical predictions, J. Roy. Statistical Soc. B, 36, (1974), 111-133.
35. Tomovic, R.: Sensitivity Analysis of Dynamic Systems, MIT Press, 1962.
36. Eslami, M.: Theory of Sensitivity in Dynamic Systems, Springer-Verlag, Berlin, 1994.
37. Murray-Smith, D.J.: Continuous System Simulation, Chapman and Hall, London, 1995.
38. Gong, M. and Murray-Smith, D.J.: A practical exercise in simulation model validation, Mathematical Modelling of Systems, this issue.
39. Hess, R.A., Gao, C. and Wang, S.H.: A generalized technique for inverse simulation applied to aircraft maneuvres, Journal of Guidance, Control and Dynamics, 14, (1991), 5, 920-92.
40. Rutherford, S. and Thomson, D.G.: Improved methodology for inverse simulation, Aeronautical Journal, 100, (1996), 79-86.
41. Findler, N.V. and Mazur, N.M.: A system for automatic model verification and validation, Trans. Society for Comp. Sim., 6, (1989), 3, 153-172.
42. Deslandres, V. and Pierreval, H.: An expert system prototype assisting the statistical validation of simulation models, Simulation, 56, (1991), 2, 79-89.
43. Birta, L.G.: A knowledge-based approach for the validation of simulation models: the foundation, ACM Transactions on Modelling and Computer Simulation, 6, (1996), 1, 76-98.
44. Murray-Smith, D.J., Bradley, R. and Leith, D.: Identifiability analysis and experimental design for simulation model validation. In Maceri, F. and lazeolla, G., (eds.), EUROSIM '92 Simulation Congress (Proceedings of the 1992 EUROSIM Conference, Capri, Italy, 28 Sept. - 4 Oct. 1992), 15-20, North-Holland, Amsterdam, 1993.
45. Federov, V. V.: Theory of Optimal Experiments, Academic Press, New York, 1972.
46. Hunter, W.G., Hill, W.J. and Henson, T.L.: Designing experiments for precise estimation of all or some of the constants in a mechanistic model, Canadian J. of Chem. Eng., 47, (1969), 76-80.
47. Mehra, R.K.: Frequency-domain synthesis of optimal inputs for linear system parameter estimation, Trans. ASME (J. Dyn. Syst. Meas., and Control, 98, (1976), 130-138.
48. Sargent, R.G.: Validation of mathematical models, in GEOVAL-1990 Symposium on Validation of Geosphere Flow and Transport Model, Stockholm, 14-17 May 1990, Proceedings, 571-579, OECD, Paris, 1991.
49. Sisle, M.E.: Validation throughout the simulation life cycle, Proceedings of the 1985 Summer Computer Simulation Conference, July 22-24, 1985, Chicago, Illinois, 168, The Society for Computer Simulation, San Diego, California, 1985.
50. Federal Aviation Authority: Airplane Simulator Qualification, FAA Advisory Circular AC 120-40B, 1991.
51. International Air Transport Association: Flight Simulator Design and Performance Data Requirements, 4th Edition, 1993.
52. Baarspul, M.A.: Review of flight simulation techniques, Progress in Aerospace Science, 27, (1990), 1-120.
53. Neville, K.W. and Stephens, A.T.: Flight update of aerodynamic mathematical model, Proceedings AIAA Flight Simulation Technology Conference, Monterey, California, 9-11 August, 1993, 289-297, AIAA, 1993. (AIAA-93-355596-CP).
54. Zhang, B.: How to consider fidelity and validity for an engineering simulator, Proceedings AIAA Flight Simulation Technology Conference, Monterey, California, 9-11 August 1993, 298-305, AIAA, 1993. (AIAA-93-3598-CP).
55. AGARD Advisory Report No. 280, Rotorcraft System Identification, AGARD, Neuillysur-Seine, France, 1991.
56. Padfield, G.D., Thorne, R., Murray-Smith, D., Black, C. and Caldwell, A.E.: UK research into system identification for helicopter flight mechanics, Vertica, 11, (1987), 4, 665-684.
57. Padfield, G.D. and DuVal, R.W.: Application areas for rotorcraft system identification: simulation model validation, AGARD Lecture Series Vol. LS-178 (Rotorcraft System Identification), Paper 12, AGARD, Neuilly-sur Seine, France, 1991.
58. Murray-Smith, D.J.: Modelling and robustness issues in rotorcraft system identification, AGARD Lecture Series Vol. LS-178 (Rotorcraft System Identification), Paper 7, AGARD, Neuilly-sur-Seine, France, 1991.
59. Patton, R., Miles, M. and Taylor, P.: Design and application of test signals for helicopter model validation in the frequency domain, in Godfrey, K. (ed.), Perturbation Signals for System Identification, 298-320, Prentice Hall, Hemel Hempstead, 1993.
60. Gray, G.J. and Murray-Smith, D.J.: The external validation of nonlinear models for helicopter dynamics, in Pooley, R. and Zobel, R. (eds.), UKSS'93: Proceedings of the 1993 Conference of the United Kingdom Simulation Society, Keswick, 13th-15th September 1993, 143-147, UKSS, Edinburgh, 1993.
61. Gray, G.J. and von Grünhagen, W.: An investigation of open loop and inverse simulation as nonlinear model validation tools for helicopter flight mechanics, Mathematical Modelling of Systems, this issue.
62. Montgomery, D.C. and Conrad, R.G.: Comparison of simulation and flight data for missile systems, Simulation, 34, (1980), 63-72.
63. Shakir,L.A.: Validation of air combat models, AIAA Paper No. AIAA- 96-3337-CP, 1996.
64. Tseng, C.M., Dimmick, G.R. and Frketich, G.: Validation of the dynamic simulation of a natural circulation loop, Proceedings of the 1985 Summer Computer Simulation Conference, July 22-24, 1985, Chicago, Illinois, 168, The Society for Computer Simulation, San Diego, California, 1985.
65. Nuclear Energy Agency, OECD: Nuclear Safety Research in OECD Countries, OECD, Paris, 1994.
66. OECD: it GEOVAL-1990: Proceedings of Symposium on Validation of Geosphere Flow and Transport Models, Stockholm 14-17 May 1990, OECD, Paris, 1991.
67. OECD: GEOVAL '94: Proceedings of an NEA/SKI Symposium on Validation Through Model Testing, Paris, 11-14 October 1994, OECD, Paris, 1995.
68. IAEA, Safety Principles and Technical Criteria for the Underground Disposal of High-level Radioactive Wastes, IAEA Safety Standards, Safety Series No. 99, IAEA, Vienna, 1989.
69. Eisenberg, N., Federline, M., Sagar, B., Wittmeyer, G., Anderson, J. and Wingefores, S.: Model validation from a regulatory perspective: a summary, in GEOVAL '94 Validation Through Model Testing, Proceedings of an NEA/SKI Symposium, Paris, France, 11-14 October 1994, 421-434, Nuclear Energy Agency, OECD, Paris, 1995.
70. Tsang, C.F.: Validation and Technical Issues from GEOVAL-90 to GEOVAL-94, in GEOVAL '94 Validation Through Model Testing, Proceedings of an NEA/SKI Symposium, Paris, France, 11-14 October 1994, 27-33, Nuclear Energy Agency, OECD, Paris, 1995.
71. Oreskes, N., Shrader-Frechette, K. and Belitz, K.: Verification, validation and confirmation of numerical models in the earth sciences, Science, 263, (1994), 641-646.
72. Cronhjort, B. and Sheng, G.: Validation in the Swedish program for spent nuclear fuel, in Ingalls, V.W., Cyamon, J. and Saylor, A.: Procedings of the 1996 Summer Computer Simulation Conference, Portland, Oregon, July 21-25, 1996, 569-574, SCS, San Diego, California, 1996.
73. Hingorani, N.G.: Flexible a.c. transmission systems, IEEE Power Engineering Review, 8, (1988), July, 3-4.
74. Acha, E., Rico-Melgoza, J.J. and Fuerte-Esquivel, C.R.: The Impact of accurate power plant component modelling at harmonic frequencies, Proceedings 30th UPEC Conference, London, 5-7 September 1995, 122-125.
75. van Baars, G.E. and Bongers, P.M.M.: Flexible wind turbine model validation, Wind Engineering, 16, (1992), 4, 247-256.
76. Ljung, L.: System Identification: Theory for the User, Prentice Hall, Englewood Cliffs, New Jersey, 1987.
77. Bucheit, D.M. and Harrison, J.F.: Simulator qualification using engineering computer codes, Proceedings of the 1985 Summer Computer Simulation Conference, July 22-24, 1985, Chicago, Illinois, 608-613, The Society for Computer Simulation, San Diego, California, 1985.
78. Bache, R.A., Gray, W.M. and Murray-Smith, D.J.: Time-domain system identification applied to noninvasive estimation of cardiopulmonary quantities, IEE Proc. - Pt.D., 128, (1981), 56-64.
79. Brovko, O., Wiberg, D.M., Arena, L. and Belville, J.W.: The extended Kalman filter as a pulmonary blood flow estimator, Antomatica, 17, (1981), 213-220.
80. Tatnall, M.L.: The development of a system for the control of inhalational anaesthesia, Biomed. Meas. Info. Control, 1, (1986), 23-30.
81. Murray-Smith, D.J. and Carson, E.R.: Case studies of respiratory system models, in Cramp, D.G. and Carson, E.R., The Respiratory System, (Measurement in Medicine Series, Vol. 2), 334-389, Croom Helm, London, 1988.
82. Carson, E.R., Hovorka, R., Roudsari, A.V. and Summers, R.: Modelling and decision support in physiology and medicine: a methodological framework with illustrations, Mathematical Modelling of Systems, this issue.
83. Skelton, R.E. (ed.): Model Validation for Control Design, Special Issue, Mathematical Modelling of Systems, 3, 1, (1997).
84. Smith, R.S. and Doyle, J.C.: Model validation - a connection between robust control and identification, IEEE Trans. On Automatic Control, 37, (1992), 942-952.
85. Mäkilä, P.M., Gotthold, D. and Fischer, B.R.: Modelling and validation for control design - uncertain systems, monotone Bode plots and process control, Mathematical Modelling of Systems, 3, (1997), 3-26.
86. Ljung, L. and Guo, L.: Classical model validation for control design purposes, Mathematical Modelling of Systems, 3, (1997), 27-42.
87. Pace, D.K.: Verification, validation and accreditation issues for computer simulations which employ artificial intelligence techniques, In Ingalls, R.W., Cynamon, J. and Saylor, A.V., (eds.), The Proceedings of the 1996 Summer Computer Simulation Conference, Portland, Oregon, July 21-25 1996, 3-7, Society for Computer Simulation, San Diego, California, 1996.
88. Murray-Smith, D.J.: Enhanced environments for the development and validation of dynamic system models, Mathematics and Computers in Simulation, 39, (1995), 459-464.