Sensitivity analysis and related analyses: A review of some statistical techniques

Journal of Statistical Computation and Simulation

Volumn 57, Issue 1-4, 1997, Pages 111-142

  Kleijnen, Jack P C ^a  

^a TILBURG UNIVERSITY   (Netherlands)

Author keywords

Design of experiments; Latin hypercube sampling; Least squares; Optimization; Perturbation; Regression analysis; Risk analysis; Screening; Sensitivity analysis; Uncertainty analysis; Validation; What if

Indexed keywords

EID: 0031380935     PISSN: 00949655     EISSN: None     Source Type: Journal    
DOI: 10.1080/00949659708811805     Document Type: Article

References (88)

1. Andres, T. H. (1996) Sampling methods and sensitivity analysis for large parameter sets. This issue.
2. Balson, W. E., Welsh, J. L. and Wilson, D. S. (1992) Using decision analysis and risk analysis to manage utility environmental risk. Interfaces, 22(6), 126-139.
3. Bankes, S. (1993) Exploratory modeling for policy analysis. Operations Research, 41(3), 435-449.
4. Bankes, S. and Lucas, T. (1995) Statistical approaches for the exploratory modeling of large complex models. This issue.
5. Banks, J. (1989) Testing, understanding and validating complex simulation models, Proceedings of the 1989 Winter Simulation Conference.
6. Beckman, R. J. and McKay, M. D. (1987) Monte Carlo estimation under different distributions using the same simulation. Technometrics, 29(2), 153-160.
7. Bedford, T. and Meeuwissen, A. M. H. (1996) The distribution of maximum entropy distributions given fixed rank correlation, and numerical approximations for use in sensitivity analyses. This issue.
8. Bettonvil, B. (1990) Detection of Important Factors by Sequential Bifurcation, Tilburg University Press, Tilburg.
9. Bettonvil, B. and Kleijnen, J. P. C. (1996) Searching for important factors in simulation models with many factors: sequential bifurcation. European Journal of Operational Research (accepted).
10. 2 emission policy. European Journal of Operational Research, 83, 466-488.
11. Brady, S. D. and Hillestad, R. J. (1995) Modeling the external risks of airports for policy analysis. (MR-605-EAC/VW) RAND European-American Center for Policy Analysis, Delft (Netherlands).
12. Breeding, R. J. et al. (1992) Summary description of the methods used in the probabilistic risk assessments for NUREG-1150. Nuclear Engineering and Design, 135, 1-27.
13. Brehmer, B., Eriksson, E. A. and Wulff, P. (1994) Risk management (feature issue), European Journal of Operational Research, 75, 477-566.
14. Cardenosa, J. (1995) Preface (to Special Issue: European verification and validation of knowledge-based systems). Expert Systems With Applications, 8(3), 321-397.
15. Cheng, R. C. H. and Holland, W. (1996) The sensitivity of computer simulation experiments to errors in input data. This issue.
16. Cochran, J. K. and Chang, J. (1990) Optimization of multivariate simulation output models using a group screening mothod. Computers Industrial Engineering, 18(1), 95-103.
17. Cooke, R. M. (1995) UNICORN: methods and code for uncertainty analysis. The SRD Association, AEA Technology, Thomson House, Risley, Warrington WA3 6AT, UK.
18. De Wit, M. S. (1995), Uncertainty analysis in building thermal modelling. SAMO95 Proceedings.
19. Downing, D. J., Gardner, R. H. and Hoffman, F. O. (1985) An examination of responsesurface methodologies for uncertainty analysis in assessment models. Technometrics, 27(1), 151-163.
20. Downing, D. J., Gardner, R. H. and Hoffman, F. O. (1986) Response to Robert G. Easterling. Technometrics, 28(1), 92-93.
21. Draper, D. (1995) Assessment and propagation of model uncertainty. Journal Royal Statistical Society, Series B, 57, 45-97.
22. Easterling, R. G., (1986) Discussion of Downing, Gardner, and Hoffman (1985). Technometrics, 28(1), 91-92.
23. Engemann, K. J. and Miller, H. E. (1992) Operations risk management at a major bank. Interfaces, 22(6), 140-149.
24. FIPS (1979) Guidelines for automatic data processing risk analysis. FIPS PUB 65 (Federal Information Processing Standards Publication), Washington.
25. Harbers, A. (1993) Onzekerheidsana.yse op een simulatiemodel voor het broeikaseffect. (Uncertainty analysis of a simulation model for the greenhouse effect.) Working Paper, School of Maragement and Economics, Tilburg University, Tilburg, Netherlands.
26. Haverkort, B. R. and Meeuwissen, A. M. H. (1995) Sensitivity and uncertainty analysis of Markov-reward models. IEEE Transactions on Reliability, 44(1), 147-154.
27. Helton, J. C. (1993) Uncertainty and sensitivity analysis techniques for use in performance assessment for radioactive waste disposal. Reliability Engineering and System Safety, 42, 327-367.
28. Helton, J. C. (1996) Uncertainty and sensitivity analysis in the presence of stochastic and subjective uncertainty. This issue.
29. Helton, J. C. and Breeding, R. J. (1993) Calculation of reactor accident safety goals. Reliability Engineering and System Safety, 39, 129-158.
30. Helton, J. C. et al. (1995a) Effect of alternative conceptual models in a preliminary performance assessment for the waste isolation pilot plant. Nuclear Engineering and Design, 251-344.
31. Helton, J. C. et al. (1995b) Robustness of an uncertainty and sensitivity analysis of early exposure results with the MACCS reactor accident consequence model. Reliability Engineering and System Safety, 18, 129-148.
32. Helton, J. C. Garner, J. W., McCurley, R. D. and Rudeen, D. K. (1991) Sensitivity analysis techniques and results fir performance assessment at the waste isolation pilot plant. Sandia Report, SAND90-7103.
33. Helton, J. C., Garner, J. W., Rechard, R. P., Rudeen, D. K. and Swift, P. N. (1992) Preliminary comparison with 40 CFR part 191, subpart B for the waste isolation pilot plant, 4, uncertainty and sensitivity analysis. Sandia Report, SAND91-0893/4.
34. Hertz, D. B. (1964) Risk analysis in capital investments. Harvard Business Review, 95-106.
35. Ho, Y. and Cao, X. (1991) Perturbation analysis of discrete event systems. Dordrecht: Kluwer.
36. Hora, S. C. (1995) Sensitivity, uncertainty, and decision analyses in the prioritization of research. This issue.
37. Iman, R. L and Conover, W. J. (1980) Small sample sensitivity analysis techniques for computer models, with an application to risk assessment. (Including comments and rejoinder). Communications in Statistics, A9(17), 1749-1874.
38. Jacoby, J. E. and Harrison, S. (1962) "Multi-variable experimentation and simulation models". Naval Research Logistic Quarterly, 9, 121-136.
39. Janssen, P. H. M., Heuberger, P. S. C. and Sanders, R. (1992) UNCSAM 1.0: a software package for sensitivity and uncertainty analysis manual. National Institute of Public Health and Environmental Protection (RIVM), Bilthoven, The Netherlands.
40. Karplus, W. J. (1983) The spectrum of mathematical models. Perspectives in Computing, 3, 4-13.
41. Kleijnen, J. P. C. (1974/1975) Statistical Techniques in Simulation, I and II. Marcel Dekker Inc., New York. (Russian translation, Publishing House "Statistics", Moscow, 1978.)
42. Kleijnen, J. P. C. (1979), Regression metamodels for generalizing simulation results. IEEE Transactions on Systems. Man, and Cybernetics. SMC, 9(2), 93-96.
43. Kleijnen, J. P.C. (1983) Risk analysis and sensitivity analysis: antithesis or synthesis? Simuletter, 14(1-4), 64-72.
44. Kleijnen, J. P. C. (1987) Statistical Tools for Simulation Practitioners. Marcel Dekker, Inc., New York.
45. Kleijnen, J. P. C. (1993) Simulation and optimization in production planning: a case study, Decision Support Systems, 9, 269-280.
46. Kleijnen, J. P. C. (1994) Sensitivity analysis versus uncertainty analysis: when to use what? Predictability and nonlinear modeling in natural sciences and economics, edited by J. Grasman and G. van Straten, Kluwer, Dordrecht, The Netherlands, 1994, 322-333. (Preprinted in Kwantitatieve Methoden, 15, 45, February 1994, 3-15.)
47. Kleijnen, J. P. C. (1994) Sensitivity analysis versus uncertainty analysis: when to use what? Predictability and nonlinear modeling in natural sciences and economics, edited by J. Grasman and G. van Straten, Kluwer, Dordrecht, The Netherlands, 1994, 322-333. (Preprinted in Kwantitatieve Methoden, 15, 45, February 1994, 3-15.)
48. Kleijnen, J. P. C. (1995a) Verification and validation of simulation models. European Journal of Operational Research, 82, 145-162.
49. Kleijnen, J. P. C. (1995b) Sensitivity analysis and optimization in simulation: design of experiments and case studies. Proceedings of the 1995 Winter Simulation Conference (edited by C. Alexopoulos, K. Kang, W.R. Lilegdon, D. Goldsman).
50. Kleijnen, J. P. C. (1995c) Case-study: statistical validation of simulation models. European Journal of Operational Research, 87, 21-34.
51. Kleijnen, J. P. C. (1995d) Sensitivity analysis and optimization of system dynamics models: regression analysis and statistical design of experiments. System Dynamics Review, 11(4), 1-14.
52. Kleijnen, J. P. C. and Rubinstein, R. Y. (1996) Optimization and sensitivity analysis of computer simulation models by the score function method. European Journal of Operational Research, 1, 1-15.
53. Kleijnen, J. P. C., Bettonvil, B. and Van Groenendaal, W. (1996) Validation of simulation models: a novel regression test. Management Science (accepted).
54. Kleijnen, J. P. C. and Van Groenendaal, W. (1992) Simulation: a Statistical Perspective. Wiley, Chichester.
55. 2 greenhouse effect. Simulation, 58(6), 410-417.
56. Kleijnen, J. P. C. and Sargent, R. (1996) Metamodelin methodology, Tilburg University, Tilburg, Netherlands.
57. Kraan, B. C. P. and Cooke, R. M. (1995) Joint CEC/USNRC accident consequence code uncertainty analysis using expert judgement. This issue
58. Krumm, F. V. and Rolle, C. F. (1992), Management and application of decision and risk analysis in Du Pont. Interface, 22(6), 84-93.
59. Li, C. H. (1962) A sequential method for screening experimental variables, American Statistical Association Journal, 57, 455-477.
60. McKay, M. D. (1992) Latin hypercube sampling as a tool in uncertainty analysis of computer models. 1992 Winter Simulation Conference Proceedings, Association for Computing Machinery, New York.
61. McKay, M. D. (1995) Evaluating prediction uncertainty. Los Alamos National Laboratory, NUREG/CR-6311 (LA-12915-MS).
62. Moors, J. J. A., Schuld, M. H. and Mathijssen, A. C. A. (1995) A new method for assessing judgmental distributions. Research Memorandum FEW 708, Tilburg University, Tilburg, Netherlands.
63. Morris, M. D. (1987) Two-stage factor screening procedures using multiple grouping assignments, Communications in Statistics: Theory and Methods. 16, 3051-3067.
64. Morris, M. D. (1991) Factorial plans for preliminary computational experiments, Technometrics, 33(2), 161-174.
65. Olivi, L. (1980) Response Surface Methodology in Risk Analysis; Synthesis and Analysis Methods for Safety and Reliability Studies. Edited by G. Apostolakis, S. Garibra and G. Volta, Plenum Publishing Corporation, New York.
66. Oren, T. I. (1993) Three simulation experimentation environments: SIMAD, SIMGEST, and E/SLAM. In Proceedings of the 1993 European Simulation Symposium. La Jolla: Society for Computer Simulation.
67. Pacheco, N. S. (1988) Session III: simulation certification, verification and validation, SDI Testing: the Road to Success: 1988 Symposium Proceedings International Test & Evaluation Association. ITEA. Fairfax (Virginia 22033).
68. Patel, M. S. (1962) Group screening with more than two stages, Technometrics, 4(2), 209-217.
69. Rahni, N., Ramdani, N., Candau, Y. and Dalicieux, P. (1995) Sensitivity analysis of dynamic buildings energy simulation models using group screening and sampling methods. This issue.
70. Rao, G. P. and Sarkar, P. K. (1995) Sensitivity studies of air scattered neutron dose from particle accelerators. This issue.
71. 2 emissions model. The Energy Journal. 8(3), 1-29.
72. Reilly, T. (1996) The effect of correlated variables on a one-way sensitivity analysis. This issue.
73. Rotmans, J. (1990) IMAGE: An Integrated Model to Assess the Greenhouse Effect. Kluwer, Dordrecht, Netherlands.
74. Rubinstein, R. Y. and Shapiro, A. (1993) Discrete Event Systems: Sensitivity Analysis and Stochastic Optimization via the Score Function Method, Wiley, New York.
75. Sacks, J., Welch, W. J., Mitchell, T. J. and Wynn, H. R. (1989) Design and analysis of computer experiment (includes comments and rejoinder). Statistical Science, 4(4), 409-431
76. Saltelli, A. (1996) On the use of rank transfoimation in sensitivity analysis. This issue.
77. Saltelli, A., Andres, T. H. and Homma, T. (1993) Sensitivity analysis of model output: an investigation of new techniques. Computational Statistics and Data Analysis, 15, 211-238.
78. Saltelli, A. and Homma, T. (1992) Sensitivity analysis for model output: performance of black box techniques on three international benchmark exercises. Computational Statistics and Data Analysis, 13 73-94.
79. Sarkar, P. K. and Rief, H. (1995) Differential operator sampling for self-optimizing Monte Carlo simulations. This issue.
80. Sobol, I. M. (1995) Sensitivity analysis of non-linear models using sensitivity indices. This issue.
81. Van Asselt, M. and Rotmans, J. (1995) Uncertainty in integrated assessment modelling: a cultural perspective based approach. RIVM, Bilthoven, Netherlands.
82. Van Groenendaal, W. (1994) Investment Analysis and DSS for Gas Transmission on Java. Tilburg(Netherlands): Tilburg University.
83. Van Groenendaal, W. and Kleijnen J. P. C. (1996) Economic risk versus technological risk in large investment projects Journal of Reliability and Systems Safety (accepted).
84. Van Meel, J. (1994) The Dynamics of Business Engineering. Delft(Netherlands): Delft University.
85. Vollebregt, T. (1996) Experimental Design for Simulation. Canterbury (New Zealand): University of Canterbury.
86. Watson, G. S. (1961) A study of the group screening method, Technometrics, 3(3), 371-388.
87. Winkler, R. L. (1996) Uncertainty ir probabilistic risk assessment. Reliability Engineering and System Safety, Special Issue, edited by J. Helton and D. Burmaster.
88. Zeigler, B. (1976) Theory of Modelling and Simulation. Wiley Interscience, New York.