Statistical analyses of scatterplots to identify important factors in large-scale simulations, 1: Review and comparison of techniques

Reliability Engineering and System Safety

Volumn 65, Issue 2, 1999, Pages 147-185

  Kleijnen, J P C ^a   Helton, J C ^b  

^a TILBURG UNIVERSITY   (Netherlands)

^b ARIZONA STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CORRELATION METHODS; MONTE CARLO METHODS; REGRESSION ANALYSIS; RISK ASSESSMENT; SENSITIVITY ANALYSIS;

EPISTEMIC UNCERTAINTY; KRUSKAL-WALLIS STATISTICS; LATIN HYPERCUBE SAMPLING; SCATTERPLOTS;

ACCIDENT PREVENTION;

EID: 0032632420     PISSN: 09518320     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0951-8320(98)00091-X     Document Type: Article

References (58)

1. National Research Council. Risk assessment in the Federal Government: managing the process. Washington, DC: National Academy Press, 1983.
2. National Research Council. Issues in risk assessment. Washington, DC: National Academy Press, 1993.
3. National Research Council. Science and judgment in risk assessment. Washington, DC: National Academy Press, 1994.
4. International Atomic Energy Agency. Evaluating the reliability of predictions made using environmental transfer models. Safety Series Report No. 100. Vienna, Austria: International Atomic Energy Agency, 1989.
5. U.S. Environmental Protection Agency. An SAB report: multi-media risk assessment for radon, review of uncertainty analysis of risks associated with exposure to radon. EPA-SAB-RAC-93-014. Washington, DC: U.S. Environmental Protection Agency, 1993.
6. National Council on Radiation Protection and Measurements. A guide for uncertainty analysis in dose and risk assessments related to environmental contamination. NCRP Commentary No. 14. Bethesda, MD: National Council on Radiation Protection and Measurements, 1996.
7. Iman R.L., Helton J.C. An investigation of uncertainty and sensitivity analysis techniques for computer models. Risk Analysis. 8:1988;71-90.
8. Helton J.C. Uncertainty and sensitivity analysis techniques for use in performance assessment of radioactive waste disposal. Reliability Engineering and System Safety. 42:1993;327-367.
9. Hamby D.M. A review of techniques for parameter sensitivity analysis of environmental models. Environmental Monitoring and Assessment. 32:1994;125-154.
10. McKay M.D., Beckman R.J., Conover W.J. A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics. 21:1979;239-245.
11. Iman R.L., Helton J.C., Campbell J.E. An approach to sensitivity analysis of computer models, part 1. Introduction, input variable selection and preliminary variable assessment. Journal of Quality Technology. 13:1981;174-183.
12. Iman R.L., Helton J.C., Campbell J.E. An approach to sensitivity analysis of computer models, part 2. Ranking of input variables, response surface validation, distribution effect and technique synopsis. Journal of Quality Technology. 13:1981;232-240.
13. Saltelli A., Marivoet J. Non-parametric statistics in sensitivity analysis for model output: a comparison of selected techniques. Reliability Engineering and System Safety. 28:1990;229-253.
14. Saltelli A., Homma T. Sensitivity analysis for model output: performance of black box techniques on three international benchmark exercises. Computational Statistics and Data Analysis. 13:1992;73-94.
15. Saltelli A., Andres T.H., Homma T. Sensitivity analysis of model output: an investigation of new techniques. Computational Statistics and Data Analysis. 15:1993;211-238.
16. Hamby D.M. A comparison of sensitivity analysis techniques. Health Physics. 68:1995;195-204.
17. Kleijnen J.P.C. Experimental design. Banks J. Handbook of simulation. 1998;173-223 John Wiley and Sons Inc, New York.
18. Helton JC, Bean JE, Berglund JW, Davis FJ, Economy K, Garner JW, Johnson JD, MacKinnon RJ, Miller J, O'Brien DG, Ramsey JL, Schreiber JD, Shinta A, Smith LN, Stoelzel DM, Stockman C, Vaughn P. Uncertainty and sensitivity analysis results obtained in the 1996 performance assessment for the Waste Isolation Pilot Plant. SAND98-0365. Albuquerque, NM: Sandia National Laboratories, 1998.
19. Iman R.L., Conover W.J. The use of the rank transform in regression. Technometrics. 21:1979;499-509.
20. Iman R.L., Conover W.J. A measure of top-down correlation. Technometrics. 29:(3):1987;351-357.
21. Kleijnen J.P.C., Helton J.C. Statistical analyses of scatterplots to identify important factors in large-scale simulations, 2: robustness of techniques. Reliability Engineering and System Safety. 65:1998;187-198.
22. U.S. Department of Energy. Title 40 CFR Part 191 compliance certification application for the Waste Isolation Pilot Plant. DOE/CAO-1996-2184: Carlsbad, NM: U.S. Department of Energy, Carlsbad Area Office, 1996.
23. U.S. Environmental Protection Agency. 40 CFR Part 194: Criteria for the certification and re-certification of the Waste Isolation Pilot Plant's compliance with the 40 CFR Part 191 disposal regulations; final rule. Federal Register 61: 5224-5245, 1996.
24. Iman R.L., Conover W.J. A distribution-free approach to inducing rank correlation among input variables. Communications in Statistics B. 11:1982;311-334.
25. Iman RL, Shortencarier MJ. A FORTRAN 77 program and user's guide for the generation of Latin hypercube and random samples for use with computer models. NUREG/CR-2624, SAND83-2365. Albuquerque, NM: Sandia National Laboratories, 1984.
26. Iman RL, Davenport JM, Frost EL, Shortencarier MJ. Stepwise regression with PRESS and rank regression (program user's guide). SAND79-1472. Albuquerque, NM: Sandia National Laboratories, 1980.
27. Draper N.R., Smith H. Applied regression analysis. 2nd edition :1981;John Wiley and Sons, Inc, New York.
28. Myers R.H. Classical and modern regression with applications. 1989;Duxbury Press, Belmont, CA.
29. Press W.H., Teukolsky S.A., Vetterlings W.T., Flannery B.P. Numerical recipes: the art of scientific computing (FORTRAN version). 2nd edition :1992;Cambridge University Press, Cambridge.
30. Chan M.S. The consequences of uncertainty for the prediction of the effects of schistosomiasis control programmes. Epidemiology and Infection. 117:1996;537-550.
31. Ma J.Z., Ackerman E., Yang J.J. Parameter sensitivity of a model of viral epidemics simulated with Monte Carlo techniques. I. Illness Attack Rates. International Journal of Biomedical Computing. 32:1993;237-253.
32. Ma J.Z., Ackerman E. Parameter sensitivity of a model of viral epidemics simulated with Monte Carlo techniques, II. Durations and peaks. International Journal of Biomedical Computing. 32:1993;255-268.
33. Whiting W.B., Tong T.M., Reed M.E. Effect of uncertainties in thermodynamic data and model parameters on calculated process performance. Industrial and Engineering Chemistry Research. 32:1993;1367-1371.
34. Breshears D., Kirchner T.B., Whicker F.W. Contaminant transport through agro-ecosystems: assessing relative importance of environmental, physiological and management factors. Ecological Applications. 2:(3):1992;285-297.
35. Iman RL, Shortencarier MJ, Johnson JD. A FORTRAN 77 program and user's guide for the calculation of partial correlation and standardized regression coefficients. NUUREG/CR-4122. SAND85-0044. Albuquerque, NM: Sandia National Laboratories, 1985.
36. Helton JC, Garner JW, McCurley RD, Rudeen DK. Sensitivity analysis techniques and results for performance assessment at the Waste Isolation Pilot Plant. SAND90-7103. Albuquerque, NM: Sandia National Laboratories, 1991.
37. Conover W.J. Practical non-parametric statistics. 2nd edition :1980;John Wiley and Sons Inc, New York.
38. Sanchez M.A., Blower S.M. Uncertainty and sensitivity analysis of the basic reproductive rate: tuberculosis as an example. American Journal of Epidemiology. 145:(12):1997;1127-1137.
39. Gwo J.P., Toran L.E., Morris M.D., Wilson G.V. Subsurface stormflow modeling with sensitivity analysis using a Latin-hypercube sampling technique. Groundwater. 34:1996;811-818.
40. Helton J.C., et al. Uncertanity and sensitivity analysis results obtained in the 1992 performance assessment for the Waste Isolation Pilot Plant. Reliability Engineering and System Safety. 51:1996;53-100.
41. Blower S.M., Dowlatabadi H. Sensitivity and uncertainty analysis of complex models of disease transmission: an HIV model, as an example. International Statistical Review. 62:(2):1994;229-243.
42. MacDonald R.C., Campbell J.E. Valuation of supplemental and enhanced oil recovery projects with risk analysis. Journal of Petroleum Technology. 38:1986;57-69.
43. Andrews D.F., Bickel P.J., Hample F.R., Huber P., Rogers W., Tukey J. Robust estimates of location. 1972;Princeton University Press, Princeton.
44. Scheffé H. The analysis of variance. 1959;John Wiley and Sons, Inc, New York.
45. Kleijnen J.P.C. Statistical tools for simulation practitioners. 1987;Marcel Dekker, New York.
46. D'Agostino R.D., Stephens H.A. Goodness-of-fit distributions. 1986;Marcel Dekker, New York.
47. David H.A. Order statistics. 2nd edition :1981;John Wiley and Sons, New York.
48. Sargent RG. Some subjective validation methods using graphical displays of data. In: Charens JM, Morrice DJ, Brunner DT, Swain JJ, editors. 1996 Winter Simulation Conference Proceedings. Piscataway, NJ: Institute of Electrical and Electronics Engineers, 1996, pp. 345-351.
49. Conover W.J., Johnson M.E., Johnson M.M. A comparative study of tests for homogeneity of variances, with applications to the outer continental shelf bidding data. Technometrics. 23:(1):1981;351-361.
50. Piepho H.-P. Tests for equality of dispersion in bivariate samples - review and empirical comparison. Journal of Statistical Computation and Simulation. 56:1997;353-372.
51. Wludyka P.S., Nelson P.R. Analysis of means type tests for variances using subsampling and jackknifing. American Journal of Mathematical and Management Sciences. 17:1997;31-60.
52. Efron B. The jackknife, the bootstrap and other resampling plans. Philadelphia: CBMS-NSF Series, Siam, 1982.
53. Miller R.G. The jackknife - a review. Biometrika. 61:1974;1-15.
54. Archer G.E.B., Saltelli A., Sobol T.M. Sensitivity measures, ANOVA-like techniques and the use of bootstrap. Journal of Statistical Computation and Simulation. 58:1997;99-120.
55. Helton J.C. Uncertainty and sensitivity analysis in the presence of stochastic and subjective uncertainty. Journal of Statistical Computation and Simulation. 57:1997;3-76.
56. Iman R.L. Tables of the exact quantiles of the top-down correlation coefficient for n=3(1)14. Communications in Statistics - Theory and Methods. 16:1987;1513-1540.
57. Efron B., Tibshirani R.J. Introduction to the bootstrap. 1993;Chapman and Hall, London.
58. Kleijnen JPC, Helton JC. Statistical analyses of scatterplots to identify important factors in large-scale simulations. SAND98-2202. Albuquerque, NM: Sandia National Laboratories, 1999.