Survey of sampling-based methods for uncertainty and sensitivity analysis

Reliability Engineering and System Safety

Volumn 91, Issue 10-11, 2006, Pages 1175-1209

  Helton, J C ^a   Johnson, J D ^b   Sallaberry, C J ^c   Storlie, C B ^d  

^a ARIZONA STATE UNIVERSITY   (United States)

^b ProStat   (United States)

^c SANDIA NATIONAL LABORATORIES   (United States)

^d Department of Environmental and Radiological Health Sciences   (United States)

Author keywords

Aleatory uncertainty; Epistemic uncertainty; Latin hypercube sampling; Monte Carlo; Sensitivity analysis; Uncertainty analysis

Indexed keywords

AUTOMATIC PROGRAMMING; CORRELATION THEORY; DISTANCE MEASUREMENT; ENTROPY; MONTE CARLO METHODS; PROBABILITY DISTRIBUTIONS; REGRESSION ANALYSIS; STATISTICAL TESTS;

ALEATORY UNCERTAINTY; EPISTEMIC UNCERTAINTY; LATIN HYPERCUBE SAMPLING; UNCERTAINTY ANALYSIS;

SENSITIVITY ANALYSIS;

EID: 33745727561     PISSN: 09518320     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ress.2005.11.017     Document Type: Article

References (217)

1. Christie M.A., Glimm J., Grove J.W., Higdon D.M., Sharp D.H., and Wood-Schultz M.M. Error analysis and simulations of complex phenomena. Los Alamos Sci 29 (2005) 6-25
2. In: Nikolaidis E., Ghiocel D.M., and Singhal S. (Eds). Engineering design reliability handbook (2004), CRC Press, Boca Raton, FL
3. Sharp D.H., and Wood-Schultz M.M. QMU and nuclear weapons certification: what's under the hood?. Los Alamos Sci 28 (2003) 47-53
4. Wagner R.L. Science, uncertainty and risk: the problem of complex phenomena. APS News 12 1 (2003) 8
5. Oberkampf W.L., DeLand S.M., Rutherford B.M., Diegert K.V., and Alvin K.F. Error and uncertainty in modeling and simulation. Reliab Eng Syst Safety 75 3 (2002) 333-357
6. Roache P.J. Verification and validation in computational science and engineering (1998), Hermosa Publishers, Albuquerque, NM
7. In: Ayyub B.M. (Ed). Uncertainty modeling and analysis in civil engineering (1997), CRC Press, Boca Raton, FL
8. Risk Assessment Forum. Guiding principles for Monte Carlo analysis, EPA/630/R-97/001 (1997), RiskAssessmentForum Environmental Protection Agency, Washington, DC Available from the NTIS as PB97-188106/XAB
9. NCRP (National Council on Radiation Protection and Measurements). A guide for uncertainty analysis in dose and risk assessments related to environmental contamination, NCRP Commentary No. 14 (1996), National Council on Radiation Protection and Measurements, Bethesda, MD
10. NRC (National Research Council). Science and judgment in risk assessment (1994), National Academy Press, Washington, DC
11. NRC (National Research Council). Issues in risk assessment (1993), National Academy Press, Washington, DC
12. US EPA (US Environmental Protection Agency). An SAB Report: multi-media risk assessment for radon, review of uncertainty analysis of risks associated with exposure to radon, EPA-SAB-RAC-93-014 (1993), US Environmental Protection Agency, Washington, DC
13. IAEA (International Atomic Energy Agency). Evaluating the reliability of predictions made using environmental transfer models, safety series no. 100 (1989), International Atomic Energy Agency, Vienna
14. Beck M.B. Water-quality modeling: a review of the analysis of uncertainty. Water Resour Res 23 8 (1987) 1393-1442
15. Helton J.C. Uncertainty and sensitivity analysis in the presence of stochastic and subjective uncertainty. J Stat Comput Simul 57 1-4 (1997) 3-76
16. In: Helton J.C., and Burmaster D.E. (Eds). Guest editorial: treatment of aleatory and epistemic uncertainty in performance assessments for complex systems. Reliab Eng Syst Safety 54 2-3 (1996) 91-94
17. Paté-Cornell M.E. Uncertainties in Risk Anal: six levels of treatment. Reliab Eng Syst Safety 54 2-3 (1996) 95-111
18. Winkler R.L. Uncertainty in probabilistic risk assessment. Reliab Eng Syst Safety 54 2-3 (1996) 127-132
19. Hoffman F.O., and Hammonds J.S. Propagation of uncertainty in risk assessments: the need to distinguish between uncertainty due to lack of knowledge and uncertainty due to variability. Risk Anal 14 5 (1994) 707-712
20. Helton J.C. Treatment of uncertainty in performance assessments for complex systems. Risk Anal 14 4 (1994) 483-511
21. Apostolakis G. The concept of probability in safety assessments of technological systems. Science 250 4986 (1990) 1359-1364
22. Haan C.T. Parametric uncertainty in hydrologic modeling. Trans ASAE 32 1 (1989) 137-146
23. Parry G.W., and Winter P.W. Characterization and evaluation of uncertainty in probabilistic Risk Anal. Nucl Safety 22 1 (1981) 28-42 [DIRS 159059]
24. Kaplan S., and Garrick B.J. On the quantitative definition of risk. Risk Anal 1 1 (1981) 11-27
25. Cacuci D.G. Sensitivity and uncertainty analysis, vol. 1: theory (2003), Chapman & Hall/CRC Press, Boca Raton, FL
26. Griewank A. Evaluating derivatives: principles and techniques of algorithmic differentiation (2000), Society for Applied and Industrial Mathematics, Philadelphia, PA
27. Berz M., Bischof C., Corliss G., and Griewank A. Computational differentiation: techniques, applications, and tools (1996), Society for Industrial and Applied Mathematics, Philadelphia, PA
28. Cacuci D.G., and Schlesinger M.E. On the application of the adjoint method of sensitivity analysis to problems in the atmospheric sciences. Atmósfera 7 1 (1994) 47-59
29. Turányi T. Sensitivity analysis of complex kinetic systems, tools and applications. J Math Chem 5 3 (1990) 203-248
30. Rabitz H., Kramer M., and Dacol D. Sensitivity analysis in chemical kinetics. In: Rabinovitch B.S., Schurr J.M., and Strauss H.L. (Eds). Annual review of physical chemistry vol. 34 (1983), Annual Reviews Inc., Palo Alto, CA 419-461
31. In: Lewins J., and Becker M. (Eds). Sensitivity and uncertainty analysis of reactor performance parameters. Advances in nuclear science and technology vol. 14 (1982), Plenum Press, New York, NY
32. Frank P.M. Introduction to system sensitivity theory (1978), Academic Press, New York, NY
33. Tomovic R., and Vukobratovic M. General sensitivity theory (1972), Elsevier, New York, NY
34. Myers R.H., Montgomery D.C., Vining G.G., Borror C.M., and Kowalski S.M. Response surface methodology: a retrospective and literature review. J Quality Technol 36 1 (2004) 53-77
35. Myers R.H. Response surface methodology-current status and future directions. J Quality Technol 31 1 (1999) 30-44
36. Andres T.H. Sampling methods and sensitivity analysis for large parameter sets. J Stat Comput Simul 57 1-4 (1997) 77-110
37. Kleijnen J.P.C. Sensitivity analysis and related analyses: a review of some statistical techniques. J Stat Comput Simul 57 1-4 (1997) 111-142
38. Kleijnen J.P.C. Sensitivity analysis of simulation experiments: regression analysis and statistical design. Math Comput Simul 34 3-4 (1992) 297-315
39. Myers R.H., Khuri A.I., Carter J., and Walter H. Response surface methodology: 1966-1988. Technometrics 31 2 (1989) 137-157
40. Sacks J., Welch W.J., Mitchel T.J., and Wynn H.P. Design and analysis of computer experiments. Stat Sci 4 4 (1989) 409-435
41. Morton R.H. Response surface methodology. Math Sci 8 (1983) 31-52
42. Mead R., and Pike D.J. A review of response surface methodology from a biometric viewpoint. Biometrics 31 (1975) 803-851
43. Myers R.H. Response surface methodology (1971), Allyn & Bacon, Boston, MA
44. Helton J.C., and Davis F.J. Latin hypercube sampling and the propagation of uncertainty in analyses of complex systems. Reliab Eng Syst Safety 81 1 (2003) 23-69
45. Helton J.C., and Davis F.J. Illustration of sampling-based methods for uncertainty and sensitivity analysis. Risk Anal 22 3 (2002) 591-622
46. Helton J.C., and Davis F.J. Sampling-based methods. In: Saltelli A., Shan K., and Scott E.M. (Eds). Sensitivity analysis (2000), Wiley, New York, NY 101-153
47. Kleijnen J.P.C., and Helton J.C. Statistical analyses of scatterplots to identify important factors in large-scale simulations, 1: review and comparison of techniques. Reliab Eng Syst Safety 65 2 (1999) 147-185
48. Blower S.M., and Dowlatabadi H. Sensitivity and uncertainty analysis of complex models of disease transmission: an HIV model, as an example. Int Stat Rev 62 2 (1994) 229-243
49. Saltelli A., Andres T.H., and Homma T. Sensitivity analysis of model output. An investigation of new techniques. Comput Stat Data Anal 15 2 (1993) 445-460
50. Iman R.L. Uncertainty and sensitivity analysis for computer modeling applications. In: Cruse T.A. (Ed). Reliability technology-1992, the winter annual meeting of the American society of mechanical engineers, Anaheim, California, November 8-13 vol. 28 (1992), American Society of Mechanical Engineers, Aerospace Division, New York, NY 153-168
51. Saltelli A., and Marivoet J. Non-parametric statistics in sensitivity analysis for model output. A comparison of selected techniques. Reliab Eng Syst Safety 28 2 (1990) 229-253
52. Iman R.L., Helton J.C., and Campbell J.E. An approach to sensitivity analysis of computer models, Part 1. Introduction, input variable selection and preliminary variable assessment. J Quality Technol 13 3 (1981) 174-183
53. Iman R.L., Helton J.C., and Campbell J.E. An approach to sensitivity analysis of computer models, Part 2. Ranking of input variables, response surface validation, distribution effect and technique synopsis. J Quality Technol 13 4 (1981) 232-240
54. Iman R.L., and Conover W.J. Small sample sensitivity analysis techniques for computer models, with an application to risk assessment. Commun Stat: Theory Methods A 9 17 (1980) 1749-1842
55. McKay M.D., Beckman R.J., and Conover W.J. A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21 2 (1979) 239-245
56. Li G., Rosenthal C., and Rabitz H. High-dimensional model representations. J Phys Chem 105 33 (2001) 7765-7777
57. Rabitz H., and Alis O.F. General foundations of high-dimensional model representations. J Math Chem 25 2-3 (1999) 197-233
58. Saltelli A., Tarantola S., and Chan K.P.-S. A quantitative model-independent method for global sensitivity analysis of model output. Technometrics 41 1 (1999) 39-56
59. Sobol' I.M. Sensitivity estimates for nonlinear mathematical models. Math Modeling Comput Exp 1 4 (1993) 407-414
60. Cukier R.I., Levine H.B., and Shuler K.E. Nonlinear sensitivity analysis of multiparameter model systems. J Comput Phys 26 1 (1978) 1-42
61. Ionescu-Bujor M., and Cacuci D.G. A comparative review of sensitivity and uncertainty analysis of large-scale systems-I: deterministic methods. Nucl Sci Eng 147 3 (2004) 189-2003
62. Cacuci D.G., and Ionescu-Bujor M. A comparative review of sensitivity and uncertainty analysis of large-scale systems-II: statistical methods. Nucl Sci Eng 147 3 (2004) 204-217
63. Frey H.C., and Patil S.R. Identification and review of sensitivity analysis methods. Risk Anal 22 3 (2002) 553-578
64. In: Saltelli A., Chan K., and Scott E.M. (Eds). Sensitivity analysis (2000), Wiley, New York, NY
65. Hamby D.M. A review of techniques for parameter sensitivity analysis of environmental models. Environ Monitor Assess 32 2 (1994) 135-154
66. Helton J.C. Uncertainty and sensitivity analysis techniques for use in performance assessment for radioactive waste disposal. Reliab Eng Syst Safety 42 2-3 (1993) 327-367
67. Ronen Y. Uncertainty analysis (1988), CRC Press, Inc., Boca Raton, FL
68. Iman R.L., and Helton J.C. An investigation of uncertainty and sensitivity analysis techniques for computer models. Risk Anal 8 1 (1988) 71-90
69. Blower S.M., Gershengorn H.B., and Grant R.M. A tale of two futures: HIV and antiretroviral therapy in San Francisco. Science 287 5453 (2000) 650-654
70. Cohen C., Artois M., and Pontier D. A discrete-event computer model of feline herpes virus within cat populations. Prevent Vet Med 45 3-4 (2000) 163-181
71. Hofer E. Sensitivity analysis in the context of uncertainty analysis for computationally intensive models. Comput Phys Commun 117 1-2 (1999) 21-34
72. Caswell H., Brault S., Read A.J., and Smith T.D. Harbor porpoise and fisheries: an uncertainty analysis of incidental mortality. Ecol Appl 8 4 (1998) 1226-1238
73. Sanchez M.A., and Blower S.M. Uncertainty and sensitivity analysis of the basic reproductive rate: tuberculosis as an example. Am J Epidemiol 145 12 (1997) 1127-1137
74. Chan M.S. The consequences of uncertainty for the prediction of the effects of schistosomiasis control programmes. Epidemiol Infect 117 3 (1996) 537-550
75. Gwo J.P., Toran L.E., Morris M.D., and Wilson G.V. Subsurface stormflow modeling with sensitivity analysis using a latin-hypercube sampling technique. Ground Water 34 5 (1996) 811-818
76. Helton J.C., Anderson D.R., Baker B.L., Bean J.E., Berglund J.W., Beyeler W., et al. Uncertainty and sensitivity analysis results obtained in the 1992 performance assessment for the waste isolation pilot plant. Reliab Eng Syst Safety 51 1 (1996) 53-100
77. Kolev N.I., and Hofer E. Uncertainty and sensitivity analysis of a postexperiment simulation of nonexplosive melt-water interaction. Exp Thermal Fluid Sci 13 2 (1996) 98-116
78. Whiting W.B., Tong T.-M., and Reed M.E. Effect of uncertainties in thermodynamic data and model parameters on calculated process performance. Ind Eng Chem Res 32 7 (1993) 1367-1371
79. Ma J.Z., Ackerman E., and Yang J.-J. Parameter sensitivity of a model of viral epidemics simulated with Monte Carlo techniques. I. Illness attack rates. Int J Biomed Comput 32 3-4 (1993) 237-253
80. Ma J.Z., and Ackerman E. Parameter sensitivity of a model of viral epidemics simulated with Monte Carlo techniques. II. Durations and peaks. Int J Biomed Comput 32 3-4 (1993) 255-268
81. Breshears D.D., Kirchner T.B., and Whicker F.W. Contaminant transport through agroecosystems: assessing relative importance of environmental, physiological, and management factors. Ecol Appl 2 3 (1992) 285-297
82. Breeding R.J., Helton J.C., Gorham E.D., and Harper F.T. Summary description of the methods used in the probabilistic risk assessments for NUREG-1150. Nucl Eng Des 135 1 (1992) 1-27
83. MacDonald R.C., and Campbell J.E. Valuation of supplemental and enhanced oil recovery projects with risk analysis. J Petroleum Technol 38 1 (1986) 57-69
84. Helton J.C., Johnson J.D., and Oberkampf W.L. An exploration of alternative approaches to the representation of uncertainty in model predictions. Reliab Eng Syst Safety 85 1-3 (2004) 39-71
85. Klir G.J. Generalized information theory: aims, results, and open problems. Reliab Eng Syst Safety 85 1-3 (2004) 21-38
86. Ross T.J. Fuzzy logic with engineering applications. 2nd ed (2004), Wiley, New York, NY
87. Halpern J.Y. Reasoning about uncertainty (2003), MIT Press, Cambridge, MA
88. In: Ross T.J., Booker J.M., and Parkinson W.J. (Eds). Fuzzy logic and probability applications: bridging the gap (2002), Society for Industrial and Applied Mathematics, Philadelphia, PA
89. Jaulin L., Kieffer M., Didrit O., and Walter E. Applied interval analysis (2001), Springer, New York, NY
90. Wolkenhauer O. Data engineering: fuzzy mathematics in systems theory and data analysis (2001), Wiley, New York, NY
91. Klir G.J., and Wierman M.J. Uncertainty-based information (1999), Physica-Verlag, New York, NY
92. In: Yager R.R., Kacprzyk J., and Fedrizzi M. (Eds). Advances in the dempster-shafer theory of evidence (1994), Wiley, New York, NY
93. Cooke R.M., and Goossens L.H.J. Expert judgement elicitation for risk assessment of critical infrastructures. J Risk Res 7 6 (2004) 643-656
94. Ayyub B.M. Elicitation of expert opinions for uncertainty and risks (2001), CRC Press, Boca Raton, FL
95. McKay M., and Meyer M. Critique of and limitations on the use of expert judgements in accident consequence uncertainty analysis. Radiat Protect Dosim 90 3 (2000) 325-330
96. Budnitz R.J., Apostolakis G., Boore D.M., Cluff L.S., Coppersmith K.J., Cornell C.A., et al. Use of technical expert panels: applications to probabilistic seismic hazard analysis. Risk Anal 18 4 (1998) 463-469
97. Thorne M.C., and Williams M.M.R. A review of expert judgement techniques with reference to nuclear safety. Progr Nucl Safety 27 2-3 (1992) 83-254
98. Cooke R.M. Experts in uncertainty: opinion and subjective probability in science (1991), Oxford University Press, Oxford; NY
99. Meyer M.A., and Booker J.M. Eliciting and analyzing expert judgment: a practical guide (1991), Academic Press, New York, NY
100. Hora S.C., and Iman R.L. Expert opinion in risk analysis: the NUREG-1150 methodology. Nucl Sci Eng 102 4 (1989) 323-331
101. Helton J.C., and Davis F.J. Sampling-based methods for uncertainty and sensitivity analysis, SAND99-2240 (2000), Sandia National Laboratories, Albuquerque, NM
102. Vaughn P., Bean J.E., Helton J.C., Lord M.E., MacKinnon R.J., and Schreiber J.D. Representation of two-phase flow in the vicinity of the repository in the 1996 performance assessment for the waste isolation pilot plant. Reliab Eng Syst Safety 69 1-3 (2000) 205-226
103. Helton J.C., Bean J.E., Economy K., Garner J.W., MacKinnon R.J., Miller J., et al. Uncertainty and sensitivity analysis for two-phase flow in the vicinity of the repository in the 1996 performance assessment for the waste isolation pilot plant: undisturbed conditions. Reliab Eng Syst Safety 69 1-3 (2000) 227-261
104. Helton J.C., Bean J.E., Economy K., Garner J.W., MacKinnon R.J., Miller J., et al. Uncertainty and sensitivity analysis for two-phase flow in the vicinity of the repository in the 1996 performance assessment for the waste isolation pilot plant: disturbed conditions. Reliab Eng Syst Safety 69 1-3 (2000) 263-304
105. US DOE (US Department of Energy). Title 40 CFR Part 191 compliance certification application for the waste isolation pilot plant, DOE/CAO-1996-2184 (1996), US Department of Energy, Carlsbad Area Office, [DIRS 100975]
106. Helton J.C., Bean J.E., Berglund J.W., Davis F.J., Economy K., Garner J.W., et al. Uncertainty and sensitivity analysis results obtained in the 1996 performance assessment for the waste isolation pilot plant, SAND98-0365 (1998), Sandia National Laboratories, Albuquerque, NM
107. Helton J.C., and Marietta M.G. Special issue: the 1996 performance assessment for the waste isolation pilot plant. Reliab Eng Syst Safety 69 1-3 (2000) 1-451
108. Helton J.C., Martell M.-A., and Tierney M.S. Characterization of subjective uncertainty in the 1996 performance assessment for the waste isolation pilot plant. Reliab Eng Syst Safety 69 1-3 (2000) 191-204
109. Goossens L.H.J., Harper F.T., Kraan B.C.P., and Metivier H. Expert judgement for a probabilistic accident consequence uncertainty analysis. Radiat Protect Dosim 90 3 (2000) 295-301
110. Goossens L.H.J., and Harper F.T. Joint EC/USNRC expert judgement driven radiological protection uncertainty analysis. J Radiol Protect 18 4 (1998) 249-264
111. Siu N.O., and Kelly D.L. Bayesian parameter estimation in probabilistic risk assessment. Reliab Eng Syst Safety 62 1-2 (1998) 89-116
112. Evans J.S., Gray G.M., Sielken Jr. R.L., Smith A.E., Valdez-Flores C., and Graham J.D. Use of probabilistic expert judgement in uncertainty analysis of carcinogenic potency. Regul Toxicol Pharmacol 20 1, pt. 1 (1994) 15-36
113. Chhibber S., Apostolakis G., and Okrent D. A taxonomy of issues related to the use of expert judgments in probabilistic safety studies. Reliab Eng Syst Safety 38 1-2 (1992) 27-45
114. Kaplan S. Expert information versus expert opinions: another approach to the problem of eliciting combining using expert knowledge in PRA. Reliab Eng Syst Safety 35 1 (1992) 61-72
115. Otway H., and Winterfeldt D.V. Expert judgement in risk analysis and management: process, context, and pitfalls. Risk Anal 12 1 (1992) 83-93
116. Bonano E.J., and Apostolakis G.E. Theoretical foundations and practical issues for using expert judgments in uncertainty analysis of high-level radioactive waste disposal. Radioactive Waste Manage Nucl Fuel Cycle 16 2 (1991) 137-159
117. Keeney R.L., and Winterfeldt D.V. Eliciting probabilities from experts in complex technical problems. IEEE Trans Eng Manage 38 3 (1991) 191-201
118. Svenson O. on expert judgments in safety analyses in the process industries. Reliab Eng Syst Safety 25 3 (1989) 219-256
119. Mosleh A., Bier V.M., and Apostolakis G. A critique of current practice for the use of expert opinions in probabilistic risk assessment. Reliab Eng Syst Safety 20 1 (1988) 63-85
120. Breeding R.J., Helton J.C., Murfin W.B., Smith L.N., Johnson J.D., Jow H.-N., et al. The NUREG-1150 probabilistic risk assessment for the surry nuclear power station. Nucl Eng Des 135 1 (1992) 29-59
121. Payne Jr. A.C., Breeding R.J., Helton J.C., Smith L.N., Johnson J.D., Jow H.-N., et al. The NUREG-1150 probabilistic risk assessment for the peach bottom atomic power station. Nucl Eng Des 135 1 (1992) 61-94
122. Gregory J.J., Breeding R.J., Helton J.C., Murfin W.B., Higgins S.J., and Shiver A.W. The NUREG-1150 probabilistic risk assessment for the sequoyah nuclear plant. Nucl Eng Des 135 1 (1992) 92-115
123. Brown T.D., Breeding R.J., Helton J.C., Jow H.-N., Higgins S.J., and Shiver A.W. The NUREG-1150 probabilistic risk assessment for the grand gulf nuclear station. Nucl Eng Des 135 1 (1992) 117-137
124. Helton J.C., and Breeding R.J. Calculation of reactor accident safety goals. Reliab Eng Syst Safety 39 2 (1993) 129-158
125. Morris M.D. Three technometrics experimental design classics. Technometrics 42 1 (2000) 26-27
126. Evans M., and Swartz T. Approximating integrals via Monte Carlo and deterministic methods (2000), Oxford University Press, Oxford, NY
127. Hurtado J.E., and Barbat A.H. Monte Carlo techniques in computational stochastic mechanics. Arch Comput Methods Eng 5 1 (1998) 3-330
128. Nicola V.F., Shahabuddin P., and Nakayama M.K. Techniques for fast simulation of models of highly dependable systems. IEEE Trans Reliab 50 3 (2001) 246-264
129. Owen A., and Zhou Y. Safe and effective importance sampling. J Am Stat Assoc 95 449 (2000) 135-143
130. Heidelberger P. Fast simulation of rare events in queueing and reliability models. ACM Trans Modeling Comput Simul 5 1 (1995) 43-85
131. Shahabuddin P. Importance sampling for the simulation of highly reliable Markovian systems. Manage Sci 40 3 (1994) 333-352
132. Goyal A., Shahabuddin P., Heidelberger P., Nicola V.F., and Glynn P.W. A unified framework for simulating Markovian models of highly dependable systems. IEEE Trans Comput 41 1 (1992) 36-51
133. Melchers R.E. Search-based importance sampling. Struct Safety 9 2 (1990) 117-128
134. Glynn P.W., and Iglehart D.L. Importance sampling for stochastic simulations. Manage Sci 35 11 (1989) 1367-1392
135. Iman R.L., and Conover W.J. A distribution-free approach to inducing rank correlation among input variables. Commun Stat: Simul Comput B 11 3 (1982) 311-334
136. Iman R.L., and Davenport J.M. Rank correlation plots for use with correlated input variables. Commun Stat: Simul Comput B 11 3 (1982) 335-360
137. Helton J.C., and Davis F.J. Latin hypercube sampling and the propagation of uncertainty in analyses of complex systems, SAND2001-0417 (2002), Sandia National Laboratories, Albuquerque, NM
138. Helton J.C., Davis F.J., and Johnson J.D. A comparison of uncertainty and sensitivity analysis results obtained with random and Latin hypercube sampling. Reliab Eng Syst Safety 89 3 (2005) 305-330
139. Kleinjen J.P.C. An overview of the design and analysis of simulation experiments for sensitivity analysis. Europ J Operat Res 164 2 (2005) 287-300
140. CRWMS M&O (Civilian Radioactive Waste Management System Management and Operating Contractor). Total system performance assessment (TSPA) model for site recommendation, MDL-WIS-PA-000002 REV 00 (2000), CRWMS M&O, Las Vegas, Nevada
141. CRWMS M&O (Civilian Radioactive Waste Management System Management and Operating Contractor). Total system performance assessment for the site recommendation, TDR-WIS-PA-000001 REV 00 (2000), CRWMS M&O, Las Vegas, Nevada
142. U.S. DOE (U.S. Department of Energy). Viability assessment of a repository at Yucca mountain, DOE/RW-0508 (1998), US Department of Energy, Office of Civilian Radioactive Waste Management, Washington, DC
143. Ibrekk H., and Morgan M.G. Graphical communication of uncertain quantities to nontechnical people. Risk Anal 7 4 (1987) 519-529
144. Tufte E.R. The visual display of quantitative information (1983), Graphics Press, Cheshire, CN
145. Berglund J.W., Garner J.W., Helton J.C., Johnson J.D., and Smith L.N. Direct releases to the surface and associated complementary cumulative distribution functions in the 1996 performance assessment for the waste isolation pilot plant: cuttings, cavings and spallings. Reliab Eng Syst Safety 69 1-3 (2000) 305-330
146. Cooke R.M., and van Noortwijk J.M. Graphical methods. In: Saltelli A., Chan K., and Scott E.M. (Eds). Sensitivity analysis (2000), Wiley, New York, NY 245-264
147. Myers R.H. Classical and modern regression with applications. 2nd ed (1989), Duxbury Press, Boston, MA
148. Draper N.R., and Smith H. Applied regression analysis. 2nd ed (1981), Wiley, New York, NY
149. Daniel C., Wood F.S., and Gorman J.W. Fitting equations to data: computer analysis of multifactor data. 2nd ed (1980), Wiley, New York, NY
150. Seber G.A. Linear regression analysis (1977), Wiley, New York, NY
151. Neter J., and Wasserman W. Applied linear statistical models: regression, analysis of variance, and experimental designs (1974), Richard D. Irwin, Homewood
152. Iman R.L., Shortencarier M.J., and Johnson J.D. A FORTRAN 77 program and user's guide for the calculation of partial correlation and standardized regression coefficients, NUREG/CR-4122, SAND85-0044 (1985), Sandia National Laboratories, Albuquerque
153. Iman R.L., and Conover W.J. The use of the rank transform in regression. Technometrics 21 4 (1979) 499-509
154. Scheffé H. The analysis of variance (1959), Wiley, New York, NY
155. Conover W.J. Practical nonparametric statistics. 2nd ed (1980), Wiley, New York, NY
156. Barnard G.A. Contribution to the discussion of professor Bartlett's paper. J R Stat Soc Ser B 26 (1963) 294
157. Press W.H., Teukolsky S.A., Vetterling W.T., and Flannery B.P. Numerical recipes in FORTRAN: the art of scientific computing. 2nd ed (1992), Cambridge University Press, Cambridge, NY
158. Kleijnen J.P.C., and Helton J.C. Statistical analyses of scatterplots to identify important factors in large-scale simulations, 2: robustness of techniques. Reliab Eng Syst Safety 65 2 (1999) 187-197
159. Kleijnen J.P.C., and Helton J.C. Statistical analyses of scatterplots to identify important factors in large-scale simulations, SAND98-2202 (1999), Sandia National Laboratories, Albuquerque
160. Wagner H.M. Global sensitivity analysis. Oper Res 43 (1995) 948-969
161. Granger C., and Lin J.-L. Using the mutual information coefficient to identify lags in nonlinear models. J Time Ser Anal 15 4 (1994) 371-384
162. Mishra S, Knowlton RG. Testing for input-output dependence in performance assessment models. In: Proceedings of the 10th international high-level radioactive waste management conference (IHLRWM), March 30-April 2, 2003. Las Vegas, Nevada. La Grange Park, IL: American Nuclear Society; p. 882-7.
163. Bonnlander BV, Weigend AS. Selecting input variables using mutual information and nonparametric density estimation. In: Proceedings of the international symposium on artificial neural networks (ISANN'94), Tainan, Taiwan, 1994. p. 42-50.
164. Moddemeijer R. On the estimation of entropy and mutual information of continuous distributions. Signal Process 16 3 (1989) 233-248
165. Simonoff J.S. Smoothing methods in statistics (1996), Springer, New York, NY
166. Hastie T.J., and Tibshirani R.J. Generalized additive models (1990), Chapman & Hall, London
167. Chambers J.M., and Hastie T.J. Statistical models in S (1992), Wadsworth & Brooks, Pacific Grove, CA
168. Storlie CB, Helton JC. Multiple Predictor smoothing methods for sensitivity analysis, SAND2005-???? Albuquerque, NM: Sandia National Laboratories; to appear.
169. Ruppert D., Ward M.P., and Carroll R.J. Semiparametric regression (2003), Cambridge University Press, Cambridge, MA
170. Breiman L., Friedman J.H., Olshen R.A., and Stone C.J. Classification and regression trees (1984), Wadsworth Intl, Belmont, CA
171. Mishra S., Deeds N.E., and RamaRao B.S. Application of classification trees in the sensitivity analysis of probabilistic model results. Reliab Eng Syst Safety 79 2 (2003) 123-129
172. Hora S.C., and Helton J.C. A distribution-free test for the relationship between model input and output when using Latin hypercube sampling. Reliab Eng Syst Safety 79 3 (2003) 333-339
173. Peacock J.A. Two-dimensional goodness-of-fit testing in astronomy. Monthly Notices R Astronom Soc 202 2 (1983) 615-627
174. Fasano G., and Franceschini A. A multidimensional version of the Kolmogorov-Smirnov test. Monthly Notices R Astronom Soc 225 1 (1987) 155-170
175. Garvey J.E., Marschall E.A., and Wright R.A. From star charts to stoneflies: detecting relationships in continuous bivariate data. Ecology 79 2 (1998) 442-447
176. Ripley B.D. Spatial point pattern analysis in ecology. In: Legendre P., and Legendre L. (Eds). Developments in numerical ecology. NATO ASI series, series G: ecological sciences vol. 14 (1987), Springer, Berlin; NY 407-430
177. Zeng G., and Dubes R.C. A comparison of tests for randomness. Pattern Recogn 18 2 (1985) 191-198
178. Diggle P.J. Statistical analysis of spatial point patterns (1983), Academic Press, New York, NY
179. Diggle P.J., and Cox T.F. Some distance-based tests of independence for sparsely sampled multivariate spatial point patterns. Int Stat Rev 51 1 (1983) 11-23
180. Byth K. On robust distance-based intensity estimators. Biometrics 38 1 (1982) 127-135
181. Byth K., and Ripley B.D. On sampling spatial patterns by distance methods. Biometrics 36 2 (1980) 279-284
182. Diggle P.J. On parameter estimation and goodness-of-fit testing for spatial point patterns. Biometrics 35 1 (1979) 87-101
183. Diggle P.J. Statistical methods for spatial point patterns in ecology. In: Cormack R.M., and Ord J.K. (Eds). Spatial and temporal analysis in ecology (1979), International Co-operative Publishing House, Fairfield, MD 95-150
184. Ripley B.D. Tests of randomness for spatial point patterns. J R Stat Soc 41 3 (1979) 368-374
185. Besag J., and Diggle P.J. Simple Monte Carlo tests for spatial pattern. Appl Stat 26 3 (1977) 327-333
186. Diggle P.J., Besag J., and Gleaves J.T. Statistical analysis of spatial point patterns by means of distance methods. Biometrics 32 (1976) 659-667
187. Cox T.F., and Lewis T. A conditioned distance ratio method for analyzing spatial patterns. Biometrika 63 3 (1976) 483-491
188. Holgate P. The use of distance methods for the analysis of spatial distribution of points. In: Lewis P.A.W. (Ed). Stochastic point processes: statistical analysis, theory, and applications (1972), Wiley-Interscience, New York 122-135
189. Holgate P. Tests of randomness based on distance methods. Biometrika 52 3-4 (1965) 345-353
190. Clark P.J., and Evans F.C. Distance to nearest neighbour as a measure of spatial relationships in populations. Ecology 35 (1954) 23-30
191. Hopkins B., and Shellam J.G. A new method for determining the type of distribution of plant individuals. Ann Botany 18 70 (1954) 213-227
192. Cressie N.A.C. Statistics for spatial data (1993), Wiley, New York, NY
193. Iman R.L., and Conover W.J. A measure of top-down correlation. Technometrics 29 3 (1987) 351-357
194. Campolongo F., Saltelli A., Sorensen T., and Tarantola S. Hitchhiker's guide to sensitivity analysis. In: Saltelli A., Chan K., and Scott M. (Eds). Sensitivity analysis (2000), Wiley, New York 15-47
195. Saltelli A., Tarantola S., Campolongo F., and Ratto M. Sensitivity analysis in practice (2004), Wiley, New York, NY
196. Cukier R.I., Fortuin C.M., Shuler K.E., Petschek A.G., and Schiably J.H. Study of the sensitivity of coupled reaction systems to uncertainties in rate coefficients, I. Theory. J Chem Phys 59 8 (1973) 3873-3878
197. Schaibly J.H., and Shuler K.E. Study of the sensitivity of coupled reaction systems to uncertainties in rate coefficients, II. Applications. J Chem Phys 59 8 (1973) 3879-3888
198. McRae G.J., Tilden J.W., and Seinfeld J.H. Global sensitivity analysis-a computational implementation of the Fourier amplitude sensitivity test (FAST). Comput Chem Eng 6 1 (1981) 15-25
199. Saltelli A., and Sobol' I.M. About the use of rank transformation in sensitivity analysis of model output. Reliab Eng Syst Safety 50 3 (1995) 225-239
200. Homma T., and Saltelli A. Importance measures in global sensitivity analysis of nonlinear models. Reliab Eng Syst Safety 52 1 (1996) 1-17
201. Archer G.E.B., Saltelli A., and Sobol' I.M. Sensitivity measures, ANOVA-like techniques and the use of bootstrap. J Stat Comput Simul 58 2 (1997) 99-120
202. Saltelli A., and Bolado R. An alternative way to compute Fourier amplitude sensitivity test (FAST). Comput Stat Data Anal 26 4 (1998) 267-279
203. Saltelli A., Tarantola S., and Campolongo F. Sensitivity analysis as an ingredient of modeling. Stat Sci 15 4 (2000) 377-395
204. Chan K., Saltelli A., and Tarantola S. Winding stairs: a sampling tool to compute sensitivity indices. Stat Comput 10 3 (2000) 187-196
205. Rabitz H., Alis O.F., Shorter J., and Shim K. Efficient input-output model representations. Comput Phys Commun 117 1-2 (1999) 11-20
206. Jansen M.J.W., Rossing W.A.H., and Daamen R.A. Monte Carlo estimation of uncertainty contributions from several independent multivariate sources. In: Grasman J., and van Straten G. (Eds). Predictability and nonlinear modeling in natural sciences and economics (1994), Kluwer Academic Publishers, Boston 334-343
207. Jansen M.J.W. Analysis of variance designs for model output. Comput Phys Commun 117 1-2 (1999) 35-43
208. McKay M.D. Evaluating prediction uncertainty, LA-12915-MS; NUREG/CR-6311 (1995), Los Alamos National Laboratory, Los Alamos, NM
209. McKay M.D. Nonparametric variance-based methods of assessing uncertainty importance. Reliab Eng Syst Safety 57 3 (1997) 267-279
210. McKay M.D., Morrison J.D., and Upton S.C. Evaluating prediction uncertainty in simulation models. Comput Phys Commun 117 1-2 (1999) 44-51
211. Helton J.C. Mathematical and numerical approaches in performance assessment for radioactive waste disposal: dealing with uncertainty. In: Scott E.M. (Ed). Modelling radioactivity in the environment (2003), Elsevier Science, New York, NY 353-390
212. Helton J.C., Anderson D.R., Basabilvazo G., Jow H.-N., and Marietta M.G. Conceptual structure of the 1996 performance assessment for the waste isolation pilot plant. Reliab Eng Syst Safety 69 1-3 (2000) 151-165
213. Diaz NJ. Realism and conservatism, remarks by chairman diaz at the 2003 nuclear safety research conference. NRC News. No. S-03-023, Washington, DC: US Nuclear Regulatory Commission; October 20, 2003.
214. Paté-Cornell E. Risk and uncertainty analysis in government safety decisions. Risk Anal 22 3 (2002) 633-646
215. Caruso M.A., Cheok M.C., Cunningham M.A., Holahan G.M., King T.L., Parry G.W., et al. An approach for using risk assessment in risk-informed decisions on plant-specific changes to the licensing basis. Reliab Eng Syst Safety 63 (1999) 231-242
216. Sielken Jr. R.L., Bretzlaff R.S., and Stevenson D.E. Challenges to default assumptions stimulate comprehensive realism as a new tier in quantitative cancer risk assessment. Regul Toxicol Pharmacol 21 (1995) 270-280
217. Nichols A.L., and Zeckhauser R.J. The perils of prudence: how conservative risk assessments distort regulation. Regul Toxicol Pharmacol 8 (1988) 61-75