Chapter 18 Metamodel-Based Simulation Optimization

Handbooks in Operations Research and Management Science

Volumn 13, Issue C, 2006, Pages 535-574

  Barton, Russell R ^a,b   Meckesheimer, Martin ^a,b  

^a PENNSYLVANIA STATE UNIVERSITY   (United States)

^b BOEING COMPANY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 77950468650     PISSN: 09270507     EISSN: None     Source Type: Book Series    
DOI: 10.1016/S0927-0507(06)13018-2     Document Type: Review

References (97)

1. Alexandrov N., Dennis Jr. J.E., Lewis R.M., and Torczon V. A trust region framework for managing the use of approximate models in optimization. Structural Optimization 15 (1998) 16-23
2. Andradóttir S. A review of simulation optimization techniques. In: Medeiros D.J., Watson E.F., Carson J.S., and Manivannan M.S. (Eds). Proceedings of the 1998 Winter Simulation Conference (1998), IEEE Press, Piscataway, NJ 151-158
3. April J., Glover F., and Kelly J.P. Practical introduction to simulation optimization. In: Chick S., Sánchez P.J., Ferrin D., and Morrice D.J. (Eds). Proceedings of the 2003 Winter Simulation Conference (2003), IEEE Press, Piscataway, NJ 71-78
4. Banks J., Carson II J.S., Nelson B.L., and Nicol D.M. Discrete-Event System Simulation. 4th edition (2005), Prentice Hall, Upper Saddle River, NJ
5. Barton R.R. Metamodels for simulation input-output relations. In: Swain J.J., Goldsman D., Crain R.C., and Wilson J.R. (Eds). Proceedings of the 1992 Winter Simulation Conference (1992), IEEE Press, Piscataway, NJ 289-299
6. Barton R.R. Simulation metamodels. In: Medeiros D.J., Watson E.F., Carson J.S., and Manivannan M.S. (Eds). Proceedings of the 1998 Winter Simulation Conference (1998), IEEE Press, Piscataway, NJ 167-174
7. Barton R.R. Issues in development of simultaneous forward-inverse metamodels. In: Kuhl M.E., Steiger N.M., Armstrong F.B., and Joines J.A. (Eds). Proceedings of the 2005 Winter Simulation Conference (2005), IEEE Press, Piscataway, NJ 209-217
8. Bazaraa M.S., Sherali H.D., and Shetty C.M. Nonlinear Programming: Theory and Algorithms. 2nd edition (1993), Wiley, New York
9. Bertsekas D.P. Nonlinear Programming. 2nd edition (1999), Athena, Nashua, NH
10. Biles W.E. A gradient regression search procedure for simulation experimentation. In: Highland H.J., Steinberg H., and Morris M.F. (Eds). Proceedings of the 1974 Winter Simulation Conference (1974), Association for Computing Machinery, New York 491-497
11. Boender C.G.E., and Rinooy Kan A.H.G. Bayesian stopping rules for multistart global optimization methods. Mathematical Programming 37 (1987) 59-80
12. Booker A.J., Dennis J.E., Frank P.D., Serafini D.B., Torczon V., and Trosset M.W. A rigorous framework for optimization of expensive functions by surrogates. Structural Optimization 17 (1999) 1-13
13. Box G.E.P. The exploration and exploitation of response surfaces. Some general considerations and examples. Biometrics 10 (1954) 16-60
14. Box G.E.P., and Behnken D.W. Some new three-level designs for the study of qualitative variables. Technometrics 2 (1960) 455-475
15. Box G.E.P., and Cox D.R. An analysis of transformations (with discussion). Journal of the Royal Statistical Society B 26 (1964) 211-246
16. Box G.E.P., and Draper N.R. Empirical Model-Building and Response Surfaces (1987), Wiley, New York
17. Box G.E.P., and Wilson K.B. On the experimental attainment of optimum conditions. Journal of the Royal Statistical Society B 13 (1951) 1-45
18. Breiman L. The Π method for estimating multivariate functions from noisy data (with discussion). Technometrics 33 (1991) 125-160
19. Charnes A., and Cooper W.W. Goal programming and multiple objective optimization - part I. European Journal of Operational Research 1 (1977) 39-54
20. Cheng R.C.H., and Currie C.S.M. Optimization by simulation metamodelling methods. In: Ingalls R.G., Rossetti M.D., Smith J.S., and Peters B.A. (Eds). Proceedings of the 2004 Winter Simulation Conference (2004), IEEE Press, Piscataway, NJ 485-490
21. Chick S.E. Bayesian methods for discrete event simulation. In: Ingalls R.G., Rossetti M.D., Smith J.S., and Peters B.A. (Eds). Proceedings of the 2004 Winter Simulation Conference (2004), IEEE Press, Piscataway, NJ 89-100
22. Craven P., and Wahba G. Smoothing noisy data with spline functions. Numerische Mathematik 31 (1979) 377-403
23. de Boor C. A Practical Guide to Splines (1978), Springer-Verlag, New York
24. del Castillo E. Stopping rules for steepest ascent in experimental optimization. Communications in Statistics: Simulation and Computation 26 (1997) 1599-1615
25. del Castillo E., and Cahya S. A tool for computing confidence regions on the stationary point of a response surface. The American Statistician 55 (2001) 358-365
26. Dierckx P. An algorithm for surface fitting with spline functions. IMA Journal of Numerical Analysis 1 (1981) 267-283
27. Dierckx P. Curve and Surface Fitting with Splines. Monographs on Numerical Analysis (1993), Oxford University Press
28. Donohue J.M. The use of variance reduction techniques in the estimation of simulation metamodels. In: Alexopoulos C., Kang K., Goldsman D., and Lilegdon W. (Eds). Proceedings of the 1995 Winter Simulation Conference (1995), IEEE Press, Piscataway, NJ 194-200
29. Donohue J.M., Houck E.C., and Myers R.H. A sequential experimental design procedure for the estimation of first- and second-order simulation metamodels. ACM Transactions on Modeling and Computer Simulation 3 (1993) 190-224
30. Donohue J.M., Houck E.C., and Myers R.H. Simulation designs and correlation induction for reducing second-order bias in first-order response surfaces. Operations Research 41 (1993) 880-902
31. Donohue J.M., Houck E.C., and Myers R.H. Simulation designs for the estimation of quadratic response surface gradients in the presence of model misspecification. Management Science 41 (1995) 244-262
32. Draper N.R., and Guttman I. Response surface designs in flexible regions. Journal of the American Statistical Association 81 (1986) 1089-1094
33. Draper N.R., and Lin D.K.J. Small response-surface designs. Technometrics 32 2 (1990) 187-194
34. Eubank R.L. Spline Smoothing and Nonparametric Regression (1988), Dekker, New York
35. In: Floudas C.A., and Pardalos P.M. (Eds). State of the Art in Global Optimization (1996), Kluwer, Dordrecht, The Netherlands
36. Franke R. Scattered data interpolation: Tests of some methods. Mathematics of Computation 38 (1982) 181-200
37. Friedman J.H. Multivariate adaptive regression splines. The Annals of Statistics 19 (1991) 1-141
38. Fu M.C. Optimization via simulation: A review. Annals of Operations Research 53 (1994) 199-248
39. Fu M.C. Simulation optimization. In: Peters B.A., Smith J.S., Medeiros D.J., and Rohrer M.W. (Eds). Proceedings of the 2001 Winter Simulation Conference (2001), IEEE Press, Piscataway, NJ 53-61
40. Fu M.C. Optimization for simulation: Theory vs. practice. INFORMS Journal on Computing 14 (2003) 192-215
41. Glover F., Kelly J.P., and Laguna M. New advances and applications of combining simulation and optimization. In: Charnes J.M., Morrice D.J., Brunner D.T., and Swain J.J. (Eds). Proceedings of the 1996 Winter Simulation Conference (1996), IEEE Press, Piscataway, NJ 144-152
42. In: Grossman I.E. (Ed). Global Optimization in Engineering Design (1996), Kluwer, Dordrecht, The Netherlands
43. Hardy R.L. Multiquadric equations of topography and other irregular surfaces. Journal of Geophysical Research 76 (1971) 1905-1915
44. Hussain M.F., Barton R.R., and Joshi S.B. Metamodeling: Radial basis functions versus polynomials. European Journal of Operational Research 138 (2002) 142-154
45. Jin R., Chen W., and Simpson T.W. Comparative studies of metamodeling techniques under multiple modeling criteria. Proceedings of the 8th AIAA/NASA/USAF/ISSMO Symposium on Multidisciplinary Analysis and Optimization, Long Beach, CA (2000), AIAA, St. Louis, MO (CD-ROM)
46. Joshi S., Sherali H.D., and Tew J.D. An enhanced response surface methodology (RSM) algorithm using gradient deflection and second-order search strategies. Computers and Operations Research 25 (1998) 531-541
47. Kelton W.D., Sadowski R.P., and Sturrock D.T. Simulation with Arena. 3rd edition (2004), McGraw-Hill, New York
48. Khuri A.I., and Cornell J.A. Response Surfaces, Designs, and Analyses (1987), Dekker, New York
49. Kleijnen J.P.C. A comment on Blanning's metamodel for sensitivity analysis: The regression metamodel in simulation. Interfaces 5 (1975) 21-23
50. Kleijnen J.P.C. Statistical Techniques in Simulation, part II (1975), Dekker, New York
51. Kleijnen J.P.C. Statistical Tools for Simulation Practitioners (1987), Dekker, New York
52. Kleijnen J.P.C. An overview of the design and analysis of simulation experiments for sensitivity analysis. European Journal of Operational Research 164 (2005) 287-300
53. Kleijnen J.P.C., and Sargent R.G. A methodology for fitting and validating metamodels in simulation. European Journal of Operational Research 120 (2000) 14-29
54. Kleijnen J.P.C., and van Beers W.C.M. Application-driven sequential designs for simulation experiments: Kriging metamodeling. Journal of the Operational Research Society 55 (2004) 876-883
55. Kleijnen J.P.C., and van Beers W.C.M. Robustness of kriging when interpolating in random simulation with heterogeneous variances: Some experiments. European Journal of Operational Research 165 (2005) 826-834
56. Kleijnen J.P.C., and van Groenendaal W. Two-stage versus sequential sample-size determination in regression analysis of simulation experiments. American Journal of Mathematical and Management Sciences 15 1/2 (1995) 83-114
57. Kolmogorov A.N. On the representation of continuous functions of several variables by superposition of continuous functions of one variable and addition. American Mathematical Society Translations Series 2 17 (1961) 369-373
58. Law A.M., and Kelton W.D. Simulation Modeling and Analysis. 3rd edition (2000), McGraw-Hill, New York
59. Law A., and McComas M.G. Simulation-based optimization. In: Yücesan E., Chen H., Snowdon J.L., and Charnes J.M. (Eds). Proceedings of the 2002 Winter Simulation Conference (2002), IEEE Press, Piscataway, NJ 41-44
60. Másson E., and Wang Y.-J. Introduction to computation and learning in artificial neural networks. European Journal of Operational Research 47 (1990) 1-28
61. MATLAB. Neural network toolbox for MATLAB. Available at (June 30, 2005). http://www.mathworks.com/products/neuralnet/
62. McAllister C.D., Altuntas B., Frank M., and Potoradi J. Implementation of response surface methodology using variance reduction techniques in semiconductor manufacturing. In: Peters B.A., Smith J.S., Medeiros D.J., and Rohrer M.W. (Eds). Proceedings of the 2001 Winter Simulation Conference (2001), IEEE Press, Piscataway, NJ 1225-1230
63. Meckesheimer M., Booker A.J., Barton R.R., and Simpson T.W. Computationally inexpensive metamodel assessment strategies. AIAA Journal 40 (2002) 2053-2060
64. Miró-Quesada G., and del Castillo E. An enhanced recursive stopping rule for steepest ascent searches in response surface methodology. Communications in Statistics: Simulation and Computation 33 (2004) 201-228
65. Mitchell T.J., and Morris M.D. The spatial correlation function approach to response surface estimation. In: Swain J.J., Goldsman D., Crain R.C., and Wilson J.R. (Eds). Proceedings of the 1992 Winter Simulation Conference (1992), IEEE Press, Piscataway, NJ 565-571
66. Montgomery D.C. Design and Analysis of Experiments. 5th edition (2001), Wiley, New York
67. Morris M.D. A class of three-level experimental designs for response surface modeling. Technometrics 42 (2000) 111-121
68. Myers R.H. Response surface methodology - Current status and future directions. Journal of Quality Technology 31 (1999) 30-44
69. Myers R.H., and Khuri A.I. A new procedure for steepest ascent. Communications in Statistics: Theory and Methods 8 (1979) 1359-1376
70. Myers R.H., and Montgomery D.C. Response Surface Methodology: Process and Product Optimization Using Designed Experiments. 2nd edition (2002), Wiley, New York
71. Neddermeijer H.G., van Oortmarssen G.J., Piersma N., and Dekker R. A framework for response surface methodology for simulation optimization. In: Joines J.A., Barton R.R., Kang K., and Fishwick P.A. (Eds). Proceedings of the 2000 Winter Simulation Conference (2000), IEEE Press, Piscataway, NJ 129-136
72. Netlab. How to use Netlab. Available at (June 30, 2005). http://homepages.cae.wisc.edu/~ece539/software/netlab/intro.htm
73. NETLIB. DIERCKX: Fortran subroutines for calculating smoothing splines for various kinds of data and geometries, with automatic knot selection. Available at (June 30, 2005). http://www.netlib.org/dierckx/
74. Nicolai R.P., Dekker R., Piersma N., and van Oortmarssen G.J. Automated response surface methodology for stochastic optimization models with unknown variance. In: Ingalls R.G., Rossetti M.D., Smith J.S., and Peters B.A. (Eds). Proceedings of the 2004 Winter Simulation Conference (2004), IEEE Press, Piscataway, NJ 491-499
75. Nozari A., Arnold S.F., and Pegden C.D. Statistical analysis for use with the Schruben and Margolin correlation induction strategy. Operations Research 35 (1987) 127-139
76. Peterson J.J., Cahya S., and del Castillo E. A general approach to confidence regions for optimal factor levels of response surfaces. Biometrics 58 (2002) 422-431
77. Ramberg J.S., Sanchez S.M., Sanchez P.J., and Hollick L.J. Designing simulation experiments: Taguchi methods and response surface metamodels. In: Nelson B.L., Kelton W.D., and Clark G.M. (Eds). Proceedings of the 1991 Winter Simulation Conference (1991), IEEE Press, Piscataway, NJ 167-176
78. Sacks J., Welch W.J., Mitchell T.J., and Wynn H.P. Design and analysis of computer experiments. Statistical Science 4 (1989) 409-435
79. Sanchez S. Robust design: Seeking the best of all possible worlds. In: Joines J.A., Barton R.R., Kang K., and Fishwick P.A. (Eds). Proceedings of the 2000 Winter Simulation Conference (2000), IEEE Press, Piscataway, NJ 69-76
80. Santner T.J., Williams B.J., and Notz W.I. Sequential design of computer experiments to minimize integrated response functions. Statistica Sinica 10 (2000) 1133-1152
81. Santner T.J., Williams B.J., and Notz W.I. The Design and Analysis of Computer Experiments (2003-June 30, 2005), Spinger-Verlag, New York. http://www.stat.ohio-state.edu/~comp_exp/index.html PeRK code available at
82. Schruben L.W., and Margolin B.H. Pseudorandom number assignment in statistically designed simulation and distribution sampling experiments. Journal of the American Statistical Association 73 (1978) 504-525
83. Shao J. Linear model selection by cross-validation. Journal of the American Statistical Association 88 (1993) 486-494
84. Shin M., Sargent R.G., and Goel A.L. Gaussian radial basis functions for simulation metamodeling. In: Yücesan E., Chen C.-H., Snowdon J.L., and Charnes J.M. (Eds). Proceedings of the 2002 Winter Simulation Conference (2002), IEEE Press, Piscataway, NJ 483-488
85. Simpson T.W., Mauery T.M., Korte J.J., and Mistree F. Comparison of response surface and kriging models for multidisciplinary design optimization. 7th Symposium on Multidisciplinary Analysis and Optimization. AIAA-98-4755 (1998), AIAA, St. Louis, MO 381-391
86. Simpson T.W., Frecker M., Rothrock L., Stump G., Barton R.R., and Ligetti C. Assessing the impact of graphical design interfaces on design efficiency and effectiveness. ASME Journal of Computer and Information Science and Engineering 3 (2003) 144-154
87. Stone M. Cross-validatory choice and assessment of statistical predictions (with discussion). Journal of the Royal Statistical Society B 36 (1974) 111-133
88. Tew J.D., and Wilson J.R. Validation of simulation analysis methods for the Schruben-Margolin correlation-induction strategy. Operations Research 40 (1992) 87-103
89. Tew J.D., and Wilson J.R. Estimating simulation metamodels using combined correlation-based variance reduction techniques. IIE Transactions 26 (1994) 2-16
90. Thurston D.L., Carnahan J.V., and Liu T. Optimization of design utility. ASME Journal of Mechanical Design 116 (1994) 801-807
91. Tibshirani R. A comparison of some error estimates for neural network models. Neural Computation 8 (1996) 152-163
92. Tomick J., Arnold S., and Barton R.R. Sample size selection for improved Nelder-Mead performance. In: Alexopolous C., Kang K., Lilegdon W.R., and Goldsman D. (Eds). Proceedings of the 1995 Winter Simulation Conference (1995), IEEE Press, Piscataway, NJ 341-345
93. Tu C.-H., and Barton R.R. Production yield estimation by the metamodel method with a boundary-focused experiment design. Proceedings of DETC'97, 1997 ASME Design Engineering Technical Conference. DETC97/DTM3870 (1997), ASME, Fairfield, NJ
94. van Beers, W.C.M., Kleijnen, J.P.C. (2004). Customized sequential designs for random simulation experiments: Kriging metamodeling and bootstrapping. CentER Discussion Paper No. 2004-63, Tilburg University
95. Watlington J.A. Polynomial and radial basis function example code. Available at (June 30, 2005). http://web.media.mit.edu/~wad/mas864/src/polynomial.html
96. Yesilyurt S., and Patera A.T. Surrogates for numerical simulations; optimization of eddy-promoter heat exchangers. Computer Methods in Applied Mechanics and Engineering 121 (1995) 231-257
97. Zionts S. The state of multiple criteria decision making: Past, present, and future. In: Goicoechea A., Duckstein L., and Zionts S. (Eds). Multiple Criteria Decision Making (1992), Springer-Verlag, New York 33-43