Numerical assessment of metamodelling strategies in computationally intensive optimization

Environmental Modelling and Software

Volumn 34, Issue , 2012, Pages 67-86

  Razavi, Saman ^a   Tolson, Bryan A ^a   Burn, Donald H ^a  

^a UNIVERSITY OF WATERLOO   (Canada)

Author keywords

Computationally intensive simulation models; Kriging; Metamodelling; Neural networks; Optimization; Radial basis functions

Indexed keywords

COMPARATIVE ASSESSMENT; COMPLEX RESPONSE; COMPUTATIONAL BUDGET; COMPUTATIONAL BURDEN; COMPUTATIONALLY INTENSIVE PROBLEMS; ENVIRONMENTAL MODEL; FUNCTION APPROXIMATION; KRIGING; META MODEL; META-MODELLING; NUMERICAL EXPERIMENTS; NUMERICAL RESULTS; OPTIMIZATION PROBLEMS; OPTIMIZATION STRATEGY; OPTIMIZERS; RADIAL BASIS FUNCTIONS; RESPONSE SURFACE; RESPONSE SURFACE METHODOLOGY; SIMULATION MODEL; TEST FUNCTIONS; WATER DISTRIBUTION NETWORKS; WATER RESOURCES SYSTEMS;

BUDGET CONTROL; COMPUTER SIMULATION; INTERPOLATION; NEURAL NETWORKS; OPTIMIZATION; RADIAL BASIS FUNCTION NETWORKS; RATING; SURFACE PROPERTIES; SYSTEMS ANALYSIS;

WATER RESOURCES;

ARTIFICIAL NEURAL NETWORK; COMPUTER SIMULATION; ENVIRONMENTAL ASSESSMENT; ENVIRONMENTAL MODELING; KRIGING; NUMERICAL MODEL; OPTIMIZATION; WATER RESOURCE;

EID: 84859099633     PISSN: 13648152     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.envsoft.2011.09.010     Document Type: Article

References (90)

1. Bau D.A., Mayer A.S. Stochastic management of pump-and-treat strategies using surrogate functions. Advances in Water Resources 2006, 29(12):1901-1917.
2. Beale M.H., Hagan M.T., Demuth H.B. Neural Network Toolbox™ 7-User's Guide 2010, downloadable at:, MathWorks, Inc. http://www.mathworks.com/help/pdf_doc/nnet/nnet.pdf.
3. Behzadian K., Kapelan Z., Savic D., Ardeshir A. Stochastic sampling design using a multi-objective genetic algorithm and adaptive neural networks. Environmental Modelling & Software 2009, 24(4):530-541.
4. Bliznyuk N., Ruppert D., Shoemaker C., Regis R., Wild S., Mugunthan P. Bayesian calibration and uncertainty analysis for computationally expensive models using optimization and radial basis function approximation. Journal of Computational and Graphical Statistics 2008, 17(2):270-294.
5. Broad D.R., Dandy G.C., Maier H.R. Water distribution system optimization using metamodels. Journal of Water Resources Planning and Management-ASCE 2005, 131(3):172-180.
6. Cheng C.T., Wu X.Y., Chau K.W. Multiple criteria rainfall-runoff model calibration using a parallel genetic algorithm in a cluster of computers. Hydrological Sciences Journal-Journal Des Sciences Hydrologiques 2005, 50(6):1069-1087.
7. Cressie N.A.C. Statistics for Spatial Data 1993, John Wiley & Sons, New York.
8. Crestaux T., Le Maitre O., Martinez J.-M. Polynomial chaos expansion for sensitivity analysis. Reliability Engineering & System Safety 2009, 94(7):1161-1172.
9. di Pierro F., Khu S.T., Savic D., Berardi L. Efficient multi-objective optimal design of water distribution networks on a budget of simulations using hybrid algorithms. Environmental Modelling & Software 2009, 24(2):202-213.
10. Doherty J. PEST: Model Independent Parameter Estimation 2005, Watermark Numerical Computing, Brisbane, Australia. fifth ed.
11. Fen C.S., Chan C.C., Cheng H.C. Assessing a response surface-based optimization approach for soil vapor extraction system design. Journal of Water Resources Planning and Management-ASCE 2009, 135(3):198-207.
12. Feyen L., Vrugt J.A., Nuallain B.O., van der Knijff J., De Roo A. Parameter optimisation and uncertainty assessment for large-scale streamflow simulation with the LISFLOOD model. Journal of Hydrology 2007, 332(3-4):276-289.
13. Foresee F.D., Hagan M.T. Gauss-Newton approximation to Bayesian regularization. Proceedings of the 1997 International Joint Conference on Neural Networks, Houston, Texas 1997, 1930-1935. IEEE, New York.
14. Gu C. Smoothing Spline ANOVA Models 2002, Springer, New York, USA.
15. Hagan M.T., Menhaj M.B. Training feedforward networks with Marquardt algorithm. IEEE Transactions on Neural Networks 1994, 5(6):989-993.
16. Hamm L., Brorsen B.W., Hagan M.T. Comparison of stochastic global optimization methods to estimate neural network weights. Neural Processing Letters 2007, 26:145-158.
17. Hansen N., Muller S.D., Koumoutsakos P. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evolutionary Computation 2003, 11(1):1-18.
18. He Y., Hui C.W. Genetic algorithm based on heuristic rules for high-constrained large-size single-stage multi-product scheduling with parallel units. Chemical Engineering and Processing 2007, 46(11):1175-1191.
19. Hornik K., Stinchcombe M., White H. Multilayer feedforward networks are universal approximators. Neural Networks 1989, 2(5):359-366.
20. Hsu K.L., Gupta H.V., Sorooshian S. Artificial neural network modeling of rainfall-runoff process. Water Resources Research 1995, 31(10):2517-2530.
21. Hussain M.F., Barton R.R., Joshi S.B. Metamodeling: radial basis functions, versus polynomials. European Journal of Operational Research 2002, 138(1):142-154.
22. Jin Y. A comprehensive survey of fitness approximation in evolutionary computation. Soft Computing 2005, 9(1):3-12.
23. Jin Y.C., Olhofer M., Sendhoff B. A framework for evolutionary optimization with approximate fitness functions. IEEE Transactions on Evolutionary Computation 2002, 6(5):481-494.
24. Johnson V.M., Rogers L.L. Accuracy of neural network approximators in simulation-optimization. Journal of Water Resources Planning and Management-ASCE 2000, 126(2):48-56.
25. Jones D.R. A taxonomy of global optimization methods based on response surfaces. Journal of Global Optimization 2001, 21(4):345-383.
26. Jones D.R., Schonlau M., Welch W.J. Efficient global optimization of expensive black-box functions. Journal of Global Optimization 1998, 13(4):455-492.
27. Joslin, D., Dragovich, J., Vo, H., Terada, J., 2006. Opportunistic fitness evaluation in a genetic algorithm for civil engineering design optimization, In: Evolutionary Computation, C.I.C.o. (Ed.), IEEE Congress on Evolutionary Computation, pp. 2904-2911.
28. Karamouz M., Razavi S., Araghinejad S. Long-lead seasonal rainfall forecasting using time-delay recurrent neural networks: a case study. Hydrological Processes 2008, 22(2):229-241.
29. Keating E.H., Doherty J., Vrugt J.A., Kang Q.J. Optimization and uncertainty assessment of strongly nonlinear groundwater models with high parameter dimensionality. Water Resources Research 2010, 46.
30. Khan M.S., Coulibaly P. Bayesian neural network for rainfall-runoff modeling. Water Resources Research 2006, 42(7).
31. Khu S.T., Savic D., Liu Y., Madsen H. A fast evolutionary-based metamodelling approach for the calibration of a rainfall-runoff model. The First Biennial Meeting of the International Environmental Modelling and Software Society 2004, 147-152. Osnabruck, Germany.
32. Khu S.T., Werner M.G.F. Reduction of Monte-Carlo simulation runs for uncertainty estimation in hydrological modelling. Hydrology and Earth System Sciences 2003, 7(5):680-692.
33. Kleijnen J.P.C. An overview of the design and analysis of simulation experiments for sensitivity analysis. European Journal of Operational Research 2005, 164(2):287-300.
34. Koch P.N., Simpson T.W., Allen J.K., Mistree F. Statistical approximations for multidisciplinary design optimization: the problem of size. Journal of Aircraft 1999, 36(1):275-286.
35. Kourakos G., Mantoglou A. Pumping optimization of coastal aquifers based on evolutionary algorithms and surrogate modular neural network models. Advances in Water Resources 2009, 32(4):507-521.
36. Kuzmin V., Seo D.J., Koren V. Fast and efficient optimization of hydrologic model parameters using a priori estimates and stepwise line search. Journal of Hydrology 2008, 353(1-2):109-128.
37. Leshno M., Lin V.Y., Pinkus A., Schocken S. Multilayer feedforward networks with a non-polynomial activation function can approximate any function. Neural Networks 1993, 6(6):861-867.
38. Liong S.Y., Khu S.T., Chan W.T. Derivation of Pareto front with genetic algorithm and neural network. Journal of Hydrologic Engineering 2001, 6(1):52-61.
39. Locatelli M. A note on the Griewank test function. Journal of Global Optimization 2003, 25(2):169-174.
40. Lophaven S.N., Nielsen H.B., Sondergaard J. DACE: AMATLAB Kriging Toolbox Version 2.0 2002, Technical Report IMM-TR-2002-12, Technical University of Denmark, Copenhagen.
41. Luk K.C., Ball J.E., Sharma A. A study of optimal model lag and spatial inputs to artificial neural network for rainfall forecasting. Journal of Hydrology 2000, 227(1-4):56-65.
42. Mackay D.J.C. Bayesin interpolation. Neural Computation 1992, 4(3):415-447.
43. Maier H.R., Dandy G.C. Neural networks for the prediction and forecasting of water resources variables: a review of modelling issues and applications. Environmental Modelling & Software 2000, 15(1):101-124.
44. MathWorks MATLAB Global Optimization Toolbox 3 2010.
45. Mugunthan P., Shoemaker C.A. Assessing the impacts of parameter uncertainty for computationally expensive groundwater models. Water Resources Research 2006, 42(10).
46. Mugunthan P., Shoemaker C.A., Regis R.G. Comparison of function approximation, heuristic, and derivative-based methods for automatic calibration of computationally expensive groundwater bioremediation models. Water Resources Research 2005, 41(11):1-17.
47. Mullur A.A., Messac A. Metamodeling using extended radial basis functions: a comparative approach. Engineering with Computers 2006, 21(3):203-217.
48. Nakayama H., Arakawa M., Sasaki R. Simulation-based optimization using computational intelligence. Optimization and Engineering 2002, 3(2):201-214.
49. Nguyen D., Widrow B. Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights. IJCNN International Joint Conference on Neural Networks 1990, 1-3:C21-C26.
50. Nicklow J., Reed P., Savic D., Dessalegne T., Harrell L., Chan-Hilton A., Karamouz M., Minsker B., Ostfeld A., Singh A., Zechman E., Evolutionary A.T.C. State of the art for genetic algorithms and beyond in water resources planning and management. Journal of Water Resources Planning and Management-ASCE 2010, 136(4):412-432.
51. Ostfeld A., Salomons S. A hybrid genetic - instance based learning algorithm for CE-QUAL-W2 calibration. Journal of Hydrology 2005, 310(1-4):122-142.
52. Papadrakakis M., Lagaros N.D., Tsompanakis Y. Structural optimization using evolution strategies and neural networks. Computer Methods in Applied Mechanics and Engineering 1998, 156(1-4):309-333.
53. Ratto M., Pagano A. Using recursive algorithms for the efficient identification of smoothing spline ANOVA models. ASTA-Advances in Statistical Analysis 2010, 94(4):367-388.
54. Ratto M., Pagano A., Young P. State dependent parameter metamodelling and sensitivity analysis. Computer Physics Communications 2007, 177(11):863-876.
55. Razavi S., Araghinejad S. Reservoir inflow modeling using temporal neural networks with forgetting factor approach. Water Resources Management 2009, 23(1):39-55.
56. Razavi S., Karamouz M. Adaptive neural networks for flood routing in river systems. Water International 2007, 32(3):360-375.
57. Razavi S., Tolson B.A. A new formulation for feedforward neural networks. IEEE Transactions on Neural Networks 2011, 22(10):1588-1598. doi: 1510.1109/TNN.2011.2163169.
58. Razavi S., Tolson B.A., Matott L.S., Thomson N.R., MacLean A., Seglenieks F.R. Reducing the computational cost of automatic calibration through model pre-emption. Water Resources Research 2010.
59. Regis R.G., Shoemaker C.A. Local function approximation in evolutionary algorithms for the optimization of costly functions. IEEE Transactions on Evolutionary Computation 2004, 8(5):490-505.
60. Regis R.G., Shoemaker C.A. Improved strategies for radial basis function methods for global optimization. Journal of Global Optimization 2007, 37(1):113-135.
61. Regis R.G., Shoemaker C.A. A stochastic radial basis function method for the global optimization of expensive functions. Informs Journal on Computing 2007, 19(4):497-509.
62. Regis R.G., Shoemaker C.A. Parallel stochastic global optimization using radial basis functions. Informs Journal on Computing 2009, 21(3):411-426.
63. Roos G.N. Development of the Dipole Flow and Reactive Tracer Test (DFRTT) for aquifer parameter estimation, Civil and Environmental Engineering, University of Waterloo 2009, University of Waterloo, Waterloo, ON.
64. Sacks J., Welch W.J., Mitchell T.J., Wynn H.P. Design and analysis of computer experiments. Statistical Science 1989, 4(4):409-435.
65. Sakata S., Ashida F., Zako M. Structural optimization using Kriging approximation. Computer Methods in Applied Mechanics and Engineering 2003, 192(7-8):923-939.
66. Santner T.J., Williams B.J., Notz W. The Design and Analysis of Computer Experiments 2003, Springer, New York.
67. Sexton R.S., Dorsey R.E., Johnson J.D. Toward global optimization of neural networks: a comparison of the genetic algorithm and backpropagation. Decision Support Systems 1998, 22(2):171-185.
68. Shamseldin A.Y. Application of a neural network technique to rainfall-runoff modelling. Journal of Hydrology 1997, 199(3-4):272-294.
69. Shoemaker C.A., Regis R.G., Fleming R.C. Watershed calibration using multistart local optimization and evolutionary optimization with radial basis function approximation. Hydrological Sciences Journal-Journal Des Sciences Hydrologiques 2007, 52(3):450-465.
70. Simpson T.W., Mistree F. Kriging models for global approximation in simulation-based multidisciplinary design optimization. AIAA Journal 2001, 39(12):2233-2241.
71. Simpson T.W., Toropov V., Balabanov V., Viana F.A.C. Design and analysis of Computer Experiments in Multidisciplinary Design Optimization: A Review of How Far We Have Come - or Not 2008, 12th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, Victoria, British Columbia, Canada, 1-22.
72. Sobester A., Leary S.J., Keane A.J. On the design of optimization strategies based on global response surface approximation models. Journal of Global Optimization 2005, 33(1):31-59.
73. Storlie C.B., Bondell H.D., Reich B.J., Zhang H.H. Surface estimation, variable selection, and the nonparametric oracle property. Statistica Sinica 2011, 21(2):679-705.
74. Sudret B. Global sensitivity analysis using polynomial chaos expansions. Reliability Engineering & System Safety 2008, 93(7):964-979.
75. Tan C.C., Tung C.P., Chen C.H., Yeh W.W.G. An integrated optimization algorithm for parameter structure identification in groundwater modeling. Advances in Water Resources 2008, 31(3):545-560.
76. Tolson B.A., Asadzadeh M., Maier H.R., Zecchin A. Hybrid discrete dynamically dimensioned search (HD-DDS) algorithm for water distribution system design optimization. Water Resources Research 2009, 45.
77. Tolson B.A., Shoemaker C.A. Dynamically dimensioned search algorithm for computationally efficient watershed model calibration. Water Resources Research 2007, 43(1).
78. 2O fluxes and N leaching from corn crops. Environmental Modelling & Software 2011, (to appear: this Thematic Issue).
79. Vrugt J.A., Nuallain O., Robinson B.A., Bouten W., Dekker S.C., Sloot P.M.A. Application of parallel computing to stochastic parameter estimation in environmental models. Computers & Geosciences 2006, 32(8):1139-1155.
80. Vrugt J.A., Robinson B.A., Hyman J.M. Self-adaptive multimethod search for global optimization in real-parameter spaces. IEEE Transactions on Evolutionary Computation 2009, 13(2):243-259.
81. Wahba G. Spline Models for Observational Data 1990, Society for Industrial and Applied Mathematics, Philadelphia, Pennsylvania, USA.
82. Wang G.G., Shan S. Review of metamodeling techniques in support of engineering design optimization. Journal of Mechanical Design 2007, 129(4):370-380.
83. Welch W.J., Buck R.J., Sacks J., Wynn H.P., Mitchell T.J., Morris M.D. Screening, predictiong, and computer experiments. Technometrics 1992, 34(1):15-25.
84. Willmes L., Back T., Jin Y.C., Sendhoff B. Comparing neural networks and Kriging for fitness approximation in evolutionary optimization. Proceedings. IEEE 2003, vol. 1-4:663-670. New York.
85. Xiang C., Ding S.Q., Lee T.H. Geometrical interpretation and architecture selection of MLP. IEEE Transactions on Neural Networks 2005, 16(1):84-96.
86. Yan S.Q., Minsker B. Optimal groundwater remediation design using an adaptive neural network genetic algorithm. Water Resources Research 2006, 42(5).
87. Young P.C., Ratto M. A unified approach to environmental systems modeling. Stochastic Environmental Research and Risk Assessment 2009, 23(7):1037-1057.
88. Zhang X.S., Srinivasan R., Van Liew M. Approximating SWAT model using artificial neural network and support vector machine. Journal of the American Water Resources Association 2009, 45(2):460-474.
89. Zou R., Lung W.S., Wu J. An adaptive neural network embedded genetic algorithm approach for inverse water quality modeling. Water Resources Research 2007, 43(8).
90. Zou R., Lung W.S., Wu J. Multiple-pattern parameter identification and uncertainty analysis approach for water quality modeling. Ecological Modelling 2009, 220(5):621-629.