Evolutionary model type selection for global surrogate modeling

Journal of Machine Learning Research

Volumn 10, Issue , 2009, Pages 2039-2078

  Gorissen, Dirk ^a   Dhaene, Tom ^a   De Turck, Filip ^a  

^a GHENT UNIVERSITY   (Belgium)

Author keywords

Active learning; Adaptive sampling; Function approximation; Genetic algorithms; Global surrogate modeling; Model type selection

Indexed keywords

ACTIVE LEARNING; ADAPTIVE SAMPLING; APPROXIMATION MODEL; BUILDING BLOCKES; COMPACT FORMULATIONS; COMPUTATIONAL COSTS; DATA SETS; DESIGN PACKAGES; DESIGN SPACE EXPLORATION; DESIGN SPACES; EVOLUTIONARY MODELS; FUNCTION APPROXIMATION; GLOBAL SURROGATE MODELING; INDISPENSABLE TOOLS; KRIGING; META MODEL; MODEL ACCURACY; OPTIMAL MODEL; SEQUENTIAL EXECUTION; SIMULATION CODE; SIMULATION ENVIRONMENT; SURROGATE MODEL; SURROGATE MODELING; TYPE SELECTION;

APPROXIMATION ALGORITHMS; COMPUTATIONAL COMPLEXITY; CONCURRENT ENGINEERING; FUEL ADDITIVES; GENETIC ALGORITHMS; JOB ANALYSIS; LEARNING ALGORITHMS; OPTIMIZATION; RAPID PROTOTYPING; SENSITIVITY ANALYSIS; SPACE RESEARCH;

SIMULATORS;

EID: 70449364091     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (89)

1. E. Alba and M. Tomassini. Parallelism and evolutionary algorithms. IEEE Trans. Evolutionary Computation, 6(5):443-462, 2002.
2. C. A. Ankenbrandt. An extension to the theory of convergence and a proof of the time complexity of genetic algorithms. In Foundations of Genetic Algorithms, pages 53-68, 1990.
3. Y. B., R. El-Yaniv, and K. Luz. Online choice of active learning algorithms. Journal of Machine Learning Research, 5:255-291, 2004.
4. D. Busby, C. L. Farmer, and A. Iske. Hierarchical nonlinear approximation for experimental design and statistical data fitting. SIAM Journal on Scientific Computing, 29(1):49-69, 2007. doi: 10.1137/050639983.
5. P-W. Chen, J-Y. Wang, and H-M. Lee. Model selection of SVMs using GA approach. In Neural Networks, 2004. Proceedings. 2004 IEEE International Joint Conference on, volume 3, pages 2035-2040, 25-29 July 2004.
6. V. Chen, K-L. Tsui, R. Barton, and M. Meckesheimer. A review on design, modeling and applications of computer experiments. HE Transactions, 38:273-291, 2006. (Pubitemid 43315497)
7. H-S. Chung and J. J. Alonso. Comparison of approximation models with merit functions for design optimization. In 8th AIAA/USAF/NASA/ISSMO Symposium on Multidisciplinary Analysis and Optimization, Long Beach, CA, September 2000. AIAA Paper 2000-4754.
8. S. Conti and A. O'Hagan. Bayesian emulation of complex multi-output and dynamic computer models, research report no. 569/07, submitted to Journal of Statistical Planning and Inference. Technical report, Department of Probability and Statistics, University of Sheffield, 2007.
9. I. Couckuyt, D. Gorissen, H. Rouhani, E. Laermans, and T. Dhaene. Evolutionary regression modeling with active learning: An application to rainfall runoff modeling. In International Conference on Adaptive and Natural Computing Algorithms, volume LNCS 5495, pages 548-558, 2009.
10. K. Crombecq, D. Gorissen, L. De Tommasi, and T. Dhaene. A novel sequential design strategy for global surrogate modeling. In Proceedings of the 41th Conference on Winter Simulation, accepted, 2009.
11. D. Deschrijver and T Dhaene. Rational modeling of spectral data using orthonormal vector fitting. In Signal Propagation on Interconnects, 2005. Proceedings. 9th IEEE Workshop on, pages 111-114, 10-13 May 2005. doi: 10.1109/SPI.2005.1500915.
12. M. S. Eldred and D. M. Dunlavy. Formulations for surrogate-based optimization wiht data fit, multifidelity, and reduced-order models. In 11th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, Protsmouth, Virginia, 2006.
13. M. T. M. Emmerich, K. Giannakoglou, and B. Naujoks. Single- and multiobjective evolutionary optimization assisted by gaussian random field metamodels. IEEE Trans. Evolutionary Computation, 10(4):421-439, 2006. (Pubitemid 44251376)
14. H. J. Escalante, M. M. Gomez, and L. E. Sucar. PSMS for neural networks. In The IJCNN 2007 Agnostic vs Prior Knowledge Challenge, pages 678-683, 2007.
15. H. Jair Escalante, M. Montes, and E. Sucar. Particle swarm model selection. Journal of Machine Learning Research (special issue on model selection), 10:405-440, 2008.
16. H. Fang, M. Rais-Rohani, Z. Liu, and M.F. Horstemeyer. A comparative study of metamodeling methods for multiobjective crashworthiness optimization. Computers and Structures, 83:2121-2136, 2005. (Pubitemid 41084679)
17. F.D. Foresee and M.T Hagan. Gauss-newton approximation to bayesian regularization. In Proceedings of the 1997 International Joint Conference on Neural Networks, pages 1930-1935, 1997.
18. A. Forrester, A. Sobester, and A. Keane. Engineering Design Via Surrogate Modelling: A Practical Guide. Wiley, 2008.
19. F. Friedrichs and C. Igel. Evolutionary tuning of multiple svm parameters. Neurocomputing, 64: 107-117, 2005.
20. S.E. Gano, H. Kim, and D.E. Brown. Comparison of three surrogate modeling techniques: Datascape, kriging, and second order regression. In Proceedings of the 11th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, AIAA-2006-7048, Portsmouth, Virginia, 2006.
21. M. Ganser, K. Grossenbacher, M. Schutz, L. Willmes, and T Back. Simulation meta-models in the early phases of the product development process. In Proceedings of Efficient Methods for Robust Design and Optimization (EUROMECH 07), 2007.
22. K. C. Giannakoglou, M. K. Karakasis, and I. C. Kampolis. Evolutionary algorithms with surrogate modeling for computationally expensive optimization problem. In Proceedings of ERCOFTAC 2006 Design Optimization International Conference, Gran Canaria, Spain, 2006.
23. T Goel, R. Haftka, W. Shyy, and N. Queipo. Ensemble of surrogates. Structural and Multidisciplinary Optimization, 33:199-216, 2007. doi: doi:10.1007/s00158-006-0051-9.
24. D.Goldberg. Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley Professional, January 1989. ISBN 0201157675.
25. D. Gorissen. Heterogeneous evolution of surrogate models. Master's thesis, Master of AI, Katholieke Universiteit Leuven (KUL), 2007.
26. D. Gorissen, L. De Tommasi, J. Croon, and T. Dhaene. Automatic model type selection with heterogeneous evolution: An application to RF circuit block modeling. In Proceedings of the IEEE Congress on Evolutionary Computation, WCCI2008, Hong Kong, 2008a.
27. D. Gorissen, L. De Tommasi, W. Hendrickx, J. Croon, and T. Dhaene. RF circuit block modeling via kriging surrogates. In Proceedings of the 17th International Conference on Microwaves, Radar and Wireless Communications (MIKON 2008), 2008b.
28. D. Gorissen, I. Couckuyt, K. Crombecq, and T. Dhaene. Pareto-based multi-output model type selection. In Proceedings of the 4th International Conference on Hybrid Artificial Intelligence (HAIS 2009), Salamanca, Spain, pages 442-449. Springer - Lecture Notes in Artificial Intelligence, Vol.LNCS 5572, 2009a.
29. D. Gorissen, I. Couckuyt, E. Laermans, and T. Dhaene. Multiobjective global surrogate modeling,dealing with the 5-percent problem - in press. Engineering with Computers, 2009b.
30. D. Gorissen, L. De Tommasi, K. Crombecq, and T Dhaene. Sequential modeling of a low noise amplifier with neural networks and active learning. Neural Computing and Applications, 18(5): 485-494, 2009c.
31. R. Gramacy, H. Lee, and W. Macready. Parameter space exploration with gaussian process trees. In ICML '04: Proceedings of the 21st International Conference on Machine Learning, page 45, New York, NY, USA, 2004. ACM Press. ISBN 1-58113-828-5. doi: http://doi.acm.org/10.1145/1015330.1015367.
32. L. Gu. A comparison of polynomial based regression models in vehicle safety analysis. In A. Diaz, editor, 2001 ASME Design Automation Conference, ASME, Pittsburgh, PA, 2001.
33. D. Harrison and D.L. Rubinfeld. Hedonic prices and the demand for clean air. Journal of Environmental Economics & Management, 5:81-102, 1978.
34. W. Hendrickx, D. Gorissen, and T. Dhaene. Grid enabled sequential design and adaptive metamodeling. In WSC '06: Proceedings of the 37th Conference on Winter simulation, pages 872-881. Winter Simulation Conference, 2006. ISBN 1-4244-0501-7.
35. C. Hocaoglu and A. Sanderson. Planning multiple paths with evolutionary speciation. IEEE Trans. Evolutionary Computation, 5(3): 169-191, 2001.
36. J. Holland. Adaptation in Natural and Artificial Systems. University of Michigan Press, Ann Arbor, 1975.
37. R. Jin, W. Chen, and T.W. Simpson. Comparative studies of metamodelling techniques under multiple modelling criteria. Structural and Multidisciplinary Optimization, 23(1): 1-13, December 2001. doi: 10.1007/s00158-001-0160-4.
38. Yaochu Jin and B. Sendhoff. Pareto-based multiobjective machine learning: An overview and case studies. Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on, 38(3):397-415, May 2008. ISSN 1094-6977. doi: 10.1109/TSMCC.2008.919172.
39. Yaochu Jin, Markus Olhofer, and Bernhard Sendhoff. A framework for evolutionary optimization with approximate fitness functions. IEEE Trans. Evolutionary Computation, 6(5):481-494, 2002. (Pubitemid 35327564)
40. D. R. Jones, M. Schonlau, and W. J. Welch. Efficient global optimization of expensive blackbox functions. J. of Global Optimization, 13(4):455-492, 1998. ISSN 0925-5001. doi: http://dx.doi.org/10.1023/A:1008306431147.
41. A. C. Keys, L. P. Rees, and A. G. Greenwood. Performance measures for selection of metamodels to be used in simulation optimization. Decision Sciences, 33:31-58, 2007.
42. J. Kleijnen, S. Sanchez, T Lucas, and T Cioppa. State-of-the-art review: A user's guide to the brave new world of designing simulation experiments. INFORMS Journal on Computing, 17(3): 263-289, 2005.
43. J. Knowles and H. Nakayama. Meta-modeling in multiobjective optimization. In Multiobjective Optimization: Interactive and Evolutionary Approaches, pages 245-284. Springer-Verlag, Berlin, Heidelberg, 2008. ISBN 978-3-540-88907-6. doi: http://dx.doi.org/10.1007/978-3-540-88908-3-10.
44. R. Lehmensiek. Efficient Adaptive Sampling Applied to Multivariate, Multiple Output Rational Interpolation Models, with Applications in Electromagnetics-based Device Modeling. PhD thesis, University of Stellenbosch, 2001.
45. S. Lessmann, R. Stahlbock, and S.F. Crone. Genetic algorithms for support vector machine model selection. In Proceedings of the International Joint Conference on Neural Networks, 2006. IJCNN '06., pages 3063-3069, 16-21 July 2006.
46. X. Rong Li and Z. Zhao. Evaluation of estimation algorithms part I: incomprehensive measures of performance. IEEE Transactions on Aerospace and Electronic Systems, 42(4): 1340-1358, October 2006. ISSN 0018-9251. doi: 10.1109/TAES.2006.314576.
47. D. Lim, Y-S. Ong, Y Jin, and B. Sendhoff. A study on metamodeling techniques, ensembles, and multi-surrogates in evolutionary computation. In Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation (GECCO 07), pages 1288-1295, New York, NY, USA, 2007. ACM. ISBN 978-1-59593-697-4. doi: http://doi.acm.org/10.1145/1276958.1277203.
48. S. N. Lophaven, H. B. Nielsen, and J. Søndergaard. Aspects of the matlab toolbox DACE. Technical report, Informatics and Mathematical Modelling, Technical University of Denmark, DTU, Richard Petersens Plads, Building 321, DK-2800 Kgs. Lyngby, 2002.
49. D. MacKay. Bayesian model comparison and backprop nets. In J. E. Moody, S. J. Hanson, and R. P. Lippmann, editors, Advances in Neural Information Processing Systems 4, pages 839-846, San Mateo, California, 1992. Morgan Kaufmann.
50. T Nakama. Theoretical analysis of genetic algorithms in noisy environments based on a markov model. In Proceedings of the 10th Annual Conference on Genetic and Evolutionary Computation (GECCO 08), pages 1001-1008, New York, NY, USA, 2008. ACM. ISBN 978-1-60558-130-9. doi: http://doi.acm.org/ 10.1145/1389095.1389283.
51. A. Neubauer. A theoretical analysis of the non-uniform mutation operator for the modified genetic algorithm. In Proceedings of the IEEE International Conference on Evolutionary Computation, pages 93-96, Apr 1997. doi: 10.1109/ICEC.1997.592275.
52. M. Nowostawski and R. Poli. Parallel genetic algorithm taxonomy. In Knowledge-Based Intelligent Information Engineering Systems, 1999. Third International Conference, pages 88-92, 31 Aug.-1 Sept. 1999. doi: 10.1109/KES.1999.820127.
53. Y-S. Ong, P.B. Nair, and K.Y. Lum. Max-min surrogate-assisted evolutionary algorithm for robust design. Evolutionary Computation, IEEE Transactions on, 10(4):392-404, Aug. 2006. doi: 10.1109/TEVC.2005.859464.
54. I. Paenke, J. Branke, and Y Jin. Efficient search for robust solutions by means of evolutionary algorithms and fitness approximation. IEEE Trans. Evolutionary Computation, 10(4):405-420, 2006.
55. B. Pamadi, R Covell, P. Tartabini, and K. Murphy. Aerodynamic characteristics and glide-back performance of langley glide-back booster. In Proceedings of 22nd Applied Aerodynamics Conference and Exhibit, Providence, Rhode Island, 2004.
56. H. Pei and E. Goodman. A comparison of cohort genetic algorithms with canonical serial and island-model distributed ga's. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2001), pages 501-510, San Francisco, California, USA, 7-11 July 2001. Morgan Kaufmann. ISBN 1-55860-774-9.
57. R. Poll. Exact schema theory for genetic programming and variable-length genetic algorithms with one-point crossover. Genetic Programming and Evolvable Machines, 2(2):123-163, 2001. ISSN 1389-2576. doi: http://dx.doi.org/10.1023/A: 1011552313821.
58. X. Qi and F. Palmieri. Theoretical analysis of evolutionary algorithms with an infinite population size in continuous space, part I: Basic properties of selection and mutation. Neural Networks, IEEE Transactions on, 5(1):102-119, Jan 1994a. ISSN 1045-9227. doi: 10.1109/72.265965.
59. X. Qi and F. Palmieri. Theoretical analysis of evolutionary algorithms with an infinite population size in continuous space, part II: Analysis of the diversification role of crossover. Neural Networks, IEEE Transactions on, 5(1):120-129, Jan 1994b. ISSN 1045-9227. doi:10.1109/72.265966.
60. N. Queipo, R. Haftka, W Shyy, T Goel, R. Vaidyanathan, and P. Tucker. Surrogate-based analysis and optimization. Progress in Aerospace Sciences, 41:1-28, 2005.
61. C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
62. G. Rawlins, editor. Foundations of Genetic Algorithms. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1991. ISBN 1558605592.
63. R.G. Regis and C.A. Shoemaker. Local function approximation in evolutionary algorithms for the optimization of costly functions. IEEE Trans. Evolutionary Computation, 8(5):490-505, 2004.
64. S. Rogers, M. Aftosmis, S. Pandya, and N. Chaderjian. Automated CFD parameter studies on distributed parallel computers. In Proc of.16th AIAA Computational Fluid Dynamics Conference, Orlando, Florida, 2003.
65. E. Sanchez, S. Pintos, and N.V. Queipo. Toward an optimal ensemble of kernel-based approximations with engineering applications. In In Proceedings of the International Joint Conference on Neural Networks, 2006. IJCNN '06., pages 2152-2158, 2006. doi: 10.1109/IJCNN.2006.246987.
66. T Santner, B. Williams, and W Notz. The design and analysis of computer experiments. Springer series in statistics. Springer, 2003.
67. A. Sharkey. On combining artificial neural nets. Connectionist Science, 8(3):299-314, 1996.
68. T. Simpson, J. D. Poplinski, P. N. Koch, and J. K. Allen. Metamodels for computer-based engineering design: Survey and recommendations. Eng. Comput. (Lond.), 17(2): 129-150, 2001.
69. T W Simpson, V. Toropov, V. Balabanov, and F. A. C. Viana. Design and analysis of computer experiments in multidisciplinary design optimization: a review of how far we have come or not. In Proceedings of the 12th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, 2008 MAO, Victoria, Canada, 2008.
70. G. Smits and M. Kotanchek. Pareto-front exploitation in symbolic regression. In Genetic Programming Theory and Practice II. Springer, Ann Arbor, USA, 2004.
71. D. P. Solomatine and A. Ostfeld. Data-driven modelling : some past experiences and new approaches. Journal of hydroinformatics, 10(1):3-22, 2008.
72. K. Stanley and R. Miikkulainen. Evolving neural networks through augmenting topologies. Evolutionary Computation, 10(2):99-127, 2002. ISSN 1063-6560. doi: http://dx.doi.org/10.1162/106365602320169811.
73. M. Streeter and L. Becker. Automated discovery of numerical approximation formulae via genetic programming. Genetic Programming and Evolvable Machines, 4(3):255-286, 2003. ISSN 1389-2576. doi: http://dx.doi.org/10.1023/A: 1025176407779.
74. J.A.K. Suykens, T Van Gestel, J. De Brabanter, B. De Moor, and J. Vandewalle. Least Squares Support Vector Machines. World Scientific Publishing Co., Pte, Ltd., Singapore, 2002. ISBN 981-238-151-1.
75. D. J.J. Toal, N. W. Bressloff, and A. J. Keane. Kriging hyperparameter tuning strategies. AIAA Journal, 46(5): 1240-1252, 2008.
76. S. Tomioka, S. Nisiyama, and T. Enoto. Nonlinear least square regression by adaptive domain method with multiple genetic algorithms. IEEE Transactions on Evolutionary Computation, 11 (1):1-16, February 2007.
77. P. Triverio, S. Grivet-Talocia, M.S. Nakhla, F. G. Canavero, and R. Achar. Stability, causality, and passivity in electrical interconnect models. IEEE Transactions on Advanced Packaging, 30(4): 795-808, 2007.
78. C. Turner, R. Crawford, and M. Campbell. Multidimensional sequential sampling for nurbs-based metamodel development. Engineering with Computers, 23(3): 155-174, 2007. ISSN 0177-0667. doi: http://dx.doi.org/10.1007/s00366-006- 0051-9.
79. E.J. Vladislavleva, G.F. Smits, and D. den Hertog. Order of nonlinearity as a complexity measure for models generated by symbolic regression via pareto genetic programming. Evolutionary Computation, IEEE Transactions on, 13(2):333-349, April 2009. ISSN 1089-778X. doi: 10. 1109/TEVC.2008.926486.
80. I. Voutchkov and A.J. Keane. Multiobjective Optimization using Surrogates. In I.C. Parmee, editor, Adaptive Computing in Design and Manufacture 2006. Proceedings of the Seventh International Conference, pages 167-175, Bristol, UK, April 2006.
81. G. Wang and S. Shan. Review of metamodeling techniques in support of engineering design optimization. Journal of Mechanical Design, 129(4):370-380, 2007. doi: 10.1115/1.2429697.
82. D. Whitley, S. Rana, and R. B. Heckendorn. The island model genetic algorithm: On separability, population size and convergence. Journal of Computing and Information Technology, 7(1):33-47, 1999.
83. Y. Xiong, W . Chen, D. Apley, and X. Ding. A nonstationary covariance-based kriging method for metamodeling in engineering design. International Journal for Numerical Methods in Engineering, 71(6):733-756, 2007.
84. R.J. Yang, N. Wang, C. H. Tho, and J. P. Bobinaeu. Metamodeling development for vehicle frontal impact simulation. Journal of Mechanical Design, 127(5):1014-1020, September 2005.
85. X. Yao. Evolving artificial neural networks. Proceedings of the IEEE, 87(9): 1423-1447, Sept. 1999. doi: 10.1109/5.784219.
86. X. Yao and Y Xu. Recent advances in evolutionary computation. Journal of Computer Science and Technology, 21(1): 1-18, 2006.
87. K. Ye, W. Li, and A. Sudjianto. Algorithmic construction of optimal symmetric latin hypercube designs. Journal of Statistical Planning and Inference, 90:145-159, 2000.
88. Y-S. Yeun, W-S. Ruy, Y-S. Yang, and N-J. Kim. Implementing linear models in genetic programming. IEEE Trans. Evolutionary Computation, 8(6):542-566, 2004. (Pubitemid 40085394)
89. C. Zhang, H. Shao, and Y. Li. Particle swarm optimisation for evolving artificial neural network. In Systems, Man, and Cybernetics, 2000 IEEE International Conference on, volume 4, pages 2487-2490vol.4, 8-11 Oct. 2000. doi: 10.1109/ICSMC.2000.884366.