Resampling methods for meta-model validation with recommendations for evolutionary computation

Evolutionary Computation

Volumn 20, Issue 2, 2012, Pages 249-275

  Bischl, B ^a   Mersmann, O ^a   Trautmann, H ^a   Weihs, C ^a  

^a TU DORTMUND UNIVERSITY   (Germany)

Author keywords

Evolutionary computation; Evolutionary optimization; Meta models; Model validation; Regression; Resampling

Indexed keywords

EVOLUTIONARY OPTIMIZATIONS; META MODEL; MODEL VALIDATION; REGRESSION; RESAMPLING;

EVOLUTIONARY ALGORITHMS; OPTIMIZATION;

REGRESSION ANALYSIS;

ARTICLE; REGRESSION ANALYSIS; STATISTICAL MODEL; THEORETICAL MODEL; VALIDATION STUDY;

MODELS, STATISTICAL; MODELS, THEORETICAL; REGRESSION ANALYSIS; VALIDATION STUDIES AS TOPIC;

EID: 84860796007     PISSN: 10636560     EISSN: 15309304     Source Type: Journal    
DOI: 10.1162/EVCO_a_00069     Document Type: Article

References (71)

1. Alpaydin, E. (1999). Combined 5 × 2CV F test for comparing supervised classification learning algorithms. Neural Computation, 11:1885-1892.
2. Bartlett, P., Boucheron, S., and Lugosi, G. (2002). Model selection and error estimation. Machine Learning, 48(1-3):85-113.
3. Bartz-Beielstein, T. (2006). Experimental research in evolutionary computation; The new experimentalism. Berlin: Springer.
4. Bengio, Y., and Grandvalet, Y. (2004). No unbiased estimator of the variance of k-fold cross-validation. Journal of Machine Learning Research, 5:1089-1105.
5. Bhattacharya, M. (2008). Meta model based EA for complex optimization. International Journal of Computational Intelligence, 1(4):36-45.
6. Binder, H., and Schumacher, M. (2008). Adapting prediction error estimates for biased complexity selection in high-dimensional bootstrap samples. Statistical Applications in Genetics and Molecular Biology, 7(1):12.
7. Birattari, M., Stützle, T., Paquete, L., and Varrentrapp, K. (2002). A racing algorithm for configuring metaheuristics. In W. Langdon (Ed.), Genetic and Evolutionary Computation Conference (GECCO), pp. 11-18.
8. Bischl, B. (2010). MLR: Machine learning in R. Retrieved from http://mlr.r-forge.r-project.org
9. B üche, D., Schraudolph, N. N., and Koumoutsakos, P. (2004). Accelerating evolutionary algorithms with Gaussian process fitness function models. IEEE Transactions on Systems, Man and Cybernetics, 35:183-194.
10. Cawley, G., and Talbot, N. (2004). Fast exact leave-one-out cross-validation of sparse least-squares support vector machines. Neural Networks, 17(10):1467-1475.
11. Diamantidis, N., Karlis, D., and Giakoumakis, E. (2000). Unsupervised stratification of cross-validation for accuracy estimation. Artificial Intelligence, 116(1-2):1-16.
12. Dietterich, T. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7):1895-1923.
13. Efron, B. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1):1-26.
14. Efron, B. (1983). Estimating the error rate of a prediction rule: Improvement on cross-validation. Journal of the American Statistical Association, 78(382):316-331.
15. Efron, B., and Tibshirani, R. (1997). Improvements on cross-validation: The 0.632 + bootstrap method. Journal of the American Statistical Association, 92(438):548-560.
16. El-Beltagy, M., Nair, P. B., and Keane, A. J. (1999). Metamodeling techniques for evolutionary optimization of computationally expensive problems: Promises and limitations. In Genetic and Evolutionary Computation Conference (GECCO), pp. 196-203.
17. Emmerich, M., Giannakoglou, K., and Naujoks, B. (2006). Single-and multiobjective evolutionary optimization assisted by Gaussian random field metamodels. IEEE Transactions on Evolutionary Computation, 10(4):421-439.
18. Fiebrink, R., and Fujinaga, I. (2006). Feature selection pitfalls and music classification. In Proceedings of the Eleventh International Society for Music Information Retrieval Conference (ISMIR), pp. 340-341.
19. Fu, W. J., Carroll, R. J., and Wang, S. (2005). Estimating misclassification error with small samples via bootstrap cross-validation. Bioinformatics, 21(9):1979-1986.
20. Giotis, A. P., and Giannakoglou, K. C. (1999). Single-and multi-objective airfoil design using genetic algorithms and artificial intelligence. In EUROGEN 99, Evolutionary Algorithms in Engineering and Computer Science.
21. Good, P. (2005). Resampling methods: A practical guide to data analysis. Basel, Switzerland: Birkhauser.
22. Hastie, T., Tibshirani, R., and Friedman, J. (2001). The elements of statistical learning. Berlin: Springer.
23. Hothorn, T., Leisch, F., Zeileis, A., and Hornik, K. (2005). The design and analysis of benchmark experiments. Journal of Computational and Graphical Statistics, 14:675-699.
24. Huang, D., Allen, T., Notz, W., and Zeng, N. (2006). Global optimization of stochastic black-box systems via sequential Kriging meta-models. Journal of Global Optimization, 34(3):441-466.
25. Jin, R., Chen, W., and Simpson, T. (2000). Comparative studies of metamodeling techniques under multiple modeling criteria. Structural and Multidisciplinary Optimization, 23:1-13.
26. Jin, Y. (2005). A comprehensive survey of fitness approximation in evolutionary computation. Soft Computing, 9(1):3-12.
27. Jin, Y., and Brank, J. (2005). Evolutionary optimization in uncertain environments: A survey. IEEE Transactions on Evolutionary Computation, 9(3):303-318.
28. Jin, Y., Olhofer, M., and Sendhoff, B. (2000). On evolutionary optimization with approximate fitness functions. In Genetic and Evolutionary Computation Conference (GECCO), pp. 786-793.
29. Jin, Y., Olhofer, M., and Sendhoff, B. (2002). A framework for evolutionary optimization with approximate fitness functions. IEEE Transactions on Evolutionary Computation, 6:481-494.
30. Jones, D., Schonlau, M., and Welch, W. (1998). Efficient global optimization of expensive black-box functions. Journal of Global Optimization, 13(4):455-492.
31. Kampolis, I. C., Zymaris, A. S., Asouti, V. G., and Giannakoglou, K. C. (2007). Multilevel optimization strategies based on metamodel-assisted evolutionary algorithms, for computationally expensive problems. In K. C. Tan and J.-X. Xu (Eds.), IEEE Congress on Evolutionary Computation (CEC), pp. 4116-4123.
32. Kern, S., Hansen, N., and Koumoutsakos, P. (2006). Local meta-models for optimization using evolution strategies. In Parallel problem solving from nature, PPSN IX, pp. 939-948.
33. Kim, J.-H. (2009). Estimating classification error rate: Repeated cross-validation, repeated holdout and bootstrap. Computational Statistics and Data Analysis, 53(11):3735-3745.
34. Knowles, J., and Nakayama, H. (2008). Meta-modeling in multiobjective optimization. In Multi-objective optimization: Interactive and evolutionary approaches (pp. 245-284). Berlin: Springer.
35. Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. In International Joint Conference on Artificial Intelligence (IJCAI), pp. 1137-1143.
36. Kruisselbrink, J. W., Emmerich, M. T. M., Deutz, A. H., and Bäck, T. (2010). A robust optimization approach using Kriging metamodels for robustness approximation in the CMA-ES. In IEEE Congress on Evolutionary Computation (CEC), pp. 1-8.
37. Lim, D., Jin, Y., Ong, Y.-S., and Sendhoff, B. (2010). Generalizing surrogate-assisted evolutionary computation. IEEE Transactions on Evolutionary Computation, 14(3):329-355.
38. Loshchilov, I., Schoenauer, M., and Sebag, M. (2010). Comparison-based optimizers need comparison-based surrogates. In Proceedings of the 11th International Conference on Paral-lel Problem Solving from Nature (pp. 364-373). Berlin: Springer.
39. Mersmann, O., Bischl, B., Trautmann, H., Preuss, M., Weihs, C., and Rudolph, G. (2011). Exploratory landscape analysis. In Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation (GECCO), pp. 829-836.
40. Mersmann, O., Preuss, M., and Trautmann, H. (2010). Benchmarking evolutionary algorithms: Towards exploratory landscape analysis. In Parallel problem solving from nature (PPSN). Lecture notes in computer science, Vol. 6238 (pp. 73-82). Berlin: Springer.
41. Molinaro, A., Simon, R., and Pfeiffer, R. (2005). Prediction error estimation: A comparison of resampling methods. Bioinformatics, 21(15):3301-3307.
42. Mukhopadhyay, N., and de Silva, B. M. (2009). Sequential methods and their applications. New York: Chapman & Hall/CRC.
43. Myers, R., and Montgomery, D. (1995). Response surface methodology. New York: Wiley.
44. Nadeau, C., and Bengio, Y. (2003). Inference for the generalization error. Machine Learning, 52(3):239-281.
45. Nakayama, H., Yun, Y., and Yoon, M. (2009). Sequential approximate multiobjective optimization using computational intelligence. Berlin: Springer.
46. Ong, Y., Nair, P., and Keane, A. (2003). Evolutionary optimization of computationally expensive problems via surrogate modeling. AIAA Journal, 41(4):687-696.
47. Ong, Y., Zhou, Z., and Lim, D. (2006). Curse and blessing of uncertainty in evolutionary algorithms using approximation. In IEEE Congress on Evolutionary Computation (CEC), pp. 2928-2935.
48. Paenke, I., Branke, J., and Jin, Y. (2006). Efficient search for robust solutions by means of evolutionary algorithms and fitness approximation. IEEE Transactions on Evolutionary Computation, 10(4):405-420.
49. Salzberg, S. (1997). On comparing classifiers: Pitfalls to avoid and a recommended approach. Data Mining and Knowledge Discovery, 1(3):317-328.
50. Sanchez, E., Pintos, S., and Queipo, N. (2008). Toward an optimal ensemble of kernel-based approximations with engineering applications. Structural and Multidisciplinary Optimization, 36(3):247-261.
51. Santers, T., Williams, N., and Notz, W. (2003). The design and analysis of computer experiments. Berlin: Springer.
52. Shao, J. (1993). Linear model selection by cross-validation. Journal of the American Statistical Association, 88(422):486-494.
53. Shi, L., and Rasheed, K. (2010). A survey of fitness approximation methods applied in evolutionary algorithms. Computational intelligence in expensive optimization problems (pp. 3-28). Berlin: Springer.
54. Simon, R. (2007). Resampling strategies for model assessment and selection. In Fundamentals of data mining in genomics and proteomics (pp. 173-186). Berlin: Springer.
55. Simon, R., Radmacher, M. D., Dobbin, K., and McShane, L. M. (2003). Pitfalls in the use of DNA microarray data for diagnostic and prognostic classification. Journal of the National Cancer Institute, 95(1):14-18.
56. Simpson, T., Mauery, T., Korte, J., and Mistree, F. (1998). Comparison of response surface and Kriging models for multidisciplinary design optimization. In Proceedings of the 7th AIAA/USAF/NASA/ISSMO Symposium on Multidisciplinary Analysis and Optimization, pp. 98-4755.
57. Smit, S., and Eiben, A. (2009). Comparing parameter tuning methods for evolutionary algorithms. In A. Tyrrell (Ed.), IEEE Congress on Evolutionary Computation (CEC), pp. 399-406.
58. Stone, M. (1974). Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society, Series B, 36(1):111-147.
59. Stone, M. (1977). An asymptotic equivalence of choice of model by cross-validation and Akaike's criterion. Journal of the Royal Statistical Society, Series B, 39:44-47.
60. Sundararajan, S., and Keerthi, S. (2001). Predictive approaches for choosing hyperparameters in Gaussian processes. Neural Computation, 13(5):1103-1118.
61. Tenne, Y., and Armfield, S. (2008). Metamodel accuracy assessment in evolutionary optimization. In IEEE Congress on Evolutionary Computation (CEC), pp. 1505-1512.
62. Ulmer, H., Streichert, F., and Zell, A. (2003). Evolution strategies assisted by Gaussian processes with improved pre-selection criterion. In IEEE Congress on Evolutionary Computation (CEC), pp. 692-699.
63. Vladislavleva, E., Smits, G., and Kotanchek, M. (2007). Better solutions faster: Soft evolution of robust regression models in Pareto genetic programming. In R. L. Riolo, T. Soule, and B. Worzel (Eds.), Genetic programming theory and practice V, Genetic and evolutionary computation (chap. 2, pp. 13-32). Berlin: Springer.
64. Wagner, T., Emmerich, M., Deutz, A., and Ponweiser, W. (2010). On expected-improvement criteria for model-based multi-objective optimization. In R. Schaefer (Ed.), Parallel Problem Solving from Nature (PPSN), pp. 718-727.
65. Wahba, G. (1980). Spline bases, regularization, and generalized cross validation for solving approximation problems with large quantities of noisy data. In International Conference on Approximation Theory in Honour of George Lorenz.
66. Weihs, C. (1993). Canonical discriminant analysis: Comparison of resampling methods and convex-hull approximation. Information and Classification (pp. 225-238). Berlin: Springer.
67. Weiss, S., and Kulikowski, C. (1991). Computer systems that learn: Classification and prediction methods from statistics, neural nets, machine learning, and expert systems. San Mateo,CA: Morgan Kaufmann.
68. Wu, C. (1986). Jackknife, bootstrap and other resampling methods in regression analysis. Annals of Statistics, 14:1261-1295.
69. Younis, A., and Dong, Z. (2010). Metamodelling using search space exploration and unimodal region elimination for design optimization. Engineering Optimization, 6(42):517-533.
70. Yun, Y., Yoon, M., and Nakayama, H. (2009). Multi-objective optimisation based on meta-modelling using support vector regression. Optimization and Engineering, 10(2):167-181.
71. Zhou, Z., Ong, Y.-S., Nair, P., Keane, A., and Lum, K. (2007). Combining global and local surrogate models to accelerate evolutionary optimization. IEEE Transactions on Systems, Man, and Cybernetics, Part C, 37(1):66-76.