Ensemble Classification and Regression-Recent Developments, Applications and Future Directions [Review Article]

IEEE Computational Intelligence Magazine

Volumn 11, Issue 1, 2016, Pages 41-53

  Ren, Ye ^a   Zhang, Le ^a   Suganthan, P N ^a  

^a NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

Author keywords

Bagging; Boosting; Computational modeling; Machine learning; Optimization; Performance evaluation

Indexed keywords

ADAPTIVE BOOSTING; ARTIFICIAL INTELLIGENCE; DECISION TREES; LEARNING ALGORITHMS; LEARNING SYSTEMS; MULTIOBJECTIVE OPTIMIZATION; OPTIMIZATION;

BAGGING; BOOSTING; COMPUTATIONAL MODEL; ENSEMBLE CLASSIFICATION; FUTURE RESEARCH DIRECTIONS; MULTIPLE KERNEL LEARNING; NEGATIVE CORRELATION LEARNING; PERFORMANCE EVALUATION;

CORRELATION METHODS;

EID: 84962026438     PISSN: 1556603X     EISSN: None     Source Type: Journal    
DOI: 10.1109/MCI.2015.2471235     Document Type: Review

References (186)

1. T. G. Dietterich, "Ensemble methods in machine learning," in Multiple Classifier Systems. Berlin, Germany: Springer-Verlag, 2000, pp. 1-15.
2. R. Kohavi and D. H. Wolpert, "Bias plus variance decomposition for zero-one loss functions," in Proc. Int. Conf. Machine Learning, 1996, pp. 275-283.
3. L. Breiman, "Random forests," Mach. Learn., vol. 45, no. 1, pp. 5-32, Oct. 2001.
4. Y. Zhao, J. Gao, and X. Yang, "A survey of neural network ensembles," in Proc. Int. Conf. Neural Networks Brain, Oct. 2005, vol. 1, pp. 438-442.
5. L. Rokach, "Ensemble-based classifiers," Artif. Intell. Rev., vol. 33, nos. 1-2, pp. 1-39, Feb. 2010.
6. D. Gopika and B. Azhagusundari, "An analysis on ensemble methods in classification tasks," Int. J. Adv. Res. Comput. Commun. Eng., vol. 3, no. 7, pp. 7423-7427, July 2014.
7. L. Rokach, "Ensemble-based classifiers," Artif. Intell. Rev., vol. 33, no. 1-2, pp. 1-39, 2010.
8. P. Yang, Y. H. Yang, B. B. Zhou, and A. Y. Zomaya, "A review of ensemble methods in bioinformatics," Curr. Bioinform., vol. 5, no. 4, pp. 296-308, Dec. 2010.
9. J. Mendes-Moreira, C. Soares, A. M. Jorge, and J. F. de Sousa, "Ensemble approaches for regression: A survey," ACM Comput. Surv., vol. 45, no. 1, pp. 1-10, Nov. 2012.
10. Y. Ren, P. N. Suganthan, and N. Srikanth, "Ensemble methods for wind and solar power forecasting: A state-ofthe-art review," Renew. Sustain. Energy Rev., vol. 50, pp. 82-91, Oct. 2015.
11. A. Krogh and J. Vedelsby, "Neural network ensembles, cross validation, and active learning," Adv. Neural Inform. Process. Syst., vol. 7, pp. 231-238, 1995.
12. G. Brown, J. Wyatt, R. Harris, and X. Yao, "Diversity creation methods: A survey and categorisation," Inform. Fusion, vol. 6, no. 1, pp. 5-20, Mar. 2005.
13. S. Geman, E. Bienenstock, and R. Doursat, "Neural networks and the bias/variance dilemma," Neural Comput., vol. 4, no. 1, pp. 1-58, Jan. 1992.
14. G. Brown, J. L. Wyatt, and P. Tino, "Managing diversity in regression ensembles," J. Mach. Learn. Res., vol. 6, pp. 1621-1650, Dec. 2005.
15. P. Domingos, "A unified bias-variance decomposition and its applications," in Proc Int. Conf. Machine Learning, 2000, pp. 231-238.
16. E. Kleinberg, "Stochastic discrimination," Ann. Math. Artif. Intell., vol. 1, no. 1, pp. 207-239, 1990.
17. R. E. Schapire and Y. Freund, "Boosting the margin: A new explanation for the effectiveness of voting methods," Ann. Stat., vol. 26, no. 5, pp. 1651-1686, 1998.
18. V. Pisetta, "New insights into decision tree ensembles," Ph.D. dissertation, Lumie Univ. Lyon 2, Lyon, France, 2012.
19. E. B. Kong and T. G. Dietterich, "Error-correcting output coding corrects bias and variance," in Proc. Int. Conf. Machine Learning, 1995, pp. 313-321.
20. J. H. Friedman, "On bias, variance, 0/1-Loss, and the curse-of-dimensionality," Data Mining Knowl. Disc., vol. 1, no. 1, pp. 55-77, 1997.
21. L. Breiman, "Arcing classifier (with discussion and a rejoinder by the author)," Ann. Stat., vol. 26, no. 3, pp. 801-849, 1998.
22. G. M. James, "Variance and bias for general loss functions," Mach. Learn., vol. 51, no. 2, pp. 115-135, 2003.
23. L. Zhang, Y. Ren, and P. N. Suganthan, "Towards generating random forest with extremely randomized trees," in Proc. IEEE Int. Joint Conf. Neural Networks, Beijing, China, July 2014, pp. 2645-2652.
24. L. Zhang and P. N. Suganthan, "Oblique decision tree ensemble via multisurface proximal support vector machine," IEEE Trans. Cybern., pp. 2168-2267, Nov. 2014.
25. L. Breiman, "Bias, variance, and arcing classifiers," Univ. California, Berkeley, CA, Tech. Rep. 460, 1996.
26. C.-X. Zhang and J.-S. Zhang, "RotBoost: A technique for combining rotation forest and adaboost," Pattern Recognit. Lett., vol. 29, no. 10, pp. 1524-1536, 2008.
27. Y. Freund and R. E. Schapire, "Experiments with a new boosting algorithm," in Proc. Int. Conf. Machine Learning, 1996, vol. 96, pp. 148-156.
28. E. K. Tang, P. N. Suganthan, and X. Yao, "An analysis of diversity measures," Mach. Learn., vol. 65, no. 1, pp. 247-271, 2006.
29. L. Breiman, "Bagging predictors," Mach. Learn., vol. 24, no. 2, pp. 123-140, 1996.
30. T. K. Ho, "The random subspace method for constructing decision forests," IEEE Trans. Pattern Anal. Mach. Intell., vol. 20, no. 8, pp. 832-844, 1998.
31. C. Tan, M. Li, and X. Qin, "Random subspace regression ensemble for near-infrared spectroscopic calibration of tobacco samples," Anal. Sci., vol. 24, no. 5, pp. 647-654, 2008.
32. L. Breiman, "Randomizing outputs to increase prediction accuracy," Mach. Learn., vol. 40, no. 3, pp. 229-242, 2000.
33. D. Opitz and R. Maclin, "Popular ensemble methods: An empirical study," J. Artif. Intell. Res., vol. 11, pp. 169-196, 1999.
34. M. Gnen and E. Alpaydn, "Multiple kernel learning algorithms," J. Mach. Learn. Res., vol. 12, pp. 2211-2268, July 2011.
35. Y. Ren and P. N. Suganthan, "A comparative study of empirical mode decomposition based short-term wind speed forecasting methods," IEEE Trans. Sustain. Energy, vol. 6, no. 1, pp. 236-244, 2015.
36. A. Chandra and X. Yao, "Multi-objective ensemble construction, learning and evolution," in Proc. PPSN Workshop Multi-objective Problem Solving from Nature (Part 9th Int. Conf. Parallel Problem Solving from Nature: PPSN-IX). Citeseer, 2006, pp. 9-13.
37. K. Deb, Multi-Objective Optimization Using Evolutionary Algorithms, vol. 16. Hoboken, NJ: Wiley, 2001.
38. L. I. Kuncheva, Fuzzy Classifier Design, vol. 49. Berlin, Germany: Springer-Verlag, 2000.
39. L. I. Kuncheva, "Fuzzy classifiers," Scholarpedia, vol. 3, no. 1, p. 2925, 2008, revision 133818.
40. H. Ishibuchi, T. Morisawa, and T. Nakashima, "Voting schemes for fuzzy-rule-based classification systems," in Proc. IEEE Int. Conf. Fuzzy Systems, 1996, vol. 1, pp. 614-620.
41. H. Ishibuchi, T. Nakashima, and T. Morisawa, "Voting in fuzzy rule-based systems for pattern classification problems," Fuzzy sets Syst., vol. 103, no. 2, pp. 223-238, 1999.
42. P. Bonissone, J. M. Cadenas, M. C. Garrido, and R. A. Dz-Valladares, "A fuzzy random forest," Int. J. Approx. Reason., vol. 51, no. 7, pp. 729-747, 2010.
43. M. Korytkowski, R. Nowicki, R. Scherer, and L. Rutkowski, "Ensemble of rough-neuro-fuzzy systems for classification with missing features," in Proc. IEEE Int. Conf. Fuzzy Systems, 2008, pp. 1745-1750.
44. N. C. Oza, "Online bagging and boosting," in Proc. IEEE Int. Conf. Systems, Man Cybernetics, 2005, vol. 3, pp. 2340-2345.
45. T. Hothorn and B. Lausen, "Double-bagging: Combining classifiers by bootstrap aggregation," Pattern Recognit., vol. 36, no. 6, pp. 1303-1309, 2003.
46. E. Bauer and R. Kohavi, "An empirical comparison of voting classification algorithms: Bagging, boosting, and variants," Mach. Learn., vol. 36, nos. 1-2, pp. 105-139, 1999.
47. D.-P. Niu, F.-l. Wang, L.-l. Zhang, D.-K. He, and M.-X. Jia, "Neural network ensemble modeling for nosiheptide fermentation process based on partial least squares regression," Chemometr. Intell. Lab. Syst., vol. 105, no. 1, pp. 125-130, Jan. 2011.
48. W. Kim, J. Park, J. Yoo, H. Kim, and C. G. Park, "Target localization using ensemble support vector regression in wireless sensor networks," IEEE Trans. Cybern., vol. 43, no. 4, pp. 1189-1198, Aug. 2013.
49. S. B. Kotsiantis, D. Kanellopoulos, and I. D. Zaharakis, "Bagged averaging of regression models," in Artificial Intelligence Applications and Innovations. Berlin, Germany: Springer-Verlag, 2006, pp. 53-60.
50. A. Scherbart and T. W. Nattkemper, "The diversity of regression ensembles combining bagging and random subspace method," in Proc. Advances Neuro-Information Processing, 2009, pp. 911-918.
51. R. E. Schapire and Y. Singer, "Improved boosting algorithms using confidence-rated predictions," Mach. Learn., vol. 37, no. 3, pp. 297-336, 1999.
52. G. Rsch, T. Onoda, and K.-R. Mler, "Soft margins for AdaBoost," Mach. Learn., vol. 42, no. 3, pp. 287-320, 2001.
53. A. Vezhnevets and V. Vezhnevets, "Modest Adaboostteaching Adaboost to generalize better," Graphicon, vol. 12, no. 5, pp. 987-997, 2005.
54. C. Domingo and O. Watanabe, "MadaBoost: A modification of AdaBoost," in Proc. Annu. Conf. Computational Learning Theory, 2000, pp. 180-189.
55. S. Avidan, "SpatialBoost: Adding spatial reasoning to adaboost," in Proc. Computer Vision, 2006, pp. 386-396.
56. J. Friedman, T. Hastie, and R. Tibshirani, "Additive logistic regression: A statistical view of boosting (with discussion and a rejoinder by the authors)," Ann. Stat., vol. 28, no. 2, pp. 337-407, 2000.
57. L. Breiman, "Pasting small votes for classification in large databases and on-line," Mach. Learn., vol. 36, nos. 1-2, pp. 85-103, 1999.
58. J. H. Friedman, "Stochastic gradient boosting," Comput. Stat. Data Anal., vol. 38, no. 4, pp. 367-378, 2002.
59. J. R. Lloyd, "GEFCom2012 hierarchical load forecasting: Gradient boosting machines and Gaussian processes," Int. J. Forecasting, vol. 30, no. 2, pp. 369-374, Apr. 2014.
60. R. Feely, "Predicting stock market volatility using neural networks," M.S. thesis, Ph.D.(B.A.) dissertation, Trinity College, Dublin, Ireland, 2000.
61. Y. Freund and R. E. Schapire, "A decision-theoretic generalization of on-line learning and an application to boosting," J. Comput. Syst. Sci., vol. 55, no. 1, pp. 119-139, Aug. 1997.
62. D. Shrestha and D. Solomatine, "Experiments with AdaBoost. RT, an improved boosting scheme for regression," Neural Comput., vol. 18, no. 7, pp. 1678-1710, 2006.
63. Y. Freund and R. E. Schapire, "Experiments with a new boosting algorithm," in Proc. Int. Conf. Machine Learning, 1996, vol. 96, pp. 148-156.
64. P. Kankanala, S. Das, and A. Pahwa, "Adaboost+: An ensemble learning approach for estimating weather-related outages in distribution systems," IEEE Trans. Power Syst., vol. 29, no. 1, pp. 359-367, Jan. 2014.
65. Y. Lin and Y. Jeon, "Random forests and adaptive nearest neighbors," J. Amer. Stat. Assoc., vol. 101, no. 474, pp. 578-590, 2006.
66. H. Jiang, Y. Deng, H.-S. Chen, L. Tao, Q. Sha, J. Chen, C.-J. Tsai, and S. Zhang, "Joint analysis of two microarray gene-expression data sets to select lung adenocarcinoma marker genes," BMC Bioinform, vol. 5, no. 1, p. 81, 2004.
67. K.-Q. Shen, C.-J. Ong, X. P. Li, Z. Hui, and E. P. Wilder-Smith, "A feature selection method for multilevel mental fatigue EEG classification," IEEE Trans. Biomed. Eng., vol. 54, no. 7, pp. 1231-1237, 2007.
68. Y. Ye, Q. Wu, J. Zhexue Huang, M. K. Ng, and X. Li, "Stratified sampling for feature subspace selection in random forests for high dimensional data," Pattern Recognit., vol. 46, no. 3, pp. 769-787, 2013.
69. L. Zhang, Y. Ren, and P. N. Suganthan, "Instance based random forest with rotated feature space," in Proc. IEEE Symp. Computational Intelligence Ensemble Learning, 2013, pp. 31-35.
70. P. Geurts, D. Ernst, and L. Wehenkel, "Extremely randomized trees," Mach. Learn., vol. 63, no. 1, pp. 3-42, 2006.
71. B. H. Menze, B. M. Kelm, D. N. Splitthoff, U. Koethe, and F. A. Hamprecht, "On oblique random forests," in Machine Learning and Knowledge Discovery in Databases. Berlin, Germany: Springer-Verlag, 2011, pp. 453-469.
72. L. Zhang and P. N. Suganthan, "Random forests with ensemble of feature spaces," Pattern Recognit., vol. 47, no. 10, pp. 3429-3437, Oct. 2014.
73. O. L. Mangasarian and E. W. Wild, "Multisurface proximal support vector machine classification via generalized eigenvalues," IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 1, pp. 69-74, 2006.
74. J. J. Rodriguez, L. I. Kuncheva, and C. J. Alonso, "Rotation forest: A new classifier ensemble method," IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 10, pp. 1619-1630, 2006.
75. L. I. Kuncheva and J. J. Rodruez, "An experimental study on rotation forest ensembles," in Multiple Classifier Systems. Berlin, Germany: Springer-Verlag, 2007, pp. 459-468.
76. K.-H. Liu and D.-S. Huang, "Cancer classification using rotation forest," Comput. Biol. Med., vol. 38, no. 5, pp. 601-610, 2008.
77. A. Ozcift, "SVM feature selection based rotation forest ensemble classifiers to improve computer-aided diagnosis of Parkinson disease," J. Med. Syst., vol. 36, no. 4, pp. 2141-2147, 2012.
78. J.-F. Xia, K. Han, and D.-S. Huang, "Sequence-based prediction of protein-protein interactions by means of rotation forest and autocorrelation descriptor," Protein Pept. Lett., vol. 17, no. 1, pp. 137-145, 2010.
79. G. Stiglic and P. Kokol, "Effectiveness of rotation forest in meta-learning based gene expression classification," in Proc. IEEE Int. Symp. Computer-Based Medical Systems, 2007, pp. 243-250.
80. D. H. Wolpert, "Stacked generalization," Neural Netw., vol. 5, no. 2, pp. 241-259, 1992.
81. L. Bo, L. Xinjun, and Z. Zhiyan, "Novel algorithm for constructing support vector machine regression ensemble," J. Syst. Eng. Electron., vol. 17, no. 3, pp. 541-545, 2006.
82. K. Lu and L. Wang, "A novel nonlinear combination model based on support vector machine for rainfall prediction," in Proc. IEEE Int. Joint Conf. Computational Sciences Optimization, 2011, pp. 1343-1346.
83. L. Wang and J. Wu, "Application of hybrid RBF neural network ensemble model based on wavelet support vector machine regression in rainfall time series forecasting," in Proc. IEEE Int. Joint Conf. Computational Sciences Optimization, 2012, pp. 867-871.
84. J.-S. Chou and A.-D. Pham, "Enhanced artificial intelligence for ensemble approach to predicting high performance concrete compressive strength," Const. Building Mater., vol. 49, pp. 554-563, Dec. 2013.
85. J. Wu, "A novel artificial neural network ensemble model based on k-nearest neighbor nonparametric estimation of regression function and its application for rainfall forecasting," in Proc. Int. Joint Conf. Computational Sciences Optimization, 2009, vol. 2, pp. 44-48.
86. S. Wang, L. Yu, L. Tang, and S. Wang, "A novel seasonal decomposition based least squares support vector regression ensemble learning approach for hydropower consumption forecasting in China," Energy, vol. 36, no. 11, pp. 6542-6554, Nov. 2011.
87. H. Liu, H.-Q. Tian, C. Chen, and Y.-F. Li, "A hybrid statistical method to predict wind speed and wind power," Renew. Energy, vol. 35, no. 8, pp. 1857-1861, Aug. 2010.
88. J. Catalao, H. Pousinho, and V. Mendes, "Hybrid wavelet-PSO-ANFIS approach for short-term wind power forecasting in Portugal," IEEE Trans. Sustain. Energy, vol. 2, no. 1, pp. 50-59, Jan. 2011.
89. A. U. Haque, P. Mandal, J. Meng, A. K. Srivastava, T.-L. Tseng, and T. Senjyu, "A novel hybrid approach based on wavelet transform and fuzzy ARTMAP network for predicting wind farm power production," in Proc. IEEE Industry Applications Society Annu. Meeting, Las Vegas, NV, Oct. 2012, pp. 1-8.
90. S. H. Cao and J. C. Cao, "Forecast of solar irradiance using recurrent neural networks combined with wavelet analysis," Appl. Thermal Eng., vol. 25, nos. 2-3, pp. 161-172, Feb. 2005.
91. N. E. Huang, Z. Shen, S. R. Long, M. Wu, H. H. Shih, Q. Zheng, N.-C. Yen, C. C. Tung, and H. H. Liu, "The empirical mode decomposition and Hilbert spectrum for nonlinear and nonstationary time series analysis," Proc. Roy. Soc. London A, vol. 454, no. 1971, pp. 903-995, 1998.
92. X. Wang and H. Li, "One-month ahead prediction of wind speed and output power based on EMD and LSSVM," in Proc. Int. Conf. Energy Environment Technology, 2009, vol. 3, pp. 439-442.
93. Y. Zhang, U. J. Lu, Y. Meng, H. Yan, and H. Li, "Wind power short-term forecasting based on empirical mode decomposition and chaotic phase space reconstruction," Automat. Electric Power Syst., vol. 36, no. 5, pp. 24-28, 2012.
94. X. Fan and Y. Zhu, "The application of empirical mode decomposition and gene expression programming to short-term load forecasting," in Proc. Int. Conf. Natural Computation, Yantai, China, Aug. 10-12, 2010, vol. 8, pp. 4331-4334.
95. Y. Ren and P. N. Suganthan, "Empirical mode decomposition-k nearest neighbor models for wind speed forecasting," J. Power Energy Eng., vol. 2, no. 4, pp. 176-185, 2014.
96. Y. Ren, P. N. Suganthan, and N. Srikanth, "A novel empirical mode decomposition with support vector regression for wind speed forecasting," IEEE Trans. Neural Netw. Learn. Syst., 10.1109/TNNLS.2014.2351391.
97. Y. Ren, X. Qiu, and P. N. Suganthan, "Empirical mode decomposition based adaboost-backpropagation neural network method for wind speed forecasting," in Proc. IEEE Symp. Computational Intelligence Ensemble Learning, Orlando, FL, Dec. 2014, pp. 1-6.
98. J. Shi, J. Guo, and S. Zheng, "Evaluation of hybrid forecasting approaches for wind speed and power generation time series," Renew. Sustain. Energy Rev., vol. 16, no. 5, pp. 3471-3480, June 2012.
99. J. Wu and C. C. Keong, "Prediction of hourly solar radiation using a novel hybrid model of ARMA and TDNN," Solar Energy, vol. 85, no. 5, pp. 808-817, 2011.
100. Y. Ren, P. N. Suganthan, N. Srikanth, and S. Sarkar, "A hybrid ARIMA-DENFIS method for wind speed forecasting," in Proc. IEEE Int. Conf. Fuzzy Systems, 2013, pp. 1-6.
101. M. Nguyen, H. A. Abbass, and R. I. McKay, "A novel mixture of experts model based on cooperative coevolution," Neurocomputing, vol. 70, nos. 1-3, pp. 155-163, 2006.
102. M. Nguyen, H. A. Abbass, and R. I. McKay, "Analysis of CCME: Coevolutionary dynamics, automatic problem decomposition and regularization," IEEE Trans. Syst. Man, Cybern. C, vol. 38, no. 1, pp. 100-109, 2008.
103. D. L. Gonzalez, Z. Chen, I. K. Tetteh, T. Pansombut, F. Semazzi, V. Kumar, A. Melechko, and N. F. Samatova, "Hierarchical classifierregression ensemble for multi-phase non-linear dynamic system response prediction: Application to climate analysis," in Proc. IEEE Int. Conf. Data Mining Workshops, 2012, pp. 781-788.
104. S. Kohli, S. Prakash, and P. Gupta, "Hierarchical age estimation with dissimilarity-based classification," Neurocomputing, vol. 120, pp. 164-176, Nov. 2013.
105. Y. Liu and X. Yao, "Ensemble learning via negative correlation," Neural Netw., vol. 12, no. 10, pp. 1399-1404, Dec. 1999.
106. G. Brown, J. L. Wyatt, and P. Tino, "Managing diversity in regression ensembles," J. Mach. Learn. Res., vol. 6, pp. 1621-1650, Dec. 2005.
107. M. Alhamdoosh and D. Wang, "Fast decorrelated neural network ensembles with random weights," Inform. Sci., vol. 264, pp. 104-117, Apr. 2014.
108. H. H. Dam, H. A. Abbass, C. Lokan, and X. Yao, "Neural based learning classifier systems," IEEE Trans. Knowl. Data Eng., vol. 20, no. 1, pp. 26-39, Jan. 2008.
109. F. Ferndez-Navarro, P. Antonio Gutirez, C. Herv-Martinez, and X. Yao, "Negative correlation ensemble learning for ordinal regression," IEEE Trans. Neural Netw. Learn. Syst, vol. 24, no. 11, pp. 1836-1849, Nov. 2013.
110. Y. Jin, T. Okabe, and B. Sendhoff, "Neural network regularization and ensembling using multi-objective evolutionary algorithms," in Proc. Congr. Evolutionary Computation, Portland, OR, 2004, pp. 1-8.
111. A. Chandra and X. Yao, "DIVACE: Diverse and accurate ensemble learning algorithm," in Intelligent Data Engineering and Automated Learning-IDEAL. Berlin, Germany: Springer-Verlag, 2004, pp. 619-625.
112. A. Chandra and X. Yao, "Ensemble learning using multi-objective evolutionary algorithms," J. Math. Model. Algorithms, vol. 5, no. 4, pp. 417-445, 2006.
113. H. Abbass, "Pareto neuro-ensembles," in AI 2003: Advances in Artificial Intelligence, vol. 2903, T. Gedeon and L. Fung, Eds. Berlin, Germany: Springer-Verlag, 2003, pp. 554-566.
114. H. A. Abbass, "Pareto neuro-evolution: Constructing ensemble of neural networks using multi-objective optimization," in Proc. IEEE Congr. Evolutionary Computation, 2003, vol. 3, pp. 2074-2080.
115. H. A. Abbass, R. Sarker, and C. Newton, "PDE: A Pareto-frontier differential evolution approach for multiobjective optimization problems," in Proc. IEEE Congr. Evolutionary Computation, 2001, vol. 2, pp. 971-978.
116. S. Gu and Y. Jin, "Generating diverse and accurate classifier ensembles using multi-objective optimization," in Proc. IEEE Symp. Computational Intelligence Multi-Criteria Decision-Making, 2014, pp. 9-15.
117. A. Chandra and X. Yao, "Evolving hybrid ensembles of learning machines for better generalisation," Neurocomputing, vol. 69, no. 7, pp. 686-700, 2006.
118. D. W. Opitz and J. W. Shavlik, "Generating accurate and diverse members of a neural-network ensemble," in Advances in Neural Information Processing Systems, vol. 8. Cambridge, MA: MIT Press, 1996, pp. 535-541.
119. H. Chen and X. Yao, "Evolutionary multiobjective ensemble learning based on Bayesian feature selection," in Proc. IEEE Congr. Evolutionary Computation, 2006, pp. 267-274.
120. H. Chen and X. Yao, "Multiobjective neural network ensembles based on regularized negative correlation learning," IEEE Trans. Knowl. Data Eng., vol. 22, no. 12, pp. 1738-1751, 2010.
121. W.-C. Chen, L.-Y. Tseng, and C.-S. Wu, "A unified evolutionary training scheme for single and ensemble of feedforward neural network," Neurocomputing, vol. 143, pp. 347-361, Nov. 2014.
122. C. J. Tan, C. P. Lim, and Y.-N. Cheah, "A multiobjective evolutionary algorithm-based ensemble optimizer for feature selection and classification with neural network models," Neurocomputing, vol. 125, pp. 217-228, Feb. 2014.
123. U. Bhowan, M. Johnston, and M. Zhang, "Ensemble learning and pruning in multi-objective genetic programming for classification with unbalanced data," in AI 2011: Advances in Artificial Intelligence. Berlin, Germany: Springer-Verlag, 2011, pp. 192-202.
124. D. Kocev, C. Vens, J. Struyf, and S. Džeroski, Ensembles of Multiobjective Decision Trees. Berlin, Germany: Springer-Verlag, 2007.
125. J.-C. Lesque, A. Durand, C. Gagn and R. Sabourin, "Multi-objective evolutionary optimization for generating ensembles of classifiers in the ROC space," in Proc. Genetic Evolutionary Computation, ACM, 2012, pp. 879-886.
126. N. Kondo, T. Hatanaka, and K. Uosaki, "RBF networks ensemble construction based on evolutionary multi-objective optimization," J. Adv. Comput. Intell. Informat., vol. 12, no. 3, pp. 297-303, 2008.
127. F. Gullo, A. Talukder, S. Luke, C. Domeniconi, and A. Tagarelli, "Multiobjective optimization of coclustering ensembles," in Proc. Annu. Conf. Companion Genetic Evolutionary Computation, ACM, 2012, pp. 1495-1496.
128. L. L. Minku and X. Yao, "An analysis of multiobjective evolutionary algorithms for training ensemble models based on different performance measures in software effort estimation," in Proc. Int. Conf. Predictive Models Software Engineering, ACM, 2013, p. 8.
129. C. Smith and Y. Jin, "Evolutionary multi-objective generation of recurrent neural network ensembles for time series prediction," Neurocomputing, vol. 143, pp. 302-311, Nov. 2014.
130. C. Smith, J. Doherty, and Y. Jin, "Multi-objective evolutionary recurrent neural network ensemble for prediction of computational f luid dynamic simulations," in Proc. IEEE Congr. Evolutionary Computation, 2014, pp. 2609-2616.
131. W. Du, S. Y. S. Leung, and C. K. Kwong, "Time series forecasting by neural networks: A knee pointbased multiobjective evolutionary algorithm approach," Expert Syst. Aplicat., vol. 41, no. 18, pp. 8049-8061, 2014.
132. L. I. Kuncheva, "'Fuzzy' versus 'nonfuzzy' in combining classifiers designed by boosting," IEEE Trans. Fuzzy Syst., vol. 11, no. 6, pp. 729-741, 2003.
133. R. Scherer, Multiple Fuzzy Classification Systems. Berlin, Germany: Springer-Verlag, 2014.
134. A.-M. Yang, Y.-M. Zhou, and M. Tang, "A classifier ensemble method for fuzzy classifiers," in Fuzzy Systems and Knowledge Discovery. Berlin, Germany: Springer-Verlag, 2006, pp. 784-793.
135. O. Cord, A. Quirin, and L. Schez, "A first study on bagging fuzzy rule-based classification systems with multicriteria genetic selection of the component classifiers," in Proc. Int. Workshop Genetic Evolving Systems, 2008, pp. 11-16.
136. Z. Fu, D. Zhang, L. Wang, and X. Li, "Ensemble learning algorithm in classification of fuzzy-classes," J. Comput. Inform. Syst., vol. 9, no. 22, pp. 8929-8938, 2013.
137. Y. Nojima and H. Ishibuchi, "Designing fuzzy ensemble classifiers by evolutionary multiobjective optimization with an entropy-based diversity criterion," in Proc. Int. Conf. Hybrid Intelligent Systems, 2006, pp. 59-59.
138. H. Ishibuchi and T. Yamamoto, "Evolutionary multiobjective optimization for generating an ensemble of fuzzy rule-based classifiers," in Proc. Genetic Evolutionary Computation, 2003, pp. 1077-1088.
139. J. Cao, H. Wang, S. Kwong, and K. Li, "Combining interpretable fuzzy rule-based classifiers via multiobjective hierarchical evolutionary algorithm," in Proc. IEEE Int. Conf. Systems, Man, Cybernetics, 2011, pp. 1771-1776.
140. Y. Zhang, H. Yang, S. Prasad, E. Pasolli, J. Jung, and M. Crawford, "Ensemble multiple kernel active learning for classification of multisource remote sensing data," IEEE J. Select. Topics Appl. Earth Observ. Remote Sensing, vol. 8, no. 2, pp. 845-858, Feb. 2015.
141. X. Wang, X. Liu, N. Japkowicz, and S. Matwin, "Ensemble of multiple kernel SVM classifiers," in Advances in Artificial Intelligence. Berlin, Germany: Springer-Verlag, 2014, pp. 239-250.
142. N. Cristianini, J. Kandola, A. Elisseeff, and J. Shawe-Taylor, "On kernel target alignment," in Advances in Neural Information Processing Systems, vol. 14. Cambridge, MA: MIT Press, 2002, pp. 367-373.
143. G. R. Lanckriet, N. Cristianini, P. Bartlett, L. E. Ghaoui, and M. I. Jordan, "Learning the kernel matrix with semidefinite programming," J. Mach. Learn. Res., vol. 5, pp. 27-72, Jan. 2004.
144. S. Sonnenburg, G. Rsch, and C. Scher, "A general and efficient multiple kernel learning algorithm," in Proc. Neural Information Processing Systems, Dec. 2005, pp. 1275-1282.
145. C. Cortes, M. Mohri, and A. Rostamizadeh, "Two-stage learning kernel algorithms," in Proc. Int. Conf. Machine Learning, 2010, pp. 239-246.
146. C. Cortes, M. Kloft, and M. Mohri, "Learning kernels using local rademacher complexity," in Proc. Advances in Neural Information Processing Systems, 2013, pp. 2760-2768.
147. F. Orabona, L. Jie, and B. Caputo, "Multi kernel learning with online-batch optimization," J. Mach. Learn. Res., vol. 13, no. 1, pp. 227-253, 2012.
148. X. Xu, I. W. Tsang, and D. Xu, "Soft margin multiple kernel learning," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 5, pp. 749-761, 2013.
149. T. Suzuki and R. Tomioka, "SpicyMKL: A fast algorithm for multiple kernel learning with thousands of kernels," Mach. Learn., vol. 85, nos. 1-2, pp. 77-108, 2011.
150. M. Kloft, U. Brefeld, S. Sonnenburg, and A. Zien, "lp-norm multiple kernel learning," J. Mach. Learn. Res., vol. 12, pp. 953-997, Mar. 2011.
151. J. Zhuang, I. W. Tsang, and S. Hoi, "Two-layer multiple kernel learning," in Proc. Int. Conf. Artificial Intelligence Statistics, 2011, pp. 909-917.
152. Z. Sun, N. Ampornpunt, M. Varma, and S. Vishwanathan, "Multiple kernel learning and the SMO algorithm," in Proc. Advances Neural Information Processing Systems, 2010, pp. 2361-2369.
153. F. R. Bach, G. R. Lanckriet, and M. I. Jordan, "Multiple kernel learning, conic duality, and the SMO algorithm," in Proc. Int. Conf. Machine Learning, 2004, p. 6.
154. A. Rakotomamonjy, F. Bach, S. Canu, and Y. Grandvalet, "SimpleMKL," J. Mach. Learn. Res., vol. 9, pp. 2491-2521, Nov. 2008.
155. A. Zien and C. S. Ong, "Multiclass multiple kernel learning," in Proc. Int. Conf. Mach. Learn., 2007, pp. 1191-1198. attend the panel "IEEE and CIS in the Next Decade", that will be held during WCCI 2016, the World Congress on Computational Intelligence, that will be held on July 25-29, in Vancouver, Canada (http://www.wcci2016.org/).
156. K. Crammer, J. Keshet, and Y. Singer, "Kernel design using boosting," in Proc. Advances Neural Information Processing Systems, 2002, pp. 537-544.
157. V. Pisetta, P.-E. Jouve, and D. A. Zighed, "Learning with ensembles of randomized trees: New insights," in Machine Learning and Knowledge Discovery in Databases, Berlin, Germany: Springer-Verlag, 2010, pp. 67-82.
158. Y. Bengio, "Learning deep architectures for AI," Found. Trends Mach. Learn., vol. 2, no. 1, pp. 1-127, 2009.
159. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proc. IEEE, vol. 86, no. 11, pp. 2278-2324, 1998.
160. D. Ciresan, U. Meier, and J. Schmidhuber, "Multicolumn deep neural networks for image classification," in Proc. IEEE Conf. Computer Vision Pattern Recognition, 2012, pp. 3642-3649.
161. D. Ciresan, U. Meier, J. Masci, and J. Schmidhuber, "Multi-column deep neural network for traffic sign classification," Neural Netw., vol. 32, pp. 333-338, Aug. 2012.
162. G. E. Hinton and R. R. Salakhutdinov, "Reducing the dimensionality of data with neural networks," Science, vol. 313, no. 5786, pp. 504-507, 2006.
163. F. Agostinelli, M. R.,erson, and H. Lee, "Adaptive multi-column deep neural networks with application to robust image denoising," in Proc. Advances Neural Information Processing Systems, 2013, pp. 1493-1501.
164. P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P.-A. Manzagol, "Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion," J. Mach. Learn. Res., vol. 11, pp. 3371-3408, Dec. 2010.
165. A. Abdullah, R. C. Veltkamp, and M. A. Wiering, "An ensemble of deep support vector machines for image categorization," in Proc. IEEE Int. Conf. Soft Computing Pattern Recognition, 2009, pp. 301-306.
166. D. M. J. Tax, R. P. W. Duin, and M. van Breukelen, "Comparison between product and mean classifier combination rules," in Proc. Workshop Statistical Pattern Recognition, 1997, pp. 165-170.
167. A. Abdullah, R. C. Veltkamp, and M. A. Wiering, "Spatial pyramids and two-layer stacking SVM classifiers for image categorization: A comparative study," in Proc. IEEE Int. Joint Conf. Neural Networks, 2009, pp. 5-12.
168. G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov, "Improving neural networks by preventing co-adaptation of feature detectors," arXiv preprint arXiv:1207.0580, July 2012.
169. J. Ba and B. Frey, "Adaptive dropout for training deep neural networks," in Proc. Advances Neural Information Processing Systems, 2013, pp. 3084-3092.
170. L. Wan, M. Zeiler, S. Zhang, Y. L. Cun, and R. Fergus, "Regularization of neural networks using dropconnect," in Proc. Int. Conf. Machine Learning, 2013, pp. 1058-1066.
171. A. Berg, J. Deng, and L. Fei-Fei, "Large scale visual recognition challenge 2010," 2010.
172. E. Busseti, I. Osband, and S. Wong, "Deep learning for time series modeling," Univ. Stanford, Stanford, CA, Tech. Rep. CS 229, Dec. 2012.
173. M. Wiering, M. Schutten, A. Millea, A. Meijster, and L. R. B. Schomaker, "Deep support vector machines for regression problems," in Proc. Int. Workshop Advances Regularization, Optimization, Kernel Methods, Support Vector Machines: Theory Applications, 2013.
174. X. Qiu, L. Zhang, Y. Ren, P. N. Suganthan, and G. Amaratunga, "Ensemble deep learning for regression and timeseries forecasting," in Proc. IEEE Symp. Computational Intelligence Ensemble Learning, Orlando, FL, Dec. 2014, pp. 1-6.
175. S. M. Halawani, I. A. Albidewi, and A. Ahmad, "A novel ensemble method for regression via classification problems," J. Comput. Sci., vol. 7, no. 3, pp. 387-393, 2011.
176. Z.-H. Zhou, N. V. Chawla, Y. Jin, and G. J. Williams, "Big data opportunities and challenges: Discussions from data analytics perspectives [discussion forum]," IEEE Comput. Intell. Mag., vol. 9, no. 4, pp. 62-74, 2014.
177. S. Alam, G. Dobbie, Y. S. Koh, P. Riddle, and S. U. Rehman, "Research on particle swarm optimization based clustering: A systematic review of literature and techniques," Swarm Evol. Comput., vol. 17, pp. 1-13, Aug. 2014.
178. S. J. Nanda and G. Panda, "A survey on nature inspired metaheuristic algorithms for partitional clustering," Swarm Evol. Comput., vol. 16, pp. 1-18, June 2014.
179. P. Mc Leod and B. Verma, "Variable hidden neuron ensemble for mass classification in digital mammograms [application notes]," IEEE Comput. Intell. Mag., vol. 8, no. 1, pp. 68-76, 2013.
180. M. Ferndez-Delgado, E. Cernadas, S. Barro, and D. Amorim, "Do we need hundreds of classifiers to solve real world classification problems" J. Mach. Learn. Res., vol. 15, no. 1, pp. 3133-3181, 2014.
181. R. Mallipeddi, S. Jeyadevi, P. N. Suganthan, and S. Baskar, "Efficient constraint handling for optimal reactive power dispatch problems," Swarm Evol. Comput., vol. 5, pp. 28-36, Aug. 2012.
182. R. Mallipeddi, P. N. Suganthan, Q.-K. Pan, and M. F. Tasgetiren, "Differential evolution algorithm with ensemble of parameters and mutation strategies," Appl. Soft Comput., vol. 11, no. 2, pp. 1679-1696, 2011.
183. Y. Chen, W. Xie, and X. Zou, "How can surrogates inf luence the convergence of evolutionary algorithms" Swarm Evol. Comput., vol. 12, pp. 18-23, Oct. 2013.
184. Y. Jin, "Surrogate-assisted evolutionary computation: Recent advances and future challenges," Swarm Evol. Comput., vol. 1, no. 2, pp. 61-70, 2011.
185. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database," in Proc. IEEE Conf. Computer Vision Pattern Recognition, 2009, pp. 248-255.
186. G. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, "Labeled faces in the wild: A database for studying face recognition in unconstrained environments," Univ. Massachusetts, Amherst, MA, Tech. Rep. 07-49, Oct. 2007.