Comparison of a genetic algorithm and simulated annealing for automatic neural network ensemble development

Neurocomputing

Volumn 121, Issue , 2013, Pages 498-511

  Soares, Symone ^a   Antunes, Carlos Henggeler ^b   Araújo, Rui ^a  

^a UNIVERSITY OF COIMBRA   (Portugal)

^b INESC COIMBRA   (Portugal)

Author keywords

Ensemble learning; Genetic algorithm; Neural network; Simulated annealing

Indexed keywords

COMBINATION STRATEGIES; ENSEMBLE LEARNING; GENERALIZATION ABILITY; MACHINE LEARNING COMMUNITIES; NEGATIVE CORRELATION LEARNING; NEURAL NETWORK ENSEMBLES; OPTIMIZATION TECHNIQUES; PREDICTION PERFORMANCE;

ADAPTIVE BOOSTING; GENETIC ALGORITHMS; NEURAL NETWORKS; PERSONNEL TRAINING;

SIMULATED ANNEALING;

ANALYTICAL PARAMETERS; ARTICLE; ARTIFICIAL NEURAL NETWORK; AUTOMATIC NEURAL NETWORK ENSEMBLE; AUTOMATION; DATA PROCESSING; DECISION MAKING; ENSEMBLE LEARNING; GENETIC ALGORITHM; INTERMETHOD COMPARISON; MACHINE LEARNING; METHODOLOGY; MODEL; PRIORITY JOURNAL; PROBLEM SOLVING; PROCESS OPTIMIZATION; REGRESSION ANALYSIS; SIMULATED ANNEALING; SIMULATION;

EID: 84884161728     PISSN: 09252312     EISSN: 18728286     Source Type: Journal    
DOI: 10.1016/j.neucom.2013.05.024     Document Type: Article

References (56)

1. Wang H., Khoshgoftaar T.M., Napolitano A. Software measurement data reduction using ensemble techniques. Neurocomputing 2012, 92:124-132.
2. S. Soares, R. Araújo, P. Sousa, F. Souza, Design and application of soft sensor using ensemble methods, in: Proceedings of 2011 IEEE 16th Conference on Emerging Technologies Factory Automation (ETFA 2011), 2011, pp. 1-8.
3. Kadlec P., Gabrys B. Local learning-based adaptive soft sensor for catalyst activation prediction. AIChE J. 2011, 57(5):1288-1301.
4. J.Z. Kolter, M.A. Maloof, Dynamic weighted majority: a new ensemble method for tracking concept drift, in: Proceedings of the Third IEEE International Conference on Data Mining (ICDM 2003), 2003, pp. 123-130.
5. T. G. Dietterich, Ensemble methods in machine learning, in: J. Kittler, F. Roli (Eds.), Proceedings of the First International Workshop on (MCS 2000) Multiple Classifier Systems, Lecture Notes in Computer Science, vol. 1857, Springer-Verlag, Berlin, Heidelberg, 2000, pp. 1-15.
6. Brown G., Wyatt J., Harris R., Yao X. Diversity creation methods. a survey and categorisation. Inf. Fusion 2005, 6(1):5-20.
7. Liu Y., Yao X., Higuchi T. Evolutionary ensembles with negative correlation learning. IEEE Trans. Evol. Comput. 2000, 4(4):380-387.
8. Coelho A.L.V., Nascimento D.S.C. On the evolutionary design of heterogeneous bagging models. Neurocomputing 2010, 73(16-18):3319-3322.
9. Xie Z., Xu Y., Hu Q., Zhu P. Margin distribution based bagging pruning. Neurocomputing 2012, 85(May (15)):11-19.
10. Lee H., Hong S., Kim E. Neural network ensemble with probabilistic fusion and its application to gait recognition. Neurocomputing 2009, 72(7-9):1557-1564.
11. Polikar R. Ensemble learning. Ensemble Machine Learning 2012, 1-34. Springer. C. Zhang, Y. Ma (Eds.).
12. M.D. Redel-Macías, F. Fernández-Navarro, P.A. Gutiérrez, A.J. Cubero-Atienza, C. Hervás-Martínez, Ensembles of evolutionary product unit or RBF neural networks for the identification of sound for pass-by noise test in vehicles, Neurocomputing 109 (0) (2013) 56-65, http://dx.doi.org/10.1016/j.neucom.2012.03.022.
13. Breiman L. Bagging predictors. Mach. Learn. 1996, 24(2):123-140.
14. Freund Y., Schapire R. A short introduction to boosting. Jpn. Soc. Artif. Intell. 1999, 14(5):771-780.
15. Wang D., Alhamdoosh M. Evolutionary extreme learning machine ensembles with size control. Neurocomputing 2013, 102:98-110.
16. R. Caruana, A. Niculescu-Mizil, G. Crew, A. Ksikes, Ensemble selection from libraries of models, in: Proceedings of the Twenty-First International Conference on Machine Learning (ICML 2004), ACM, Banff, Alberta, Canada, 2004, pp. 18-25.
17. Martínez-Muñoz G., Hernández-Lobato D., Suárez A. An analysis of ensemble pruning techniques based on ordered aggregation. IEEE Trans. Pattern Anal. Mach. Intell. 2009, 31(2):245-259.
18. G. Martínez-Muñoz, A. Suárez, Pruning in ordered bagging ensembles, in: Proceedings of the 23rd International Conference on Machine Learning (ICML 2006), ACM, Pittsburgh, PA, 2006, pp. 609-616.
19. Castro P.A.D., Zuben F.J.V. Learning ensembles of neural networks by means of a Bayesian artificial immune system. IEEE Trans. Neural Networks 2011, 22(2):304-316.
20. Kim Y.-W., Oh I.-S. Classifier ensemble selection using hybrid genetic algorithms. Pattern Recognition Lett. 2008, 29(6):796-802.
21. Zhou Z.-H., Wu J., Tang W. Ensembling neural networks. many could be better than all. Artif. Intell. 2002, 137(1-2):239-263. code available at: http://lamda.nju.edu.cn/files/Gasen.zip
22. Dondeti S., Kannan K., Manavalan R. Genetic algorithm optimized neural networks ensemble for estimation of mefenamic acid and paracetamol in tablets. Acta Chim. Slov. 2005, 52(4):440-449.
23. Chandra A., Chen H., Yao X. Trade-off between diversity and accuracy in ensemble generation. Multi-Objective Machine Learning, Studies in Computational Intelligence 2006, vol. 16:429-464. Springer, Berlin, Heidelberg.
24. Jia Y., Culver T.B. Bootstrapped artificial neural networks for synthetic flow generation with a small data sample. J. Hydrol. 2006, 331(3-4):580-590.
25. D. Cristinacce, T.F. Cootes, Boosted regression active shape models, in: Proceedings of British Machine Vision Conference, BMVA Press, 2007, pp. 79.1-79.10.
26. M. Ries, O. Nemethova, M. Rupp, Performance evaluation of mobile video quality estimators, in: Proceedings of the 15th European Signal Processing Conference, Poznan, Poland, 2007, pp. 159-163.
27. Fortuna L., Graziani S., Xibilia M.G. Comparison of soft-sensor design methods for industrial plants using small data sets. IEEE Trans. Instrum. Meas. 2009, 58(8):2444-2451.
28. Yu-Bo T., Zhi-Bin X. Particle-swarm-optimization-based selective neural network ensemble and its application to modeling resonant frequency of microstrip antenna. Microstrip Antennas 2011, 69-82. InTech. N. Nasimuddin (Ed.).
29. Re M., Valentini G. Ensemble Methods. A Review, Chapman & Hall/CRC Data Mining and Knowledge Discovery Series 2012, 563-582. CRC Press, Boca Raton, (Ch. 26).
30. J. Torres-Sospedra, M. Fernández-Redondo, C. Hernández-Espinosa, A research on combination methods for ensembles of multilayer feedforward, in: Proceedings of IEEE International Joint Conference on Neural Networks (IJCNN 2005), vol. 2, 2005, pp. 1125-1130.
31. Kasabov N.K. Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering 1996, MIT Press, Cambridge, MA, USA. 1st Edition.
32. Aladag C.H. A new architecture selection method based on tabu search for artificial neural networks. Expert Systems with Applications 2011, 38(4):3287-3293.
33. Gómez I., Franco L., Jerez J.M. Neural network architecture selection. Can function complexity help?. Neural Process. Lett. 2009, 30(2):71-87.
34. P.L. Rosin, F. Fierens, Improving neural network generalisation, in: Proceedings of International Geoscience and Remote Sensing Symposium (IGARSS '95), vol. 2, Firenze, Italy, 1995, pp. 1255-1257.
35. Hansen L.K., Salamon P. Neural network ensembles. IEEE Trans. Pattern Anal. Mach. Intell. 1990, 12(10):993-1001.
36. Hashem S. Optimal linear combinations of neural networks. Neural Networks 1994, 10(4):599-614.
37. Efron B., Tibshirani R.J. An Introduction to the Bootstrap 1993, Chapman & Hall, New York.
38. Hagan M.T., Demuth H.B., Beale M. Neural Network Design 1996, PWS Publishing, Boston, MA, USA.
39. D. Nguyen, B. Widrow, Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights, in: Proceedings of International Joint Conference on Neural Networks (IJCNN 1990), vol. 3, 1990, pp. 21-26.
40. J. Škutová, Weights initialization methods for MLP neural networks, Transactions of the VŠB-Technical University of Ostrova, Mechanical Series LIV (2) (2008) 147-152.
41. Polikar R. Ensemble based systems in decision making. IEEE Circuits Syst. Mag. 2006, 6(3):21-45.
42. Holland J.H. Adaptation in Natural and Artificial Systems 1992, MIT Press, Cambridge, MA, USA.
43. Sivanandam S.N., Deepa S.N. Introduction to Genetic Algorithms 2007, Springer.
44. S. Soares, C. Antunes, R. Araújo, A genetic algorithm for designing neural network ensembles, in: Proceedings of Fourteenth International Conference on Genetic and Evolutionary Computation Conference (GECCO 2012), ACM, Philadelphia, PA, USA, 2012, pp. 681-688.
45. Haupt R.L, Haupt S.E. Practical Genetic Algorithms 2004, Wiley-Interscience. 2nd Edition.
46. R. Chibante (Ed.), Simulated Annealing, Theory with Applications, Global Optimization, Sciyo, 2010.
47. Michalewicz Z., Fogel D.B. How to Solve it. Modern Heuristics 2000, Springer-Verlag, Berlin, Germany.
48. L. Torgo, Regression Datasets, Laboratory of Artificial Intelligence and Decision Support (LIAAD), University of Porto 2011. http://www.liaad.up.pt/~ltorgo/Regression/DataSets.html.
49. Friedman J.H. Multivariate adaptive regression splines. Ann. Stat. 1991, 19(1):1-67.
50. Belsley D.A., Kuh E., Welsch R.E. Regression Diagnostics. Identifying Influential Data and Sources of Collinearity 1980, Wiley.
51. R. Caruana, S. Lawrence, L. Giles, Overfitting in neural nets: backpropagation, conjugate gradient, and early stopping, in: Proceedings of Neural Information Processing Systems Conference (NIPS 2000), 2000, pp. 402-408.
52. Jeong D.-I., Kim Y.-O. Rainfall-runoff models using artificial neural networks for ensemble streamflow prediction. Hydrol. Process. 2005, 19(19):3819-3835.
53. C.R. Houck, J.A. Joines, M.G. Kay, A genetic algorithm for function optimization: A matlab implementation, Technical Report, North Carolina State University, Raleigh, NC, USA, 1996. http://www.daimi.au.dk/~pmn/Matlab/dochome/toolbox/GAOT/.
54. Liu Y., Yao X. Ensemble learning via negative correlation. Neural Networks 1999, 12(10):1399-1404.
55. Brown G., Wyatt J.L., Tiňo P. Managing diversity in regression ensembles. J. Mach. Learn. Res. 2005, 6:1621-1650.
56. A. Cordiner, Adaboost Toolbox-A Matlab Toolbox For Adaptive Boosting, Technical Report, School of Computer Science and Software Engineering, University of Wollongong, Wollongong, Australia, 2009. http://dl.dropbox.com/u/6830023/blog/adaboost_toolbox/AdaBoost.zip.