Comparing evolutionary hybrid systems for design and optimization of multilayer perceptron structure along training parameters

Information Sciences

Volumn 177, Issue 14, 2007, Pages 2884-2905

  Castillo, P A ^a   Merelo, J J ^a   Arenas, M G ^b   Romero, G ^a  

^a UNIVERSITY OF GRANADA   (Spain)

^b UNIVERSITY OF JAÉN   (Spain)

Author keywords

Artificial neural networks; Co evolution; Evolutionary algorithms; G Prop; Hybrid algorithms; Multilayer perceptrons; Optimization

Indexed keywords

ALGORITHM PARAMETERS; CO-EVOLUTION; G-PROP; HYBRID ALGORITHMS;

CLASSIFICATION (OF INFORMATION); EVOLUTIONARY ALGORITHMS; OPTIMIZATION; PARAMETER ESTIMATION;

MULTILAYER NEURAL NETWORKS;

EID: 34247536736     PISSN: 00200255     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ins.2007.02.021     Document Type: Article

References (108)

1. E. Alpaydim, GAL: Networks that grow when they learn and shrink when they forget, International Journal of Pattern Recognition and Artificial Intelligence 8 (1) (1994) 391-414.
2. P.J. Angeline, G.M. Saunders, J.B. Pollack, An evolutionary algorithm that construct recurrent neural networks, IEEE Transactions on Neural Networks 5 (1994) 54-64.
3. Giuliano Armano, Michele Marchesi, Andrea Murru, A hybrid genetic-neural architecture for stock indexes forecasting, Information Sciences 170 (1) (2005) 3-33.
4. G. Bebis, M. Georgiopoulos, T. Kasparis, Coupling weight elimination with genetic algorithms to reduce network size and preserve generalization, Neurocomputing 17 (1997) 167-194.
5. I. Bellido, G. Fernandez, Backpropagation Growing Networks: Towards Local Minima Elimination, Lecture Notes in Computer Science, vol. 540, Springer-Verlag, 1991, pp. 130-135.
6. C.M. Bishop, Neural Networks for Pattern Recognition, Clarendon Press, Oxford University Press Inc., New York, 1996.
7. A. Cangelosi, D. Parisi, S. Nolfi, Cell Division and Migration in a Genotype for Neural Networks, Network: Computation in Neural Systems 5 (1994) 497-515.
8. E. Cantú -Paz, C. Kamath, Inducing oblique decision trees with evolutionary algorithms, IEEE Transactions on Evolutionary Computation 7 (1) (2003) 54-68.
9. P.A. Castillo, M.G. Arenas, J.G. Castellano, M. Cillero, J.J. Merelo, A. Prieto, V. Rivas, G. Romero, Function approximation with evolved multilayer perceptrons, in: Nikos E. Mastorakis (Ed.), Advances in Neural Networks and Applications, Artificial Intelligence Series, World Scientific and Engineering Society Press, 2001, ISBN 960-8052-26-2, pp. 195-200.
10. P.A. Castillo, J. Carpio, J.J. Merelo, V. Rivas, G. Romero, A. Prieto, Evolving multilayer perceptrons, Neural Processing Letters 12 (2) (2000) 115-127.
11. P.A. Castillo, J.G. Castellano, J.J. Merelo, A. Prieto, Diseño de Redes Neuronales Artificiales Mediante Algoritmos Evolutivos. Inteligencia Artificial, Revista Iberoamericana de Inteligencia Artificial. No.14, pp. 2-32 (ISSN: 1137-3601). (c) AEPIA 〈http://aepia.dsic.upv.es/〉, 2001.
12. P.A. Castillo, J. González, J.J. Merelo, V. Rivas, G. Romero, A. Prieto, G-Prop-II: Global optimization of multilayer perceptrons using GAs, in: Congress on Evolutionary Computation, Washington, DC, USA, vol. III, 1999, pp. 2022-2027, ISBN 0-7803-5536-9.
13. P.A. Castillo, J. González, J.J. Merelo, V. Rivas, G. Romero, A. Prieto, G-Prop-III: Global optimization of multilayer perceptrons using an evolutionary algorithm, in: Congress on Evolutionary Computation, Genetic and Evolutionary Computation Conference, Orlando, USA, vol. I, 1999, p. 942, ISBN 1-55860-611-4.
14. P.A. Castillo, J. González, J.J. Merelo, V. Rivas, G. Romero, A. Prieto, SA-Prop: Optimization of Multilayer Perceptron Parameters Using Simulated Annealing, Lecture Notes in Computer Science, vol. 1606, Springer-Verlag, 1999, ISBN 3-540-66069-0, pp. 661-670.
15. P.A. Castillo, J.J. Merelo, V. Rivas, G. Romero, A. Prieto, G-Prop: Global optimization of multilayer perceptrons using GAs, Neurocomputing 35 (1-4) (2000) 149-163.
16. P.A. Castillo, M.G. Arenas, J.J. Castillo-Valdivieso, A. Prieto, J.J. Merelo, G. Romero, Artificial Neural Networks Design Using Evolutionary Algorithms, Advances in Soft-Computing, Springer-Verlag, 2003, ISBN 1-85233-755-9, pp. 43-52.
17. P.A. Castillo, M.G. Arenas, J.J. Merelo, V. Rivas, G. Romero, Optimisation of Multilayer Perceptrons Using a Distributed Evolutionary Algorithm with SOAP, Lecture Notes in Computer Science, vol. 2439, Springer-Verlag, 2002, pp. 676-685.
18. P.A. Castillo, J.J. Merelo, G. Romero, A. Prieto, I. Rojas, Statistical analysis of the parameters of a neuro-genetic algorithm, IEEE Transactions on Neural Networks, 1045-9227 13 (6) (2002) 1374-1394.
19. A. Chandra, X. Yao, Evolutionary framework for the construction of diverse hybrid ensembles, in: Proceedings of the 13th European Symposium on Artificial Neural Networks, Brugge, Belgium, 2005, pp. 253-258.
20. A. Chandra, X. Yao, Ensemble learning using multi-objective evolutionary algorithms, Journal of Mathematical Modelling and Algorithms 5 (4) (2006) 417-445.
21. A. Chandra, X. Yao, Evolving hybrid ensembles of learning machines for better generalisation, Neurocomputing 69 (7-9) (2006) 686-700.
22. Yuehui Chen, Bo Yang, Jiwen Dong, Ajith Abraham, Time-series forecasting using flexible neural tree model, Information Sciences 174 (3-4) (2005) 219-235.
23. G. Cybenko, Approximation by superpositions of sigmoids, Mathematics of Control, Signals, and Systems 2 (1989) 303-314.
24. C. Darwin, On the Origin of Species by Means of Natural Selection, John Murray, London, 1859.
25. L. Davis, Handbook of Genetic Algorithms, Van Nostrand Reinhold, New York, 1991.
26. I. de Falco, A. Iazzetta, P. Natale, E. Tarantino, Evolutionary neural networks for nonlinear dynamics modeling, in: Parallel Problem Solving from Nature 98, Lectures Notes in Computer Science, vol. 1498, 1998, pp. 593-602.
27. C. Domeniconi, J. Peng, D. Gunopulos, Adaptive metric nearest neighbor classification, in: Proceedings of IEEE Conference on CVPR, Hilton Head Island, SC, 2000, pp. 517-522.
28. W. Duch, Datasets used for classification: comparison of results, 2004. 〈http://www.phys.uni.torun.pl/kmk/projects/datasets.html〉.
29. A.F. Duprat, T. Huynh, G. Dreyfus, Toward a principled methodology for neural network design and performance evaluation in QSAR. Application to the prediction of LogP, Journal of Chemical Information and Computer Sciences 38 (4) (1998) 586-594.
30. P. Durr, C. Mattiussi, D. Floreano. Neuroevolution with Analog Genetic Encoding, Lecture Notes in Computer Science, vol. 4193, 2006, pp. 671-680, ISBN 978-3-540-38990-3.
31. S. Dzeroski, B. Zenko, Is combining classifiers with stacking better than selecting the best one? Machine Learning 54 (3) (2004) 255-273.
32. A.E. Eiben, J.E. Smith, Introduction to Evolutionary Computing, Springer, 2003, ISBN 3-540-40184-9.
33. S. Fahlman, Faster-learning variations on back-propagation: an empirical study, in: Proceedings of the 1988 Connectionist Models Summer School, Morgan Kaufmann, 1988, pp. 38-51.
34. S.E. Fahlman, An empirical study of learning speed in back-propagation networks, Technical report, Carnegie Mellon University, 1988.
35. R.A. Fisher, Theory of statistical estimation, in: Proceedings of the Cambridge Philosophical Society, vol. 22, 1925, pp. 700-725.
36. R.A. Fisher, The comparison of samples with possibly unequal variances, Annals of Eugenics 9 (1936) 174-180.
37. K. Funahashi, On the approximate realization of continuous mappings by neural networks, Neural Networks 2 (3) (1989) 183-192.
38. A.R. Gallant, H. White, There exists a neural network that does not make avoidable mistakes, Proceedings of the IEEE International Conference on Neural Networks (San Diego), vol. 1, IEEE, New York, 1988, pp. 657-664.
39. N. García-Pedrajas, C. Hervás-Martínez, J. Munoz-Pérez, Multiobjective cooperative coevolution of artificial neural networks, Neural Networks 15 (10) (2002) 1255-1274.
40. N. García-Pedrajas, C. Hervás-Martínez, J. Munoz-Pérez. SYMBIONT: A cooperative coevolutionary model for evolving artificial neural networks for classification, in: Studies in Fuzziness and Soft Computing Archive, 2002, pp. 341-354, ISBN 3-7908-1455-5.
41. N. García-Pedrajas, C. Hervás-Martínez, J. Munoz-Pérez, COVNET: A cooperative coevolutionary model for evolving artificial neural networks, IEEE Transactions on Neural Networks 14 (3) (2003) 575-596.
42. N. García-Pedrajas, C. Hervás-Martínez, D. Ortiz, Cooperative coevolution of artificial neural network ensembles for pattern classification, IEEE Transactions on Evolutionary Computation 9 (3) (2005) 271-302.
43. N. García-Pedrajas, D. Ortiz-Boyer, A cooperative constructive method for neural networks for pattern recognition, Pattern Recognition 40 (1) (2007) 80-98.
44. N. García-Pedrajas, D. Ortiz-Boyer, R. del Castillo- Gomáriz, C. Hervás-Martínez, Cascade ensembles, in: J. Cabestany, A. Prieto, D.F. Sandoval (Eds.), IWANN2005, LNCS, vol. 3512, Springer-Verlag, Berlin, Heidelberg, 2005, pp. 598-603.
45. J.J. Grefenstette, Optimization of control parameters for genetic algorithms, IEEE Transactions on Systems, Man, and Cybernetics SMC-16 (1) (1986) 122-128.
46. M.A. Grönroos, Evolutionary Design of Neural Networks, Master of Science Thesis in Computer Science, Dept. of Mathematical Sciences, University of Turku, 1998.
47. F. Gruau, Genetic synthesis of boolean neural networks with a cell rewriting developmental process, in: D. Whitley, J.D. Schaffer (Eds.), Proceedings of the International Workshop on Combinations of Genetic Algorithms and Neural Networks (COGANN'92), IEEE Computer Society Press, Los Alamitos, CA, 1992, pp. 55-74.
48. F. Gruau, Neural Network Synthesis Using Cellular Encoding and the Genetic Algorithm, PhD thesis, Ecole Normale de Lyon, France, 1994.
49. F.C. Gruau, Cellular encoding of genetic neural networks, Technical Report, LIP-IMAG Ecole Normale Superieure de Lyon, 46 Allée d'Italie 69007 Lyon, France, 1992.
50. J. Hallinan, P. Jackway, Co-operative evolution of a neural classifier and feature subset, Lecture Notes in Computer Science 1585 (1999) 397-404.
51. D. Hillis, Co-evolving parasites improve simulated evolution as an optimization procedure, in: Artificial Life II, Addison Wesley, 1992, pp. 313-323.
52. P. Husband, F. Mill, Simulated co-evolution as the mechanism for emergent planning and scheduling, in: Belew, Booker (Eds.), Proceedings of the Fourth International Conference on Genetic Algorithms, Morgan Kaufmann, San Mateo, CA, 1991, pp. 264- 270.
53. P. Husbands, Distributed coevolutionary genetic algorithms for multi-criteria and multi-constraint optimisation, in: T. Fogarty et al. (Eds.), Evolutionary Computing, Lecture Notes in Computer Science, vol. 865, Springer-Verlag, 1994, pp. 150-165.
54. J. Hwang, S. You, S. Lay, I. Jou, The cascade-correlation learning: a projection pursuit learning perspective, IEEE Transactions on Neural Networks 7 (2) (1996) 278-289.
55. I. Jagielska, C. Matthews, T. Whitfort, An investigation into the application of neural networks, fuzzy logic, genetic algorithms, and rough sets to automated knowledge acquisition problems, Neurocomputing 24 (1999) 37-54.
56. T. Jasic, H. Poh, Analysis of Pruning in Backpropagation Networks for Artificial and Real World Mapping Problems, Lecture Notes in Computer Science, vol. 930, Springer-Verlag, 1995, pp. 239-245.
57. R. Keesing, D.G. Stork, Evolution and learning in neural networks: the number and distribution of learning trials affect the rate of evolution, in: R. Lippmann, J.E. Moody, D.S. Touretzky (Eds.), Advances in Neural Information Processing Systems 3
58. [NIPS Conference, Denver, Colorado, USA, November 26-29, 1990], Morgan Kaufmann, 1991, ISBN 1-55860-184-8, pp. 804-810.
59. D. Kim, C. Kim, Forecasting time series with genetic fuzzy predictor ensemble, IEEE Transactions on Fuzzy Systems 5 (4) (1997) 523-535.
60. W. Kinnebrock, Accelerating the standard backpropagation method using a genetic approach, Neurocomputing 6 (1994) 583-588.
61. A.N. Kolmogorov, On the representation of continuous functions of several variables by superpositions of continuous functions of one variable and addition, American Mathematical Society Translations 28 (1963) 55-59.
62. P. Kuchenko, SOAP::Lite. 〈http://www.soaplite.com〉.
63. Z.-J. Lee, C.-Y. Lee, A hybrid search algorithm with heuristics for resource allocation problem, Information Sciences 173 (1-3) (2005) 155-167.
64. A. León-Barranco, C.A. Reyes-García, ARGEN + AREPO: Improving the search process with artificial genetic engineering, in: J. Cabestany, A. Prieto, D.F. Sandoval (Eds.), IWANN2005, LNCS, vol. 3512, Springer-Verlag, Berlin, Heidelberg, 2005, pp. 637- 645.
65. F.H.F. Leung, H.K. Lam, S.H. Ling, P.K.S. Tam, Tuning of the structure and parameters of a neural network using an improved genetic algorithm, IEEE Transactions on Neural Networks 14 (1) (2003) 79-88.
66. R.P. Lippmann, An introduction to computing with neural nets, IEEE ASSP Magazine 3 (4) (1987) 4-22.
67. Y. Liu, X. Yao, T. Higuchi, Evolutionary ensembles with negative correlation learning, IEEE Transactions on Evolutionary Computation 4 (4) (2000) 380-387.
68. O.L. Mangasarian, R. Setiono, W.H. Wolberg, Pattern recognition via linear programming: theory and application to medical diagnosis, in: Thomas F. Coleman, Yuying Li (Eds.), Large-Scale Numerical Optimization, SIAM Publications, Philadelphia, 1990, pp. 22-30.
69. H.A. Mayer, R. Schwaiget, R. Huber, Evolving topologies of artificial neural networks adapted to image processing tasks, in: Proceedings of the 26th International Symposium on Remote Sensing of Environment, Vancouver, BC, Canada, 1996, pp. 71-74.
70. J.J. Merelo, Evolutionary computation in Perl, in: Münich Perl Mongers (Ed.), YAPC: Europe: 2002, 2002, pp. 2-22.
71. J.J. Merelo, M. Patón, A. Cañ as, A. Prieto, F. Morán, Genetic optimization of a multilayer neural network for cluster classification tasks, Neural Network World 3 (1993) 175-186.
72. J.J. Merelo, M. Pató n, A. Cañ as, A. Prieto, F. Morán, Optimization of a Competitive Learning Neural Network by Genetic Algorithms, Lecture Notes in Computer Science, vol. 686, Springer-Verlag, 1993, pp. 185-192.
73. J.J. Merelo, A. Prieto, G-LVQ, a combination of genetic algorithms and LVQ, in: N.C. Steele, D.W. Pearson, R.F. Albrecht (Eds.), Artificial Neural Nets and Genetic Algorithms, Springer-Verlag, 1995, pp. 92-95.
74. J.J. Merelo, P.A. Castillo, M.G. Arenas, G. Romero, Specifying Evolutionary Algorithms in XML, Lecture Notes in Computer Science, LNCS, 0302-9743 2686 (2003) 502-509.
75. J.J. Merelo, A. Prieto, F. Morán, Optimization of classifiers using genetic algorithms, in: Mukesh J. Patel, Vasant Honavar, Karthik Balakrishnan (Eds.), Advances in the Evolutionary Synthesis of Intelligent Agents, MIT Press, 2001, ISBN 0262162016, pp. 91-108 (Chapter 4).
76. Z. Michalewicz, Genetic Algorithms + Data Structures = Evolution Programs, third extended ed., Springer-Verlag, 1996.
77. M. Monirul-Islam, X. Yao, K. Murase, A constructive algorithm for training cooperative neural network ensembles, IEEE Transactions on Neural Networks 14 (4) (2003) 820-834.
78. D.J. Montana, L. Davis, Training feedforward neural networks using genetic algorithms, in: Proceedings of the 11th International Joint Conference on Artificial Intelligence, 1989, pp. 762-767.
79. D. Moriarty, R. Miikkulainen, Hierarchical evolution of neural networks, in: Proceedings of the 1998 IEEE Conference on Evolutionary Computation (ICEC-98, Anchorage, AK), IEEE, Piscataway, NJ, 1998, pp. 428-433.
80. D.E. Moriarty, R. Miikkulainen, Efficient reinforcement learning through symbiotic evolution, Machine Learning 22 (1996) 11-32.
81. D.E. Moriarty, R. Miikkulainen, Forming neural networks through efficient and adaptive coevolution, Evolutionary Computation 4 (5) (1998) 373-399.
82. B. Olsson, Co-evolutionary search in asymmetric spaces, Information Sciences 133 (3-4) (2001) 103-125.
83. P. Palmes, S. Usui, Robustness, evolvability and optimality in evolutionary neural networks, Biosystems 82 (2) (2005) 168-188.
84. J. Paredis, Coevolutionary computation, Artificial Life 2 (1995) 355-375.
85. J. Paredis, The symbiotic evolution of solutions and their representations, in: Eshelman (Ed.), Proceedings of the Sixth International Conference on Genetic Algorithms, Morgan Kaufmann, San Francisco, CA, 1995, pp. 359-365.
86. R. Parekh, J. Yang, V. Honavar, MUpstart - A constructive neural network learning algorithm for multi-category pattern classification, in: Proceedings of the IEEE/INNS International Conference on Neural Networks (ICNN'97), Houston, TX, vol. III, 1997, pp. 1924-1929.
87. R. Parekh, J. Yang, V. Honavar, Pruning strategies for the MTiling constructive learning algorithm, in: Proceedings of the IEEE/INNS International Conference on Neural Networks (ICNN'97), Houston, TX, vol. III, 1997, pp. 1960-1965.
88. R. Parekh, J. Yang, V. Honavar, Constructive theory refinement in knowledge based neural networks, in: Proceedings of the IEEE/INNS International Joint Conference on Neural Networks (IJCNN'98), Anchorage, AK, 1998, pp. 2318-2323.
89. R. Parekh, J. Yang, V. Honavar, Constructive neural network learning algorithms for pattern classification, IEEE Transactions on Neural Networks 11 (2) (2000) 436-451.
90. M. Pelillo, A. Fanelli, AMethod of Pruning Layered Feed-Forward Neural Networks, Lecture Notes in Computer Science, vol. 686, Springer-Verlag, 1993, pp. 278-283.
91. V. Petridis, S. Kazarlis, A. Papaikonomu, A. Filelis, A hybrid genetic algorithm for training neural networks, Artificial Neural Networks 2 (1992) 953-956.
92. M.A. Potter, K.A. De Jong, Cooperative coevolution: an architecture for evolving coadapted subcomponents, Evolutionary Computation 8 (1) (2000) 1-29.
93. L. Prechelt, PROBEN1 - A set of benchmarks and benchmarking rules for neural network training algorithms. Technical Report 21/94, Fakultät fü r Informatik, Universität Karlsruhe, D-76128 Karlsruhe, Germany, September 1994.
94. R.D. Reed, Pruning algorithms - a survey, IEEE Transactions on Neural Networks 4 (5) (1993) 740-744.
95. R.D. Reed, R.J. Marks II, Neural Smithing, Bradford, The MIT Press, Cambridge, MA, London, England, 1999.
96. M. Riedmiller, H. Braun, A direct adaptive method for faster backpropagation learning: the RPROP algorithm, IEEE International Conference on Neural Networks (San Francisco), vol. 1, IEEE, New York, 1993, pp. 586-591.
97. C.D. Rosin, R.K. Belew, New methods for competitive coevolution, Evolutionary Computation 5 (1) (1997) 1-29.
98. D.E. Rumelhart, G.E. Hinton, R.J. Williams, Learning internal representations by error backpropagation, in: D.E. Rumelhart, J.L. McClelland, The PDP Research Group (Eds.), Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol. 1, MIT Press, Cambridge, MA, 1986, pp. 318-362.
99. R. Smalz, M. Conrad, Combining evolution with credit apportionment: a new learning algorithm for neural nets, Neural Networks 7 (2) (1994) 341-351.
100. D. Thierens, J. Suykens, J. Vandewalle, B. De Moor, Genetic weight optimization of a feedforward neural network controller, in: Proceedings of the Conference on Artificial Neural Nets and Genetic Algorithms, Springer-Verlag, 1993, pp. 658-663.
101. D. Whitley, The GENITOR algorithm and selection pressure: Why rank-based allocation of reproductive trials is best? in: J.D. Schaffer (Ed.), Proceedings of the 3rd International Conference on Genetic Algorithms, Morgan Kauffmann, 1989, pp. 116-121.
102. D.H. Wolpert, W.G. Macready, No free lunch theorems for optimization, IEEE Transactions on Evolutionary Computation 1 (1) (1997) 67-82.
103. S. Yang, X. Yao, Experimental study on population-based incremental learning algorithms for dynamic optimization problems, Soft Computing 9 (11) (2005) 815-834.
104. X. Yao, Evolving artificial neural networks, Proceedings of the IEEE 87 (9) (1999) 1423-1447.
105. X. Yao, Y. Liu, Towards designing artificial neural networks by evolution, Applied Mathematics and Computation 91 (1) (1998) 83-90.
106. X. Yao, Y. Xu, Recent advances in evolutionary computation, Journal of Computer Science and Technology 21 (1) (2006) 1-18.
107. Q. Zhao, Co-evolutionary learning of neural networks, Journal of Intelligent and Fuzzy Systems, 1064-1246 6 (1998) 83-90.
108. Q.F. Zhao, O. Hammami, K. Kuroda, K. Saito, Cooperative co-evolutionary algorithm - how to evaluate a module? in: Proceedings of the 1st IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks, San Antonio, 2000, pp. 150-157.