Meta-learning approach to neural network optimization

Neural Networks

Volumn 23, Issue 4, 2010, Pages 568-582

  Kordík, Pavel ^a   Koutník, Jan ^a   Drchal, Jan ^a   Kovářík, Oleg ^a   Čepek, Miroslav ^a   Šnorek, Miroslav ^a  

^a CZECH TECHNICAL UNIVERSITY IN PRAGUE   (Czech Republic)

Author keywords

Ant colony; Diversity promotion; Evolutionary algorithms; Inductive modeling; Meta learning; Meta optimization; Neural network optimization; Optimization algorithm benchmark; Self organization

Indexed keywords

ANT COLONIES; ANT COLONY; INDUCTIVE MODELING; META-OPTIMIZATION; METALEARNING; NEURAL NETWORK OPTIMIZATION; OPTIMIZATION ALGORITHMS; SELF-ORGANIZATIONS;

ELECTRIC NETWORK TOPOLOGY; EVOLUTIONARY ALGORITHMS; LEARNING ALGORITHMS; STRUCTURAL OPTIMIZATION;

NEURAL NETWORKS;

ALGORITHM; ARTICLE; ARTIFICIAL NEURAL NETWORK; CLASSIFIER; EVOLUTION; LEARNING; NERVE CELL; PRIORITY JOURNAL; PROBLEM SOLVING; QUALITY CONTROL;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; LEARNING; MODELS, BIOLOGICAL; NERVE NET; NEURAL NETWORKS (COMPUTER); NEURONS; PATTERN RECOGNITION, AUTOMATED;

EID: 77950296566     PISSN: 08936080     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neunet.2010.02.003     Document Type: Article

References (94)

1. Adeney, K., & Korenberg, M. (2000). An easily calculated bound on condition for orthogonal algorithms. In IEEE-INNS-ENNS international joint conference on neural networks (p. 3620). Vol. 3.
2. Aine S., Chakrabarti P.P., Kumar R. An automated meta-level control framework for optimizing the quality-time tradeoff of VLSI algorithms. IEEE Transactions on CAD of Integrated Circuits and Systems 1992, 26(11). 2008, 2007.
3. Alpaydin E., Kaynak C. Cascading classifiers. Kybernetika 1998, 34:369-374.
4. Angeline P.J., Saunders G.M., Pollack J.P. An evolutionary algorithm that constructs recurrent neural networks. IEEE Transactions on Neural Networks 1994, 5(1):54-65.
5. Bilchev G., Parmee I.C. The ant colony metaphor for searching continuous design spaces. Selected papers from AISB workshop on evolutionary computing 1995, 25-39. Springer-Verlag, London, UK.
6. Blum C., Socha K. Training feed-forward neural networks with ant colony optimization: An application to pattern classification. Proceedings of hybrid intelligent systems conference 2005, 233-238. IEEE Computer Society, Los Alamitos, CA, USA.
7. Boyan, J. (1998). Learning evaluation functions for global optimization. Ph.D. thesis. Robotics Institute, Carnegie Mellon University, Pittsburgh, PA.
8. Brazdil P., Giraud-Carrier C., Soares C., Vilalta R. Metalearning, applications to data mining. Cognitive technologies 2009, Springer, Berlin, Heidelberg.
9. Breiman L. Bagging predictors. Machine Learning 1996, 24(2):123-140.
10. Brown, G. (2004). Diversity in neural network ensembles. Ph.D. thesis. The University of Birmingham, School of Computer Science, Birmingham B15 2TT, United Kingdom.
11. Buk Z., Koutník J., Šnorek M. NEAT in HyperNEAT substituted with genetic programming. LNCS 2009, Vol. 5495. Springer-Verlag, Berlin, Heidelberg.
12. Burke E., Kendall G. Search methodologies: Introductory tutorials in optimization and decision support techniques 2005, Springer.
13. Burke E.K., Petrovic S., Qu R. Case-based heuristic selection for timetabling problems. Journal of Scheduling 2006, 9(2):115-132.
14. Cicirello, V. A. (2003). Boosting stochastic problem solvers through online self-analysis of performance. Ph.D. thesis. Carnegie Mellon University, Pittsburgh, PA, USA, 2003. Adviser-Smith, Stephen F.
15. Cramer N.L. A representation for the adaptive generation of simple sequential programs. Proceedings of the 1st international conference on genetic algorithms 1985, 183-187. L. Erlbaum Associates Inc, Hillsdale, NJ, USA.
16. D'Ambrosio D.B., Stanley K.O. A novel generative encoding for exploiting neural network sensor and output geometry. GECCO'07: Proceedings of the 9th annual conference on genetic and evolutionary computation 2007, 974-981. ACM, New York, NY, USA.
17. D'Ambrosio D.B., Stanley K.O. Generative encoding for multiagent learning. GECCO'08: Proceedings of the 10th annual conference on genetic and evolutionary computation 2008, 819-826. ACM, New York, NY, USA.
18. Dennis J.E., Schnabel R.B. Numerical methods for unconstrained optimization and nonlinear equations (classics in applied mathematics, 16) 1996, Society for Industrial and Applied Mathematics.
19. de Oca, M. A. M., Stützle, T., Birattari, M., & Dorigo, M. (2006). A comparison of particle swarm optimization algorithms based on run-length distributions. In ANTS workshop (pp. 1-12).
20. Deugo D., Ferguson D. Evolution to the xtreme: Evolving evolutionary strategies using a meta-level approach. Proceedings of the 2004 IEEE congress on evolutionary computation 2004, 31-38. IEEE Press, Portland, Oregon.
21. Diosan L.S., Oltean M. Evolving evolutionary algorithms using evolutionary algorithms. GECCO'07: Proceedings of the 2007 GECCO conference companion on genetic and evolutionary computation 2007, 2442-2449. ACM, New York, NY, USA.
22. Drchal J., Kučerová A., Němeček J. Optimizing synaptic weights of neural networks. ICECT'03: Proceedings of the third international conference on engineering computational technology 2002, 211-212. Civil-Comp press, Edinburgh, UK.
23. Drummond A., Strimmer K. Pal: An object-oriented programming library for molecular evolution and phylogenetics. Bioinformatics 2001, 17(7):662-663.
24. Eiben A.E., Michalewicz Z., Schoenauer M., Smith J.E. Parameter control in evolutionary algorithms. Parameter setting in evolutionary algorithms 2007, 19-46. Springer.
25. Engelbrecht A. Computational intelligence: An introduction 2002, Halsted Press, New York, NY, USA.
26. Fahlman, S. E. (1988). An empirical study of learning speed in back-propagation networks. Technical report CMU-CS-88-162. Carnegie Mellon Univ.
27. Fahlman S., Lebiere C. The cascade-correlation learning architecture. Advances in neural information processing systems 2 1991, 524-532. Morgan Kaufmann.
28. Ferri C., Flach P., Hernández-Orallo J. Delegating classifiers. ICML'04: Proceedings of the twenty-first international conference on machine learning 2004, 37. ACM, New York, NY, USA.
29. Fogel, D., Fogel, L., & Atmar, J. (1991). Meta-evolutionary programming. In Signals, systems and computers, 1991. 1991 Conference record of the twenty-fifth asilomar conference on (pp. 540-545). Vol. 1.
30. Freund Y., Schapire R. A decision-theoretic generalization of on-line learning and an application to boosting. Proceedings of the second European conference on computational learning theory 1995, 23-37. Springer Verlag.
31. Gama J., Brazdil P. Cascade generalization. Machine Learning 2000, 41(3):315-343.
32. Gambier D., Bruzek J., Schmitt A., Houet F., Murail P. Age at death and sex diagnosis of the cro-magnon fossils (Dordogne, France) based on the pelvic bone. Comptes Rendus Palevol 2006, 5:735-741.
33. Gauci J., Stanley K. Generating large-scale neural networks through discovering geometric regularities. GECCO'07: Proceedings of the 9th annual conference on genetic and evolutionary computation 2007, 997-1004. ACM, New York, NY, USA.
34. Glover F.W., Kochenberger G.A. Handbook of metaheuristics (international series in operations research & management science) 2003, Springer.
35. Gomez F., Miikkulainen R. Incremental evolution of complex general behavior. Adaptive Behavior 1996, 5:5-317.
36. Gomez F.J., Schmidhuber J. Co-evolving recurrent neurons learn deep memory pomdps. GECCO'05: Proceedings of the 2005 conference on genetic and evolutionary computation 2005, 491-498. ACM, New York, NY, USA.
37. Gomez F., Schmidhuber J., Miikkulainen R. Accelerated neural evolution through cooperatively coevolved synapses. Journal of Machine Learning Research 2008, 9:937-965.
38. Grefenstette J. Optimization of control parameters for genetic algorithms. IEEE Transactions on System Man, and Cybernetics 1986, 16(1):122-128.
39. Gruau, F. (1994). Neural network synthesis using cellular encoding and the genetic algorithm. Ph.D. thesis. Ecole Normale Superieure de Lyon, France.
40. Hansen N., Ostermeier A. Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation 2001, 9(2):159-195.
41. Recent advances in memetic algorithms 2004, Springer, Berlin, Heidelberg, New York. W. Hart, N. Krasnogor, J. Smith (Eds.).
42. Holland J. Adaptation in neural and artificial systems 1975, University of Michigan Press.
43. Ho Y., Pepyne D. Simple explanation of the no-free-lunch theorem and its implications. Journal of Optimization Theory and Applications 2002, 115:549.
44. Hrstka O., Kučerová A. Improvements of real coded genetic algorithms based on differential operators preventing premature convergence. Advances in Engineering Software 2004, 35(3-4):237-246.
45. Ivakhnenko A.G. Polynomial theory of complex systems. IEEE Transactions on Systems, Man, and Cybernetics 1971, SMC-1(1):364-378.
46. Juang, C.-F., & Liou, Y.-C. (2004). On the hybrid of genetic algorithm and particle swarm optimization for evolving recurrent neural network. In Proceedings of the IEEE international joint conference on neural networks (pp. 2285-2289) Vol. 3. Dept. of Electr. Eng., Nat. Chung-Hsing Univ., Taichung, Taiwan, 25-29.
47. Kaynak C., Alpaydin E. Multistage cascading of multiple classifiers: One man's noise is another man's data. ICML'00: Proceedings of the seventeenth international conference on machine learning 2000, 455-462. Morgan Kaufmann Publishers Inc, San Francisco, CA, USA.
48. Kong, M., & Tian, P. (2006). A direct application of ant colony optimization to function optimization problem in continuous domain. In ANTS workshop (pp. 324-331).
49. Kordík, P. (2006a). Fully automated knowledge extraction using group of adaptive models evolution. Ph.D. thesis. Czech Technical University in Prague, FEE, Dep. of Comp. Sci. and Computers, FEE, CTU Prague, Czech Republic.
50. Kordík, P. (2006b) Fully automated knowledge extraction using group of adaptive models evolution. Ph.D. thesis. Czech Technical University in Prague, Praha.
51. Kordík P. Hybrid self-organizing modeling systems. Studies in computational intelligence 2009, Vol. 211:290. Springer-Verlag, Berlin, Heidelberg.
52. Kordík, P., Kovářík, O., & Šnorek, M. (2007). Optimization of models: Looking for the best strategy. In Proceedings of the 6th EUROSIM congress on modelling and simulation: Vol. 2 (pp. 314-320). Vienna: ARGESIM.
53. Koza J.R. Genetic programming: On the programming of computers by means of natural selection 1992, MIT Press, Cambridge, MA, USA.
54. Li Y.-j., Wu T.-j. An adaptive ant colony system algorithm for continuous-space optimization problems. Journal of Zhejiang University Science 2003, 4(1):40-46.
55. Ivakhnenko A.G., Müller J.-A. Self-organization of nets of active neurons. System Analysis Modelling and Simulation 1995, 20(1-2):93-106.
56. Mahfoud, S. W. (1995). Niching methods for genetic algorithms. Technical report 95001. Illinois Genetic Algorithms Laboratory (IlliGaL), University of Ilinios at Urbana-Champaign.
57. Mahfoud, S. W. (1995). A comparison of parallel and sequential niching methods. In Sixth international conference on genetic algorithms (pp. 136-143).
58. Meissner M., Schmuker M., Schneider G. Optimized Particle Swarm Optimization (OPSO) and its application to artificial neural network training. BMC Bioinformatics 2006, 7:125.
59. Merz P., Freisleben B. Fitness landscape analysis and memetic algorithms for the quadratic assignment problem. IEEE Transactions on Evolutionary Computation 2000, 4(4):337-352.
60. Moriarty D.E., Miikkulainen R. Forming neural networks through efficient and adaptive coevolution. Evolutionary Computation 1997, 5(4):373-399.
61. Muller, J. A., & Lemke, F. (2000). Self-organising data mining. Berlin.
62. Murre, J. M. J. (1992). Learning and categorization in modular neural networks. Harvester Wheatsheaf.
63. Ong Y.-S., Keane A.J. Meta-lamarckian learning in memetic algorithms. IEEE Transactions on Evolutionary Computation 2004, 8(2):99-110.
64. Onwubolu G.C., Davendra D. Differential evolution: A handbook for global permutation-based combinatorial optimization 2009, Springer Publishing Company, Incorporated.
65. Ortega, J. (1996). Making the most of what youve got: Using models and data to improve prediction accuracy. Ph.D. thesis. Vanderbilt University.
66. Ortega J., Koppel M., Argamon S. Arbitrating among competing classifiers using learned referees. Knowledge and Information System 2001, 3(4):470-490.
67. Poli R., Graff M. Free lunches for neural network search. GECCO'09: Proceedings of the 11th annual conference on genetic and evolutionary computation 2009, 1291-1298. ACM, New York, NY, USA.
68. Poli R., Graff M., McPhee N.F. Free lunches for function and program induction. FOGA'09: Proceedings of the tenth ACM SIGEVO workshop on foundations of genetic algorithms 2009, 183-194. ACM, New York, NY, USA.
69. Prechelt, L. (1994). PROBEN1 - A set of benchmarks and benchmarking rules for neural network training algorithms. Technical report 21/94. Fakultät für Informatik, Universität Karlsruhe, D-76128 Karlsruhe, Germany.
70. Rechenberg I. Evolutionsstrategie: Optimierung technischer systeme nach prinzipien der biologischen evolution 1973, Frommann-Holzboog.
71. Riedmiller, M., & Braun, H. (1992). Rprop - A fast adaptive learning algorithm. Technical report. Universitat Karlsruhe.
72. Ritchie M.D., White B.C., Parker J.S., Hahn L.W., Moore J.H. Optimization of neural network architecture using genetic programming improves detection and modeling of gene-gene interactions in studies of human diseases. BMC Bioinformatics 2003, 4(1):28+.
73. Roli A., Milano M. MAGMA: A multiagent architecture for metaheuristics. IEEE Transactions on Systems, Man and Cybernetics - Part B 2004, 34. 2002.
74. Rosenblatt, F. (1957). The perceptron: A perceiving and recognizing automaton. Technical report 85-460-1.
75. Salane, Tewarson A unified derivation of symmetric quasi-newton update formulas. Applied Math 1980, 25:29-36.
76. Satoru Hiwa, T. H., & Miki, M. (2007). Hybrid optimization using direct, ga, and sqp for global exploration. In IEEE Proceedings of 2007 congress on evolutionary computation (pp. 1709-1716).
77. Schapire R.E. The strength of weak learnability. Machine Learning 1990, 5(2):197-227.
78. Schmidhuber J. Discovering neural nets with low Kolmogorov complexity and high generalization capability. Neural Networks 1997, 10(5):857-873.
79. Schmidhuber, J. (1987). Evolutionary principles in self-referential learning. on learning now to learn: The meta-meta-meta...-hook. Diploma thesis. Technische Universitat Munchen, Germany.
80. Shewchuk, J. R. (1994). An introduction to the conjugate gradient method without the agonizing pain. Technical report. School of Computer Science Carnegie Mellon University, Pittsburgh, PA 15213.
81. Smith J., Fogarty T. Operator and parameter adaptation in genetic algorithms. Soft Computing 1997, 1(2):81-87.
82. Stanley K.O. Compositional pattern producing networks: A novel abstraction of development. Genetic Programming and Evolvable Machines 2007, 8(2):131-162.
83. Stanley, K. O. (2004). Efficient evolution of neural networks through complexification. Ph.D. thesis.
84. Stanley K.O., Miikkulainen R. Evolving neural networks through augmenting topologies. Evolutionary Computation 2002, 10:99-127.
85. Storn R., Price K. Differential evolution - A simple and efficient heuristic for global optimization over continuous spaces. Journal of Global Optimization 1997, 11:341-359.
86. Tsoukalas L.H., Uhrig R.E. Fuzzy and neural approaches in engineering 1996, John Wiley & Sons, Inc., New York, NY, USA.
87. Tsutsui, S., Pelikan, M., & Ghosh, A. (2005). Performance of aggregation pheromone system on unimodal and multimodal problems. In The IEEE congress on evolutionary computation, 2005 (pp. 880-887). Vol. 1 IEEE.
88. Venturini G., Slimane M. On how pachycondyla apicalis ants suggest a new search algorithm. Future Generation Computer Systems 2000, 16:937-946.
89. Vesterstrom, J., & Thomsen, R. (2004). A comparative study of differential evolution, particle swarm optimization, and evolutionary algorithms on numerical benchmark problems. In Proceedings of the 2004 congress on evolutionary computation (pp. 1980-1987). Vol. 2.
90. Wade, J. G. (2006). Convergence properties of the conjugate gradient method. Available at http://www-math.bgsu.edu/~gwade/tex_examples/example2.txt.
91. Wolpert D.H. Stacked generalization. Neural Networks 1992, 5:241-259.
92. Wolpert D., Macready W. No free lunch theorems for optimization. Evolutionary Computation, IEEE Transactions on 1997, 1(1):67-82.
93. Yao X., Liu Y. Epnet for chaotic time-series prediction. Selected papers from the first Asia-Pacific conference on simulated evolution and learning 1997, 146-156. Springer-Verlag.
94. Yuan B., Gallagher M. Combining meta-EAs and racing for difficult EA parameter tuning tasks. Parameter setting in evolutionary algorithms 2007, 121-142. Springer. F. Lobo, C. Lima, Z. Michalewicz (Eds.).