A heuristically enhanced gradient approximation (HEGA) algorithm for training neural networks

Neurocomputing

Volumn 73, Issue 7-9, 2010, Pages 1303-1323

  Panagiotopoulos, Dimokritos ^a   Orovas, Christos ^a   Syndoukas, Dimitrios ^a  

^a UNIVERSITY OF WESTERN MACEDONIA   (Greece)

Author keywords

Gradient approximation; Local search; Neural networks learning; Weight perturbation

Indexed keywords

ARTIFICIAL NEURAL NETWORK; BENCH-MARK PROBLEMS; GRADIENT APPROXIMATION; LOCAL SEARCH; TRAINING ALGORITHMS; WEIGHT PERTURBATION;

APPROXIMATION ALGORITHMS; COMPUTATIONAL EFFICIENCY;

NEURAL NETWORKS;

ANALYTIC METHOD; ARTICLE; ARTIFICIAL NEURAL NETWORK; BACK PROPAGATION; CLASSIFICATION ALGORITHM; INTERMETHOD COMPARISON; MATHEMATICAL ANALYSIS; MATHEMATICAL COMPUTING; PRIORITY JOURNAL; QUALITY CONTROL;

EID: 77949267043     PISSN: 09252312     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neucom.2009.12.014     Document Type: Article

References (20)

1. Melkikh A.V. Can organism pick up new valuable information from the environment?. Biophysics (Biofizika) 47 6 (2002) 1053-1058
2. Rumelhart D.E., and McClelland J.L. Parallel Distributed Processing, Foundations vol. 1 (1986), MIT Press, Cambridge, MA
3. Y. Chauvin, D.E. Rumelhart (Eds.), Backpropagation: Theory, Architectures, and Applications. Lawrence Erlbaum Associates, 1995.
4. Fletcher R., and Reeves C.M. Function minimization by conjugate gradients. Computer Journal 7 (1964) 149-154
5. Moller M.F. A scaled conjugate gradient algorithm for fast supervised learning. Neural Networks 6 (1993) 525-533
6. Hagan M.T., and Menhaj M. Training feedforward networks with the marquardt algorithm. IEEE Transactions on Neural Networks 5 6 (1994) 989-993
7. Kirkpatrik S., Gelatt C., and Vecchi M.P. Optimization by simulated annealing. Science 220 (1983) 671-680
8. Jabri M., and Flower B. Weight perturbation: an optimal architecture and learning technique for analog vlsi feedforward and recurrent multilayer networks. IEEE Transactions on Neural Networks 3 1 (1992) 154-157
9. Flower B., and Jabri M. Summed weight neuron perturbation: an o(n) improvement over weight perturbation. Advances in Neural Information Processing Systems vol. 5 (1993), Morgan Kaufman Publishers, San Mateo, CA 212-219
10. Gauwenberghs G. A fast stochastic error-descent algorithm for supervised learning and optimization. Advances in Neural Information Processing Systems vol. 5 (1993), Morgan Kaufman Publishers, San Mateo, CA 244-251
11. Alspector J., Meir R., Yuhasa B., Jayakumar A., and Lippe D. A parallel gradient descent method for learning in analog vlsi neural networks. Advances in Neural Information Processing Systems vol. 5 (1993), Morgan Kaufman Publishers, San Mateo, CA 836-844
12. Unnikrishnan K.P., and Venugopal K.P. Alopex: a correlation-based learning algorithm for feedforward and recurrent neural networks. Neural Computation 6 (1994) 469-490
13. Bia A. Alopex-b: a new, simpler, but yet faster version of the alopex training algorithm. International Journal of Neural Systems 11 6 (2001) 497-507
14. B.A. Rowland, A.S. Maida, I.S.N. Berkeley, Synaptic noise as a means of implementing weight-perturbation learning. CACS Technical Report TR-2005-2-1, University of Louisiana at Lafayette, Center for Advanced Computer Studies, 2005.
15. Haykin S., Chen Z., and Becker S. Stochastic correlative learning algorithms. IEEE Transactions on Signal Processing 52 8 (2004) 2200-2209
16. Werfel J., Xie X., and Seung H.S. Learning curves for stochastic gradient descent in linear feedforward networks. Neural Computation 17 12 (2005) 2699-2718
17. Haykin S. Neural Networks: A Comprehensive Foundation. second ed. (1999), Prentice-Hall, NJ
18. Widrow B., and Lehr M.A. 30 years of adaptive neural networks: perceptron, madaline, and backpropagation. Proceedings of the IEEE 78 9 (1990) 1415-1442
19. Russell S., and Norvig P. Artificial Intelligence: A Modern Approach. Second ed. (2003), Prentice-Hall, NJ
20. S.B. Thrun, et al., The monk's problems-a performance comparison of different learning algorithms. Technical Report CMU-CS-91197, Carnegie Mellon University, December 1991.