Convergence of Learning Algorithms with Constant Learning Rates

IEEE Transactions on Neural Networks

Volumn 2, Issue 5, 1991, Pages 484-489

  Kuan, Chung Ming ^a   Hornik, Kurt ^b  

^a UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

^b VIENNA UNIVERSITY OF TECHNOLOGY   (Austria)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER PROGRAMMING - ALGORITHMS; LEARNING SYSTEMS; MATHEMATICAL TECHNIQUES - DIFFERENTIAL EQUATIONS; PROBABILITY - RANDOM PROCESSES;

BACKPROPAGATION; CONVERGENCE; FEATURE EXTRACTION ALGORITHMS; FEEDFORWARD NETWORKS; LEARNING ALGORITHMS;

NEURAL NETWORKS;

EID: 0026222695     PISSN: 10459227     EISSN: 19410093     Source Type: Journal    
DOI: 10.1109/72.134285     Document Type: Article

References (23)

1. L. Ljung, “Analysis of recursive stochastic algorithms,” IEEE Trans. Automat. Contr., vol. AC-22, pp. 551–575, 1977.
2. H. J. Kushner and D. S. Clark, Stochastic Approximation Methods for Constrained and Unconstrained Systems. New York: Springer Verlag, 1978.
3. E. Oja, “A simplified neuron model as a principal component analyzer,” J. Math, and Biol., vol. 15, pp. 267–273, 1982.
4. E. Oja and J. Karhunen, “On stochastic approximation of the eigenvectors and the eigenvalues of the expectation of a random matrix,” J. Math. Anal. and Appl., vol. 106, pp. 69–84, 1985.
5. H. White, “Some asymptotic results for learning in single hidden-layer feedforward network models,” J. Amer. Statist. Ass., vol. 84, pp. 1003–1013, 1989.
6. T. D. Sanger, “Optimal unsupervised learning in a single-layer linear feedforward neural network,” Neural Networks, vol. 2, pp. 459–473, 1989.
7. K. Hornik and C.-M. Kuan, “Convergence analysis of local feature extraction algorithms,” BEBR Discussion Paper 90–1717, College of Commerce, University of Illinois at Urbana-Champaign, 1990.
8. C.-M. Kuan and H. White, “Recursive M-estimation, nonlinear regression and neural network learning with dependent observations,” BEBR Working Paper 90–1703, College of Commerce, University of Illinois, Urbana-Champaign, 1990.
9. H. J. Kushner, Approximation and Weak Convergence Methods for Random Processes. Cambridge, MA: MIT Press, 1984.
10. P. Billingsley, Convergence of Probability Measures. New York: Wiley, 1968.
11. H. J. Kushner and A. Shwartz, “Weak convergence and asymptotic properties of adaptive filters with constant gains,” IEEE Trans. Inform. Theory, vol. IT-30, pp. 177–182, 1984.
12. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning internal representations by error propagation,” in Parallel Distributed Processing: Explorations in the Microstructures of Cognition, D. E. Rumelhart et al., Eds. Cambridge, MA: MIT Press, 1986, vol. 1, ch. 8, 318–362.
13. P. Baldi and K. Hornik, “Neural networks and principal component analysis: Learning from examples without local minima,” Neural Networks, vol. 2, pp. 53–58, 1989.
14. H. Bourlard and Y. Kamp, “Auto-association by multilayer perceptrons and singular value decomposition,” Biol. Cybern., vol. 59, pp. 291–294, 1988.
15. R. Linsker, “Self-organization in a perceptual network,” Computer, vol. 21, pp. 105–117, 1988.
16. P. Baldi and K. Hornik, “Back-propagation and unsupervised learning in linear networks,” in Back Propagation: Theory, Architectures and Applications, Y. Chauvin and D. E. Rumelhart, Eds. Earlbaum Associates, 1991.
17. R. J. Williams, “Feature discovery through error-correction learning,” Tech. Rep. 8501, Institute of Cognitive Science, University of California, San Diego, 1985.
18. P. Baldi, “Linear learning: Landscapes and algorithms,” in Proc. 1988 NIPS Conf (Denver), 1988.
19. E. Oja, “Neural networks, principal components, and subspaces,” Int. J. Neural Systems, vol. 1, pp. 61–68, 1989.
20. A. Krogh and J. A. Hertz, “Hebbian learning of principal components,” in Parallel Processing in Neural Systems and Computers, R. Eckmiller et al., Eds. New York: Elsevier, 1990, pp. 183–186.
21. J. Rubner and P. Tavian, “A self-organizing network for principal component analysis,” Preprint, Physics Department, Technische Universitat München, Munich, Germany, 1990.
22. P. Földiák, “Adaptive network for optimal linear feature extraction,” in Proc. Int. Joint Conf. Neural Networks, 1989, pp. I:401 – 405.
23. J. L. Doob, Stochastic Processes. New York: Wiley, 1953.