Learning in neural networks by normalized stochastic gradient algorithm: Local convergence

Proceedings of the 5th Seminar on Neural Network Applications in Electrical Engineering, NEUREL 2000

Volumn , Issue , 2000, Pages 11-17

  Tadic V ^a,b   Stankovic S ^c  

^a MIHAJLO PUPIN INSTITUTE   (Serbia)

^b UNIVERSITY OF MELBOURNE   (Australia)

^c School of Electrical Engineering University of Belgrade   (Serbia)

Author keywords

Algorithm design and analysis; Automatic control; Backpropagation algorithms; Convergence; Feedforward neural networks; Intelligent networks; Multi layer neural network; Multilayer perceptrons; Neural networks; Stochastic processes

Indexed keywords

ALGORITHMS; AUTOMATION; BACKPROPAGATION ALGORITHMS; CONTROL; CYBERNETICS; ELECTRIC NETWORK PARAMETERS; INTELLIGENT NETWORKS; LEARNING ALGORITHMS; MULTILAYER NEURAL NETWORKS; MULTILAYERS; NETWORK LAYERS; NEURAL NETWORKS; RANDOM PROCESSES; STOCHASTIC SYSTEMS;

ALGORITHM DESIGN AND ANALYSIS; ASYMPTOTIC PROPERTIES; CONVERGENCE; CONVERGENCE RESULTS; CRITERION FUNCTIONS; LOCAL CONVERGENCE; STOCHASTIC GRADIENT ALGORITHMS;

FEEDFORWARD NEURAL NETWORKS;

EID: 67649130661     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/NEUREL.2000.902375     Document Type: Conference Paper

References (19)

1. Chen, H.-F.: Recursive Estimation and Control for Stochastic Systems, New York: Wiley, 1985.
2. Ermoliev, Y.M.: Methods of Stochastic Programming, Moscow: Nauka, 1976. (in Russian)
3. Haykin, S.: Neural Networks: a Comprehensive Foundation, New York: MacMillan, 1994.
4. Hu, S.-T.: Elements of General Topology, San Francisco: Holden-Day, 1965.
5. Kuan, C.-M., Hornik, K. and White, H.: "A convergence result for learning in recurrent neural networks", Neural Computation, Vol. 6, pp. 420-440, 1994.
6. Kuan, C.-M. and White, H.: "Adaptive learning with nonlinear dynamics driven by dependent processes", Econometrica, Vol. 62, pp. 1087-1114, 1994.
7. Kuan, C.-M. and White, H.: "Artificial neural networks: an econometric perspective", Econometric Reviews, Vol. 13, pp. 1-91, 1994.
8. Kushner, H.J. and Clark, D.S.: Stochastic Aapproximation Methods for Constrained and Unconstrained Systems, New York: Springer Verlag, 1978.
9. Ljung, L, Pflug, G. and Walk, H.: Stochastic Approximation and Optimization of Random Systems, Boston: Birkhäuser Verlag, 1992.
10. Nedeljković, V.: A novel multilayer neural networks training algorithm that minimizes the probability of classification error", IEEE Transactions on Neural Networks, Vol. 4, pp. 650-659, 1993.
11. Nevel'son, M.B. and Has'minski, R.Z.: Stochastic Approximation and Recursive Estimation, Providence: American Mathematical Society, 1976.
12. Pados, D.A. and Papantoni-Kazakos, P.: "New nonleast-squares neural network learning algorithms for hypothesis testing", IEEE Transactions on Neural Networks, Vol. 6, pp. 596-609, 1995.
13. Polyak, B.T. and Tsypkin, Y.Z.: "Criterial algorithms of stochastic approximation" Automation and Remote Control, Vol. 45-2, pp. 766-774, 1984.
14. Rao, N.S., Protopopescu, V., Mann, R.C., Oblow, E.M. and Iyengar, S.S.: "Learning algorithm for feedforward networks based on finite samples", IEEE Transactions on Neural Networks, Vol. 7, pp. 926-940, 1996.
15. Shiryayev, A.N.: Probability, Vew York: Springer-Verlag, 1984.
16. Stanković, S. and Tadić, V.:"On the convergence of the stochastic gradient algorithm applied to learning in neural networks", Automation and Remote Control, Vol. 58, No. 7, pp. 118-134, 1998.
17. Vapnik, V.: The Nature of Statistical Learning Theory, New York: Springer Verlag, 1995.
18. White, H.: "Some asymptotic results for learning in single hidden-layer feedforward neural network models", Journal of the American Statistical Association, Vol. 84, pp. 1003-1013, 1989.
19. White, H.: Artificial Neural Networks, Oxford: Blackwell Publishers, 1992.