Learning algorithms for feedforward networks based on finite samples

IEEE Transactions on Neural Networks

Volumn 7, Issue 4, 1996, Pages 926-940

  Rao, Nageswara S V ^a   Protopopescu, Vladimir ^a   Mann, Reinhold C ^a   Oblow, E M ^a   Iyengar, S Sitharama ^a  

^a OAK RIDGE NATIONAL LABORATORY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

APPROXIMATION THEORY; BACKPROPAGATION; CONVERGENCE OF NUMERICAL METHODS; FUNCTIONS; LEARNING ALGORITHMS; MATHEMATICAL TECHNIQUES; RANDOM PROCESSES; REGRESSION ANALYSIS;

CONVERGENT ALGORITHMS; INEQUALITIES; ROBBINS-MONRO STYLE STOCHASTIC APPROXIMATION; WAVELET NETWORKS;

FEEDFORWARD NEURAL NETWORKS;

EID: 0030190723     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/72.508936     Document Type: Article

References (59)

1. M. A. Aizerman E. M. Braverman L. I. Rozonoer The Robbins̵Monro process and the method of potential functions Automat. Remote Contr. 26 1882 1885 1965
2. M. A. Aizerman E. M. Braverman L. I. Rozonoer Extrapolative problems in automatic control and method of potential functions Amer. Math. Soc. Translations 87 281 303 1970
3. A. R. Barron Universal approximation bounds for superpositions of a sigmoidal function IEEE Trans. Inform. Theory 39 931 945 1993 18 6498 256500
4. A. Benveniste M. Metivier P. Priouret Adaptive Algorithms and Stochastic Approximations. 1990 Springer-Verlag New York
5. P. Billingsley Probability and Measure. 1986 Wiley New York
6. A. L. Blum R. L. Rivest Training a three-node neural network is NP-complete Neural Networks 5 117 127 1992
7. E. K. Blum L. K. Li Approximation theory and feedforward networks Neural Networks 4 511 515 1991
8. A. Blumer A. Ehrenfeucht D. Haussler M. Warmuth Learnability and the Vapnik̵Chervonenkis dimension J. Assoc. Comput. Machinery 36 4 929 965 1989
9. E. M. Braverman E. S. Pjatnickii Estimation of the rate of convergence of algorithm based on the potential functions method Automat. Remote Contr. 27 1 80 100 1966
10. C. H. Chen H. Lai A comparison study of the gradient descent and the conjugate gradient backpropagation neural networks Proc. World Congr. Neural Networks III 401 406 1993
11. H. F. Chen Recursive Estimation and Control for Stochastic Systems. 1985 Wiley New York
12. H. F. Chen L. Guo Identification and Stochastic Adaptive Control. 1991 Birkha¨user MA, Boston
13. T.-S. Chiang C.-R. Hwang S.-J. Sheu Diffusion for global optimization in $R^n$ SIAM J. Contr. Optimization 25 3 737 753 1987
14. G. Cybenko Approximation by superpositions of a sigmoidal function Math. Contr., Signals, Syst. 2 303 314 1989
15. J. E. Moody S. J. Hanson R. P. Lippmann Toward faster stochastic gradient search Advances In Neural Information Processing Systems 4 1010 1016 1992 Morgan Kaufmann CA, San Mateo
16. J. L. Doob Stochastic Processes. 1953 Wiley New York
17. A. Farago G. Lugosi Strong universal consistency of neural network classifier IEEE Trans. Inform. Theory 39 1146 1151 1993 18 6268 243433
18. S. T. Hanson J. D. Cowan C. L. Giles Diffusion approximations for the constant learning rate backpropagation algorithm and resistance to local minima Advances in Neural Information Processing Systems 5 459 466 1993 Morgan Kaufmann CA, San Mateo
19. L. Fisher S. J. Yakowitz Uniform convergence of the potential function algorithm SIAM J. Contr. Optimization 63 9 95 103 1976
20. J. P. Fitch S. K. Lehman F. U. Dowla S. Y. Lu E. M. Johansson D. M. Goodman Ship wake detection procedure using conjugate gradient training artificial neural networks IEEE Trans. Geosci. Remote Sensing 29 718 726 1991 36 2742 83986
21. H. L. Frisch C. Borzi G. Ord J. K. Percus G. O. Williams Approximate representation of functions of several variables in terms of functions of one variable Phys. Rev. Lett. 63 9 927 929 1989
22. K. Funahashi On the approximate realizations of continuous mappings by neural networks Neural Networks 2 183 192 1989
23. S. B. Gelfand S. J. Mitter Metropolis-type annealing algorithms for global optimization in $R^d$ SIAM J. Contr. Optimization 31 1 111 131 1993
24. L. Gerencser Rate of convergence of recursive estimators SIAM J. Contr. Optimization 30 5 1200 1227 1992
25. P. Hall On projection pursuit regression Ann. Statist. 17 2 573 588 1989
26. P. Hall C. C. Heyde Martingale Limit Theory and Its Applications. 1980 Academic New York
27. R. A. Jacobs Increased rates of convergence through learning rate adaptation Neural Networks 1 295 307 1988
28. A. Kolmogorov On the representation of continuous functions of many variables by superpositions of continuous functions of one variable and addition Doklady Akademii Nauk SSSR 114 5 953 956 1957
29. V. Kurkova Kolmogorov's theorem and multilayer neural networks Neural Networks 5 501 506 1992
30. H. J. Kushner D. S. Clark Stochastic Approximations for Constrained and Unconstrained Systems. 1979 Springer-Verlag New York
31. M. Leshno V. Ya. Lin A. Pinkus S. Schocken Multilayer feedforward networks with a nonpolynomial activation function can approximate any function Neural Networks 6 861 867 1993
32. L. Ljung Analysis of recursive stochastic algorithms IEEE Trans. Automat. Contr. AC-22 4 551 575 1977
33. G. G. Lorentz Representation of Functions. 1986 Chelsea New York
34. H. N. Mhaskar C. A. Micchelli Approximation by superposition of sigmoidal and radial basis functions Advances Appl. Math. 13 350 373 1992
35. B. K. Natarajan Machine Learning: A Theoretical Approach. 1991 Morgan Kaufmann CA, San Mateo
36. V. Nedeljkovic A novel multilayer neural networks training algorithm that minimizes the probability of classification error IEEE Trans. Neural Networks 4 650 659 1993 72 6121 238319
37. B. T. Polyak Convergence and convergence rate of iterative stochastic algorithms I: General case Automat. Remote Contr. 12 1858 1868 1976
38. B. T. Polyak A. B. Juditsky Acceleration of stochastic approximation by averaging SIAM J. Contr. Optimization 30 4 838 855 1992
39. B. L. S. Prakasa Rao Nonparametric Functional Estimation. 1983 Academic New York
40. N. S. V. Rao Fusion rule estimation in multiple sensor systems with unknown noise densities J. Franklin Inst. 331B 5 509 530 1995
41. N. S. V. Rao V. Protopopescu Algorithms for {PAC} learning of functions with smoothness properties Proc. 4th Int. Symp. Artificial Intell. Math. 134 137 1996
42. N. S. V. Rao V. R. R. Uppuluri E. M. Oblow Stochastic approximation algorithms for classes of PAC learning problem Proc. 1994 IEEE Conf. Neural Networks 791 796 3013 8557 374279
43. P. Revesz How to apply the method of stochastic approximation in the nonparametric estimation of a regression function Math. Operationsforsch. Statist. Statist. 8 1 119 126 1977
44. H. Robbins S. Monro A stochastic approximation method Ann. Math. Statist. 22 400 407 1951
45. J. S. Rustagi A convergence theorem for nonnegative almost supermartingales and some applications Optimizing Methods in Statistics 233 258 1971 Academic New York
46. D. E. Rumelhart J. L. McClelland Learning internal representations by error propagation Parallel and Distributed Processing: Explorations in the Microstructures of Cognition 1986 MIT Press MA, Cambridge
47. D. Ruppert A Newton̵Raphson version of the multivariate Robbins̵Monroe procedure Ann. Statist. 13 1 236 245 1985
48. A. Griewank G. F. Corliss Neural networks, backpropagation, and automatic differentiation Automatic Differentiation of Algorithms: Theory, Implementation, and Application 31 42 1991 SIAM PA, Philadelphia
49. E. D. Sontag Feedback stabilization using two-hidden-layer nets IEEE Trans. Neural Networks 3 981 990 1992 72 4264 165599
50. V. Milutinovic P. Antognelli Training of multilayer perceptrons by stochastic approximation Neural Networks, Concepts, Applications and Implementations IV 201 239 1991 Prentice-Hall NJ, Englewood Cliffs
51. Z. Tang G. J. Koehler Deterministic global optimal FNN training algorithms Neural Networks 7 2 301 311 1994
52. Ya. Z. Tsypkin Establishing characteristics of a function transformer from randomly observed points Automat. Remote Contr. 26 1878 1881 1965
53. L. G. Valiant A theory of the learnable Comm. ACM 27 11 1134 1142 1984
54. P. P. van der Smagt Minimization methods for training feedforward neural networks Neural Networks 7 1 1 11 1994
55. V. N. Vapnik Estimation of Dependences Based on Empirical Data. 1982 Springer-Verlag New York
56. M. T. Wasan Stochastic Approximation. 1969 Cambridge Univ. Press U.K., Cambridge
57. P. J. Werbos Backpropagation through time: What it does and how to do it Proc. IEEE 78 10 1550 1560 1990 5 2116 58337
58. H. White Some asymptotic results for learning in single hidden layer feedforward network models J. Amer. Statist. Assoc. 84 1008 1013 1989
59. Q. Zhang A. Benveniste Wavelet networks IEEE Trans. Neural Networks 3 889 898 1992 72 4264 165591