LMS learning algorithms: Misconceptions and new results on convergence

IEEE Transactions on Neural Networks

Volumn 11, Issue 1, 2000, Pages 47-56

  Wang, Zi Qin ^a,b   Manry, Michael T ^a,c   Schiano, Jeffrey L ^a,d  

^a IEEE

^b FAS Technologies Inc   (United States)

^c University of Texas at Arlington   (United States)

^d The Pennsylvania State University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BACKPROPAGATION; LEARNING ALGORITHMS; LEARNING SYSTEMS; LEAST SQUARES APPROXIMATIONS;

LEAST MEAN SQUARE (LMS) LEARNING ALGORITHMS; WIDROW-HOFF DELTA RULE;

NEURAL NETWORKS;

EID: 0033640656     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/72.822509     Document Type: Article

References (44)

1. W. S. McCulloch and W. H. Pitts, "A logical calculus of the ideas immanent in nervous activity," Bull. Math. Biophys., vol. 5, pp. 115-133, 1943.
2. R. Rosenblatt, Principles of Neurodynamics. New York: Spartan, 1959.
3. M. Minsky and S. Papert, Perceptrons: An Introduction to Computational Geometry. Cambridge, MA: MIT Press, 1969.
4. B. Widrow and M. E. Hoff, "Adaptive switching circuits," in 1960 IRE WESCON Conv. Rec., 1960, pp. 96-104.
5. B. Widrow and S. D. Stearns, Adaptive Signal Processing. Englewood Cliffs, NJ: Prentice-Hall, 1985.
6. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning internal representations by error propagation," in Parallel Distributed Processing, D. E. Rumelhart and R. McClelland, Eds. Cambridge, MA: MIT Press, 1986, vol. 1.
7. P. J. Werbos, "Beyond regression: New tools for prediction and analysis in the behavioral sciences," Ph.D. dissertation, Harvard Univ., Cambodge, MA, 1974.
8. G. Cybenko, "Approximation by superpositions of a sigmoidal function," Math. Contr., Signals, Syst., vol. 2, no. 4, pp. 303-314, 1989.
9. K. Funahashi, "On the approximate realization of continuous mappings by neural networks," Neural Networks, vol. 2, pp. 183-192, 1989.
10. R. P. Lippmann, "An introduction to computing with neural nets," IEEE ASSP Mag., pp. 4-22, 1987.
11. S. A. Barton, "A matrix method for optimizing a neural network," Neural Comput., vol. 3, no. 3, pp. 450-459, 1991.
12. M. T. Manry, S. J. Apollo, L. S. Allen, W. D. Lyle, W. Gong, M. S. Dawson, and A. K. Fung, "Fast training of neural networks for remote sensing," Remote Sensing Rev., vol. 9, pp. 77-96, 1994.
13. H.-H. Chen, M. T. Manry, and H. Chandrasekaran, "A neural-network training algorithm utilizing multiple sets of linear equations," Neurocomput., vol. 25, pp. 55-72, Apr. 1999.
14. Y. H. Pao, Adaptive Pattern Recognition and Neural Networks. Reading, MA: Addison-Wesley, 1989, pp. 197-222.
15. M. S. Chen and M. T. Manry, "Backpropagation representation theorem using power series," presented at the IJCNN'90.
16. D. S. Chen and R. C. Jain, "A robust backpropagation learning algorithm for function approximation," IEEE Trans. Neural Networks, vol. 5, pp. 467-479, 1994.
17. N. Murata, S. Yoshizawa, and S. Amari, "Network information criterion - Determining the number of hidden units for an artificial neural-network model," IEEE Trans. Neural Networks, vol. 5, pp. 865-872, 1984.
18. J. S. Albus, "A new approach to manipulator control: The cerebellar model articulation controller (CMAC)," Trans. ASME J. Dynamic Syst. Measurement Contr., vol. 97, pp. 220-227, 1975.
19. _, "Data storage in the cerebellar model articulation controller (CMAC)," Trans. ASME J. Dynamic Syst. Measurement Contr., vol. 97, pp. 228-233, 1975.
20. D. S. Broomhead and D. Löwe, "Multivariable functional interpolation and adaptive networks," Complex Syst., vol. 2, pp. 321-355, 1988.
21. I. R. H. Jackson, "Convergence properties of radial basis functions," Constr. Approx., vol. 4, pp. 243-264, 1988.
22. W. T. Miller, F. H. Glanz, and L. G. Kraft, "CMAC: An associative neural network alternative to backpropagation," Proc. IEEE, vol. 78, no. 10, pp. 1561-1567, 1990.
23. S. Elanayar and Y. C. Shin, "Radial basis function neural network for approximation and estimation of nonlinear stochastic dynamic systems," IEEE Trans. Neural Networks, vol. 5, pp. 594-603, 1994.
24. A. J. Morris, G. A. Montague, and M. J. Willis, "Artificial neural networks: Studies in process modeling and control," Trans. ICHemE, pt. A, vol. 72, pp. 3-19, 1994.
25. C. Lau, Neural Networks: Theoretical Foundations and Analysis. New York, NY: IEEE Press, 1992.
26. B. Widrow and M. A. Lehr, "30 years of adaptive neural networks: Perceptron, madaline and backpropagation," Proc. IEEE, vol. 78, no. 9, pp. 1415-1442, 1990.
27. J. Moody, "Fast learning in multi-resolution hierarchies," in Advances in Neural Information Processing Systems 1, D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1989, pp. 29-39.
28. D. Knuth, "Sorting and searching," in The Art of Computer Programming. Menlo Park, CA: Addison-Wesley, 1973, vol. 3, p. 506.
29. W. T. Miller, "Real-time application of neural networks for sensor-based control of robots with vision," IEEE Trans. Syst., Man, Cybern., vol. 19, pp. 825-831, 1989.
30. S. Chen, C. F. N. Cowan, and P. M. Grant, "Orthogonal least squares learning algorithm for radial basis function networks," IEEE Trans. Neural Networks, vol. 2, pp. 302-309, 1991.
31. T. L. Hemminger and Y. H. Pao, "Detection and classification of underwater acoustic transients using neural networks," IEEE Trans. Neural Networks, vol. 5, no. 5, p. 712, 1994.
32. M. Klassen, Y. H. Pao, and V. Chen, "Characteristics of the functional link net: A high-order delta rule net," Int. Conf. Neural Networks, vol. 1, p. 507, July 1988.
33. Y. H. Pao and Y. Takefuji, "Functional link net computing: Theory, system architecture and functionalities," Computer, vol. 25, no. 5, p. 76, May 1992.
34. R. Flether, "Conjugate direction methods," in Numerical Methods for Unconstrained Optimization, W. Murray, Ed. New York: Academic, 1972, ch. 5.
35. P. E. Gill, W. Murray, and M. H. Wright, Practical Optimization. New York: Academic, 1981.
36. W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Numerical Recipes, 2nd ed. Cambridge, U.K.: Cambridge Univ. Press, 1992.
37. T. C. Hsia, System Identification: Least Squares Methods. Lexington, MA: Lexington, 1977.
38. L. Ljung and T. Soderstrom, Theory and Practice of Recursive Identification. Cambridge, MA: MIT Press, 1983.
39. L. A. Hageman and D. M. Young, Applied Iterative Methods. San Diego, CA: Academic, 1981.
40. R. S. Varga, Matrix Iterative Analysis. Englewood Cliffs, NJ: Prentice-Hall, 1962.
41. Y. F. Wong and A. Sideris, "Learning convergence in the cerebellar model articulation controller," IEEE Trans. Neural Networks, vol. 3, no. 1, pp. 115-121, Sept. 1992.
42. Z.-Q. Wang, J. L. Schiano, and M. Ginsberg, "Analysis of CMAC neural network learning," presented at the Proc. 1995 World Congr. Neural Networks, Washington, D.C., Aug. 1995.
43. M. T. Hagen, H. B. Demuth, and M. Beale, Neural Network Design. Boston, MA: PWS, 1996.
44. H. Demuth and M. Beale, Neural Network Toolbox User's Guide. Natick, MA: The Math Works Inc., June 1994.