An Adaptive Least Squares Algorithm for the Efficient Training of Artificial Neural Networks

IEEE Transactions on Circuits and Systems

Volumn 36, Issue 8, 1989, Pages 1092-1101

  Kollias, Stefanos ^a   Anastassiou, Dimitris ^a  

^a Department of Electrical Engineering Center for Telecommunications Research Columbia University ^*  (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER SYSTEMS, DIGITAL--PARALLEL PROCESSING; IMAGE PROCESSING; LOGIC CIRCUITS; MATHEMATICAL TECHNIQUES--LEAST SQUARES APPROXIMATIONS; OPTIMIZATION;

ARTIFICIAL NEURAL NETWORKS; DIGITAL IMAGE HALFTONING; FEEDFORWARD MULTILAYER PERCEPTRONS; LEARNING ALGORITHM; LEAST SQUARES OPTIMIZATION; NEURAL NETWORKS TRAINING;

SYSTEMS SCIENCE AND CYBERNETICS;

EID: 0024715766     PISSN: 00984094     EISSN: None     Source Type: Journal    
DOI: 10.1109/31.192419     Document Type: Article

References (24)

1. P. Werbos, “Beyond regression: New tool for prediction and analysis in the behavioral sciences,” Ph.D. dissertation, Harvard Univ., Cambridge, MA, 1974.
2. D. Rumelhart, G. Hinton, and G. Williams, “Learning internal representations by error propagation,” in Parallel Distributed Processing, vol. 1, (D. Rumelhart and J. McCleland, Eds.). Cambridge, MA: MIT Press, 1986.
3. D. Parker, “Learning logic,” MIT Tech. Rep. TR-47, Center for Comp. Res. in Econ. and Manag. Sci., 1985.
4. T. Sejnowski and C. Rosenberg, “NET talk: A parallel network that learns to read aloud,” Comp. Syst., vol. 1, pp. 145-168, 1987.
5. P. Werbos, “Building and understanding adaptive systems: A statistical/numerical approach to factory automation and brain research,” IEEE Trans. Syst., Man, Cybern., vol. SMC-17, Jan./Feb. 1987.
6. K. Levenberg, “A method for the solution of certain Nonlinear problems in least squares,” Quart. Appl. Math., pp. 164-168, 1944.
7. D. Marquardt, “An algorithm for least squares estimation of Nonlinear parameters,” J. Soc. Ind. Appl. Math., pp. 431-441, 1963.
8. J. Beck and K. Arnold, Parameter Estimation in Engineering and Science. New York: Wiley, 1976.
9. D. Luenberger, Linear and Nonlinear Programming. Reading, MA: Addison-Wesley, 1984.
10. J. Ortega and W. Rheinboldt, Iterative Solution of Nonlinear Equations in Several Variables. New York: Academic, 1970.
11. R. Fletcher and T. Freeman, “A modified Newton method for minimization,” J. Opt. Theory Appl., vol. 23, pp. 357-372, 1977.
12. R. Watrous, “Learning algorithms for connectionist networks: Applied gradient methods of nonlinear optimization,” Tech. Rep. MS-CIS-87-51, LINC LAB 72, Univ. of Pennsylvania, 1986.
13. D. Parker, “Second order backpropagation: Implementing an optimal O(n) approximation to Newton's method as an artificial neural network,” presented at the IEEE Conf. on Neural Information Processing Systems, Denver, CO, Nov. 1987.
14. M. Davies and J. Whitting, “A modified form of Levenberg correction,” in Numerical Methods for Nonlinear Optimization. (Ed. F. A. Lootsma). London, England: Academic, 1972.
15. J. Cadzow, “Recursive digital filters synthesis via gradient based algorithms,” IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-24, pp. 349-355, 1976.
16. G. Watson, Approximation Theory and Numerical Methods. New York: Wiley, 1980.
17. G. Carayannis, N. Kalouptsidis, and D. Manolakis, “A fast sequential algorithm for filtering and prediction,” IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-31, pp. 1394-1402, 1983.
18. Y. Boutalis, S. Kollias, and G. Carayannis, “A fast multichannel approach to adaptive image estimation,” IEEE Trans. Acoust., Speech, Signal Processing, vol. 37, July 1989.
19. J. Dennis and R. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Englewood Cliffs, NJ: Prentice-Hall, 1983.
20. J. More, “The Levenberg-Marquardt algorithm: Implementation and theory,” in Numerical Analysis, (G. A. Watson, Ed.) Lecture Notes in Math., 630, Springer Verlag, Berlin, 1977.
21. M. Osborne, “Nonlinear least squares: The Levenberg algorithm revisited,” J. Australian Math. Soc., vol. 19 (Series B), pp. 343-357, 1976.
22. D. Anastassiou, “Digital image halftoning: A neural based approach,” in Proc. IEEE Int. Symp. on CAS, Helsinki, Finland, June 1988.
23. J. Jarvis, C. Judice, and W. Ninke, “A new technique for displaying continuous-tone images on bi-level displays,” Computer Graphics and Image Processing, vol. 5, pp. 13-40, 1976.
24. P. Werbos, “Backpropagation: Past and future,” in Proc. IEEE Int. Conf. on Neural Networks, San Diego, CA, July 1988.