Modified Quasi-Newton methods for training neural networks

Computers and Chemical Engineering

Volumn 20, Issue 9, 1996, Pages 1133-1140

  Robitaille, B ^a   Marcos, B ^a   Veillette, M ^a   Payre, G ^a  

^a UNIVERSITÉ DE SHERBROOKE   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

BACKPROPAGATION; CONVERGENCE OF NUMERICAL METHODS; LEARNING ALGORITHMS; LEARNING SYSTEMS;

CONJUGATE GRADIENT METHOD; HESSIAN MATRIX; MODIFIED QUASI NEWTON METHODS;

FEEDFORWARD NEURAL NETWORKS;

EID: 0030243770     PISSN: 00981354     EISSN: None     Source Type: Journal    
DOI: 10.1016/0098-1354(95)00228-6     Document Type: Article

References (24)

1. Agrawal P., C. Lee, H. C. Lim and D. Ramkrishna, Theoretical investigations of dynamic behavior of isothermal continuous stirred tank biological reactors. Chem. Engng Sci. 37, 453-562 (1982).
2. Armijo L., Minimization of functions having Lipschitz-continuous first partial derivatives. Pac. J. Math. 16, 1-3 (1966).
3. Barnard E., Optimization for training neural nets. IEEE Trans. Neural Networks 3, 232-240 (1992).
4. Becker S. and Y. Le Cun, Improving the convergence of back-propagation learning with second order methods. In D. Touretzky, G. Hinton and T. Sejnowski (eds). Proc. Connectionist Model Summer School, pp. 29-37. Morgan-Kaufman, San Mateo (1988).
5. Bello M. G., Enhanced training algorithms and integrated training/architecture selection for multilayer perceptron networks. IEEE Trans. Neural Networks 3, 864-875 (1992).
6. Bishop C., Exact calculation of the Hessian matrix for the multilayer perceptron. Neural Comput. 4, 494-501 (1992).
7. Dennis J. E. and R. B. Schnabel, Numerical Methods for Unconstrained Optimisation and Nonlinear Equations. Prentice-Hall, Englewood Cliffs, New Jersey (1983).
8. Fahlman S. E., An empirical study of learning speed in back-propagation networks. Internal report: CMU-CS-88/162, Carnegie Mellon University, Pittsburgh (1988).
9. Fletcher R. and C. M. Reeves, Function minimization by conjugate gradients. Comput. J. 7, 149-154 (1964).
10. Hoskins J. C. and D. M. Himmelblau, Artificial neural network model of knowledge representation in chemical engineering. Computers chem. Engng 12, 881-890 (1988).
11. Jacobs R. A., Increased rates of convergences through learning rate adaptation. Neural Networks 1, 29-36 (1988).
12. Kollias S. and D. Anastassiou, An adaptive least squares algorithm for the efficient training of artificial neural networks. IEEE Trans. Circuits Systems 36, 1092-1101 (1989).
13. Leonard J. A. and M. A. Kramer, Improvement of the backpropagation algorithm for training neural networks. Computers chem. Engng 14, 337-341 (1990).
14. Nahas E. P., M. A. Henson and D. E. Seborg, Nonlinear internal model control strategy for neural network models. Computers chem. Engng 16, 1039-1057 (1992).
15. Parker D. B., Optimal algorithms for adaptive networks: second order back propagation, second order direct propagation, and second order Hebbian learning. IEEE First Int. Conf. Neural Networks, San Diego, II-593-600 (1987).
16. Renard F., Réseaux de neurones couplés à la spectrophotométrie ultraviolette pour le contrôle automatique de la qualité des eaux. Master thesis, Université de Sherbrooke, Sherbrooke, Québec, Canada, to be published Summer 1995.
17. Rigler A. K., J. M. Irvine and K. Vogl, Rescaling of variables in backpropagation learning. Neural Networks 4, 225-229 (1991).
18. Rumelhart D. E. G. E. Hinton and R. J. Williams, Learning internal representation by error propagation. In D. E. Rumelhart and J. L. McClelland (eds), Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Ch. 8. MIT Press, Cambridge, MA (1986).
19. Tallaneare T., SuperSAB: fast adaptive backpropagation with good scaling properties. Neural Networks 3, 561-573 (1990).
20. Thomas O. and S. Gallot, Ultaviolet multiwavelength absorptiometry (UVMA) for the examination of natural waters and wastewaters; part I: general considerations. Fresenius J. Anal. Chem. 338, 234-237 (1990).
21. Van Ooyen A. and B. Nienhuis, Improving the convergence of the back-propagation algorithm. Neural Networks 5, 465-471 (1992).
22. Ungar L. H., A bioreactor benchmark for adaptive network-based process control. In W. T. Miller, III, R. S. Sutton and P. J. Werbos (eds), Neural Networks for Control, Ch. 16. MIT Press, Cambridge, MA, U.S.A. (1990).
23. Watrous R. L., Learning algorithms for connectionist networks: applied gradient methods of nonlinear optimization. IEEE First Int. Conf. Neural Networks, San Diego, II-619-627 (1987).
24. Werbos P. J., Beyond regression: new tools for prediction and analysis in the behavioral sciences. Ph.D. Thesis, Harvard University Committee in Applied Mathematics (1974).