Memory efficient BFGS neural-network learning algorithms using MLP-NETwork: A survey

Proceedings of the IEEE International Conference on Control Applications

Volumn 1, Issue , 2004, Pages 586-591

  Asirvadam, Vijanth S ^a   McLoone, Seán F ^b   Irwin, George W ^c  

^a MULTIMEDIA UNIVERSITY   (Malaysia)

^b NATIONAL UNIVERSITY OF IRELAND   (Ireland)

^c QUEEN'S UNIVERSITY BELFAST   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

BACKPROPAGATION ALGORITHMS; BENCHMARK PROBLEMS; CONJUGATE GRADIENT (CG); TRAINING ALGORITHMS;

ALGORITHMS; BACKPROPAGATION; BENCHMARKING; COMPUTER SIMULATION; LEARNING ALGORITHMS; OPTICAL DATA STORAGE; PARAMETER ESTIMATION; PROBLEM SOLVING;

NEURAL NETWORKS;

EID: 17744383055     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (23)

1. Ruraelhart D.E, Hinton G.E and Williams R.J., "Learning internal representations by error propagation", Parallel Distributed Processing: Exploration in the Microstructure of Cognition, In: D.E Rumelhart and J.L McClelland (eds.) vol. 1, chapter 8, MIT press, Cambridge MA, 1986.
2. Jacobs R.A., " Increased Rates of Convergence Through Learning Rate Adaptation", Neural Networks, Vol 1. pp. 295-307, 1998.
3. Moreira M. and Fiesler E., "Neural Networks with Adaptive Learning Rate and Momentum Terms ", Istitut Dalle Molle D'Intelligence Artificielle Perceptive, IDIAP-RR 95-04, 1995.
4. Leonard J. and Kramer M.A., "Improvement of the Backpropagation Algorithm for Training Neural Networks ", Vol. 14:3, pp. 337-341, 1990.
5. Battiti R., "First and second-order methods for learning", Neural Computation, Vol 4, pp. 141-166, 1992.
6. McLoone S.F. and Irwin G.W., "Fast gradient based offline training of multilayer perceptrons", In: Neural Network Engineering in Dynamical Systems, Hunt, K.J., Irwin, G.W. and Warwick, K (eds.), Ch. 9, pp. 179-200, Springer, 1995.
7. Ruano A.E.B, Fleming P.J. and Jones D.I, "Connectionist Approach to PID autotuning", IEE Proc. Part-D, Vol.139:3, pp. 279-285., 1992.
8. Hagan M.T. and Menhaj M.B., 'Training feedforward networks with Marquardt algorithm", IEEE Trans. Neural Networks, Vol. 1.5:6, pp. 989-993, November 1994.
9. McLoone S.F. and Irwin G.W., "A Variable Memory Quasi-Newton Training Algorithm", Neural Processing Letters, Vol. 9, pp. 77-89, 1999.
10. Robitaille B., Marcos B., Vellette M. and Payre G., "Modified Quasi-Newton Methods for Training Neural Networks", Computers and Chemical Engineering, Vol. 20:9, pp. 1133-1140, 1996.
11. Lightbody G. and Irwin G.W., "A Parallel Algorithm for Training Neural Network Based Nonlinear Models", Proc. 2nd IFAC Workshop on Algorithms and Architectures for Real-Time Control, pp.99-104, 1992.
12. McLoone S.F. and Irwin G.W., "Fast parallel off-line training of multilayer perceptrons", IEEE Trans. Neural Networks, Vol. 8:3, pp. 646-653, July 1997.
13. Barnard, E., "Optimization for Training Neural Nets", IEEE Transactions on Neural Networks, Vol. 3:2, pp.232-240, 1992.
14. McKeown J.J., Meegan, D. and Sprevak, D, "An Introduction to Unconstrained Optimisation", Adam Hilger, Bristol, 1990.
15. Jenning Alan, McKeown John.J, "Matrix Computation 2nd Edition", John Wiley & Sons, 1992, New York.
16. McLoone S.F.,"Neural Network Training for Modelling and Control" PhD thesis, Queen's University of Blefast, Advanced Control Engineering Research Group, May 1996.
17. McLoone S.F., Asirvadam V.S. and Irwin G.W., "A Memory Optimal BFGS Neural Network Training Algorithm", International Joint Neural Network Conference, World Congress on Computational Intelligence, Honolulu, Hawaii, pp. 513-18, May 12-17, 2002.
18. McLoone S.F., "An efficient learning rate selection strategy for gradient based neural network training algorithms", Preprints 2nd Int. ICSC symposium on Neural Computation, Berlin, Germany, pp. 192-198, 2000.
19. Buckley A. and Lenir A., "QN-Like variable storage conjugate gradients", Mathematical Programming 27, pp. 155-175, 1983.
20. Nazareth L., "A relationship between the BFGS and conjugate gradient algorithm and its implications for new algorithms", SIAM Journal of Numerical Analysis, Vol. 16:5, pp. 794-800, 1979.
21. Nocedal J., "Updating Quasi-Newton matrices with limited storage", Mathematics of Computation, Vol. 35:151, pp. 773-782, 1980
22. Liu C.D. and Nocedal J., "On the limited memory BFGS method for large scale optimization" Mathematical Programming 45, pp. 503-528, 1989.
23. Heckenthaler T. and Engell S., "Robust approximately time-optimal fuzzy control of a two-tank system.", IEEE Control Systems Magazine, 14:24-30, 1994.