Adaptive self-scaling non-monotone BFGS training algorithm for recurrent neural networks

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 4668 LNCS, Issue PART 1, 2007, Pages 259-268

  Peng, Chun Cheng ^a   Magoulas, George D ^a  

^a UNIVERSITY OF LONDON   (United Kingdom)

Author keywords

Adaptive learning; BFGS; Line search; Nonmonotone strategy; Recurrent networks; Self scaling factor

Indexed keywords

NETWORK ARCHITECTURE;

ADAPTIVE LEARNING; BFGS; LINE SEARCHES; NONMONOTONE; RECURRENT NETWORKS; SCALING FACTORS;

RECURRENT NEURAL NETWORKS;

EID: 38149085445     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/978-3-540-74690-4_27     Document Type: Conference Paper

References (50)

1. Abraham, A.: Meta learning evolutionary artificial neural networks. Neurocomputing 56, 1-38 (2004)
2. Al-Baali, M.: Numerical experience with a class of self-scaling quasi-Newton algorithms. J. Optim. Theory Appl. 96, 533-553 (1998)
3. Alpsan, D., Towsey, M., Ozdamar, O., Tsoi, A.C., Ghista, D.N.: Efficiency of modified backpropagation and optimisation methods on a real-world medical problem. Neural Networks 8(6), 945-962 (1995)
4. Asirvadam, V.S., McLoone, S.F., Irwin, G.W.: Memory efficient BFGS neural-network learning algorithms using MLP-network: a survey. In: Proc. IEEE Int'l Conf. Control Application, pp. 586-591. IEEE Computer Society Press, Los Alamitos (2004)
5. Bajramovic, F., Gruber, C., Sick, B.: A comparison of first- and second- order training algorithms for dynamic neural networks. In: Proc. IEEE Int'l Joint Conf. Neural Networks, pp. 837-842. IEEE Computer Society Press, Los Alamitos (2004)
6. Baldi, P.: Gradient descent learning algorithm overview: a general dynamical systems perspective. IEEE Trans. Neural Networks 6(1), 182-195 (1993)
7. Becerikli, Y., Konar, A.F., Samad, T.: Intelligent optimal control with dynamic neural networks. Neural Networks 16, 251-259 (2003)
8. Becerikli, Y., Oysal, Y., Konar, A.F.: Trajectory priming with dynamic fuzzy networks in nonlinear optimal control. IEEE Tr. Neu. Net. 15(2), 383-394 (2004)
9. Beigi, H.S.M.: Neural networks learning through optimally conditioned quadratically convergent methods requiring no line search. In: Proc. 36th Midwest Symposium on Circuits and Systems, pp. 109-112 (1993)
10. Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Networks 5(2), 157-166 (1994)
11. Charalambous, C.: A conjugate gradient algorithm for the efficient training of artificial neural networks. IEE Proceedings Part G 139, 301-310 (1992)
12. Chella, A., Gentile, A., Sorbello, F., Tarantino, A.: Supervised learning for feed-forward neural networks: a new minimax approach for fast convergence. In: Proc. IEEE Int'l Conf. Neural Networks, pp. 605-609. IEEE Computer Society Press, Los Alamitos (1993)
13. Chen, O.T.-C., Sheu, B.J.: Optimization schemes for neural network training. In: Proc. IEEE Int'l Conf. Neural Networks, pp. 817-822. IEEE Computer Society Press, Los Alamitos (1994)
14. Chen, K., Xu, L., Chi, H.: Improved learning algorithms for mixture of experts in multiclass classification. Neural Networks 12, 1229-1252 (1999)
15. Denton, J.W., Hung, M.S.: A comparison of nonlinear methods for supervised learning in multilayer feedforward neural nets. Eu. J. Ope. Res. 93, 358-368 (1996)
16. Dietterich, T.G.: Machine learning for sequential data: a review. In: Caelli, T.M., Amin, A., Duin, R.P.W., Kamel, M.S., de Ridder, D. (eds.) SPR 2002 and SSPR 2002. LNCS, vol. 2396, pp. 15-30. Springer, Heidelberg (2002)
17. dos Santos, E.P., von Zuben, F.J.: Efficient second-order learning algorithms for discrete-time recurrent neural networks. In: Medsker, L.R., Jain, L.C. (eds.) Rec. neural nets: design and applications, pp. 47-75. CRC Press, USA (2000)
18. Duch, W., Korczak, J.: Optimisation and global minimisation methods suitable for neural networks. Neural Computing Surveys 2, 163-212 (1999)
19. Fischer, M.M., Staufer, P.: Optimisation in an error backpropagation neural network environment with a performance test on a spectral pattern classification problem. Geographical Analysis 31(2), 89-108 (1999)
20. Gill, P.E., Murray, W., Wright, M.H.: Practical Optimization. Academic Press, London (1981)
21. Gordienko, P.: Construction of efficient neural networks: algorithms and tests. In: Proc. IEEE Int'l Joint Conf. Neural Networks, pp. 313-316. IEEE Computer Society Press, Los Alamitos (1993)
22. Grippo, L., Lampariello, F., Lucidi, S.: A nonmonotone line search technique for Newton's method. SIAM J. Numerical Analysis 23, 707-716 (1986)
23. Gruber, C., Sick, B.: Fast and efficient second-order training of the dynamic neural network paradigm. In: Proc IEEE Int'l J. Conf. Neu. Nets., vol. 4, pp. 2482-2487. IEEE Computer Society Press, Los Alamitos (2003)
24. Hui, L.C.K., Lam, K.-Y., Chea, C.W.: Global optimisation in neural network training. Neural Computing and Applications 5, 58-64 (1997)
25. Irwin, G., Lightbody, G., McLoone, S.: Comparison of gradient based training algorithms for multilayer perceptrons. In: Proc. IEE Colloquium Advances in Neural Networks for Control and Systems, 11/1-11/6 (1994)
26. Ishikawa, T., Tsukui, Y., Matsunami, M.: Optimization of electromagnetic devices using artificial neural network with quasi-Newton algorithm. IEEE Trans. Magnetics 32(3), 1226-1229 (1996)
27. Karjala, T.W., Himmelblau, D.M., Miikkulainen, R.: Data rectification using recurrent (Elman) neu. nets. In: Proc. IEEE Int'l J. Conf. Neu. Nets., pp. 901-906. IEEE Computer Society Press, Los Alamitos (1992)
28. Kremer, S.C.: Spatiotemporal connectionist networks: a taxonomy and review. Neural Computation 13, 249-306 (2001)
29. Lightbody, G., Irwin, G.W.: A novel neural internal model control structure. In: Proc. American Control Conf., pp. 350-354 (1995)
30. Lightbody, G., Irwin, G.W.: Multi-layer perceptron based modelling of nonlinear systems. Fuzzy Sets and Systems 79, 93-112 (1996)
31. Likas, A., Stafylopatis, A.: Training the random neural network using quasi-Newton methods. European J. Operational Research 126, 331-339 (2000)
32. McLeod, P., Plunkett, K., Rolls, E.T.: Introduction to connectionist modelling of cognitive processes, pp. 148-151. Oxford University Press, Oxford (1998)
33. McLoone, S., Asirvadam, V.S., Irwin, G.W.: A memory optimal BFGS neural network training algorithm. In: Proc. IEEE Int'l J. Conf. Neu. Nets., pp. 513-518. IEEE Computer Society Press, Los Alamitos (2002)
34. McLoone, S., Irwin, G.W.: Fast parallel off-line training of multilayer perceptrons. IEEE Tr. Neural Networks 8(3), 646-653 (1997)
35. McLoone, S., Irwin, G.: A variable memory quasi-Newton training algorithm. Neural Processing Letters 9, 77-89 (1999)
36. Mizutani, E.: Powell's dogleg trust-region steps with the quasi-Newton augmented Hessian for neural nonlinear least-squares learning. In: Proc. IEEE Int'l Joint Conf. Neural Networks, pp. 1239-1244. IEEE Computer Society Press, Los Alamitos (1999)
37. Nawi, N.M., Ransing, M.R., Ransing, R.S.: An improved learning algorithm based on the Broyden-Fletcher-Goldfarb-Shanno (BFGS) method for back propagation neural networks. In: Proc. 6th Int'l Conf. Intel. Sys. Design and Applications, pp. 152-157 (2006)
38. Nocedal, J., Yuan, Y.: Analysis of a self-scaling quasi-Newton method. Math. Program 61, 19-37 (1993)
39. Pearlmutter, B.A.: Gradient calculations for dynamic recurrent neural networks: a survey. IEEE Trans. Neural Networks 6(5), 1212-1228 (1995)
40. Phua, P.K.H., Ming, D.: Parallel nonlinear optimization techniques for training neural networks. IEEE Trans. Neural Networks 14(6), 1460-1468 (2003)
41. Plagianakos, V.P., Magoulas, G.D., Vrahatis, M.N.: Deterministic nonmonotone strategies for effective training of multi-layer perceptrons. IEEE Trans. Neural Networks 13(6), 1268-1284 (2002)
42. Plaut, D., McClelland, J., Seidenberg, M., Patterson, K.: Understanding normal and impaired reading: computational principles in quasi-regular domains. Psychological Review 103, 56-115 (1993)
43. Saini, L.M., Soni, M.K.: Artificial neural network based peak load forecasting using Levenberg-Marquardt and quasi-Newton methods. In: Proc. Generation, Transmission and Distribution, pp. 578-584 (2002)
44. Setiono, R., Hui, L.C.K.: Some n-bit parity problems are solvable by feed-forward networks with less than n hidden units. In: Proc. IEEE Int'l Joint Conf. Neural Networks, pp. 305-308. IEEE Computer Society Press, Los Alamitos (1993)
45. Setiono, R., Hui, L.C.K.: Use of a quasi-Newton in a feedforward neural network construction algorithm. IEEE Trans. Neural Networks 6(1), 273-277 (1995)
46. Sorensen, P.H., Norgaard, M., Ravn, O., Poulsen, N.K.: Implementation of neural network based non-linear predictive control. Neurocomputing 28, 37-51 (1999)
47. Sperduti, A., Starita, A.: Supervised neural networks for the classification of structures. IEEE Trans. Neural Networks 8(3), 714-735 (1997)
48. Tsoi, A.C.: Recurrent neural network architectures: an overview. In: Giles, C.L., Gori, M. (eds.) Adaptive Processing of Sequences and Data Structures. LNCS (LNAI), vol. 1387, pp. 1-26. Springer, Heidelberg (1998)
49. Tsoi, A.C.: Gradient based learning algorithms. In: Giles, C.L., Gori, M. (eds.) Adaptive Processing of Sequences and Data Structures. LNCS (LNAI), vol. 1387, pp. 27-62. Springer, Heidelberg (1998)
50. Yin, H.X., Du, D.L.: The global convergence of self-scaling BFGS algorithm with nonmonotone line search for unconstrained nonconvex optimization problems. Acta Mathematica Sinica, Springer, Heidelberg (2006), (Online) http://www.springerlink.com/content/17h30w8t4217mh64/fulltext.pdf