Fast and Efficient Second-Order Training of the Dynamic Neural Network Paradigm

Proceedings of the International Joint Conference on Neural Networks

Volumn 4, Issue , 2003, Pages 2482-2487

  Gruber, Christian ^a   Sick, Bernhard ^a  

^a UNIVERSITY OF PASSAU   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; APPROXIMATION THEORY; BACKPROPAGATION; COMPUTATIONAL METHODS; ERROR ANALYSIS; FIR FILTERS; REGRESSION ANALYSIS;

SECOND-ORDER TRAINING;

NEURAL NETWORKS;

EID: 0141608208     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (32)

1. G. Dorffner, "Neural networks for time series processing," in Neural Network World, vol. 6 (4), pp. 447-468. 1996.
2. C. Ulbricht, G. Dorffner, S. Canu, D. Guillemyn, G. Marijuán, J. Olarte, C. Rodríguez, and I. Martín, "Mechanisms for handling sequences with neural networks," in Intelligent Engineering Systems through Artificial Neural Networks, C. H. Dagli, L. I. Burke, and Y. C. Shin, Eds. 1992, vol. 2, pp. 273-278, ASME Press, New York.
3. A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K. J. Lang, "Phoneme recognition using time-delay neural networks," in IEEE Tr. on Acoustics, Speech, and Signal Processing, vol. 37 (3), pp. 328-339. 1989.
4. E. A. Wan, Finite Impulse Response Neural Networks with Applications in Time Series Prediction, Ph.D. thesis, Stanford University, Department of Electrical Engineering, 1993.
5. D.-T. Lin, J. E. Dayhoff, and P. A. Ligomenides, "Adaptive time-delay neural network for temporal correlation and prediction," in SPIE Intelligent Robots and Computer Vision XI, vol. 1826, pp. 170 - 181. 1992.
6. J. A Howell and H. Buxton, "Learning gestures with time-delay RBF networks," in Proc. of the 8th Int. Conf. on Artificial Neural Networks (ICANN '98), Skövde, L. Niklasson, M. Bodén, and T. Ziemke, Eds. 1998, vol. 1, pp. 239 - 244, Springer Verlag, London.
7. A. C. Tsoi, "Recurrent neural network architectures - An overview," in Adaptive Processing of Sequences and Data Structures, C. L. Giles and M. Gori, Eds., number 1387 in LNCS, pp. 1 - 26. Springer Verlag, Berlin, Heidelberg, New York, 1998, (Proc. of the Int. Summer School on Neural Networks, Vietri sul Mare, Salerno, 1997).
8. S. Haykin, Neural Networks - A Comprehensive Foundation, Macmillan College Publishing Company, New York, 1994.
9. U. Bodenhausen, "The tempo-algorithm: Learning in a neural network with variable time-delays," in Int. Joint Conf. on Neural Networks (IJCNN '90), Washington D.C., 1990, vol. 1, pp. 597 - 600.
10. T.-N. Lin, C. L. Giles, B. G. Horne, and S.-Y. Kung, "A delay damage model selection algorithm for NARX neural networks," in IEEE Tr. On Signal Processing, vol. 45 (11), pp. 2719 - 2730. 1997, (special issue on neural networks).
11. T. Lin, B. G. Horne, P. Tiňo, and C. L. Giles, "Learning long-term dependencies in NARX recurrent neural networks," in IEEE Tr. On Neural Networks, vol. 7 (6), pp. 1329 - 1338. 1996.
12. D.-T. Lin, P. A. Ligomenides, and J. E. Dayhoff, "Spatiotemporal topology and temporal sequence identification with an adaptive time-delay neural network," in SPIE - Intelligent Robots and Computer Vision XII, vol. 2055, pp. 536 - 545. 1993.
13. B. Sick, "Technische Anwendungen von Soft-Computing Methoden," Lecture Notes, University of Passau, July 2001.
14. U. Bodenhausen and A. Waibel, "Application oriented automatic structuring of time-delay neural networks for high performance character and speech recognition," in Proc. of the 1993 IEEE Int. Conf. on Neural Networks (ICNN '93), San Francisco, 1993, vol. 3, pp. 1627 - 1632.
15. E. A. Wan, "Time series prediction by using a connectionist network with internal delay lines," in Time Series Prediction: Forecasting the Future and Understanding the Past, A. S. Weigend and N. A. Gershenfeld, Eds., SFI Studies in the Sciences of Complexity, pp. 195 - 217. Addison-Wesley Publishing, Reading, 1993.
16. R. Cancelliere and R. Gemello, "Efficient training of time delay neural networks for sequential patterns," in Neurocomputing, vol. 10 (1), pp. 33-42. 1996.
17. S. P. Day and M. R. Davenport, "Continuous-time temporal backpropagation with adaptable time delays," in IEEE Tr. on Neural Networks, vol. 4 (2), pp. 348 - 354. 1993.
18. D.-T. Lin, J. E. Dayhoff, and P. A. Ligomenides, "Trajectory production with the adaptive time-delay neural network," in Neural Networks, vol. 8(3), pp. 447-461. 1995.
19. D. S. Clouse, C. L. Giles, B. G. Home, and G. W. Cottrell, "Time-delay neural networks: Representation and induction of finite-state machines," in IEEE Tr. on Neural Networks, vol. 8 (5), pp. 1065 - 1070. 1997.
20. H. T. Siegelmann, B. G. Home, and C. L. Giles, "Computional capabilities of recurrent NARX neural networks," IEEE Tr. on Systems, Man and Cybernetics - Part B, vol. 27, no. 2, pp. 208 - 215, 1997.
21. P. J. Werbos, "Backpropagation through time: What it does and how to do it," in Proc. of the IEEE, vol. 78 (10). 1990.
22. A. Zell, Simulation Neuronaler Netze, Addison-Wesley Publishing, Bonn, 2000.
23. A. J. Shepherd, Second Order Methods for Neural Networks - Fast and Reliable Training Methods for Multi-Layer Perceptrons, Springer-Verlag, London, Berlin, Heidelberg, 1997.
24. R. Battiti, "First- and second-order methods for learning: Between steepest descent and newton's method," Neural Computation, vol. 4, no. 2, 1992.
25. M. F. Møller, "A scaled conjugate algorithm for fast supervised learning," Neural Networks, vol. 6, no. 1, pp. 525-533, 1993.
26. C. M. Bishop, Neural Networks for Pattern Recognition, Clarendon Press, Oxford, 1995.
27. J. E. Dennis and R. B. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations, Prentice Hall, Englewood Cliffs, New Jersey, 1983.
28. C. Gruber, "Second-Order Lernalgorithmen zum effizienten Training dynamischer Neuronaler Netze," M.S. thesis, University of Passau, 2002.
29. M. Riedmiller and H. Braun, "A direct method for faster backpropagation learning: The RPROP algorithm," in Proc. of the 1993 IEEE Int. Conf. on Neural Networks (ICNN '93), San Francisco, 1993, vol. 1, pp. 586-591.
30. L. Prechelt, "PROBEN 1 - A Set of Neural Network Benchmark Problems and Benchmarking Rules," Tech. rep. 21/94, University of Karlsruhe, 1994.
31. B. A. Pearlmutter, "Fast exact multiplication by the Hessian," Neural Computation, vol. 6, no. 1, pp. 147 - 160, 1994.
32. K. Saito and R. Nakano, "Partial BFGS-update and efficient step-length calculation for three-layer neural networks," Neural Computation, vol. 9, no. 1, pp. 123-141, 1997.