Recurrent neural network with short-term memory and fast structural learning method

Systems and Computers in Japan

Volumn 34, Issue 6, 2003, Pages 69-79

  Kikuchi, Shinichi ^a   Nakanishi, Masakazu ^a  

^a KEIO UNIVERSITY   (Japan)

Author keywords

Convergence rate; Fast learning; Generalization ability; Recurrent neural network; Structural learning

Indexed keywords

ASYMPTOTIC STABILITY; BACKPROPAGATION; COMPUTER SIMULATION; FINITE AUTOMATA; LEARNING ALGORITHMS; LEARNING SYSTEMS;

CONVERGENCE RATE; FAST LEARNING; GENERALIZATION ABILITY; STRUCTURAL LEARNING;

RECURRENT NEURAL NETWORKS;

EID: 0038528482     PISSN: 08821666     EISSN: None     Source Type: Journal    
DOI: 10.1002/scj.1206     Document Type: Article

References (51)

1. Pearlmutter BA. Gradient calculations for dynamic recurrent neural networks: A survey. IEEE Trans Neural Networks 1995;6:1212-1228.
2. Sejnowski TJ, Rosenberg CR. Parallel networks that learn to pronounce English text. Complex Systems 1987;1:145-168.
3. Waibel A, Hanazawa T, Hinton G, Shikano K, Lang KJ. Phoneme recognition using time-delay neural networks. IEEE Trans Acoust Speech Signal Process 1989;37:328-339.
4. Jordan MI. Attractor dynamics and parallelism in a connectionist sequential machine. Proc 8th Annual Conference of the Cognitive Science Society, p 531-546, 1987.
5. Elman JL. Finding structure in time. Cogn Sci 1990;14:179-211.
6. Elman JL. Distributed representations, simple recurrent networks, and grammatical structure. Machine Learning 1991;7:195-225.
7. Frasconi P, Gori M, Soda G. Local feedback multi-layered networks. Neural Computation 1992;4:120-130.
8. Parlos AG, Chong KT, Atiya AF. Application of the recurrent multilayer perceptron in modeling complex process dynamics. IEEE Trans Neural Networks 1994;5:255-266.
9. Williams RJ, Zipser D. A learning algorithm for continually running fully recurrent neural networks. Neural Computation 1989;1:210-280.
10. Pearlmutter BA. Learning state space trajectories in recurrent neural networks. Neural Computation 1989;1:263-269.
11. Doya K, Yoshizawa S. Adaptive neural oscillator using continuous-time back-propagation learning. Neural Networks 1989;2:375-385.
12. Sato M. A learning algorithm to teach spatiotemporal patterns to recurrent neural networks. Biol Cybern 1990;62:259-263.
13. Mozer MC. Neural net architectures for temporal sequence processing. In Weigend AS, Gershenfeld NA (editors). Time series prediction: Forecasting the future and understanding the past. Addison-Wesley; 1993. p 243-264.
14. Cleeremans A, Servan-Schreiber D, McClelland JL. Finite state automata and simple recurrent networks. Neural Computation 1989;1:372-381.
15. Hanes MD, Ahalt SC, Krishnamurthy AK. Acoustic-to-phonetic mapping using recurrent neural networks. IEEE Trans Neural Networks 1994;5:659-662.
16. Port RF. Representation and recognition of temporal patterns. Connect Sci 1990;2:151-176.
17. Kremer SC. On the computational power of Elman-style recurrent networks. IEEE Trans Neural Networks 1995;6:1000-1004.
18. Rumelhart DE, Hinton GE, Williams RJ. Learning representations by back-propagating errors. Nature 1986;323:533-536.
19. Jacobs RA. Increased rates of convergence through learning rate adaptation. Neural Networks 1988;1:295-307.
20. Vogl TP, Mangis JK, Rigler AK, Zink WT, Alkon DL. Accelerating the convergence of the back-propagation method. Biol Cybern 1988;59:257-263.
21. van Ooyen A, Nienhuis B. Improving the convergence of the back-propagation algorithm. Neural Networks 1992;5:465-471.
22. Tanaka A, Komura M. Accelerating the convergence of the back-propagation method by removing singular points from BP errors. Trans IEICE 1992;J75-D-II:1000-1008. (in Japanese)
23. Jia Q, Toad N, Usui S. A study on initial value setting of the back-propagation learning algorithm. Trans IEICE 1990;J73-D-II:1179-1185. (in Japanese)
24. Mitani M, Kitsu N, Obori T, Watanabe I. A fast learning for pattern recognition neural networks. Trans IEICE 1994;J77-D-II:211-218. (in Japanese)
25. Ishikawa M. Recent topics on network learning algorithms. J SICE 1991;30:285-290. (in Japanese)
26. Reed R. Pruning algorithms-A survey. IEEE Trans Neural Networks 1993;4:740-747.
27. Kwok T-Y, Yeung D-Y. Constructive algorithms for structure learning in feedforward neural networks for regression problems. IEEE Trans Neural Networks 1997;8:630-645.
28. Ash T. Dynamic node creation in backpropagation networks. Connect Sci 1989;1:365-375.
29. Fahlman SE, Lebiere C. The cascade-correlation learning architecture. Adv Neural Inf Process Syst 1990;2:524-532.
30. Le Cun Y, Denker JS, Solla SA. Optimal brain damage. Adv Neural Inf Process Syst 1990;2:598-605.
31. Sietsma J, Dow RJF. Creating artificial neural networks that generalize. Neural Networks 1991;4:67-79.
32. Hagiwara M. A simple and effective method for removal of hidden units and weights. Neurocomputing 1994;6:207-218.
33. Hanson SJ, Pratt LY. Comparing biases for minimal network construction with backpropagation. Adv Neural Inf Process Syst 1989;1:177-185.
34. Chauvin Y. A back-propagation algorithm with optimal use of hidden units. Adv Neural Inf Process Syst 1989;1:519-526.
35. Plaut DC, Nowlan SJ, Hinton GE. Experiments on learning by back propagation. Tech Rep CMU-CS-86-126, Carnegie-Mellon University, 1986.
36. Ishikawa M. Structural learning with forgetting. Neural Networks 1996;9:509-521.
37. Fogel DB. An information criterion for optimal neural network selection. IEEE Trans Neural Networks 1991;2:490-497.
38. Murata N, Yoshizawa S, Amari S. Network information criterion-Determining the number of hidden units for an artificial neural network model. IEEE Trans Neural Networks 1994;5:865-872.
39. McClelland JL, Rumelhart DE. An interactive activation model of context effects in letter perception: Part 1. An account of basic findings. Psychol Rev 1981;88:375-407.
40. McClelland JL, Rumelhart DE. An interactive activation model of context effects in letter perception: Part 2. The contextual enhancement effect and some tests and extensions of the model. Psychol Rev 1982;89:60-94.
41. Mozer MC. A focused backpropagation algorithm for temporal pattern recognition. Complex Syst 1989;3:349-381.
42. Hinton GE. Connectionist learning procedures. Artif Intell 1989;40:185-234.
43. Tan Y, Kato Y, Ejima T. Error functions to improve learning speed in PDP models. Trans IEICE 1990;J73-D-II:2022-2028. (in Japanese)
44. Ishikawa M. A structural learning of connectionist models using forgetting. J JSAI 1989;5:595-603. (in Japanese)
45. Ishikawa M, Yamamoto H. Discovery of rules and generalization by structural learning of neural networks. Brain & Neural Networks 1994;1:57-63. (in Japanese)
46. Kinouchi M, Hagiwara M. Learning temporal sequences by complex neurons with local feedback. Trans IEE Japan 1996;116-C:748-754. (in Japanese)
47. Connor JT, Martin RD, Atlas LE. Recurrent neural networks and robust time series prediction. IEEE Trans Neural Networks 1994;5:240-254.
48. Kreider JF, Habel JS. The great energy predictor shootout-The first building data analysis and prediction, concept and summary. Proc. ASHRAE Seminar, 1993. ftp://ftp.cs.colorado.edu/pub/cs/energy-shootout
49. Ishikawa M, Moriyama T. Prediction of time series by a structural learning of neural networks. Fuzzy Sets Syst 1996;82:167-176.
50. Shimada H, Ishikawa M. Prediction of time series by structural learning of recurrent neural networks. Proc IIZUKA'98, p 610-614.
51. Demura K, Kajiura M, Anzai Y. The recurrent SOLAR algorithm. Trans IEICE 1995;J78-D-II:962-972. (in Japanese)