A generalized LSTM-like training algorithm for second-order recurrent neural networks

Neural Networks

Volumn 25, Issue , 2012, Pages 70-83

  Monner, Derek ^a   Reggia, James A ^a  

^a Institute of Advanced Computer Science   (United States)

Author keywords

Gradient based training; Long Short Term Memory (LSTM); Recurrent neural network; Sequential retrieval; Temporal sequence processing

Indexed keywords

GRADIENT BASED; LOCAL ENVIRONMENTS; LONG SHORT TERM MEMORIES; OPERATING INSTRUCTIONS; ORIGINAL ALGORITHMS; RECURRENT NETWORKS; SECOND ORDERS; SEQUENTIAL RETRIEVAL; SHORT TERM MEMORY; SINGLE LAYER; SPATIAL AND TEMPORAL LOCALITY; SPECIFIC TASKS; TEMPORAL SEQUENCES; TRAINING ALGORITHMS;

ALGORITHMS; BRAIN; MEMORY ARCHITECTURE; NETWORK LAYERS; RECURRENT NEURAL NETWORKS;

NETWORK ARCHITECTURE;

ANALYTICAL ERROR; ARTICLE; CELL ENGINEERING; GENERALIZED LONG SHORT TERM MEMORY; INFORMATION PROCESSING; INFORMATION STORAGE; INTERMETHOD COMPARISON; LEARNING ALGORITHM; MATHEMATICAL COMPUTING; MEMORY CELL; NERVE CELL MEMBRANE POTENTIAL; NERVE CELL NETWORK; PRIORITY JOURNAL; SHORT TERM MEMORY; STATISTICAL ANALYSIS; TASK PERFORMANCE;

ALGORITHMS; LEARNING; MEMORY, LONG-TERM; MEMORY, SHORT-TERM; NEURAL NETWORKS (COMPUTER); TIME FACTORS;

EID: 82355185899     PISSN: 08936080     EISSN: 18792782     Source Type: Journal    
DOI: 10.1016/j.neunet.2011.07.003     Document Type: Article

References (23)

1. Bayer, J., Wierstra, D., Togelius, J., & Schmidhuber, J. (2009). Evolving memory cell structures for sequence learning. In Proceedings of the international conference on artificial neural networks.
2. Elman J.L. Finding structure in time. Cognitive Science 1990, 14:179-211.
3. Gers F.A., Cummins F. learning to forget: continual prediction with LSTM. Neural Computation 2000, 12:2451-2471.
4. Gers F.A., Pérez-Ortiz J.A., Eck D., Schmidhuber J. Kalman filters improve LSTM network performancee in problems unsolvable by traditional recurrent nets. Neural Networks 2003, 16:241-250.
5. Gers, F. A., & Schmidhuber, J. (2000). Recurrent nets that time and count. In Proceedings of the international joint conference on neural networks (pp. 189-194).
6. Gers F.A., Schmidhuber J. LSTM recurrent networks learn simple context-free and context-sensitive languages. IEEE Transactions on Neural Networks 2001, 12:1333-1340.
7. Giles C.L., Maxwell T. Learning, invariance, and generalization in high-order neural networks. Applied Optics 1987, 26:4972-4978.
8. Graves, A., Eck, D., Beringer, N., & Schmidhuber, J. (2004). Biologically plausible speech recognition with LSTM neural nets. In Proceedings of the international workshop on biologically inspired approaches to advanced information technology (pp. 127-136).
9. Graves A., Schmidhuber J. offline handwriting recognition with multidimensional recurrent neural networks. Neural information processing systems: Vol. 21 2008, D. Koller, D. Schuurmans, Y. Bengio, L. Bottou (Eds.).
10. Hinton G.E. Connectionist learning procedures. Artificial Intelligence 1989, 40:185-234.
11. Hochreiter S., Schmidhuber J. Long short-term memory. Neural Computation 1997, 9:1735-1780.
12. Miller C.B., Giles C.L. Experimental comparison of the effect of order in recurrent neural networks. Pattern Recognition 1993, 7:849-872.
13. Monner, D., & Reggia, J. A. (2009). An unsupervised learning method for representing simple sentences. In 2009 international joint conference on neural networks (pp. 2133-2140).
14. Monner, D., & Reggia, J. A. (2011). Systematically grounding language through vision in a deep, recurrent neural network. To appear in Proceedings of the conference on artificial general intelligence.
15. Psaltis D., Park C., Hong J. Higher order associative memories and their optical implementations. Neural Networks 1988, 1:149-163.
16. Puskorius G.V., Feldkamp L.A. Neurocontrol of nonlinear dynamical systems with Kalman filter trained recurrent networks. IEEE Transactions on Neural Networks 1994, 5:279-297.
17. Rumelhart D.E., Hinton G.E., Williams R.J. Learning representations by back-propagating errors. Nature 1986, 323:533-536.
18. Schmidhuber J., Wierstra D., Gagliolo M., Gomez F. Training recurrent networks by Evolino. Neural Computation 2007, 19:757-779.
19. Shin, Y., & Ghosh, J. (1991). The pi-sigma network: an efficient higher-order neural network for pattern classification and function approximation. In Proceedings of the international joint conference on neural networks (pp. 13-18).
20. Sutton R.S., Barto A.G. Temporal-difference learning. Reinforcement learning: an introduction 1998, 167-200. MIT Press.
21. Weems S.A., Reggia J.A. Simulating single word processing in the classic aphasia syndromes based on the Wernicke-Lichtheim-Geschwind theory. Brain and Language 2006, 98:291-309.
22. Werbos P.J. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE 1990, 78:1550-1560.
23. Williams R.J., Zipser D. A learning algorithm for continually running fully recurrent neural networks. Neural Computation 1989, 1:270-280.