Temporal-Kernel Recurrent Neural Networks

Neural Networks

Volumn 23, Issue 2, 2010, Pages 239-243

  Sutskever, Ilya ^a   Hinton, Geoffrey ^a  

^a UNIVERSITY OF TORONTO   (Canada)

Author keywords

Backpropagation through time; Fixed points; Long term dependencies; Recurrent Neural Networks; Supervised learning

Indexed keywords

BACK-PROPAGATION THROUGH TIME; CONNECTIONIST MODELS; FIXED POINTS; HIDDEN UNITS; INPUT STRING; LONG-TERM DEPENDENCIES; SEQUENTIAL PROBLEMS; SERIAL RECALL; SHORT TERM MEMORY; STABLE STATE;

BACKPROPAGATION; REINFORCEMENT LEARNING;

RECURRENT NEURAL NETWORKS;

ARTICLE; ARTIFICIAL NEURAL NETWORK; CONTROLLED STUDY; LANGUAGE; PREDICTION; PRIORITY JOURNAL; RECALL; SHORT TERM MEMORY; SPEECH;

ALGORITHMS; HUMANS; MEMORY, SHORT-TERM; NEURAL NETWORKS (COMPUTER); NEUROPSYCHOLOGICAL TESTS; TIME FACTORS;

EID: 73949127981     PISSN: 08936080     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neunet.2009.10.009     Document Type: Article

References (23)

1. Amit D. The Hebbian paradigm reintegrated: Local reverberations as internal representations. Behavioral and Brain Sciences 18 4 (1995) 617-657
2. Bengio Y., Simard P., and Frasconi P. Learning long-term dependencies with gradient descent is difficult. Neural Networks, IEEE Transactions on 5 2 (1994) 157-166
3. Botvinick M., and Plaut D. Short-term memory for serial order: A recurrent neural network model. Psychological Review 113 2 (2006) 201-233
4. Camperi M., and Wang X. A model of visuospatial working memory in prefrontal cortex: Recurrent network and cellular bistability. Journal of Computational Neuroscience 5 4 (1998) 383-405
5. De Vries B., and Principe J. Gamma model-A new neural model for temporal processing. Neural Networks 5 4 (1992) 565-576
6. Feldkamp L., Prokhorov D., and Feldkamp T. Simple and conditioned adaptive behavior from Kalman filter trained recurrent networks?. Neural Networks 16 5-6 (2003) 683-689
7. Gers F., Schmidhuber J., and Cummins F. Learning to forget: Continual prediction with LSTM. Neural Computation 12 10 (2000) 2451-2471
8. Gers F., Schraudolph N., and Schmidhuber J. Learning precise timing with LSTM recurrent networks. Journal of Machine Learning Research 3 (2002) 115-143
9. Hinton, G., & Brown, A. (2000). Spiking Boltzmann machines. In Advances in neural information processing systems 12: Proceedings of the 1999 conference
10. Hochreiter, S. (1991). Untersuchungen zu dynamischen neuronalen Netzen. Master's thesis, Institut fur Informatik, Technische Universitat, Munchen
11. Hochreiter S., and Schmidhuber J. Long short-term memory. Neural Computation 9 8 (1997) 1735-1780
12. Jaeger H., and Haas H. Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication. Science 304 5667 (2004) 78-80
13. Lin T., Horne B., Tino P., and Giles C. Learning long term dependencies in narx recurrent neural networks. Recurrent Neural Networks: Design and Applications (2000) 133
14. Mayer, H., Gomez, F., Wierstra, D., Nagy, I., Knoll, A., & Schmidhuber, J. (2006). A system for robotic heart surgery that learns to tie knots using recurrent neural networks. In Intelligent robots and systems, 2006 IEEE/RSJ international conference on (pp. 543-548)
15. Natarajan R., Huys Q., Dayan P., and Zemel R. Encoding and decoding spikes for dynamic stimuli. Neural Computation (2008) 1-36
16. Prokhorov, D., Feldkamp, L., & Tyukin, I. (2002). Adaptive behavior with fixed weights in RNN: An overview. In Proceedings of international joint conference on neural networks, (pp. 2018-2023) Vol. 2
17. Rumelhart D., Hinton G., and Williams R. Learning representations by back-propagating errors. Nature 323 6088 (1986) 533-536
18. Seung H. How the brain keeps the eyes still. Proceedings of the National Academy of Sciences 93 23 (1996) 13339
19. Shepherd G. The synaptic organization of the brain (1998), Oxford University Press, New York
20. Werbos P. Backpropagation through time: What it does and how to do it. Proceedings of the IEEE 78 10 (1990) 1550-1560
21. White O., Lee D., and Sompolinsky H. Short-term memory in orthogonal neural networks. Physical Review Letters 92 14 (2004) 148102
22. Williams, R. (1992). Training recurrent networks using the extended Kalman filter. In Neural networks, 1, 1992. IJCNN. International joint conference on, Vol. 4
23. Younger, A., Hochreiter, S., & Conwell, P. (2001). Meta-learning with backpropagation. In Neural networks,, 2001. Proceedings. IJCNN'01. International joint conference on, Vol. 3