Recurrent neural networks with auxiliary memory units

IEEE Transactions on Neural Networks and Learning Systems

Volumn 29, Issue 5, 2018, Pages 1652-1661

  Wang, Jianyong ^a   Zhang, Lei ^a   Guo, Quan ^a   Yi, Zhang ^a  

^a SICHUAN UNIVERSITY   (China)

Author keywords

Learning algorithms; Memory; Recurrent neural networks (RNNs); Sequence learning

Indexed keywords

ERRORS; LEARNING SYSTEMS; NEURAL NETWORKS; NEURONS; RECURRENT NEURAL NETWORKS; SOFTWARE PROTOTYPING;

EFFICIENT LEARNING; MEMORY MECHANISM; RECURRENT NEURAL NETWORK (RNNS); SEQUENCE CLASSIFICATION; SEQUENCE GENERATION; SEQUENCE LEARNING; STABLE CONVERGENCE; STATE OF THE ART;

LEARNING ALGORITHMS;

EID: 85016125010     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2017.2677968     Document Type: Article

References (25)

1. Z. Yi and K. K. Tan, Convergence Analysis of Recurrent Neural Networks. Norwell, MA, USA: Kluwer, 2004.
2. B. Hammer, "On the approximation capability of recurrent neural networks, " Neurocomputing, vol. 31, no. 1-4, pp. 107-123, 2000.
3. Z. Yi and K. K. Tan, "Multistability of discrete-time recurrent neural networks with unsaturating piecewise linear activation functions, " IEEE Trans. Neural Netw., vol. 15, no. 2, pp. 329-336, Mar. 2004.
4. L. Zhang, Z. Yi, S. L. Zhang, and P. A. Heng, "Activity invariant sets and exponentially stable attractors of linear threshold discrete-time recurrent neural networks, " IEEE Trans. Autom. Control, vol. 54, no. 6, pp. 1341-1347, Jun. 2009.
5. Z. Yi and K. K. Tan, "Global convergence of Lotka-Volterra recurrent neural networks with delays, " IEEE Trans. Circuits Syst. I, Reg. Papers, vol. 52, no. 11, pp. 2482-2489, Nov. 2005.
6. Z. Yi, "Foundations of implementing the competitive layer model by Lotka-Volterra recurrent neural networks, " IEEE Trans. Neural Netw., vol. 21, no. 3, pp. 494-507, Mar. 2010.
7. J. L. Elman, "Finding structure in time, " Cognit. Sci., vol. 14, no. 2, pp. 179-211, Mar. 1990.
8. S. Hochreiter and J. Schmidhuber, "Long short-term memory, " Neural Comput., vol. 9, no. 8, pp. 1735-1780, 1997.
9. J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. [Online]. Available: https://arxiv.org/abs/1412.3555.
10. K. Cho, B. van Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, and Y. Bengio. (2014). "Learning phrase representations using RNN encoder-decoder for statistical machine translation." [Online]. Available: https://arxiv.org/abs/1406.1078.
11. I. Sutskever, J. Martens, and G. E. Hinton, "Generating text with recurrent neural networks, " in Proc. 28th Int. Conf. Mach. Learn. (ICML), 2011, pp. 1017-1024.
12. A. Graves, A.-R. Mohamed, and G. E. Hinton, "Speech recognition with deep recurrent neural networks, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), May 2013, pp. 6645-6649.
13. J. Donahue et al., "Long-term recurrent convolutional networks for visual recognition and description, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2015, pp. 2625-2634.
14. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, "Show and tell: A neural image caption generator, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Boston, MA, USA, Jun. 2015, pp. 3156-3164.
15. K. Cho, A. Courville, and Y. Bengio, "Describing multimedia content using attention-based encoder-decoder networks, " IEEE Trans. Multimedia, vol. 17, no. 11, pp. 1875-1886, Nov. 2015.
16. Y. Bengio, P. Simard, and P. Frasconi, "Learning long-term dependencies with gradient descent is difficult, " IEEE Trans. Neural Netw., vol. 5, no. 2, pp. 157-166, Mar. 1994.
17. R. Pascanu, T. Mikolov, and Y. Bengio, "On the difficulty of training recurrent neural networks, " in Proc. ICML, vol. 28. 2013, pp. 1310-1318.
18. F. A. Gers, J. Schmidhuber, and F. Cummins, "Learning to forget: Continual prediction with LSTM, " Neural Comput., vol. 12, no. 10, pp. 2451-2471, 2000.
19. J. G. Zilly, R. K. Srivastava, and J. Koutnik, and J. Schmidhuber. (2016). "Recurrent highway networks." [Online]. Available: https://arxiv.org/abs/1607.03474.
20. R. J. Williams and D. Zipser, "Gradient-based learning algorithms for recurrent networks and their computational complexity, " in Backpropagation: Theory, Architectures, and Applications, vol. 1. Hillsdale, NJ, USA: Lawrence Erlbaum Associates, 1995, pp. 433-486.
21. R. J. Williams and D. Zipser, "A learning algorithm for continually running fully recurrent neural networks, " Neural Comput., vol. 1, no. 2, pp. 270-280, 1989.
22. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors, " Nature, vol. 323, pp. 533-536, Oct. 1986.
23. J. Martens and I. Sutskever, "Learning recurrent neural networks with hessian-free optimization, " in Proc. 28th Int. Conf. Mach. Learn. (ICML), 2011, pp. 1033-1040.
24. Y. Bengio, N. Boulanger-Lewandowski, and R. Pascanu, "Advances in optimizing recurrent networks, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), May 2013, pp. 8624-8628.
25. J. Koutník, K. Greff, F. Gomez, and J. Schmidhuber. (2014). "A clockwork RNN." [Online]. Available: https://arxiv.org/abs/1402.3511.