Gated feedback recurrent neural networks

32nd International Conference on Machine Learning, ICML 2015

Volumn 3, Issue , 2015, Pages 2067-2075

  Chung, Junyoung ^a   Gulcehre, Caglar ^a   Cho, Kyunghyun ^a   Bengio, Yoshua ^b  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

^b CIFAR ^*  (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; COMPUTER SOFTWARE; LEARNING SYSTEMS; MODELING LANGUAGES; RECONFIGURABLE HARDWARE;

CONVENTIONAL APPROACH; DIFFERENT LAYERS; EMPIRICAL EVALUATIONS; LAYER INTERACTION; LONG SHORT TERM MEMORY; PROGRAM EVALUATION; RECURRENT LAYERS; RECURRENT NEURAL NETWORK (RNN);

RECURRENT NEURAL NETWORKS;

EID: 84969972350     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (25)

1. Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. Neural machine translation by jointly learning to align and translate. Technical report, arXiv preprint arXiv: 1409.0473, 2014.
2. Bastien, Frédéric, Lamblin, Pascal, Pascanu, Razvan, Bergstra, James, Goodfellow, Ian J., Bergeron, Arnaud, Bouchard, Nicolas, and Bengio, Yoshua. Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop, 2012.
3. Bengio, Yoshua, Simard, Patrice, and Frasconi, Paolo. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5 (2): 157-166, 1994.
4. Bengio, Yoshua, Ducharme, Rejean, and Vincent, Pascal. A neural probabilistic language model. In Adv. Neural Inf. Proc. Sys. 13, pp. 932-938, 2001.
5. Cho, Kyunghyun, Van Merriënboer, Bart, Gulcehre, Caglar, Bougares, Fethi, Schwenk, Holger, and Bengio, Yoshua. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.
6. El Hihi, Salah and Bengio, Yoshua. Hierarchical recurrent neural networks for long-term dependencies. In Advances in Neural Information Processing Systems, pp. 493-499. Citeseer, 1995.
7. Gers, Felix A., Schmidhuber, Jürgen, and Cummins, Fred A. Learning to forget: Continual prediction with LSTM. Neural Computation, 12(10):2451-2471, 2000.
8. Goodfellow, Ian J., Warde-Farley, David, Lamblin, Pascal, Dumoulin, Vincent, Mirza, Mehdi, Pascanu, Razvan, Bergstra, James, Bastien, Frédéric, and Bengio, Yoshua. Pylearn2: a machine learning research library. arXiv preprint arXiv: 1308.4214, 2013.
9. Graves, Alex. Generating sequences with recurrent neural networks. arXiv preprint arXiv: 1308.0850, 2013.
10. Hermans, Michiel and Schrauwen, Benjamin. Training and analysing deep recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 190-198, 2013.
11. Hinton, Geoffrey. Neural networks for machine learning. Coursera, video lectures, 2012.
12. Hochreiter, Sepp. Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, Institut fur Informatik, Lehrstuhl Prof. Brauer, Technische Universität München, 1991. URL http://www7. informatik.tu-muenchen.de/-Ehochreit.
13. Hochreiter, Sepp. The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 6(02):107-116, 1998.
14. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
15. Hutter, Marcus. The human knowledge compression contest. 2012. URL http://prize.hutterl.net/.
16. Kingma, Diederik and Ba, Jimmy. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
17. Koutník, Jan, Greff, Klaus, Gomez, Faustino, and Schmidhuber, Jürgen. A clockwork rnn. In Proceedings of the 31st International Conference on Machine Learning (ICML'14), 2014.
18. Mikolov, Tomas. Statistical Language Models based on Neural Networks. PhD thesis, Brno University of Technology, 2012.
19. Mikolov, Tomas, Sutskever, Ilya, Deoras, Anoop, Le, Hai-Son, Kombrink, Stefan, and Cernocky, J. Subword language modeling with neural networks. Preprint, 2012.
20. Schmidhuber, Jürgen. Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2):234-242, 1992.
21. Stollenga, Marijn F, Masci, Jonathan, Gomez, Faustino, and Schmidhuber, Jürgen. Deep networks with internal selective attention through feedback connections. In Advances in Neural Information Processing Systems, pp. 3545-3553, 2014.
22. Sutskever, Ilya, Martens, James, and Hinton, Geoffrey E. Generating text with recurrent neural networks. In Proceedings of the 28th International Conference on Machine Learning (ICML'11), pp. 1017-1024, 2011.
23. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc VV. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pp. 3104-3112, 2014.
24. Zaremba, Wojciech and Sutskever, Ilya. Learning to execute. arXiv preprint arXiv:1410.4615, 2014.
25. Zaremba, Wojciech, Sutskever, Ilya, and Vinyals, Oriol. Recurrent neural network regularization. arXiv preprint arXiv: 1409.2329, 2014.