Batch normalized recurrent neural networks

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

Volumn 2016-May, Issue , 2016, Pages 2657-2661

  Laurent, Cesar ^a   Pereyra, Gabriel ^b   Brakel, Philemon ^a   Zhang, Ying ^a   Bengio, Yoshua ^a,c  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

^b UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^c CIFAR Fellow ^*  (United States)

Author keywords

batch normalization; LSTM; optimization; RNN

Indexed keywords

EID: 84973326024     PISSN: 15206149     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICASSP.2016.7472159     Document Type: Conference Paper

References (23)

1. Sergey Ioffe and Christian Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift, " arXiv preprint arXiv: 1502. 03167, 2015.
2. Alan Graves, Abdel-rahman Mohamed, and Geoffrey Hinton, "Speech recognition with deep recurrent neural networks, " in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013, pp. 6645-6649.
3. Ilya Sutskever, Oriol Vinyals, and Quoc Le, "Sequence to sequence learning with neural networks, " in Advances in Neural Information Processing Systems, 2014, pp. 3104-3112.
4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio, "Neural machine translation by jointly learning to align and translate, " arXiv preprint arXiv: 1409. 0473, 2014.
5. Tomás Mikolov, "Statistical language models based on neural networks, " Presentation at Google, Mountain View, 2nd April, 2012.
6. Sepp Hochreiter and Jürgen Schmidhuber, "Long shortterm memory, " Neural computation, vol. 9, no. 8, pp. 1735-1780, 1997.
7. WillWilliams, Niranjani Prasad, David Mrva, Tom Ash, and Tony Robinson, "Scaling recurrent neural network language models, " arXiv preprint arXiv: 1502. 00512, 2015.
8. Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al., "Deepspeech: Scaling up end-to-end speech recognition, " arXiv preprint arXiv: 1412. 5567, 2014.
9. Yann A LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller, "Efficient backprop, " in Neural networks: Tricks of the trade, pp. 9-48. Springer, 2012.
10. Guillaume Desjardins, Karen Simonyan, Razvan Pascanu, and Koray Kavukcuoglu, "Natural neural networks, " arXiv preprint arXiv: 1507. 00210, 2015.
11. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei, "ImageNet Large Scale Visual Recognition Challenge, " International Journal of Computer Vision (IJCV), pp. 1-42, April 2015.
12. Mike Schuster and Kuldip K Paliwal, "Bidirectional recurrent neural networks, " Signal Processing, IEEE Transactions on, vol. 45, no. 11, pp. 2673-2681, 1997.
13. Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio, "How to construct deep recurrent neural networks, " arXiv preprint arXiv: 1312. 6026, 2013.
14. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio, "On the difficulty of training recurrent neural networks, " arXiv preprint arXiv: 1211. 5063, 2012.
15. Felix A Gers, Nicol N Schraudolph, and Jürgen Schmidhuber, "Learning precise timing with lstm recurrent networks, " The Journal of Machine Learning Research, vol. 3, pp. 115-143, 2003.
16. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov, "Dropout: A simple way to prevent neural networks from overfitting, " The Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929-1958, 2014.
17. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals, "Recurrent neural network regularization, " arXiv preprint arXiv: 1409. 2329, 2014.
18. Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian J. Goodfellow, Arnaud Bergeron, Nicolas Bouchard, and Yoshua Bengio, "Theano: new features and speed improvements, " Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop, 2012.
19. B. van Merriënboer, D. Bahdanau, V. Dumoulin, D. Serdyuk, D. Warde-Farley, J. Chorowski, and Y. Bengio, "Blocks and Fuel: Frameworks for deep learning, " ArXiv e-prints, June 2015.
20. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini, "Building a large annotated corpus of english: The penn treebank, " Computational linguistics, vol. 19, no. 2, pp. 313-330, 1993.
21. Douglas B Paul and Janet M Baker, "The design for the wall street journal-based csr corpus, " in Proceedings of the workshop on Speech and Natural Language. Association for Computational Linguistics, 1992, pp. 357-362.
22. Alan Graves, Navdeep Jaitly, and Abdel-rahman Mohamed, "Hybrid speech recognition with deep bidirectional lstm, " in Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on. IEEE, 2013, pp. 273-278.
23. Xavier Glorot and Yoshua Bengio, "Understanding the difficulty of training deep feedforward neural networks, " in International conference on artificial intelligence and statistics, 2010, pp. 249-256.