Advances in optimizing recurrent networks

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

Volumn , Issue , 2013, Pages 8624-8628

  Bengio, Yoshua ^a   Boulanger Lewandowski, Nicolas ^a   Pascanu, Razvan ^a  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

Author keywords

deep learning; long term dependencies; Recurrent networks; representation learning

Indexed keywords

CREDIT ASSIGNMENT; DEEP LEARNING; LONG-TERM DEPENDENCIES; PROBABILITY MODELS; RECURRENT NETWORKS; REPRESENTATION LEARNING; RESEARCH ACTIVITIES; TECHNICAL SOLUTIONS;

EXPERIMENTS; RECURRENT NEURAL NETWORKS; SIGNAL PROCESSING;

OPTIMIZATION;

EID: 84890543516     PISSN: 15206149     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICASSP.2013.6639349     Document Type: Conference Paper

References (32)

1. S. Hochreiter, " Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, T.U. Munich," 1991.
2. Y. Bengio, P. Simard, and P. Frasconi, "Learning long-term dependencies with gradient descent is difficult," IEEE T. Neural Nets, 1994.
3. D.E. Rumelhart, G.E. Hinton, and R.J. Williams, "Learning representations by back-propagating errors," Nature, vol. 323, pp. 533-536, 1986.
4. J. Martens and I. Sutskever, "Learning recurrent neural networks with Hessian-free optimization," in ICML2011, 2011.
5. T. Mikolov, S. Kombrink, L. Burget, J. Cernocky, and S. Khudanpur, "Extensions of recurrent neural network language model," in ICASSP 2011, 2011.
6. I. Sutskever, Training Recurrent Neural Networks, Ph.D. thesis, CS Dept., U. Toronto, 2012.
7. Yoshua Bengio, Learning deep architectures for AI, Now Publishers, 2009.
8. Y. Bengio, A. Courville, and P. Vincent, "Unsupervised feature learning and deep learning: A review and new perspectives," Tech. Rep., arXiv:1206.5538, 2012.
9. G. E. Hinton, S. Osindero, and Y.-W. Teh, "A fast learning algorithm for deep belief nets," Neural Computation, vol. 18, pp. 1527-1554, 2006.
10. Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle, "Greedy layer-wise training of deep networks," in NIPS2006, 2007.
11. M. Ranzato, C. Poultney, S. Chopra, and Y. LeCun, "Efficient learning of sparse representations with an energy-based model," in NIPS2006, 2007.
12. D. Erhan, Y. Bengio, A. Courville, P. Manzagol, P. Vincent, and S. Bengio, "Why does unsupervised pre-training help deep learning?," J. Machine Learning Res., (11) 2010.
13. J. Martens, "Deep learning via Hessian-free optimization," in ICML2010, 2010, pp. 735-742.
14. Q. Le, M. Ranzato, R. Monga, M. Devin, G. Corrado, K. Chen, J. Dean, and A. Ng, "Building high-level features using large scale unsupervised learning," in ICML2012, 2012.
15. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio, "Understanding the exploding gradient problem," Tech. Rep., Universit e De Montreal, 2012, arXiv:arXiv:1211.5063.
16. Tomas Mikolov, Statistical Language Models based on Neural Networks, Ph.D. thesis, Brno University of Technology, 2012.
17. T. Lin, B. G. Horne, P. Tino, and C. L. Giles, "Learning longterm dependencies is not as difficult with NARX recurrent neural networks," Tech. Rep. UMICAS-TR-95-78, U. Mariland, 1995.
18. S. ElHihi and Y. Bengio, "Hierarchical recurrent neural networks for long-term dependencies," in NIPS1995, 1996.
19. Herbert Jaeger, Mantas Lukosevicius, Dan Popovici, and Udo Siewert, "Optimization and applications of echo state networks with leaky-integrator neurons," Neural Networks, vol. 20, no. 3, pp. 335-352, 2007.
20. I. Sutskever and G. Hinton, "Temporal kernel recurrent neural networks," Neural Networks, vol. 23, no. 2, (23) 2, 2010.
21. S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Computation, vol. 9, no. 8, pp. 1735-1780, 1997.
22. Udo Siewert and Welf Wustlich, "Echo-state networks with band-pass neurons: Towards generic time-scale-independent reservoir structures," ¡p¿Preliminary Report¡/p¿, October 2007.
23. Tomas Mikolov and Geoffrey Zweig, "Context dependent reucrrent neural network language model," Workshop on Spoken Language Technology, 2012.
24. I. Sutskever, G. Hinton, and G. Taylor, "The recurrent temporal restricted Boltzmann machine," in NIPS2008. 2009.
25. N. Boulanger-Lewandowski, Y. Bengio, and P. Vincent, "Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription," in ICML2012, 2012.
26. H. Larochelle and I. Murray, "The Neural Autoregressive Distribution Estimator," in AISTATS2011, 2011.
27. X. Glorot, A. Bordes, and Y. Bengio, "Deep sparse rectifier neural networks," in AISTATS2011, 2011.
28. V. Nair and G.E. Hinton, "Rectified linear units improve restricted Boltzmann machines," in ICML2010, 2010.
29. A. Krizhevsky, I. Sutskever, and G. Hinton, "ImageNet classification with deep convolutional neural networks," in NIPS2012. 2012.
30. Yu Nesterov, "A method for unconstrained convex minimization problem with the rate of convergence o(1=k2)," Doklady AN SSSR (translated as Soviet. Math. Docl.), vol. 269, pp. 543-547, 1983.
31. James Bergstra and Yoshua Bengio, "Random search for hyper-parameter optimization," J. Machine Learning Res., vol. 13, pp. 281-305, 2012.
32. Tomas Mikolov, Stefan Kombrink, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur, "Extensions of recurrent neural network language model," in Proc. 2011 IEEE international conference on acoustics, speech and signal processing (ICASSP 2011), 2011.