Hessian-free optimization for learning deep multidimensional recurrent neural networks

Advances in Neural Information Processing Systems

Volumn 2015-January, Issue , 2015, Pages 883-891

  Cho, Minhyung ^a   Dhir, Chandra Shekhar ^a   Lee, Jaehyung ^a  

^a Applied Research Korea   (South Korea)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; APPROXIMATION ALGORITHMS; CHARACTER RECOGNITION; FISHER INFORMATION MATRIX; INFORMATION SCIENCE; SPEECH RECOGNITION;

CONVEX APPROXIMATION; EM ALGORITHMS; HANDWRITING RECOGNITION; NONCONVEXITY; SEQUENCE LABELING; TEMPORAL CLASSIFICATION;

RECURRENT NEURAL NETWORKS;

EID: 84965175669     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (25)

1. Alex Graves. Supervised sequence labelling with recurrent neural networks, volume 385. Springer, 2012.
2. Alex Graves, Marcus Liwicki, Horst Bunke, Jürgen Schmidhuber, and Santiago Fernández. Unconstrained on-line handwriting recognition with recurrent neural networks. In Advances in Neural Information Processing Systems, pages 577-584, 2008.
3. Alex Graves and Jürgen Schmidhuber. Offline handwriting recognition with multidimensional recurrent neural networks. In Advances in Neural Information Processing Systems, pages 545-552, 2009.
4. Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In Proceedings of ICASSP, pages 6645-6649. IEEE, 2013.
5. Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. CoRR, abs/1412.6550, 2014. URL http://arxiv.org/abs/1412.6550.
6. Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504-507, 2006.
7. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1-9, 2015.
8. James Martens. Deep learning via Hessian-free optimization. In Proceedings of the 27th International Conference on Machine Learning, pages 735-742, 2010.
9. James Martens and Ilya Sutskever. Learning recurrent neural networks with Hessian-free optimization. In Proceedings of the 28th International Conference on Machine Learning, pages 1033-1040, 2011.
10. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-1780, 1997.
11. Nicol N Schraudolph. Fast curvature matrix-vector products for second-order gradient descent. Neural Computation, 14(7):1723-1738, 2002.
12. Barak A Pearlmutter. Fast exact multiplication by the hessian. Neural Computation, 6(1):147-160, 1994.
13. James Martens and Ilya Sutskever. Training deep and recurrent networks with Hessian-free optimization. In Neural Networks: Tricks of the Trade, pages 479-535. Springer, 2012.
14. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd International Conference on Machine Learning, pages 369-376, 2006.
15. Stephen Boyd and Lieven Vandenberghe, editors. Convex Optimization. Cambridge University Press, 2004.
16. Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation, 10(2):251-276, 1998.
17. Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks. In International Conference on Learning Representations, 2014.
18. Hyeyoung Park, S-I Amari, and Kenji Fukumizu. Adaptive natural gradient learning algorithms for various stochastic models. Neural Networks, 13(7):755-764, 2000.
19. Christopher M. Bishop, editor. Pattern Recognition and Machine Learning. Springer, 2007.
20. Mario Pechwitz, S. Snoussi Maddouri, Volker Märgner, Noureddine Ellouze, and Hamid Amiri. IFN/ENIT-database of handwritten arabic words. In Proceedings of CIFED, pages 129-136, 2002.
21. DARPA-ISTO. The DARPA TIMIT acoustic-phonetic continuous speech corpus (TIMIT). In speech disc cd1-1.1 edition, 1990.
22. Alex Graves. Sequence transduction with recurrent neural networks. In ICML Representation Learning Workshop, 2012.
23. Kai-Fu Lee and Hsiao-Wuen Hon. Speaker-independent phone recognition using hidden markov models. IEEE Transactions on Acoustics, Speech and Signal Processing, 37(11):1641-1648, 1989.
24. Alex Graves. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, pages 2348-2356, 2011.
25. Alex Graves. Rnnlib: A recurrent neural network library for sequence learning problems, 2008.