Investigations on hessian-free optimization for cross-entropy training of deep neural networks

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Volumn , Issue , 2013, Pages 3317-3321

  Wiesler, Simon ^a   Li, Jinyu ^b   Xue, Jian ^b  

^a RWTH AACHEN UNIVERSITY   (Germany)

^b MICROSOFT   (United States)

Author keywords

Deep learning; Hessian free; Optimization

Indexed keywords

ALGORITHMS; OPTIMIZATION;

ASYMPTOTIC CONVERGENCE; DEEP LEARNING; DEEP NEURAL NETWORKS; GAUSSIAN MIXTURE MODEL; HESSIAN-FREE; OPTIMIZATION ALGORITHMS; RECOGNITION ACCURACY; STOCHASTIC GRADIENT DESCENT;

TIME VARYING NETWORKS;

EID: 84906237512     PISSN: 2308457X     EISSN: 19909772     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (14)

1. G. E. Dahl, D. Yu, L. Deng, and A. Acero, "Contextdependent pre-trained deep neural networks for largevocabulary speech recognition, " Audio, Speech, and Language Processing, IEEE Transactions on, vol. 20, no. 1, pp. 30-42, 2012.
2. F. Seide, G. Li, and D. Yu, "Conversational speech transcription using context-dependent deep neural networks, " in Proc. Inter Speech, 2011, pp. 437-440.
3. N. Jaitly, P. Nguyen, A. Senior, and V. Vanhoucke, "Application of pretrained deep neural networks to large vocabulary speech recognition, " in Proceedings of Inter Speech 2012, 2012.
4. T. N. Sainath, B. Kingsbury, B. Ramabhadran, P. Fousek, P. Novak, and A.-R. Mohamed, "Making deep belief networks effective for large vocabulary continuous speech recognition, " in Automatic Speech Recognition and Understanding (ASRU), 2011 IEEE Workshop on. IEEE, 2011, pp. 30-35.
5. D. Liu and J. Nocedal, "On the Limited Memory BFGS Method for Large-Scale Optimization, " Math. Program., vol. 45, no. 1, pp. 503-528, 1989.
6. J. Dean, G. S. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Y. Ng, "Large scale distributed deep networks, " in Advances in Neural Information Processing Systems, 2012.
7. J. Martens, "Deep learning via hessian-free optimization, " in Proceedings of the 27th International Conference on Machine Learning (ICML), vol. 951, 2010, p. 2010.
8. B. Kingsbury, "Scalable minimum bayes risk training of deep neural network acoustic models using distributed hessian-free optimization, " in Inter Speech, 2012.
9. J. Nocedal and S. Wright, Numerical Optimization. Springer, 1999.
10. M. R. Hestenes and E. Stiefel, "Methods of conjugate gradients for solving linear systems, " Journal of Research of the National Bureau of Standards, vol. 49, no. 6, pp. 409-436, 1952.
11. B. Pearlmutter, "Fast exact multiplication by the hessian, " Neural Computation, vol. 6, no. 1, pp. 147-160, 1994.
12. N. Schraudolph, "Fast curvature matrix-vector products for second-order gradient descent, " Neural computation, vol. 14, no. 7, pp. 1723-1738, 2002.
13. J. J. Moré and D. C. Sorensen, "Computing a trust region step, " SIAM Journal on Scientific and Statistical Computing, vol. 4, no. 3, pp. 553-572, 1983.
14. T. Steihaug, "The conjugate gradient method and trust regions in large scale optimization, " SIAM Journal on Numerical Analysis, vol. 20, no. 3, pp. 626-637, 1983.