An empirical study of learning rates in deep neural networks for speech recognition

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

Volumn , Issue , 2013, Pages 6724-6728

  Senior, Andrew ^a   Heigold, Georg ^a   Ranzato, Marc'Aurelio ^a   Yang, Ke ^a  

^a GOOGLE INC   (United States)

Author keywords

AdaDec; AdaGrad; Deep neural networks; large vocabulary speech recognition; learning rate; Voice Search

Indexed keywords

ADADEC; ADAGRAD; DEEP NEURAL NETWORKS; LARGE VOCABULARY SPEECH RECOGNITION; LEARNING RATES; VOICE SEARCHES;

LEARNING ALGORITHMS; NEURAL NETWORKS; SCHEDULING; SIGNAL PROCESSING; VOCABULARY CONTROL;

SPEECH RECOGNITION;

EID: 84887388950     PISSN: 15206149     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICASSP.2013.6638963     Document Type: Conference Paper

References (23)

1. G.E. Dahl, D. Yu, L. Deng, and A. Acero, "Large vocabulary continuous speech recognition with context-dependent DBNHMMs," in ICASSP, 2011
2. G. Hinton, L. Deng, D. Yu, G. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kings-bury, "Deep neural networks for acoustic modeling in speech recognition," IEEE Signal Processing Magazine, vol. 29, no. 6, November 2012
3. J.S. Bridle, "Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters," in NIPS, 1990, vol. 2, pp. 211-217
4. T. LeCun, L. Bottou, G. Orr, and K. Muller, "Efficient backprop," in Neural Networks: Tricks of the trade, G. Orr and Muller K., Eds. Springer, 1998
5. B. Kingsbury, T.N. Sainath, and H. Soltau, "Scalable minimum bayes risk training of deep neural network acoustic models using distributed hessian-free optimization," in Interspeech, 2012
6. O. Vinyals and D. Povey, "Krylov subspace descent for deep learning," in NIPS Deep learning workshop, 2011
7. T. N. Sainath, B. Kingsbury, B. Ramabhadran, P. Fousek, P. Novak, and A. Mohamed, "Making deep belief networks effective for large vocabulary continuous speech recognition," in ASRU, December 2011
8. J. Dean, G. S. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M.'A. Ranzato, A. Senior, P. Tucker, K. Yang, and A.Y. Ng, "Large scale distributed deep networks," in NIPS, 2012
9. F. Seide, G. Li, and D. Yu, "Conversational speech transcription using context-dependent deep neural networks," in Interspeech, 2011
10. N. Jaitly, P. Nguyen, A. W. Senior, and V. Vanhoucke, "Application of pretrained deep neural networks to large vocabulary speech recognition," in Interspeech, 2012
11. T. Schaul, S. Zhang, and Y. LeCun, "No more pesky learning rates," in ArXiv, June 2012
12. H. Robbins and S. Monro, "A stochastic approximation method," Annals of Mathematical Statistics, no. 22, pp. 400-407, Sep. 1951
13. A. Mohamed, T. N. Sainath, G. Dahl, B. Ramabhadran, G. Hinton, and M. Picheny, "Deep belief networks using discriminative features for phone recognition," in ICASSP, May 2011
14. Leon Bottou, "Large-scale machine learning with stochastic gradient descent," in Proceedings of the 19th International Conference on Computational Statistics (COMPSTAT'2010), Yves Lechevallier and Gilbert Saporta, Eds., Paris, France, August 2010, pp. 177-187, Springer
15. W. Xu, "Towards optimal one pass large scale learning with averaged stochastic gradient descent," CoRR, vol. abs/1107.2490, 2011
16. S. Renals, N. Morgan, M. Cohen, and H. Franco, "Connectionist probability estimation in the decipher speech recognition system," in ICASSP, 1992, pp. 601-604
17. J. Duchi, E. Hazan, and Y. Singer, "Adaptive subgradient methods for online learning and stochastic optimization," Journal of Machine Learning Research, vol. 12, pp. 2121-2159, 2010
18. M. Riedmiller and H. Braun, "A direct adaptive method for faster backpropagation learning: The RPROP algorithm," in IEEE International Conference on Neural Networks. 1993, pp. 586-591, IEEE
19. C. Igel and M. Husken, "Improving the Rprop learning algorithm," 2000
20. M. Mahajan, A. Gunawardana, and A. Acero, "Training algorithms for hidden conditional random fields," in ICASSP, 2006, pp. 273-276
21. V. Mnih, "CUDAMat: A CUDA-based matrix class for Python," Tech. Rep. 2009-004, University of Toronto, Nov. 2009
22. V. Vanhoucke, A. Senior, and Mao M., "Improving the speed of neural networks on CPUs," in NIPS Workshop on deep learning, 2011
23. B.T. Polyak and A.B. Juditsky, "Acceleration of stochastic approximation by averaging," Automation and Remote Control, vol. 30, no. 4, pp. 838-855, 1992.