A nonmonotone learning rate strategy for SGD training of deep neural networks

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

Volumn 2015-August, Issue , 2015, Pages 4974-4978

  Keskar, Nitish Shirish ^a   Saon, George ^b  

^a NORTHWESTERN UNIVERSITY   (United States)

^b IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

deep learning; learning rate; nonmonotonicity; speech recognition; stepsize

Indexed keywords

AUDIO SIGNAL PROCESSING; DEEP LEARNING; LEARNING ALGORITHMS; NONLINEAR PROGRAMMING; SITE SELECTION; SPEECH COMMUNICATION; SPEECH RECOGNITION; STOCHASTIC MODELS; STOCHASTIC SYSTEMS;

BROADCAST NEWS TRANSCRIPTIONS; LEARNING RATES; NON-LINEAR OPTIMIZATION; NON-MONOTONICITY; NONMONOTONE LINESEARCH; STEP SIZE; STOCHASTIC GRADIENT DESCENT; TELEPHONE CONVERSATION;

DEEP NEURAL NETWORKS;

EID: 84946057367     PISSN: 15206149     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICASSP.2015.7178917     Document Type: Conference Paper

References (22)

1. G. Hinton, L. Deng, D. Yu, G. Dahl, A. Mohamed, N. laitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, et aI., "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups," Signal Processing Magazine, IEEE, vol. 29, no. 6, pp. 82-97,2012.
2. J. Nocedal and S. Wright, "Numerical optimization, second edition," Numerical optimization, pp. 497-528, 2006.
3. A. Senior, G. Heigold, M. Ranzato, and K. Yang, "An empirical study of learning rates in deep neural networks for speech recognition," in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013,pp. 6724-6728.
4. M. Zeiler, "Adadelta: An adaptive learning rate method," arXiv preprint arXiv: 1212.5701,2012.
5. S. Renals and N. Morgan and H. Bourlard and M. Cohen and H. Franco and C. Wooters and P. Kohn, "Connectionist speech recognition: status and prospects," Tech. Rep. TR-OI-070, lCSI, 1991.
6. S. Wiesler, A. Richard, R. Schluter, and H. Ney, "Meannormalized stochastic gradient for large-scale deep learning," in Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on. IEEE, 2014, pp. 180-184.
7. J. Duchi, E. Hazan, and Y. Singer, "Adaptive subgradient methods for online learning and stochastic optimization," The Journal of Machine Learning Research, vol. 12, pp. 2121-2159, 2011.
8. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior, P. Tucker, K. Yang, Q. Le, et al., "Large scale distributed deep networks," in Advances in Neural Information Processing Systems, 2012, pp. 1223-1231.
9. "Quicknet," http://www1.icsi.berkeley.edu/Speech/qn.html. Accessed: 2014-09-30.
10. Y. LeCun, L. Bottou, G. Orr, and K. Muller, "Efficient backprop," in Neural networks: Tricks of the trade, pp. 9-48. Springer, 2012.
11. L. Grippo, F. Lampariello, and S. Lucidi, "A nonmonotone line search technique for Newton's method," SIAM Journal on Numerical Analysis, vol. 23, no. 4, pp. 707-716,1986.
12. H. Zhang and W. Hager, "A nonmonotone line search technique and its application to unconstrained optimization," SIAM Journal on Optimization, vol. 14, no. 4, pp. 1043-1056,2004.
13. Y. Dai, W. Hager, K. Schittkowski, and H. Zhang, "The cyclic barzilai-borwein method for unconstrained optimization," IMA Journal of Numerical Analysis, vol. 26, no. 3, pp. 604-627,2006.
14. M. Solodov, "Incremental gradient algorithms with stepsizes bounded away from zero," Computational Optimization and Applications, vol. 11, no. 1, pp. 23-35,1998.
15. P. Tseng, "An incremental gradient (-projection) method with momentum term and adaptive stepsize rule," SIAM Journal on Optimization, vol. 8, no. 2, pp. 506-531, 1998.
16. M. Schmidt and N. LeRoux, "Fast convergence of stochastic gradient descent under a strong growth condition," arXiv preprint arXiv: 1308.6370,2013.
17. A Shapiro and Y Wardi, "Convergence analysis of gradient descent stochastic algorithms," Journal of optimization theory and applications, vol. 91, no. 2, pp. 439-454,1996.
18. Saeed Ghadimi and Guanghui Lan, "Stochastic first-and zeroth-order methods for nonconvex stochastic programming," SIAM Journal on Optimization, vol. 23, no. 4, pp. 2341-2368, 2013.
19. Xiao Wang, Shiqian Ma, and Wei Liu, "Stochastic quasiNewton methods for nonconvex stochastic optimization," 2014.
20. G. Saon, H. Soltau, D. Nahamoo, and M. Picheny, "Speaker adaptation of neural network acoustic models using i-vectors," in Proceedings of ASRU, 2013.
21. F. Seide, G. Li, X. Chien, and D. Yu, "Feature engineering in context-dependent deep neural networks for conversational speech transcription," in Proceedings of ASRU, 2011.
22. G. Cawley and N. Talbot, "On over-fitting in model selection and subsequent selection bias in performance evaluation," The Journal of Machine Learning Research, vol. 11, pp. 2079-2107,2010.