A primal-dual method for training recurrent neural networks constrained by the echo-state property

2nd International Conference on Learning Representations, ICLR 2014 - Workshop Track Proceedings

Volumn , Issue , 2014, Pages

  Chen, Jianshu ^a   Deng, Li ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b MICROSOFT RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CONSTRAINT THEORY; DEEP NEURAL NETWORKS;

FEATURE EXTRACTOR; INEQUALITY CONSTRAINT; OPTIMIZATION PROBLEMS; PRIMAL-DUAL METHODS; RECOGNITION ERROR; RECURRENT NEURAL NETWORK (RNN); TEMPORAL PREDICTION; THRESHOLD PARAMETERS;

LONG SHORT-TERM MEMORY;

EID: 85083950550     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (36)

1. Beck, A. and Teboulle, M. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences, 2(1):183–202, 2009.
2. Bengio, Y., Boulanger-Lewandowski, N., and Pascanu, R. Advances in optimizing recurrent networks. In Proc. ICASSP, Vancouver, Canada, May 2013.
3. Boyd, S. P. and Vandenberghe, L. Convex Optimization. Cambridge university press, 2004.
4. Dahl, G., Yu, D., Deng, L., and Acero, A. Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. IEEE Trans. on Audio, Speech and Language Processing, 20(1):30–42, jan 2012.
5. Dahl, G. E., Yu, D., Deng, L., and Acero, A. Large vocabulary continuous speech recognition with context-dependent DBN-HMMs. In Proc. IEEE ICASSP, pp. 4688–4691, Prague, Czech, May 2011.
6. Dahl, G. E., Sainath, T. N., and Hinton, G. E. Improving deep neural networks for lvcsr using rectified linear units and dropout. In Proc. ICASSP, pp. 8609–8613, Vancouver, Canada, May 2013. IEEE.
7. Deng, L., Hassanein, K., and Elmasry, M. Analysis of the correlation structure for a neural predictive model with application to speech recognition. Neural Networks, 7(2):331–339, 1994.
8. Deng, L., Abdel-Hamid, O., and Yu, D. A deep convolutional neural network using heterogeneous pooling for trading acoustic invariance with phonetic confusion. In Proc. IEEE ICASSP, Vancouver, Canada, May 2013a.
9. Deng, L., Hinton, G., and Kingsbury, B. New types of deep neural network learning for speech recognition and related applications: An overview. In Proc. IEEE ICASSP, Vancouver, Canada, May 2013b.
10. Deng, L., Li, J., Huang, J.-T., Yao, K., Yu, D., Seide, F., Seltzer, M., Zweig, G., He, X., Williams, J., Gong, Y., and Acero, A. Recent advances in deep learning for speech research at microsoft. In Proc. ICASSP, Vancouver, Canada, 2013c.
11. Graves, A. Sequence transduction with recurrent neural networks. In Representation Learning Workshp, ICML, 2012.
12. Graves, A., Fernández, S., Gomez, F., and Schmidhuber, J. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proc. ICML, pp. 369–376, Pittsburgh, PA, June 2006. ACM.
13. Graves, A., Mohamed, A., and Hinton, G. Speech recognition with deep recurrent neural networks. In Proc. ICASSP, Vancouver, Canada, May 2013.
14. Hinton, G., Deng, L., Yu, D., Dahl, G. E., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T. N., and Kingsbury, B. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 29(6):82–97, November 2012.
15. Jaeger, H. The “echo state” approach to analysing and training recurrent neural networks. GMD Report 148, GMD - German National Research Institute for Computer Science, 2001a.
16. Jaeger, H. Short term memory in echo state networks. GMD Report 152, GMD - German National Research Institute for Computer Science, 2001b.
17. Jaeger, H. Tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the “echo state network” approach. GMD Report 159, GMD - German National Research Institute for Computer Science, 2002.
18. Kingsbury, B., Sainath, T. N., and Soltau, H. Scalable minimum bayes risk training of deep neural network acoustic models using distributed hessian-free optimization. In Proc. INTERSPEECH, Portland, OR, September 2012.
19. LeCun, Yann. Learning invariant feature hierarchies. In Proc. ECCV, pp. 496–505, Firenze, Italy, October 2012. Springer.
20. Lee, K.-F. and Hon, H.-W. Speaker-independent phone recognition using hidden Markov models. IEEE Transactions on Acoustics, Speech and Signal Processing,, 37(11):1641–1648, November 1989.
21. Maas, A. L., Le, Q., O’Neil, T. M., Vinyals, O., Nguyen, P., and Ng, A. Y. Recurrent Neural Networks for Noise Reduction in Robust ASR. In Proc. INTERSPEECH, Portland, OR, September 2012.
22. Manjunath, G. and Jaeger, H. Echo state property linked to an input: Exploring a fundamental characteristic of recurrent neural networks. Neural computation, 25(3):671–696, 2013.
23. Martens, J. and Sutskever, I. Learning recurrent neural networks with hessian-free optimization. In Proc. ICML, pp. 1033–1040, Bellevue, WA, June 2011.
24. Mikolov, T. Statistical Language Models Based on Neural Networks. PhD thesis, Ph. D. thesis, Brno University of Technology, 2012.
25. Mikolov, T., Deoras, A., Povey, D., Burget, L., and Cernocky, J. Strategies for training large scale neural network language models. In Proc. IEEE ASRU, pp. 196–201, Honolulu, HI, December 2011. IEEE.
26. Pascanu, R., Mikolov, T., and Bengio, Y. On the difficulty of training recurrent neural networks. In Proc. ICML, Atlanta, GA, June 2013.
27. Polyak, B. Introduction to Optimization. Optimization Software, NY, 1987.
28. Robinson, A. J. An application of recurrent nets to phone probability estimation. IEEE Transactions on Neural Networks, 5(2):298–305, August 1994.
29. Sainath, T.N., Kingsbury, B., Soltau, H., and Ramabhadran, B. Optimization techniques to improve training speed of deep neural networks for large speech tasks. IEEE Transactions on Audio, Speech, and Language Processing, 21(11):2267–2276, November 2013.
30. Seide, F., Li, G., and Yu, D. Conversational speech transcription using context-dependent deep neural networks. In Proc. INTERSPEECH, pp. 437–440, Florence, Italy, August 2011.
31. Sutskever, I. Training Recurrent Neural Networks. PhD thesis, Ph. D. thesis, University of Toronto, 2013.
32. Sutskever, I., Martens, J., Dahl, G., and Hinton, G. E. On the importance of initialization and momentum in deep learning. In Proc. ICML, Atlanta, GA, June 2013.
33. Tüske, Z., Sundermeyer, M., Schlüter, R., and Ney, H. Context-Dependent MLPs for LVCSR: TANDEM, Hybrid or Both? In Proc. Interspeech, Portland, OR, September 2012.
34. Vinyals, Oriol, Ravuri, Suman V, and Povey, Daniel. Revisiting recurrent neural networks for robust ASR. In Proc. ICASSP, pp. 4085–4088, Kyoto, Japan, March 2012. IEEE.
35. Yu, D., Deng, L., and Dahl, G. Roles of pre-training and fine-tuning in context-dependent DBN-HMMs for real-world speech recognition. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2010.
36. Yu, D., Deng, L., and Seide, F. The deep tensor neural network with applications to large vocabulary speech recognition. IEEE Trans. on Audio, Speech and Language Processing, 21(2):388 –396, 2013.