A study of the recurrent neural network encoder-decoder for large vocabulary speech recognition

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Volumn 2015-January, Issue , 2015, Pages 3249-3253

  Lu, Liang ^a   Zhang, Xingxing ^a   Cho, Kyunghyun ^b   Renals, Steve ^a  

^a UNIVERSITY OF EDINBURGH   (United Kingdom)

^b UNIVERSITÉ DE MONTRÉAL   (Canada)

Author keywords

Deep neural networks; Encoder decoder; End to end speech recognition; Recurrent neural networks

Indexed keywords

DECODING; HIDDEN MARKOV MODELS; MARKOV PROCESSES; RECURRENT NEURAL NETWORKS; SPEECH; SPEECH COMMUNICATION; VOCABULARY CONTROL;

AUTOMATIC SPEECH RECOGNITION; DEEP NEURAL NETWORKS; ENCODER-DECODER; END TO END; FEATURE REPRESENTATION; HIDDEN MARKOV MODELS (HMMS); LARGE VOCABULARY SPEECH RECOGNITION; RECURRENT NEURAL NETWORK (RNNS);

SPEECH RECOGNITION;

EID: 84959173420     PISSN: 2308457X     EISSN: 19909772     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (41)

1. G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury, "Deep neural networks for acoustic modeling in speechrecognition: The shared views of four research groups, " SignalProcessing Magazine, IEEE, vol. 29, no. 6, pp. 82-97, 2012.
2. H. A. Bourlard and N. Morgan, Connectionist speech recognition: A hybrid approach. Springer, 1994, vol. 247.
3. N. Morgan and H. A. Bourlard, "Neural networks for statisticalrecognition of continuous speech, " Proceedings of the IEEE, vol. 83, no. 5, pp. 742-772, 1995.
4. S. Renals, N. Morgan, H. Bourlard, M. Cohen, and H. Franco, "Connectionist probability estimators in HMM speech recognition, "IEEE Transactions on Speech and Audio Processing, vol. 2, no. 1, pp. 161-174, 1994.
5. A.-r. Mohamed, G. E. Dahl, and G. Hinton, "Acoustic modelingusing deep belief networks, " Audio, Speech, and Language Processing, IEEE Transactions on, vol. 20, no. 1, pp. 14-22, 2012.
6. F. Seide, G. Li, and D. Yu, "Conversational speech transcriptionusing context-dependent deep neural networks. " in Interspeech, 2011, pp. 437-440.
7. T. Robinson and F. Fallside, "A recurrent error propagation networkspeech recognition system, " Computer Speech and Language, vol. 5, pp. 259-274, 1991.
8. T. Robinson, M. Hochberg, and S. Renals, "The use of recurrentnetworks in continuous speech recognition, " in Automatic Speechand Speaker Recognition-Advanced Topics, C. Lee, K. Paliwal, and F. Soong, Eds. Kluwer Academic Publishers, 1996, pp. 233-258.
9. A. Robinson, G. Cook, D. Ellis, E. Fosler-Lussier, S. Renals, and D. Williams, "Connectionist speech recognition of broadcastnews, " Speech Communication, vol. 37, pp. 27-45, 2002.
10. M. Schuster and K. K. Paliwal, "Bidirectional recurrent neuralnetworks, " Signal Processing, IEEE Transactions on, vol. 45, no. 11, pp. 2673-2681, 1997.
11. H. Sak, O. Vinyals, G. Heigold, A. Senior, E. McDermott, R. Monga, and M. Mao, "Sequence discriminative distributedtraining of long short-term memory recurrent neural networks, "in Proc. INTERSPEECH, 2014.
12. J. Baker, "The DRAGON system-an overview, " IEEE Transactionson Acoustics, Speech, and Signal Processing, vol. 23, pp. 24-29, 1975.
13. L. Bahl, F. Jelinek, and R. Mercer, "A maximum likelihood approachto speech recognition, " IEEE Transactions on PatternAnalysis and Machine Intelligence, vol. 5, pp. 179-190, 1983.
14. L. Rabiner, "A tutorial on hidden markov models and selectedapplications in speech recognition, " Proceedings of the IEEE, vol. 77, no. 2, pp. 257-286, 1989.
15. J. A. Bilmes, "What HMMs can do, " IEICE TRANSACTIONS onInformation and Systems, vol. 89, no. 3, pp. 869-891, 2006.
16. M. Mohri, F. Pereira, and M. Riley, "Weighted finite-state transducersin speech recognition, " Computer Speech & Language, vol. 16, pp. 69-88, 2002.
17. D. Povey, "Discriminative training for large vocabulary speechrecognition, " Ph. D thesis Cambridge, UK: Cambridge University, 2004.
18. B. Kingsbury, T. N. Sainath, and H. Soltau, "Scalable minimumBayes risk training of deep neural network acoustic models usingdistributed Hessian-free optimization. " in INTERSPEECH, 2012.
19. K. Veselý, A. Ghoshal, L. Burget, and D. Povey, "Sequencediscriminativetraining of deep neural networks, " in Proc. INTERSPEECH, 2013.
20. M. Ostendorf, V. Digalakis, and O. Kimball, "From HMM's tosegment models: A unified view of stochastic modeling for speechrecognition, " IEEE Transactions on Speech and Audio Processing, pp. 360-378, 1996.
21. N. Smith and M. Gales, "Speech recognition using SVMs, " in Advancesin neural information processing systems, 2001, pp. 1197-1204.
22. A. Gunawardana, M. Mahajan, A. Acero, and J. C. Platt, "Hiddenconditional rand om fields for phone classification. " in INTERSPEECH, 2005, pp. 1117-1120.
23. M. DeWachter, M. Matton, K. Demuynck, P. Wambacq, R. Cools, and D. Van Compernolle, "Template-based continuous speechrecognition, " Audio, Speech, and Language Processing, IEEETransactions on, vol. 15, no. 4, pp. 1377-1390, 2007.
24. Y. Hifny and S. Renals, "Speech recognition using augmentedconditional rand om fields, " Audio, Speech, and Language Processing, IEEE Transactions on, vol. 17, no. 2, pp. 354-365, 2009.
25. A. Graves, "Sequence transduction with recurrent neural networks, "arXiv preprint arXiv: 1211. 3711, 2012.
26. I. Sutskever, O. Vinyals, and Q. V. Le, "Sequence to sequencelearning with neural networks, " in Advances in Neural InformationProcessing Systems, 2014, pp. 3104-3112.
27. K. Cho, B. van Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, and Y. Bengio, "Learning phrase representationsusing RNN encoder-decoder for statistical machine translation, "Pro. EMNLP, 2014.
28. D. Bahdanau, K. Cho, and Y. Bengio, "Neural machine translationby jointly learning to align and translate, " in Proc. ICLR, 2015.
29. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, "Show and tell: Aneural image caption generator, " arXiv preprint arXiv: 1411. 4555, 2014.
30. K. Xu, J. Ba, R. Kiros, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio, "Show, attend and tell: Neural image caption generationwith visual attention, " arXiv preprint arXiv: 1502. 03044, 2015.
31. A. Graves and N. Jaitly, "Towards end-to-end speech recognitionwith recurrent neural networks, " in Proc. ICML, 2014, pp. 1764-1772.
32. A. L. Maas, A. Y. Hannun, D. Jurafsky, and A. Y. Ng, "First-Pass Large Vocabulary Continuous Speech Recognition using Bi-Directional Recurrent DNNs, " arXiv preprint arXiv: 1408. 2873, 2014.
33. J. Chorowski, D. Bahdanau, K. Cho, and Y. Bengio, "End-to-endContinuous Speech Recognition using Attention-based RecurrentNN: First Results, " arXiv preprint arXiv: 1412. 1602, 2014.
34. A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger et al., "Deep Speech: Scaling up end-to-endspeech recognition, " in arXiv preprint arXiv: 1412. 5567, 2014.
35. M. D. Zeiler, "Adadelta: An adaptive learning rate method, " arXivpreprint arXiv: 1212. 5701, 2012.
36. A. Senior, G. Heigold, M. Ranzato, and K. Yang, "An empiricalstudy of learning rates in deep neural networks for speech recognition, "in Proc. ICASSP. IEEE, 2013, pp. 6724-6728.
37. Y. Bengio, P. Simard, and P. Frasconi, "Learning long-term dependencieswith gradient descent is difficult, " Neural Networks, IEEE Transactions on, vol. 5, no. 2, pp. 157-166, 1994.
38. S. Hochreiter and J. Schmidhuber, "Long short-term memory, "Neural computation, vol. 9, no. 8, pp. 1735-1780, 1997.
39. J. J. Godfrey, E. C. Holliman, and J. McDaniel, "SWITCHBOARD: Telephone speech corpus for research and development, "in Proc. ICASSP. IEEE, 1992, pp. 517-520.
40. C. Gulcehre, O. Firat, K. Xu, K. Cho, L. Barrault, H.-C. Lin, F. Bougares, H. Schwenk, and Y. Bengio, "On using monolingualcorpora in neural machine translation, " arXiv preprintarXiv: 1503. 03535, 2015.
41. A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K. J. Lang, "Phoneme recognition using time-delay neural networks, "Acoustics, Speech and Signal Processing, IEEE Transactions on, vol. 37, no. 3, pp. 328-339, 1989.