Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Volumn , Issue , 2013, Pages 3771-3775

  Mesnil, Grégoire ^a,c   He, Xiaodong ^b   Deng, Li ^b   Bengio, Yoshua ^a  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

^b MICROSOFT RESEARCH   (United States)

^c UNIVERSITÉ DE ROUEN   (France)

Author keywords

Recurrent neural network; Slot filling; Spoken language understanding; Word embeddings

Indexed keywords

FILLING; NETWORK ARCHITECTURE;

CONDITIONAL RANDOM FIELD; DEEP NEURAL NETWORKS; DOMAIN DETECTIONS; EMBEDDINGS; LEARNING METHODS; RECURRENT NETWORKS; RELATIVE ERRORS; SPOKEN LANGUAGE UNDERSTANDING;

RECURRENT NEURAL NETWORKS;

EID: 84906237242     PISSN: 2308457X     EISSN: 19909772     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (28)

1. J. Bergstra, O. Breuleux, F. Bastien, P. Lamblin, R. Pascanu, G. Desjardins, J. Turian, D. Warde-Farley and Y. Bengio, "Theano: A CPU and GPU Math Expression Compiler, " Proc. Python for Scientific Computing Conference (SciPy) 2010.
2. Y. Bengio, R. Ducharme and P. Vincent, "A Neural Probabilistic Language Model", in NIPS 2000.
3. R. Collobert and J. Weston, "A unified architecture for natural language processing: deep neural networks with multitask learning, " in ICML 2008.
4. R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa, "Natural language processing (almost) from scratch, " in Journal of Machine Learning Research, vol. 12, 2011.
5. CRPpp: http://crfpp.googlecode.com/svn/trunk/doc/index.
6. G. Dahl, D. Yu, L. Deng, and A. Acero, "Context- Dependent Pre-trained Deep Neural Networks for Large Vocabulary Speech Recognition, in IEEE Transactions on Audio, Speech, and Language Processing, " vol. 20, no. 1, pp. 30-42, January 2012.
7. L. Deng, G. Tur, X. He, and D. Hakkani-Tur, "Use of Kernel Deep Convex Networks and End-To-End Learning for Spoken Language Understanding, " IEEE Workshop on Spoken Language Technologies, December 2012.
8. J. Elman, "Finding structure in time, " in Cognitive Science, 14 (2), 1990.
9. Y. He and S. Young, "A data-driven spoken language understanding system, " in IEEE ASRU 2003.
10. G. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Improving neural networks by preventing co-adaptation of feature detectors, " arXiv: 1207.0580v1, 2012.
11. G. Hinton, Li Deng, Dong Yu, George Dahl, Abdelrahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara Sainath, and Brian Kingsbury, "Deep Neural Networks for Acoustic Modeling in Speech Recognition, " in IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 82-97, Nov. 2012.
12. M. Jordan, "Serial order: A parallel distributed processing approach, " in Tech. Rep. No. 8604. San Diego: University of California, Institute for Cognitive Science.
13. J. Lafferty, A. McCallum, and F. Pereira, "Conditional random fields: Probabilistic models for segmenting and labeling sequence data, " in ICML 2001.
14. T. Mikolov, http://www.fit.vutbr.cz/imikolov/rnnlm/.
15. T. Mikolov, Stefan Kombrink, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur, "Extensions of recurrent neural network based language model, " in ICASSP 2011.
16. A. Mnih and G. Hinton, "A scalable hierarchical distributed language model" in NIPS, 2008, pp. 1081- 1088.
17. C. Raymond and G. Riccardi, "Generative and discriminative algorithms for spoken language understanding, " in Interspeech 2007.
18. M. Schuster and K. Paliwal, "Bidirectional recurrent neural networks, " in IEEE Transactions on Signal Processing, November 1997.
19. H. Schwenk and J-L. Gauvain, "Training neural network language models on very large corpora, " in HLT/EMNLP 2005.
20. C. Shaoul and C. Westbury. 2010. The Westbury lab wikipedia corpus.
21. Socher, R., Lin, C., Ng, A., and Manning, C. "Learning continuous phrase representations and syntactic parsing with recursive neural networks, " Proc. ICML, 2011.
22. G. Tur, D. Hakkani-Tur, and L. Heck, "What is left to be understood in ATIS?" in IEEE SLT, 2010.
23. G. Tur and L. Deng, "Intent Determination and Spoken Utterance Classification, " in Chapter 4, Tur and De Mori (eds). Spoken Language Understanding: Systems for Extracting Semantic Information from Speech, pp. 81- 104, Wiley, 2011.
24. G. Tur, L. Deng, D. Hakkani-Tur, and X. He, "Towards Deeper Understanding Deep Convex Networks for Semantic Utterance Classification, " in ICASSP, 2012.
25. Y. Wang, L. Deng, and A. Acero, "Spoken Language Understanding - An Introduction to the Statistical Framework, " IEEE Signal Processing Magazine, vol. 22, no. 5, pp. 16-31, 2005.
26. Y. Wang, L. Deng, and A. Acero, "Semantic Frame Based Spoken Language Understanding, " in Chapter 3, Tur and De Mori (eds) Spoken Language Understanding: Systems for Extracting Semantic Information from Speech, pp. 35-80, Wiley, 2011.
27. S. Yaman, L. Deng, D. Yu, Y. Wang, and A. Acero, "An integrative and discriminative technique for spoken utterance classification, " IEEE Trans. Audio, Speech, and Language Processing, vol. 16, no. 6, pp. 1207-1214, 2008.
28. K. Yao, G. Zweig, M-Y. Hwang, Y. Shi, D. Yu, "Recurrent neural networks for language understanding", submitted to Interspeech 2013.