Recurrent neural networks for language understanding

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Volumn , Issue , 2013, Pages 2524-2528

  Yao, Kaisheng ^a   Zweig, Geoffrey ^b   Hwang, Mei Yuh ^a   Shi, Yangyang ^c   Yu, Dong ^b  

^a MICROSOFT   (United States)

^b MICROSOFT RESEARCH   (United States)

^c DELFT UNIVERSITY OF TECHNOLOGY   (Netherlands)

Author keywords

Recurrent neural networks; Spoken language understanding

Indexed keywords

COMPUTATIONAL LINGUISTICS;

LANGUAGE UNDERSTANDING; SPOKEN LANGUAGE UNDERSTANDING; WORD CONTEXTS; WORD REPRESENTATIONS;

RECURRENT NEURAL NETWORKS;

EID: 84904483474     PISSN: 2308457X     EISSN: 19909772     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (41)

1. T. Mikolov, M. Karafiat, J. Cernocky, and S. Khudanpur, "Recurrent neural network based language model, " in INTERSPEECH, 2010, pp. 1045-1048.
2. T. Mikolov, S. Kombrink, L. Burget, J. Cernocky, and S. Khudanpur, "Extensions of recurrent neural network based language model, " in ICASSP, 2011, pp. 5528-5531.
3. T. Mikolov, A. Deoras, S. Kombrink, L. Burget, and J. Cernocky, "Empirical evaluation and combination of advanced language modeling techniques, " in INTERSPEECH, 2011, pp. 605- 608.
4. T. Mikolov, A. Deoras, D. Povey, L. Burget, and J. Cernocky, "Strategies for training large scale neural network language models, " in ASRU, 2011.
5. T. Mikolov and G. Zweig, "Context dependent recurrent neural network language model, " in Proc. SLT, 2012, pp. 234-239.
6. E. Arisoy, T. N. Sainath, B. Kingsbury, and B. Ramabhadran, "Deep neural network language models, " in Proceedings of the NAACL-HLT 2012 Workshop: Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT, 2012, pp. 20-28.
7. H. Schwenk and J.L. Gauvain, "Connectionist language modeling for large vocabulary continuous speech recognition, " in ICASSP. IEEE, 2002, vol. 1, pp. I-765.
8. Y. Bengio, R. Ducharme, Vincent, P., and C. Jauvin, "A neural probabilistic language model, " Journal of Machine Learning Reseach, vol. 3, no. 6, 2003.
9. H. Schwenk, "Continuous space language models, " Computer Speech and Language, vol. 21, no. 3, pp. 492 - 518, 2007.
10. H.-S. Le, I. Oparin, A. Allauzen, J.-L. Gauvain, and F. Yvon, "Structured output layer neural network language model, " in ICASSP, 2011, pp. 5524-5527.
11. F. Morin and Y. Bengio, "Hierarchical probabilistic neural network language model, " in Proceedings of the international workshop on artificial intelligence and statistics, 2005, pp. 246-252.
12. A. Mnih and G.E. Hinton, "A scalable hierarchical distributed language model, " in Advances in neural information processing systems, 2009, vol. 21, pp. 1081-1088.
13. Mikolov, W.T. Yih, and G. Zweig, "Linguistic regularities in continuous space word representations, " in NAACL-HLT, to appear, 2013.
14. H. Schwenk, D. Dchelotte, and J.-L. Gauvain, "Continuous space language models for statistical machine translation, " in COLING/ACL, 2006.
15. L. H. Son, A. Allauzen, and F. Yvon, "Measuring the influence of long range dependencies with neural network language models, " in NAACL/HLT, 2012, pp. 1-10.
16. H. Schwenk, A. Rousseau, and M. Attik, "Large, pruned or continuous space language models on a GPU for statistical machine translation, " in NAACL/HLT, 2012, pp. 723-730.
17. C. Hemphill, J. Godfrey, and G. Doddington, "The ATIS spoken language systems pilot corpus, " in Proceedings of the DARPA speech and natural language workshop, 1990, pp. 96-101.
18. P. Price, "Evaluation of spoken language systems: The ATIS domain, " in Proceedings of the Third DARPA Speech and Natural Language Workshop. Morgan Kaufmann, 1990, pp. 91-95.
19. W. Ward et al., "The CMU air travel information service: Understanding spontaneous speech, " in Proceedings of the DARPA Speech and Natural Language Workshop, 1990, pp. 127-129.
20. Y. He and S. Young, "A data-driven spoken language understanding system, " in ASRU, 2003, pp. 583-588.
21. C. Raymond and G. Riccardi, "Generative and discriminative algorithms for spoken language understanding, " INTERSPEECH, pp. 1605-1608, 2007.
22. R. De Mori, "Spoken language understanding: A survey, " in ASRU, 2007, pp. 365-376.
23. F. Béchet, "Processing spontaneous speech in deployed spoken language understanding systems: A survey, " SLT, vol. 1, 2008.
24. V. Zue and J. Glass, "Conversational interfaces: Advances and challenges, " Proceedings of the IEEE, vol. 88, no. 8, pp. 1166- 1180, 2000.
25. J. Liu, S. Cyphers, P. Pasupat, I. McGraw, and J. Glass, "A conversational movie search system based on conditional random fields, " in INTERSPEECH, 2012.
26. J. Lafferty, A. McCallum, and F. Pereira, "Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data, " in ICML, 2001, pp. 282-289.
27. Y.-Y. Wang, A. Acero, M. Mahajan, and J. Lee, "Combining statistical and knowledge-based spoken language understanding in conditional models, " in COLING/ACL, 2006, pp. 882-889.
28. A. Moschitti, G. Riccardi, and C. Raymond, "Spoken language understanding with kernels for syntactic/semantic structures, " in ASRU, 2007, pp. 183-188.
29. T. Kudo and Y. Matsumoto, "Chunking with support vector machines, " in ACL, 2001.
30. K. Macherey, F. Och, H. Ney, et al., "Natural language understanding using statistical machine translation, " in European Conf. on Speech Communication and Technology. Citeseer, 2001, pp. 2205-2208.
31. J. Elman, "Finding structure in time, " Cognitive science, vol. 14, no. 2, pp. 179-211, 1990.
32. P.F. Brown, V.J. Della Pietra, P.V. deSouza, J. Lai, and R.L. Mercer, "Class-based n-gram models of natural language, " Computational Linguistics, vol. 18, no. 4, pp. 467-479, 1992.
33. G. Mesnil, X. He, L. Deng, and Y. Bengio, "Investigation of recurrent-neural-network architectures and learning methods for language understanding, " in submitted to INTERSPEECH, 2013.
34. Y. Shi, P. Wiggers, and C. M. Jonker, "Towards recurrent neural networks language models with linguistic and contextual features, " in INTERSPEECH, 2012.
35. G. Tur, D. Hakkani-Tur, L. Heck, and S. Parthasarathy, "Sentence simplification for spoken language understanding, " in ICASSP, 2011, pp. 5628-5631.
36. D. Dahl, M. Bates, M. Brown, W. Fisher, K. Hunicke-Smith, D. Pallett, C. Pao, A. Rudnicky, and E. Shriberg, "Expanding the scope of the ATIS task: The ATIS-3 corpus, " in Proceedings of the workshop on Human Language Technology. Association for Computational Linguistics, 1994, pp. 43-48.
37. L. Deng, G. Tur, X. He, and D. Hakkani-Tur, "Use of kernel deep convex networks and end-to-end learning for spoken language understanding, " in SLT, 2013, pp. 210-215.
38. R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa, "Natural language processing (almost) from scratch, " Journal of Machine Learning Research, vol. 12, pp. 2493-2537, 2011.
39. L. Van der Maaten and G. Hinton, "Visualizing data using t-SNE, " Journal of Machine Learning Research, vol. 9, no. 2579-2605, 2008.
40. F. Seide, G. Li, X. Chen, and D. Yu, "Feature engineering in context-dependent deep neural networks for conversational speech transcription, " in ASRU, 2011, pp. 24-29.
41. D. Yu, M. L. Seltzer, J. Li, J.-T. Huang, and F. Seide, "Feature learning in deep neural networks - studies on speech recognition tasks, " 2013, http://arxiv.org/pdf/1301.3605.