Measuring the influence of long range dependencies with neural network language models

WLM 2012 - Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT at the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2012

Volumn , Issue , 2012, Pages 1-10

  Son, Le Hai ^a   Allauzen, Alexandre ^a   Yvon, François ^a  

^a UNIVERSITÉ PARIS SACLAY   (France)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; NATURAL LANGUAGE PROCESSING SYSTEMS; NEURAL MACHINE TRANSLATION; NEURAL NETWORKS; SPEECH RECOGNITION; SPEECH TRANSMISSION;

AUTOMATIC SPEECH RECOGNITION; CONTEXT WINDOW; LANGUAGE MODEL; LOCAL CONTEXTS; LONG-RANGE DEPENDENCIES; N-GRAM LANGUAGE MODELS; NATURAL LANGUAGE PROCESSING APPLICATIONS; NETWORK LANGUAGE; NEURAL-NETWORKS; STATISTICAL MACHINE TRANSLATION;

COMPUTER AIDED LANGUAGE TRANSLATION;

EID: 84906217753     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (31)

1. Alexandre Allauzen, Gilles Adda, Hélène Bonneau- Maynard, Josep M. Crego, Hai-Son Le, Aurélien Max, Adrien Lardilleux, Thomas Lavergne, Artem Sokolov, Guillaume Wisniewski, and François Yvon. 2011. LIMSI @ WMT11. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 309-315, Edinburgh, Scotland.
2. Y Bengio, R Ducharme, P Vincent, and C Jauvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3(6):1137-1155.
3. Thorsten Brants, Ashok C. Popat, Peng Xu, Franz J. Och, and Jeffrey Dean. 2007. Large language models in machine translation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 858-867.
4. Peter F. Brown, Peter V. deSouza, Robert L. Mercer, Vincent J. Della Pietra, and Jenifer C. Lai. 1992. Classbased n-gram models of natural language. Comput. Linguist., 18(4):467-479.
5. Ciprian Chelba and Frederick Jelinek. 2000. Structured language modeling. Computer Speech and Language, 14(4):283-332.
6. Stanley F. Chen and Joshua Goodman. 1996. An empirical study of smoothing techniques for language modeling. In Proc. ACL'96, pages 310-318, San Francisco.
7. Michael Collins, Brian Roark, and Murat Saraclar. 2005. Discriminative syntactic language modeling for speech recognition. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05), pages 507-514, Ann Arbor, Michigan, June. Association for Computational Linguistics.
8. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In Proc. of ICML'08, pages 160-167, New York, NY, USA. ACM.
9. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493-2537.
10. Ahmad Emami, Imed Zitouni, and Lidia Mangu. 2008. Rich morphology based n-gram language models for arabic. In INTERSPEECH, pages 829-832.
11. R. Kuhn and R. De Mori. 1990. A cache-based natural language model for speech recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(6):570-583, june.
12. Hong-Kwang Kuo, Lidia Mangu, Ahmad Emami, and Imed Zitouni. 2010. Morphological and syntactic features for arabic speech recognition. In Proc. ICASSP 2010.
13. Raymond Lau, Ronald Rosenfeld, and Salim Roukos. 1993. Adaptive language modeling using the maximum entropy principle. In Proc HLT'93, pages 108-113, Princeton, New Jersey.
14. Hai-Son Le, Ilya Oparin, Alexandre Allauzen, Jean-Luc Gauvain, and François Yvon. 2011a. Structured output layer neural network language model. In Proceedings of ICASSP'11, pages 5524-5527.
15. Hai-Son Le, Ilya Oparin, Abdel. Messaoudi, Alexandre Allauzen, Jean-Luc Gauvain, and François Yvon. 2011b. Large vocabulary SOUL neural network language models. In Proceedings of InterSpeech 2011.
16. Xunying Liu, Mark J. F. Gales, and Philip C. Woodland. 2011. Improving lvcsr system combination using neural network language model cross adaptation. In INTERSPEECH, pages 2857-2860.
17. Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Proceedings of the 11th Annual Conference of the International Speech Communication Association (INTERSPEECH 2010), volume 2010, pages 1045-1048. International Speech Communication Association.
18. Tomáš Mikolov, Anoop Deoras, Daniel Povey, Lukáš Burget, and Jan Černocký. 2011a. Strategies for training large scale neural network language models. In Proceedings of ASRU 2011, pages 196-201. IEEE Signal Processing Society.
19. Tomáš Mikolov, Stefan Kombrink, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur. 2011b. Extensions of recurrent neural network language model. In Proc. of ICASSP'11, pages 5528-5531.
20. Andriy Mnih and Geoffrey E Hinton. 2008. A scalable hierarchical distributed language model. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems 21, volume 21, pages 1081-1088.
21. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 1, ACL '03, pages 160-167, Stroudsburg, PA, USA. Association for Computational Linguistics.
22. Ronald Rosenfeld, Stanley F. Chen, and Xiaojin Zhu. 2001. Whole-sentence exponential language models: A vehicle for linguistic-statistical integration. Computers, Speech and Language, 15:2001.
23. R. Rosenfeld. 2000. Two decades of statistical language modeling: Where do we go from here? Proceedings of the IEEE, 88(8).
24. D. E. Rumelhart, G. E. Hinton, and R. J.Williams. 1986. Parallel distributed processing: explorations in the microstructure of cognition, vol. 1. chapter Learning internal representations by error propagation, pages 318-362. MIT Press, Cambridge, MA, USA.
25. Helmut Schmid. 1994. Probabilistic part-of-speech tagging using decision trees. In Proceedings of International Conference on New Methods in Language Processing.
26. Lane Schwartz, Chris Callison-Burch, William Schuler, and Stephen Wu. 2011. Incremental syntactic language models for phrase-based translation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 620-631, Portland, Oregon, USA, June. Association for Computational Linguistics.
27. Holger Schwenk and Jean-Luc Gauvain. 2002. Connectionist language modeling for large vocabulary continuous speech recognition. In Proc. ICASSP, pages 765-768, Orlando, FL.
28. H. Schwenk and P. Koehn. 2008. Large and diverse language models for statistical machine translation. In International Joint Conference on Natural Language Processing, pages 661-666, Janv 2008.
29. Holger Schwenk. 2007. Continuous space language models. Comput. Speech Lang., 21(3):492-518.
30. Kristie Seymore and Ronald Rosenfeld. 1997. Using story topics for language model adaptation. In Proc. of Eurospeech '97, pages 1987-1990, Rhodes, Greece.
31. Yeh W. Teh. 2006. A hierarchical Bayesian language model based on Pitman-Yor processes. In Proc. of ACL'06, pages 985-992, Sidney, Australia.