Faster and smaller n-gram language models

ACL-HLT 2011 - Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies

Volumn 1, Issue , 2011, Pages 258-267

  Pauls, Adam ^a   Klein, Dan ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CACHING TECHNIQUE; COMPACT REPRESENTATION; LANGUAGE MODEL; LOSSLESS; LOSSY COMPRESSIONS; MACHINE TRANSLATIONS; N-GRAM LANGUAGE MODELS; N-GRAMS;

COMPUTATIONAL LINGUISTICS;

EID: 84455177089     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (19)

1. Paolo Boldi and Sebastiano Vigna. 2005. Codes for the world wide web. Internet Mathematics, 2.
2. Thorsten Brants and Alex Franz. 2006. Google web1t 5-gram corpus, version 1. In Linguistic Data Consortium, Philadelphia, Catalog Number LDC2006T13.
3. Thorsten Brants, Ashok C. Popat, Peng Xu, Franz J. Och, and Jeffrey Dean. 2007. Large language models in machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.
4. Bernard Chazelle, Joe Kilian, Ronitt Rubinfeld, and Ayellet Tal. 2004. The Bloomier filter: an efficient data structure for static support lookup tables. In Proceedings of the fifteenth annual ACM-SIAM symposium on Discrete algorithms.
5. David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In The Annual Conference of the Association for Computational Linguistics.
6. Kenneth Church, Ted Hart, and Jianfeng Gao. 2007. Compressing trigram language models with golomb coding. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.
7. Marcello Federico and Mauro Cettolo. 2007. Efficient handling of n-gram language models for statistical machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation.
8. Edward Fredkin. 1960. Trie memory. Communications of the ACM, 3:490-499, September.
9. Ulrich Germann, Eric Joanis, and Samuel Larkin. 2009. Tightly packed tries: how to fit large models into memory, and make them load fast, too. In Proceedings of the Workshop on Software Engineering, Testing, and Quality Assurance for Natural Language Processing.
10. S. W. Golomb. 1966. Run-length encodings. IEEE Transactions on Information Theory, 12.
11. David Guthrie and Mark Hepple. 2010. Storing the web in memory: space efficient language models with constant time retrieval. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.
12. Boulos Harb, Ciprian Chelba, Jeffrey Dean, and Sanjay Ghemawat. 2009. Back-off language model compression. In Proceedings of Interspeech.
13. Bo-June Hsu and James Glass. 2008. Iterative language model estimation: Efficient data structure and algorithms. In Proceedings of Interspeech.
14. Abby Levenberg and Miles Osborne. 2009. Streambased randomised language models for smt. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.
15. Zhifei Li and Sanjeev Khudanpur. 2008. A scalable decoder for parsing-based machine translation with equivalent language model state maintenance. In Proceedings of the Second Workshop on Syntax and Structure in Statistical Translation.
16. Zhifei Li, Chris Callison-Burch, Chris Dyer, Juri Ganitkevitch, Sanjeev Khudanpur, Lane Schwartz, Wren N. G. Thornton, JonathanWeese, and Omar F. Zaidan. 2009. Joshua: an open source toolkit for parsingbased machine translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation.
17. Franz Josef Och and Hermann Ney. 2004. The alignment template approach to statistical machine translation. Computationl Linguistics, 30:417-449, December. (Pubitemid 41399543)
18. Andreas Stolcke. 2002. SRILM: An extensible language modeling toolkit. In Proceedings of Interspeech.
19. E. W. D. Whittaker and B. Raj. 2001. Quantizationbased language model compression. In Proceedings of Eurospeech.