One billion word benchmark for measuring progress in statistical language modeling

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Volumn , Issue , 2014, Pages 2635-2639

  Chelba, Ciprian ^a   Mikolov, Tomas ^a   Schuster, Mike ^a   Ge, Qi ^a   Brants, Thorsten ^a   Koehn, Phillipp ^b   Robinson, Tony ^c  

^a GOOGLE INC   (United States)

^b UNIVERSITY OF EDINBURGH   (United Kingdom)

^c Cantab Research Ltd   (United Kingdom)

Author keywords

Benchmark; Language modeling; Reproducible research

Indexed keywords

BENCHMARKING; NATURAL LANGUAGE PROCESSING SYSTEMS; RECURRENT NEURAL NETWORKS; SPEECH COMMUNICATION;

CROSS ENTROPY; LANGUAGE MODEL; N-GRAM MODELS; REPRODUCIBLE RESEARCH; STATISTICAL LANGUAGE MODELING; TRAINING DATA;

COMPUTATIONAL LINGUISTICS;

EID: 84910091099     PISSN: 2308457X     EISSN: 19909772     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (32)

1. [Bengio et al., 2003] Y. Bengio, R. Ducharme, and P. Vincent. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137-1155.
2. [Brants et al., 2007] T. Brants, A. C. Popat, P. Xu, F. J. Och, and J. Dean. 2007. Large language models in machine translation. In Proceedings of EMNLP.
3. [Brown et al., 1992] P. F. Brown, P. V. deSouza, R. L. Mercer, V. J. Della Pietra, and J. C. Lai. 1992. Class-Based N-gram Models of Natural Language. Computational Linguistics, 18, 467-479.
4. [Elman, 1990] J. Elman. 1990. Finding Structure in Time. Cognitive Science, 14, 179-211.
5. [Emami, 2006] A. Emami. 2006. A Neural Syntactic Language Model. Ph.D. thesis, Johns Hopkins University.
6. [Goodman, 2001a] J. T. Goodman. 2001a. A bit of progress in language modeling, extended version. Technical report MSR-TR- 2001-72.
7. [Goodman, 2001b] J. T. Goodman. 2001b. Classes for fast maximum entropy training. In Proceedings of ICASSP.
8. [Jelinek et al., 1991] F. Jelinek, B. Merialdo, S. Roukos, and M. Strauss. 1991. A Dynamic Language Model for Speech Recognition. In Proceedings of the DARPA Workshop on Speech and Natural Language.
9. [Chelba and Jelinek, 2000] C. Chelba and F. Jelinek. 2000. Structured language modeling. Computer Speech & Language.
10. [Chelba et al., 2010] C. Chelba, T. Brants, W. Neveitt, and P. Xu. 2010. Study on Interaction between Entropy Pruning and Kneser-Ney Smoothing. In Proceedings of Interspeech.
11. [Chen and Goodman, 1996] S. F. Chen and J. T. Goodman. 1996. An empirical study of smoothing techniques for language modeling. In Proceedings of ACL.
12. [Chen, 2009] S. F. Chen. 2009. Shrinking exponential language models. In Proceedings of NAACL-HLT.
13. [Katz, 1995] S. Katz. 1987. Estimation of probabilities from sparse data for the language model component of a speech recognizer. In IEEE Transactions on Acoustics, Speech and Signal Processing.
14. [Kneser and Ney, 1995] R. Kneser and H. Ney. 1995. Improved Backing-Off For M-Gram Language Modeling. In Proceedings of ICASSP.
15. [Mikolov et al., 2010] T. Mikolov, M. Karafiát, L. Burget, J. Cě rnocký, and S. Khudanpur. 2010. Recurrent neural network based language model. In Proceedings of Interspeech.
16. [Mikolov et al., 2011a] T. Mikolov, S. Kombrink, L. Burget, J. Cěrnocky, and S. Khudanpur. 2011. Extensions of Recurrent Neural Network Language Model. In Proceedings of ICASSP.
17. [Mikolov et al., 2011b] T. Mikolov, A. Deoras, S. Kombrink, L. Burget, and J. Cěrnocky. 2011a. Empirical Evaluation and Combination of Advanced Language Modeling Techniques. In Proceedings of Interspeech.
18. [Mikolov et al., 2011c] T. Mikolov, A. Deoras, D. Povey, L. Burget, and J. Cěrnocky. 2011b. Strategies for Training Large Scale Neural Network Language Models. In Proceedings of ASRU.
19. [Mikolov, 2012] T. Mikolov. 2012. Statistical Language Models based on Neural Networks. Ph.D. thesis, Brno University of Technology.
20. [Mnih and Hinton, 2007] A. Mnih and G. Hinton. 2007. Three new graphical models for statistical language modelling. In Proceedings of ICML.
21. [Morin and Bengio, 2005] F. Morin and Y. Bengio. 2005. Hierarchical Probabilistic Neural Network Language Model. In Proceedings of AISTATS.
22. [Roark et al., 2004] B. Roark, M. Saralar, M. Collins, and M. Johnson. 2004. Discriminative language modeling with conditional random fields and the perceptron algorithm. In Proceedings of ACL.
23. [Rosenfeld, 1994] R. Rosenfeld. 1994. Adaptive Statistical Language Modeling: A Maximum Entropy Approach. Ph.D. thesis, Carnegie Mellon University.
24. [Rumelhart et al., 1986] D. E. Rumelhart, G. E. Hinton, and R. J. Williams. 1986. Learning internal representations by backpropagating errors. Nature, 323:533-536.
25. [Schwenk, 2007] H. Schwenk. 2007. Continuous space language models. Computer Speech and Language, vol. 21.
26. [Stolcke, 1998] A. Stolcke. 1998. Entropy-based Pruning of Backoff Language Models. In Proceedings of News Transcription and Understanding Workshop.
27. [Sundermeyer et al., 2012] M. Sundermeyer, R. Schluter, and H. Ney. 2012. LSTM Neural Networks for Language Modeling. In Proceedings of Interspeech.
28. [Teh, 2006] Y.W. Teh. 2006. A hierarchical Bayesian language model based on Pitman Yor processes. In Proceedings of Coling/ACL.
29. [Wu et al., 2012] Y. Wu, H. Yamamoto, X. Lu, S. Matsuda, C. Hori, and H. Kashioka. 2012. Factored Recurrent Neural Network Language Model in TED Lecture Transcription. In Proceedings of IWSLT.
30. [Xu, 2005] Peng Xu. 2005. Random forests and the data sparseness problem in language modeling. Ph.D. thesis, Johns Hopkins University.
31. [Xu et al., 2011] Puyang Xu, A. Gunawardana, and S. Khudanpur. 2011. Efficient Subsampling for Training Complex Language Models. In Proceedings of EMNLP.
32. [Zweig and Makarychev, 2013] G. Zweig and K. Makarychev. 2013. Speed Regularization and Optimality in Word Classing. In Proceedings of ICASSP.