Backoff inspired features for Maximum Entropy language models

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Volumn , Issue , 2014, Pages 2645-2649

  Biadsy, Fadi ^a   Hall, Keith ^a   Moreno, Pedro ^a   Roark, Brian ^a  

^a GOOGLE INC   (United States)

Author keywords

Language modeling; Linear models; Maximum entropy modeling; N gram models

Indexed keywords

MAXIMUM ENTROPY METHODS; MAXIMUM LIKELIHOOD; NATURAL LANGUAGE PROCESSING SYSTEMS; SPEECH COMMUNICATION; SPEECH RECOGNITION;

BACKOFF; LANGUAGE MODEL; MAXIMUM ENTROPY MODELING; MODEL QUALITIES; N-GRAM LANGUAGE MODELS; N-GRAM MODELS; TRAINING SCENARIO; WORD ERROR RATE;

COMPUTATIONAL LINGUISTICS;

EID: 84910028654     PISSN: 2308457X     EISSN: 19909772     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (25)

1. R. Lau, R. Rosenfeld, and S. Roukos, "Trigger-based language models: A maximum entropy approach, " in Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 1993, pp. 45- 48.
2. R. Rosenfeld, "A maximum entropy approach to adaptive statistical language modeling, " Computer Speech and Language, vol. 10, pp. 187-228, 1996.
3. S. M. Katz, "Estimation of probabilities from sparse data for the language model component of a speech recogniser, " IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 35, no. 3, pp. 400-401, 1987.
4. S. F. Chen and R. Rosenfeld, "A survey of smoothing techniques for ME models, " IEEE Transactions on Speech and Audio Processing, vol. 8, pp. 37-50, 2000.
5. R. Rosenfeld, "A whole sentence maximum entropy language model, " in Proceedings of IEEE Workshop on Speech Recognition and Understanding, 1997, pp. 230- 237.
6. R. Rosenfeld, S. F. Chen, and X. Zhu, "Whole-sentence exponential language models: A vehicle for linguisticstatistical integration, " Computer Speech and Language, vol. 15, no. 1, pp. 55-73, Jan. 2001.
7. B. Roark, M. Saraclar, and M. Collins, "Discriminative ngram language modeling, " Computer Speech & Language, vol. 21, no. 2, pp. 373-392, 2007.
8. T. Brants, A. C. Popat, P. Xu, F. J. Och, and J. Dean, "Large language models in machine translation, " in In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing (EMNLP) and Computational Natural Language Learning (CoNLL), 2007.
9. J. Wu and S. Khudanpur, "Efficient training methods for maximum entropy language modeling." in INTERSPEECH, 2000, pp. 114-118.
10. T. Alumäe and M. Kurimo, "Efficient estimation of maximum entropy language models with n-gram features: An srilm extension." in INTERSPEECH, 2010, pp. 1820- 1823.
11. P. Xu, A. Gunawardana, and S. Khudanpur, "Efficient subsampling for training complex language models, " in Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2011, pp. 1128-1136.
12. J. Wu and S. Khudanpur, "Building a topic-dependent maximum entropy model for very large corpora, " in Acoustics, Speech, and Signal Processing (ICASSP), 2002 IEEE International Conference on, vol. 1. IEEE, 2002, pp. 1-777.
13. R. Sarikaya, M. Afify, Y. Deng, H. Erdogan, and Y. Gao, "Joint morphological-lexical language modeling for processing morphologically rich languages with application to dialectal arabic, " Audio, Speech, and Language Processing, IEEE Transactions on, vol. 16, no. 7, pp. 1330- 1339, 2008.
14. M. A. B. Shaik, A. E.-D. Mousa, R. Schlüter, and H. Ney, "Feature-rich sub-lexical language models using a maximum entropy approach for german LVCSR, " in INTERSPEECH, 2013.
15. J. N. Tsitsiklis, D. P. Bertsekas, and M. Athans, "Distributed asynchronous deterministic and stochastic gradient optimization algorithms, " IEEE Transactions on Automatic Control, vol. 31:9, 1986.
16. K. Hall, S. Gilpin, and G. Mann, "Mapreduce/bigtable for distributed optimization, " in Neural Information Processing Systems Workshop on Leaning on Cores, Clusters, and Clouds, 2010.
17. R. McDonald, K. Hall, and G. Mann, "Distributed training strategies for the structured perceptron, " in Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2010, pp. 456-464.
18. M. Zinkevich, M. Weimer, A. Smola, and L. Li, "Parallelized stochastic gradient descent, " in Advances in Neural Information Processing Systems 23, J. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta, Eds., 2010, pp. 2595-2603.
19. F. Niu, B. Recht, C. Ré, and S. J. Wright, "Hogwild: A lock-free approach to parallelizing stochastic gradient descent, " in Advances in Neural Information Processing Systems, 2011.
20. P. Xu, A. Gunawardana, and S. Khudanpur, "Efficient subsampling for training complex language models." in EMNLP. ACL, 2011, pp. 1128-1136. [Online]. Available: Http://dblp.uni-trier.de/db/conf/emnlp/emnlp2011.html#XuGK11.
21. F. Morin and Y. Bengio, "Hierarchical probabilistic neural network language model, " in AISTATS05, 2005, pp. 246- 252.
22. Y. Tsuruoka, J. Tsujii, and S. Ananiadou, "Stochastic gradient descent training for l1-regularized log-linear models with cumulative penalty, " in Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL, 2009, pp. 477-485.
23. J. Dean and S. Ghemawat, "Mapreduce: Simplified data processing on large clusters, " in Proceedings of the 6th Conference on Symposium on Opearting Systems Design & Implementation - Volume 6, ser. OSDI'04, 2004, pp. 10-10.
24. J. Schalkwyk, D. Beeferman, F. Beaufays, B. Byrne, C. Chelba, M. Cohen, M. Kamvar, and B. Strope, "your word is my command: Google search by voice: A case study, " in Advances in Speech Recognition. Springer, 2010, pp. 61-90.
25. C. Allauzen and M. Riley, "Bayesian language model interpolation for mobile speech input." in INTERSPEECH, 2011, pp. 1429-1432.