A comparison of training approaches for discriminative segmental models

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Volumn , Issue , 2014, Pages 1219-1223

  Tang, Hao ^a   Gimpel, Kevin ^a   Livescu, Karen ^a  

^a TOYOTA TECHNOLOGICAL INSTITUTE AT CHICAGO   (United States)

Author keywords

Empirical bayes risk; Large margin training; Segmental conditional random fields; Speech recognition

Indexed keywords

COST FUNCTIONS; RANDOM PROCESSES; SPEECH COMMUNICATION;

CONDITIONAL LIKELIHOOD; EMPIRICAL BAYES; FLEXIBLE FRAMEWORK; LATTICE RESCORING; MULTIPLE TASKS; SEGMENTAL CONDITIONAL RANDOM FIELDS; WORD ERROR RATE;

SPEECH RECOGNITION;

EID: 84910091098     PISSN: 2308457X     EISSN: 19909772     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (35)

1. M. Ostendorf and S. Roukos, "A stochastic segment model for phoneme-based continuous speech recognition, " IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 37, no. 12, pp. 1857-1869, 1989.
2. J. R. Glass, "A probabilistic framework for segment-based speech recognition, " Computer Speech and Language, vol. 17, no. 2, pp. 137-152, 2003.
3. M. DeWachter, M. Matton, K. Demuynck, P.Wambacq, R. Cools, and D. Van Compernolle, "Template-based continuous speech recognition, " IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 15, no. 4, pp. 1377-1390, 2007.
4. G. Zweig and P. Nguyen, "A segmental CRF approach to large vocabulary continuous speech recognition, " in Proc. IEEEWorkshop on Automatic Speech Recognition and Understanding (ASRU), 2009, pp. 152-157.
5. G. Zweig, P. Nguyen, D. Van Compernolle, K. Demuynck, L. Atlas, P. Clark, G. Sell, M. Wang, F. Sha, H. Hermansky et al., "Speech recognition with segmental conditional random fields: A summary of the JHU CLSP 2010 SummerWorkshop, " in Proc. International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2011, pp. 5044-5047.
6. S. Sarawagi and W. W. Cohen, "Semi-Markov conditional random fields for information extraction, " in Proc. Neural Information Processing Systems (NIPS), 2004, pp. 1185-1192.
7. K. Demuynck, D. Seppi, D. Van Compernolle, P. Nguyen, and G. Zweig, "Integrating meta-information into exemplar-based speech recognition with segmental conditional random fields, " in Proc. International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2011, pp. 5048-5051.
8. G. Heigold, H. Ney, R. Schlüter, and S. Wiesler, "Discriminative training for automatic speech recognition: Modeling, criteria, optimization, implementation, and performance, " IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 58-69, 2012.
9. J. Kaiser, B. Horvat, and Z. Kacic, "A novel loss function for the overall risk criterion based discriminative training of HMM models, " in Proc. International Conference on Spoken Language Processing (ICSLP), 2000.
10. D. Povey and P. C. Woodland, "Minimum phone error and Ismoothing for improved discriminative training, " in Proc. International Conference on Acoustics, Speech, and Signal Processing (ICASSP), vol. 1, 2002, pp. I-105.
11. V. Doumpiotis and W. Byrne, "Lattice segmentation and minimum bayes risk discriminative training for large vocabulary continuous speech recognition, " Speech Communication, vol. 48, no. 2, pp. 142-160, 2006.
12. S.-X. Zhang and M. Gales, "Structured support vector machines for noise robust continuous speech recognition, " in Proc. Interspeech, 2011, pp. 989-992.
13. F. Sha and L. K. Saul, "Large margin hidden Markov models for automatic speech recognition, " Proc. Neural Information Processing Systems (NIPS), vol. 19, p. 1249, 2007.
14. H. Jiang, X. Li, and C. Liu, "Large margin hidden Markov models for speech recognition, " IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 14, no. 5, pp. 1584-1595, 2006.
15. D. Yu, L. Deng, X. He, and A. Acero, "Large-margin minimum classification error training for large-scale speech recognition tasks, " in Proc. International Conference on Acoustics, Speech, and Signal Processing (ICASSP), vol. 4, 2007, pp. IV- 1137.
16. H. Tang, J. Keshet, and K. Livescu, "Discriminative pronunciation modeling: A large-margin, feature-rich approach, " in Proc. Association for Computational Linguistics (ACL), 2012.
17. A. Ragni and M. Gales, "Derivative kernels for noise robust ASR, " in Proc. IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), 2011.
18. S.-X. Zhang, A. Ragni, and M. J. F. Gales, "Structured log linear models for noise robust speech recognition, " Signal Processing Letters, IEEE, vol. 17, no. 11, pp. 945-948, 2010.
19. J. Duchi, E. Hazan, and Y. Singer, "Adaptive subgradient methods for online learning and stochastic optimization, " The Journal of Machine Learning Research, vol. 12, pp. 2121-2159, 2011.
20. J. Lafferty, A. McCallum, and F. Pereira, "Conditional random fields: Probabilistic models for segmenting and labeling sequence data, " in Proc. International Conference on Machine Learning (ICML), 2001.
21. B. Taskar, V. Chatalbashev, D. Koller, and C. Guestrin, "Learning structured prediction models: A large margin approach, " in Proc. International Conference on Machine Learning (ICML), 2005.
22. C. B. Do, Q. Le, C. H. Teo, O. Chapelle, and A. Smola, "Tighter bounds for structured estimation, " in Proc. Neural Information Processing Systems (NIPS), 2008.
23. I. Tsochantaridis, T. Joachims, T. Hofmann, and Y. Altun, "Large margin methods for structured and interdependent output variables, " Journal of Machine Learning Research, vol. 6, pp. 1453- 1484, 2005.
24. K. Gimpel and N. A. Smith, "Structured ramp loss minimization for machine translation, " in Proc. Human Language Technology/Conference of the North American Chapter of the Association for Computational Linguistics (HLT/NAACL), 2012.
25. L. Bahl, P. Brown, P. V. de Souza, and R. Mercer, "Maximum mutual information estimation of hidden Markov model parameters for speech recognition, " in Acoustics, Speech, and Signal Processing, IEEE International Conference on ICASSP'86., vol. 11. IEEE, 1986, pp. 49-52.
26. L. Xiao, "Dual averaging methods for regularized stochastic learning and online optimization." Journal of Machine Learning Research, vol. 11, no. 2543-2596, p. 4, 2010.
27. Y. Tsuruoka, J. Tsujii, and S. Ananiadou, "Stochastic gradient descent training for l1-regularized log-linear models with cumulative penalty, " in ACL/AFNLP. Association for Computational Linguistics, 2009, pp. 477-485.
28. M. Riedmiller and H. Braun, "A direct adaptive method for faster backpropagation learning: The RPROP algorithm, " in Proc. IEEE International Conference on Neural Networks (ICNN), 1993, pp. 586-591.
29. S. Shalev-Shwartz and A. Tewari, "Stochastic methods for l1- regularized loss minimization, " Journal of Machine Learning Research, vol. 12, pp. 1865-1892, 2011.
30. G. Zweig, "Classification and recognition with direct segment models, " in Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference on, 2012, pp. 4161-4164.
31. T. Kim, G. Shakhnarovich, and K. Livescu, "Fingerspelling recognition with semi-Markov conditional random fields, " in Proc. International Conference on Computer Vision (ICCV), 2013.
32. G. Zweig and P. Nguyen, "SCARF: A segmental conditional random field toolkit for speech recognition, " Proc. Interspeech, 2010.
33. A. Graves, A.-R. Mohamed, and G. Hinton, "Speech recognition with deep recurrent neural networks, " in Proc. International Conference on Acoustics, Speech, and Signal Processing (ICASSP). IEEE, 2013, pp. 6645-6649.
34. D. Povey, D. Kanevsky, B. Kingsbury, B. Ramabhadran, G. Saon, and K. Visweswariah, "Boosted mmi for model and feature-space discriminative training, " in Proc. International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2008, pp. 4057-4060.
35. G. Heigold, T. Deselaers, R. Schlüter, and H. Ney, "Modified MMI/MPE: A direct evaluation of the margin in speech recognition, " in Proc. International Conference on Machine Learning (ICML), 2008, pp. 384-391.