Multitask learning of deep neural networks for low-resource speech recognition

IEEE Transactions on Audio, Speech and Language Processing

Volumn 23, Issue 7, 2015, Pages 1172-1183

  Chen, Dongpeng ^a   Mak, Brian Kan Wing ^a  

^a HONG KONG UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Hong Kong)

Author keywords

Deep neural network (DNN); low resource speech recognition; multitask learning; universal grapheme set; universal phone set

Indexed keywords

COMPUTATIONAL LINGUISTICS; LEARNING ALGORITHMS; LINEARIZATION; TELEPHONE SETS; LEARNING SYSTEMS;

DEEP NEURAL NETWORKS; LOW-RESOURCE SPEECH RECOGNITION; MULTITASK LEARNING; UNIVERSAL GRAPHEME SET; UNIVERSAL PHONE SET;

SPEECH RECOGNITION;

EID: 84929327770     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASLP.2015.2422573     Document Type: Article

References (58)

1. S. Hunnicutt, H. M. Meng, S. Seneff, and V. W. Zue, "Reversible letter-to-sound sound-to-letter generation based on parsing word morphology," in Proc. Eurospeech, 1993, pp. 763-766.
2. E. G. Schukat-Talamazzini, H. Niemann, W. Eckert, T. Kuhn, and S. Rieck, "Automatic speech recognition without phonemes," in Proc. Eurospeech, 1993.
3. S. Kanthak and H. Ney, "Context-dependent acoustic modeling using graphemes for large vocabulary speech recognition," in Proc. ICASSP, 2002, vol. 1, pp. 845-848.
4. P. Charoenpornsawat, S. Hewavitharana, and T. Schultz, "Thai grapheme-based speech recognition," in Proc. HLT-NAACL, Companion Vol.: Short Papers, 2006, pp. 17-20, ACL.
5. S. Stüker, "Acoustic modeling for under-resourced languages," Ph.D. dissertation, Univ. of Karlsruhe, Karlsruhe, Germany, 2009.
6. T. Ko and B. Mak, "Eigentrigraphemes for under-resourced languages," Speech Commun., vol. 56, pp. 132-141, Jan. 2014.
7. S. Takahashi and S. Sagayama, "Four-level tied-structure for efficient representation of acoustic modeling," in Proc. ICASSP, 1995, vol. 1, pp. 520-523.
8. K. F. Lee, "Context-dependent phonetic hidden Markov models for speaker-independent continuous speech recognition," IEEE Trans. Acoust., Speech, Signal Process., vol. 38, no. 4, pp. 599-609, Apr. 1990.
9. S. J. Young and P. C. Woodland, "The use of state tying in continuous speech recognition," in Proc. Eurospeech, 1993, vol. 3, pp. 2203-2206.
10. M. Y. Hwang and X. D. Huang, "Shared-distribution hidden Markov model for speech recognition," IEEE Trans. Speech Audio Process., vol. 1, pp. 414-420, Jan. 1993.
11. E. Bocchieri and B. Mak, "Subspace distribution clustering hidden Markov model," IEEE Trans. Speech Audio Process., vol. 9, no. 3, pp. 264-275, Mar. 2001.
12. J. R. Bellegarda and D. Nahamoo, "Tied mixture continuous parameter modeling for speech recognition," IEEE Trans. Acoust., Speech, Signal Process., vol. 38, no. 12, pp. 2033-2045, Dec. 1990.
13. X. Huang and M. A. Jack, "Semi-continuous hidden Markov models for speech signals," Comput. Speech Lang., vol. 3, no. 3, pp. 239-251, Jul. 1989.
14. D. Povey, L. Burget, M. Agarwal, P. Akyazi, K. Feng, A. Ghoshal, O. Glembek, N. K. Goel, M. Karafiát, and A. Rastrow et al., "Subspace Gaussian mixture models for speech recognition," in Proc. ICASSP, 2010, pp. 4330-4333.
15. G. Saon and J.-T. Chien, "Bayesian sensing hidden Markov models," IEEE Trans. Audio, Speech, Lang. Process., vol. 20, no. 1, pp. 43-54, Jan. 2012.
16. M. J. F. Gales and K. Yu, "Canonical state models for automatic speech recognition," in Proc. Interspeech, 2010, pp. 58-61.
17. S. J. Pan and Q. Yang, "A survey on transfer learning," IEEE Trans. Knowl. Data Eng., vol. 22, no. 10, pp. 1345-1359, 2010.
18. W. Byrne, P. Beyerlein, J. M. Huerta, S. Khudanpur, B. Marthi, J. Morgan, N. Peterek, J. Picone, D. Vergyri, and T. Wang, "Towards language independent acoustic modeling," in Proc. ICASSP, 2000, vol. 2, pp. 1029-1032.
19. V. Le and L. Besacier, "Automatic speech recognition for under-resourced languages: Application to Vietnamese language," IEEE Trans. Audio, Speech, Lang. Process., vol. 17, no. 8, pp. 1471-1482, Nov. 2009.
20. J. Kohler, "Multi-lingual phoneme recognition exploiting acoustic-phonetic similarities of sounds," in Proc. ICSLP, 1996.
21. A. Ghoshal, P. Swietojanski, and S. Renals, "Multilingual training of deep-neural networks," in Proc. ICASSP, 2013, pp. 7319-7323.
22. J.-T. Huang, J. Li, D. Yu, L. Deng, and Y. Gong, "Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers," in Proc. ICASSP, 2013, pp. 7304-7308.
23. J. Kohler, "Language adaptation of multilingual phone models for vocabulary independent speech recognition tasks," in Proc. ICASSP, 1998, vol. 1, pp. 417-420.
24. F. Grezl, M. Karafiat, and K. Vesely, "Adaptation of multilingual stacked bottle-neck neural network structure for new language," in Proc. ICASSP, 2014, pp. 7654-7658.
25. D. Imseng, H. Bourlard, J. Dines, P. Garner, and M. Magimai-Doss, "Applying multi-and cross-lingual stochastic phone space transformations to non-native speech recognition," IEEE Trans. Audio, Speech, Lang. Process., vol. 21, no. 8, pp. 1713-1726, Aug. 2013.
26. N. T. Vu, D. Imseng, D. Povey, P. Motlicek, T. Schultz, and H. Bourlard, "Multilingual deep neural network based acoustic modeling for rapid language adaptation," in Proc. ICASSP, 2014, pp. 7639-7643.
27. X. Cui, V. Goel, and B. Kingsbury, "Data augmentation for deep neural network acoustic modeling," in Proc. ICASSP, 2014, pp. 5582-5586.
28. L. Burget, P. Schwarz, M. Agarwal, P. Akyazi, K. Feng, A. Ghoshal, O. Glembek, N. Goel, M. Karafiat, D. Povey, A. Rastrow, R. Rose, and S. Thomas, "Multilingual acoustic modeling for speech recognition based on subspace Gaussian mixture models," in Proc. ICASSP, Mar. 2010, pp. 4334-4337.
29. P. Cohen, S. Dharanipragada, J. Gros, M. Monkowski, C. Neti, S. Roukos, and T. Ward, "Towards a universal speech recognizer for multiple languages," in Proc. IEEE ASRU, 1997, pp. 591-598.
30. H. Lin, L. Deng, D. Yu, Y.-F. Gong, A. Acero, and C.-H. Lee, "A study on multilingual acoustic modeling for large vocabulary ASR," in Proc. ICASSP, Apr. 2009, pp. 4333-4336.
31. R. Caruana, "Multitask learning," Ph.D. dissertation, Carnegie Mellon Univ., Pittsburgh, PA, USA, 1997.
32. A. Mohamed, G. Dahl, and G. E. Hinton, "Acoustic modeling using deep belief networks," IEEE Trans. Audio, Speech, Lang. Process., vol. 20, no. 1, pp. 14-22, Jan. 2012.
33. D. Chen, B. Mak, C.-C. Leung, and S. Sivadas, "Joint acoustic modeling of triphones and trigraphemes by multi-task learning deep neural networks for low-resource speech recognition," in Proc. ICASSP, 2014, pp. 5992-5296.
34. S. Thrun and L. Pratt, Learning to Learn. Norwell, MA, USA: Kluwer, Nov. 1997.
35. J. Baxter, "A model of inductive bias learning," J. Artif. Intell. Res., vol. 12, pp. 149-198, 2000.
36. S. Ben-David and R. Schuller, "Exploiting task relatedness for multiple task learning," in Proc. COLT, 2003, pp. 567-580.
37. R. Collobert and J. Weston, "A unified architecture for natural language processing: Deep neural networks with multitask learning," in Proc. ICML, 2008, pp. 160-167, ACM.
38. G. Tur, "Multitask learning for spoken language understanding," in Proc. ICASSP, 2006, pp. 585-588.
39. Y. Huang, W. Wang, L. Wang, and T. Tan, "Multi-task deep neural network for multi-label learning," in Proc. ICIP, 2013, pp. 2897-2900.
40. S. Parveen and P. D. Green, "Multitask learning in connectionist ASR using recurrent neural networks," in Proc. Eurospeech, 2003, pp. 1813-1816.
41. M. Seltzer and J. Droppo, "Multi-task learning in deep neural networks for improved phoneme recognition," in Proc. ICASSP, 2013, pp. 6965-6968.
42. G. Heigold, V. Vanhoucke, A. Senior, P. Nguyen, M. Ranzato, M. Devin, and J. Dean, "Multilingual acoustic models using distributed deep neural networks," in Proc. ICASSP, 2013, pp. 8619-8623.
43. J. G. Fiscus, "A post-processing system to yield reduced word error rates: Recognizer output voting error reduction (ROVER)," in Proc. IEEE ASRU, 1997, pp. 347-354.
44. I. P. Association, Handbook of the International Phonetic Association: A guide to the use of the International Phonetic Alphabet. Cambridge, U.K.: Cambridge Univ. Press, 1999.
45. Meraka-Institute, Lwazi ASR corpus [Online]. Available: http://www.meraka.org.za/lwazi 2009
46. "Lwazi phone set," [Online]. Available: ftp://hlt.mirror.ac.za/Phoneset/Lwazi.Phoneset.1.2.pdf 2009
47. T. Evgeniou and M. Pontil, "Regularized multi-task learning," in Proc. 10th ACM SIGKDD Int. Conf. Knowl. Discov. Data Mining. ACM, 2004, pp. 109-117.
48. M. M. Tempest, "Dictionarymaker 2.16 user manual," [Online]. Available: http://dictionarymaker.sourceforge.net/2009
49. G. E. Hinton, S. Osindero, and Y. Teh, "A fast learning algorithm for deep belief nets," Neural Comput., vol. 18, no. 7, pp. 1527-1554, 2006.
50. G. E. Dahl, D. Yu, L. Deng, and A. Acero, "Context-dependent pre-trained deep neural networks for large vocabulary speech recognition," IEEE Trans. Audio, Speech, Lang. Process., vol. 20, no. 1, pp. 30-42, Jan. 2012.
51. V. Zue, S. Seneff, and J. Glass, "Speech database development at MIT: TIMIT and beyond," Speech Commun., vol. 9, no. 4, pp. 351-356, Aug. 1990.
52. N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov, "Dropout: A simple way to prevent neural networks from overfitting," J. Mach. Learn. Res., vol. 15, pp. 1929-1958, 2014.
53. Y. Miao and F. Metze, "Improving low-resource CD-DNN-HMM using dropout and multilingual DNN training," in Proc. ICASSP, 2013, pp. 7304-7308.
54. D. Yu, F. Seide, G. Li, and L. Deng, "Exploiting sparseness in deep neural networks for large vocabulary speech recognition," in Proc. ICASSP, 2012, pp. 4409-4412.
55. Y. Tang, "Deep learning using linear support vector machines," arXiv:1306.0239v3 [cs.LG], 2013.
56. R. Min, Z. Yuan, D. A. Stanley, A. Bonner, and Z. Zhang, "A deep nonlinear feature mapping for large-margin kNN classification," in Proc. ICDM, 2009, pp. 357-366.
57. Y. Zhang and D.-Y. Yeung, "A convex formulation for learning task relationships in multi-task learning," in Proc. 26th Conf. UAI, Jul. 2010.
58. W. Zhong and J. Kwok, "Convex multitask learning with flexible task clusters," in Proc. ICML, 2012.