A unified approach to transfer learning of deep neural networks with applications to speaker adaptation in automatic speech recognition

Neurocomputing

Volumn 218, Issue , 2016, Pages 448-459

  Huang, Zhen ^a   Siniscalchi, Sabato Marco ^b   Lee, Chin Hui ^a  

^a GEORGIA INSTITUTE OF TECHNOLOGY   (United States)

^b KORE UNIVERSITY OF ENNA   (Italy)

Author keywords

Deep neural network; Multi task learning; Speaker adaptation; Transfer learning

Indexed keywords

DEEP LEARNING; DEEP NEURAL NETWORKS; LEARNING SYSTEMS; METADATA; MULTI-TASK LEARNING; NEURAL NETWORKS; SPEECH; TRANSFER LEARNING;

ADAPTATION ALGORITHMS; AUTOMATIC SPEECH RECOGNITION; AUTOMATIC SPEECH RECOGNITION SYSTEM; HIERARCHICAL STRUCTURES; MULTI-CONDITION TRAININGS; PERFORMANCE DEGRADATION; SPEAKER ADAPTATION; WORD ERROR RATE REDUCTIONS;

SPEECH RECOGNITION;

ARTICLE; ARTIFICIAL NEURAL NETWORK; AUTOMATIC SPEECH RECOGNITION; CONTROLLED STUDY; DEEP NEURAL NETWORK; INFORMATION PROCESSING; KERNEL METHOD; MEASUREMENT ACCURACY; MULTI CONDITION TRAINING; MULTI TASK HETEROGENEOUS TRANSFER LEARNING; PRIORITY JOURNAL; PROBABILITY; TASK PERFORMANCE; WALL STREET JOURNAL;

EID: 84994096935     PISSN: 09252312     EISSN: 18728286     Source Type: Journal    
DOI: 10.1016/j.neucom.2016.09.018     Document Type: Article

References (59)

1. [1] Li, J., Deng, L., Gong, Y., Haeb-Umbach, R., An overview of noise-robust automatic speech recognition. IEEE Trans. Audio, Speech, Lang. Process. 22:4 (2014), 745–777.
2. [2] Lee, C.-H., On stochastic feature and model compensation approaches to robust speech recognition. Speech Commun. 25:1–3 (1998), 29–47.
3. [3] Schultz, T., Waibel, A., Language independent and language adaptive acoustic modeling for speech recognition. Speech Commun. 35 (2001), 31–51.
4. [4] Siniscalchi, S.M., Lyu, D.-C., Svendsen, T., Lee, C.-H., Experiments on cross-language attribute detection and phone recognition with minimal target specific training data. IEEE Trans. Audio Speech Lang. Process. 20:3 (2012), 875–887.
5. [5] Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., Kingsbury, B., Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 29:6 (2012), 82–97.
6. [6] Hwang, M.-Y.M.-Y., Huang, X., Shared-distribution hidden markov models for speech recognition. IEEE Trans. Speech Audio Process. 1:4 (1993), 414–420.
7. [7] T. N. Sainath, B. Kingsbury, B. Ramabhadran, P. Fousek, P. Novak, A. Mohamed, Making deep belief networks effective for large vocabulary continuous speech recognition, in: Proc. ASRU, 2011, pp. 30–35.
8. [8] Dahl, G.E., Yu, D., Deng, L., Acero, A., Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. IEEE Trans. Audio, Speech Lang. Proc. 20:1 (2012), 30–42.
9. [9] K. Vesely`, A. Ghoshal, L. Burget, D. Povey, Sequence-discriminative training of deep neural networks, in: Proc. Interspeech, 2013, pp. 2345–2349.
10. [10] M. Franch, Catastrophic forgetting in connectionist networks: causes, consequences and solutions, Trends in Cognitive Sciences, vol. 3 (4).
11. [11] D. Yu, K. Yao, H. Su, G. Li, F. Seide, KL-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition, in: Proc. ICASSP, 2013, pp. 7893–7897.
12. [12] Gemello, R., Mana, F., Scanzio, S., Laface, P., De Mori, R., Linear hidden transformations for adaptation of hybrid ANN/HMM models. Speech Commun. 49:10–11 (2007), 827–835.
13. [13] Deng, L., Li, X., Machine learning paradigms for speech recognition: an overview. IEEE Trans. Audio, Speech Lang. Process. 21 (2013), 1060–1089.
14. [14] Kalinli, O., Seltzer, M.L., Droppo, J., Acero, A., Noise adaptive training for robust automatic speech recognition. IEEE Trans. Audio Speech Lang. Process. 18:8 (2010), 1889–1901.
15. [15] Pan, S.J., Yang, Q., A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22 (2010), 1245–1359.
16. [16] Gauvain, J., Lee, C., Maximum a posteriori estimation for multivariate gaussian mixture observations of Markov chains. IEEE Trans. Speech Audio Process. 2:2 (1994), 291–298.
17. [17] R. Caruna, Multitask learning: A knowledge-based source of inductive bias, in: Proc. ICML, 1993, pp. 41–48.
18. [18] S. Parveen, P. Green, Multitask learning in connectionist robust asr using recurrent neural networks., in: Proc. INTERSPEECH, 2003.
19. [19] Y. Lu, F. Lu, S. Sehgal, S. Gupta, J. Du, C. Tham, P. Green, V. Wan, Multitask learning in connectionist speech recognition, in: Proc. Australian International Conference on Speech Science and Technology, 2004.
20. [20] M. Seltzer, J. Droppo, Multi-task learning in deep neural networks for improved phoneme recognition, in: Proc. ICASSP, 2013, pp. 6965–6969.
21. [21] Y. Xu, J. Du, Z. Huang, L.-R. Dai, C.-H. Lee, Multi-objective learning and mask-based post-processing for deep neural network based speech enhancement, 2015, submitted to INTERSPEECH.
22. [22] D.B. Paul, J.M. Baker, The design for the wall street journal-based CSR corpus, in: Proc. Workshop on Speech and Natural Language, Banff, Canada, 1992, pp. 899–902.
23. [23] J. Neto, L. Almeida, M. Hochberg, C. Martins, L. Nunes, S. Renals, T. Robinson, Speaker-adaptation for hybrid HMM-ANN continuous speech recognition system, in: Proc. Eurospeech, Madrid, Spain, 1995, pp. 2171–2174.
24. [24] B. Li, K.C. Sim, Comparison of discriminative input and output transformations for speaker adaptation in the hybrid NN/HMM systems, in: Proc. INTERSPEECH, 2010, pp. 526–529.
25. [25] Rumelhart, D.E., Hinton, G.E., Williams, R.J., Learning representations by back-propagating errors. Cogn. Model., 5(3), 1988, 1.
26. [26] Z. Huang, S.M. Siniscalchi, I.-F. Chen, J. Li, J. Wu, C.-H. Lee, Maximum a posteriori adaptation of network parameters in deep models, in: Proc. Interspeech, 2015.
27. [27] Z. Huang, J. Li, S.M. Siniscalchi, I.-F. Chen, J. Wu, C.-H. Lee, Rapid adaptation for deep neural networks through multi-task learning, in: Proc. Interspeech, 2015.
28. [28] Haykin, S., Neural Networks: A Comprehensive Foundation. 1994, Macmillan.
29. [29] O. Dekel, R. Gilad-Bachrach, O. Shamir, L. Xiao, Optimal distributed online prediction, in: Proc. ICML, 2011, pp. 713–720.
30. [30] Hinton, G.E., Osindero, S., Teh, Y., A fast learning algorithm for deep belief nets. Neural Comput. 18:7 (2006), 1527–1554.
31. [31] X. Zhang, J. Trmal, D. Povey, S. Khudanpur, Improving deep neural network acoustic models using generalized maxout networks, in: Proc. ICASSP 2014, 2006, pp. 215–219.
32. [32] F. Seide, G. Li, D. Yu, Conversational speech transcription using context-dependent deep neural networks, in: Proc. Interspeech, Florence, Italy, 2011, pp. 437–440.
33. [33] Xue, S., Abdel-Hamid, O., Jiang, H., Dai, L., Liu, Q., Fast adaptation of deep neural network based on discriminant codes for speech recognition. IEEE/ACM Trans. Audio Speech Lang. Proc. 22:12 (2014), 1713–1725.
34. [34] S. Xue, H. Jiang, L. Dai, Speaker adaptation of hybrid NN/HMM model for speech recognition based on singular value decomposition, in: Proc. ISCSLP, 2014.
35. [35] A. Mohamed, T. Sainath, G. Dahl, B. Ramabhadran, G. Hinton, M. Picheny, Deep belief networks using discriminative features for phone recognition, in: Proc. ICASSP, 2011, p. 5060–5063.
36. [36] Hain, T., Burget, L., Dines, J., Garner, P., Grézl, F., Hannani, A.E., Karafíat, M., Lincoln, M., Wan, V., Transcribing meetings with the amida systems. IEEE Trans. Audio Speech Lang. Process. 20:2 (2012), 486–498.
37. [37] G. Saon, H. Soltau, D. Nahamoo, M. Picheny, Speaker adaptation of neural network acoustic models using i-vectors, in: Proc. ASRU, 2013, pp. 55–59.
38. [38] Lee, C.-H., Huo, Q., On adaptive decision rules and decision parameter adaptation for automatic speech recognition. Proc. IEEE 88:8 (2000), 1241–1269.
39. [39] Chomsky, N., Halle, M., The Sound Pattern of English. 1968, Harper & Row.
40. [40] J. Li, R. Zhao, J.-T. Huang, Y. Gong, Learning small-size dnn with output-distribution-based criteria, in: Proc. Interspeech, 2014.
41. [41] Heigold, G., Ney, H., Lehnen, P., Gass, T., Schluter, R., Equivalence of generative and log-linear models. IEEE Trans. Audio Speech Lang. Process. 19:5 (2011), 1138–1148.
42. [42] Shinoda, K., Lee, C.-H., A structural Bayes approach to speaker adaptation. IEEE Trans. Speech Audio Process. 9:3 (2001), 276–287.
43. [43] Bellegarda, J.R., Nahamoo, D., Tied mixture continuous parameter modeling for speech recognition. IEEE Trans. Acoust., Speech Signal Process. 38:12 (1990), 2033–2045.
44. [44] Huang, X., Jack, M.A., Semi-continuous hidden markov models for speech signal. Comput. Speech Lang. 3:3 (1989), 239–251.
45. [45] X. Li, X. Wu, Decision tree based state tying for speech recognition using DNN derived embeddings, in: Proc. ISCSLP, 2014, pp. 123–127.
46. [46] N. Srivastava, R. Salakhutdinov, Discriminative transfer learning with tree-based priors, in: Proc. NIST, 2013.
47. [47] Morris, J., Fosler-Lussier, E., Conditional random fields for integrating local discriminative classifiers. IEEE Trans. Audio Speech Lang. Process. 16:3 (2008), 617–628.
48. [48] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, J. Silovsky`, G. Stemmer, K. Vesely`, The Kaldi speech recognition toolkit, in: Proc. ASRU, 2011.
49. [49] S. Young, J. Odell, P. Woodland, Tree-based state tying for high accuracy acoustic modeling, in: Proc. ARPA Human Language Technology Workshop, Plainsboro, NJ, USA, 1994, pp. 307–312.
50. [50] Gauvain, J.-L., Lamel, L., Large vocabulary continuous speech recognition: advances and applications. Proc. IEEE 88:8 (2000), 1181–1200.
51. [51] T. N. Sainath, B. Kingsbury, V. Sindhwani, E. Arisoy, B. Ramabhadran, Low-rank matrix factorization for deep neural network training with high-dimensional output targets, in: Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, IEEE, 2013, pp. 6655–6659.
52. [52] J. Xue, J. Li, Y. Gong, Restructuring of deep neural network acoustic models with singular value decomposition, in: Proc. Interspeech 2014, 2013, pp. 2365–2369.
53. [53] Digalakis, V.V., Rtischev, D., Neumeye, L.G., Speaker adaptation using constrained estimation of gaussian mixtures. IEEE Trans. Speech Audio Process. 3:4 (1995), 357–366.
54. [54] Gales, M.J.F., Maximum likelihood linear transformations for HMM-based speech recognition. Comput. Speech Lang. 12 (1998), 75–98.
55. [55] M. Rosenstein, Z. Marx, L. Kaelbling, To transfer or not to transfer, in: Neural Information Processing Systems (NIPS ?05) Workshop Inductive Transfer: 10 Years Late, 2005.
56. [56] J. J. Godfrey, E. Holliman, Switchboard-1 release 2, Linguistic Data Consortium, Philadelphia.
57. [57] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, J. Silovsky`, G. Stemmer, K. Vesely`, The Kaldi speech recognition toolkit, in: Proc. ASRU, 2011.
58. [58] J. Fiscus, W.M. Fisher, A.F. Martin, M.A. Przybocki, D.S. Pallett, 2000 NIST evaluation of conversational speech recognition over the telephone: English and mandarin performance results, in: Proc. Speech Transcription Workshop, 2000.
59. [59] N.T. Vu, W. Breiter, F. Metze, T. Schultz, Initialization schemes for multilayer perceptron training and their impact on ASR performance using multilingual data, in: Proc. Interspeech, Portland, OR, USA, 2012.