Learning hidden unit contributions for unsupervised acoustic model adaptation

IEEE/ACM Transactions on Audio Speech and Language Processing

Volumn 24, Issue 8, 2016, Pages 1450-1463

  Swietojanski, Pawel ^a   Li, Jinyu ^b   Renals, Steve ^a  

^a UNIVERSITY OF EDINBURGH   (United Kingdom)

^b MICROSOFT   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ACOUSTIC MODEL ADAPTATION; ADAPTATION TECHNIQUES; FEATURE EXTRACTOR; SPEAKER ADAPTIVE TRAININGS; SPEAKER DEPENDENTS; SPEAKER INDEPENDENTS; UNSUPERVISED ADAPTATION; WORD ERROR RATE REDUCTIONS;

SPEECH RECOGNITION;

EID: 84976435936     PISSN: 23299290     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASLP.2016.2560534     Document Type: Article

References (84)

1. G. Hinton et al., "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups, " IEEE Signal Process. Mag., vol. 29, no. 6, pp. 82-97, Nov. 2012
2. H. Bourlard and N. Morgan, Connectionist Speech Recognition: A Hybrid Approach. Norwell, MA, USA: Kluwer, 1994
3. S. Renals, N. Morgan, H. Bourlard, M. Cohen, and H. Franco, "Connectionist probability estimators in HMM speech recognition, " IEEE Trans Speech Audio Process, vol. 2, no. 1, pp. 161-174, Jan. 1994
4. F. Seide, X. Chen, and D. Yu, "Feature engineering in context-dependent deep neural networks for conversational speech transcription, " in Proc. 12th IEEE Workshop Automatic Speech Recog. Understanding, 2011, pp. 24-29
5. G. Dahl, D. Yu, L. Deng, and A. Acero, "Context-dependent pretrained deep neural networks for large-vocabulary speech recognition, " IEEE Trans. Audio, Speech, Lang. Process., vol. 20, no. 1, pp. 30-42, Jan. 2012
6. H. Hermansky, D. Ellis, and S. Sharma, "Tandem connectionist feature extraction for conventional HMM systems, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2000, pp. 1635-1638
7. F.Grezl, M. Karafiat, S.Kontar, and J. Cernocky, "Probabilistic and bottleneck features for LVCSR of meetings, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2007, pp. IV-757-IV-760
8. T. Sainath, A.Mohamed, B. Kingsbury, and B. Ramabhadran, "Deep convolutional neural networks for LVCSR, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2013, pp. 8614-8618
9. P. Swietojanski, A. Ghoshal, and S. Renals, "Revisiting hybrid and GMMHMM system combination techniques, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2013, pp. 6744-6748
10. P. Woodland et al., "Cambridge University transcription systems for the multi-genre broadcast challenge, " in Proc. IEEE Workshop Automatic Speech Recog. Understanding, 2015, pp. 639-646
11. P. Woodland, "Speaker adaptation for continuous density HMMs: A review, " in Proc. ISCAWorkshop AdaptationMethods Speech Recog., 2001, pp. 11-19
12. O. Abdel-Hamid and H. Jiang, "Rapid and effective speaker adaptation of convolutional neural network based models for speech recognition, " in Proc. 14th Annu. Conf. Int. SpeechCommun. Assoc., 2013, pp. 1248-1252
13. P. Swietojanski and S. Renals, "Learning hidden unit contributions for unsupervised speaker adaptation of neural network acoustic models, " in Proc. IEEE Spoken Lang. Technol. Workshop, 2014, pp. 171-176
14. P. Swietojanski and S. Renals, "SAT-LHUC: Speaker adaptive training for learning hidden unit contributions, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2016, pp. 5010-5014
15. M. Cettolo, C. Girardi, and M. Federico, "Wit3 : Web inventory of transcribed and translated talks, " in Proc. 16th Conf. Eur. Assoc. Mach. Translation, 2012, pp. 261-268
16. J. Carletta, "Unleashing the killer corpus: Experiences in creating the multi-everything AMI meeting corpus." Language Resources Eval., vol. 41, no. 2, pp. 181-190, 2007
17. J. Godfrey, E. Holliman, and J. McDaniel, "SWITCHBOARD: Telephone speech corpus for research and development, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 1992, pp. 517-520
18. N. Parihar, J. Picone, D. Pearce, and H. Hirsch, "Performance analysis of the Aurora large vocabulary baseline system, " in Proc. 12th Eur. Signal Process. Conf., 2004, pp. 553-556
19. M. Gales, "Maximum likelihood linear transformations for HMMbased speech recognition, " Comput. Speech Lang., vol. 12, pp. 75-98, Apr. 1998
20. A. Mohamed, T. Sainath, G. Dahl, B. Ramabhadran, G. Hinton, and M. Picheny, "Deep belief networks using discriminative features for phone recognition, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2011, pp. 5060-5063
21. T. Hain, et al., "Transcribing meetings with the AMIDA systems, " IEEE Trans. Audio, Speech, Lang. Process., vol. 20, no. 2, pp. 486-498, Feb. 2012
22. T. Sainath, B. Kingsbury, and B. Ramabhadran, "Auto-encoder bottleneck features using deep belief networks." in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2012, pp. 4153-4156
23. P. Bell, P. Swietojanski, and S. Renals, "Multi-level adaptive networks in tandem and hybrid ASR systems, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2013, pp. 6975-6979
24. J. Neto, L. Almeida, M. Hochberg, C. Martins, L. Nunes, S. Renals, and T. Robinson, "Speaker adaptation for hybrid HMM-ANN continuous speech recognition system, " in Proc. 4th Eur. Conf. Speech Commun. Technol., 1995, pp. 2171-2174
25. V. Abrash, H. Franco, A. Sankar, and M. Cohen, "Connectionist speaker normalization and adaptation, " in Proc. 4th Eur. Conf. Speech Commun. Technol., 1995, pp. 2183-2186
26. B. Li and K. Sim, "Comparison of discriminative input and output transformations for speaker adaptation in the hybrid nn/hmm systems, " in Proc. 11th Annu. Conf. Int. Speech Commun. Assoc., 2010
27. N. Dehak, P. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet, "Front end factor analysis for speaker verification, " IEEE Trans. Audio, Speech, Lang. Process., vol. 19, no. 4, pp. 788-798, May 2010
28. M. Karafiat, L. Burget, P. Matejka, O. Glembek, and J. Cernozky, "iVectorbased discriminative adaptation for automatic speech recognition, " in Proc. IEEE Workshop Automatic Speech Recog. Understanding, 2011, pp. 152-157
29. G. Saon, H. Soltau, D. Nahamoo, and M. Picheny, "Speaker adaptation of neural network acoustic models using i-vectors." in Proc. IEEE Automatic Speech Recog. Understanding, 2013, pp. 55-59. [Online].Available: http://dblp.uni-Trier.de/db/conf/asru/asru2013.html#SaonSNP13
30. V. Gupta, P. Kenny, P. Ouellet, and T. Stafylakis, "I-vector based speaker adaptation of deep neural networks for french broadcast audio transcription, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2014, pp. 6334-6338
31. P. Karanasou, Y. Wang, M. Gales, and P. Woodland, "Adaptation of deep neural network acoustic models using factorised i-vectors, " in Proc. Annu. Conf. Int. Speech Commun. Assoc., 2014, pp. 2180-2184
32. Y. Miao, H. Zhang, and F. Metze, "Speaker adaptive training of deep neural network acoustic models using i-vectors, " IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 23, no. 11, pp. 1938-1949, Nov. 2015
33. Y. Liu, P. Zhang, and T. Hain, "Using neural network front-ends on far field multiple microphones based speech recognition, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2014, pp. 5542-5546
34. J. Bridle and S. Cox, "Recnorm: Simultaneous normalisation and classification applied to speech recognition, " in Proc. Adv. Neural Inf. Process Sys 3, 1990, pp. 234-240. [Online]. Available: http://papers.nips.cc/paper/328-recnorm-simultaneous-normalisationand-classification-applied-To-speech-recognition.pdf
35. O. Abdel-Hamid and H. Jiang, "Fast speaker adaptation of hybrid NN/HMM model for speech recognition based on discriminative learning of speaker code, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2013, pp. 4277-4280
36. S. Xue, O. Abdel-Hamid, J. Hui, L. Dai, and Q. Liu, "Fast adaptation of deep neural network based on discriminant codes for speech recognition, " IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 22, no. 12, pp. 1713-1725, Dec. 2014
37. H. Liao, "Speaker adaptation of context dependent deep neural networks, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2013, pp. 7947-7951
38. D. Yu, K. Yao, H. Su, G. Li, and F. Seide, "KL-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition." in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2013, pp. 7893-7897
39. Y. Huang and Y. Gong, "Regularized sequence-level deep neural network model adaptation, " in Proc. Annu. Conf. Int. Speech Commun. Assoc., 2015, pp. 1081-1085
40. J. Xue, J. Li, D. Yu, M. Seltzer, and Y. Gong, "Singular value decomposition based low-footprint speaker adaptation and personalization for deep neural network, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2014, pp. 6359-6363
41. L. Samarakoon and K. C. Sim, "Learning factorized feature transforms for speaker normalization, " in Proc. IEEE Workshop Automatic Speech Recog. Understanding, 2015, pp. 145-152
42. T. Ochiai, S. Matsuda, X. Lu, C. Hori, and S. Katagiri, "Speaker adaptive training using deep neural networks, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2014, pp. 6349-6353
43. K. Yao, D. Yu, F. Seide, H. Su, L. Deng, and Y. Gong, "Adaptation of context-dependent deep neural networks for automatic speech recognition, " in Proc. IEEE Spoken Language Technol. Workshop, 2012, pp. 366-369
44. Y. Zhao, J. Li, J. Xue, and Y. Gong, "Investigating online lowfootprint speaker adaptation using generalized linear regression and clickthrough data, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2015, pp. 4310-4314
45. P. Swietojanski and S. Renals, "Differentiable pooling for unsupervised speaker adaptation, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2015, pp. 4305-4309
46. S. Siniscalchi, J. Li, and C. Lee, "Hermitian polynomial for speaker adaptation of connectionist speech recognition systems, " IEEE Trans. Audio, Speech, Lang. Process., vol. 21, no. 10, pp. 2152-2161, Oct. 2013
47. Z. Huang, S. M. Siniscalchi, I.-F. Chen, J.Wu, and C.-H. Lee, "Maximum a posteriori adaptation of network parameters in deep models, " in Proc. 16th Annu. Conf. Int. Speech Commun. Assoc., 2015, pp. 1076-1080
48. Z. Huang, J. Li, S.M. Siniscalchi, I.-F. Chen, J.Wu, and C.-H. Lee, "Rapid adaptation for deep neural networks through multi-Task learning, " in Proc. 16th Annu. Conf. Int. Speech Commun. Assoc., 2015, pp. 3625-3629
49. P. Swietojanski, P. Bell, and S. Renals, "Structured output layer with auxiliary targets for context-dependent acoustic modelling, " in Proc.Annu. Conf. Int. Speech Commun. Assoc., 2015, pp. 3605-3609
50. R. Price, K. Iso, and K. Shinoda, "Speaker adaptation of deep neural networks using a hierarchy of output layers, " in Proc. IEEE Spoken Language Technol. Workshop, 2014, pp. 153-158
51. K. Hornik, M. Stinchcombe, and H. White, "Multilayer feedforward networks are universal approximators, " Neural Netw., vol. 2, no. 5, pp. 359-366, 1989
52. K. Hornik, "Approximation capabilities of multilayer feedforward networks, " Neural Netw., vol. 4, no. 2, pp. 251-257, 1991
53. A. Barron, "Universal approximation bounds for superpositions of a sigmoidal function, " IEEE Trans. Inf. Theory, vol. 39, no. 3, pp. 930-945, May 1993
54. C. Zhang and P. Woodland, "Parameterised sigmoid and ReLU hidden activation functions for DNN acoustic modelling, " in Proc. 16th Annu. Conf. Int. Speech Commun. Assoc., 2015, pp. 3224-3228
55. E. Trentin, "Networks with trainable amplitude of activation functions, " Neural Netw., vol. 14, pp. 471-493, 2001
56. C. Wu and M. Gales, "Multi-basis adaptive neural network for rapid adaptation in speech recognition, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2015, pp. 4315-4319
57. T. Tan, Y. Qian, M. Yin, Y. Zhuang, and K. Yu, "Cluster adaptive training for deep neural network, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2015, pp. 4325-4329
58. M. Delcroix, K. Kinoshita, T. Hori, and T. Nakatani, "Context adaptive deep neural networks for fast acoustic model adaptation, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2015, pp. 4535-4539
59. T. Anastasakos, J. McDonough, R. Schwartz, and J. Makhoul, "A compact model for speaker-adaptive training, " in Proc. 4th Int. Conf. Spoken Language, 1996, pp. 1137-1140
60. S. Renals, T. Hain, and H. Bourlard, "Recognition and understanding of meetings: The AMI and AMIDA projects, " in Proc. IEEE Workshop Aut. Speech Recog. Understanding, Dec. 2007, pp. 238-247
61. M. Federico, M. Cettolo, L. Bentivogli, M. Paul, and S. Stüker, "Overview of the IWSLT 2012 evaluation campaign, " in Proc. 9th Int. Workshop Spoken Language Translation, 2012, pp. 12-33
62. P. Bell, P. Swietojanski, J. Driesen, M. Sinclair, F.McInnes, and S. Renals, "The UEDIN ASR systems for the IWSLT 2014 evaluation, " in Proc. Int. Workshop Spoken Language Translation, 2014, pp. 26-33
63. P. Swietojanski, A. Ghoshal, and S. Renals, "Hybrid acoustic models for distant and multichannel large vocabulary speech recognition, " in Proc. IEEE Workshop Aut. Speech Recog. Understanding, 2013, pp. 285-290
64. T. Sainath et al., "Improvements to deep convolutional neural networks for LVCSR, " in Proc. IEEEWorkshop Aut. Speech Recog. Understanding, 2013, pp. 315-320
65. Y. LeCun, L. Bottou, G. Orr, and K. M?uller, "Efficient backprop, " in Neural Networks: Tricks of the Trade. New York, NY, USA: Springer, 1998, ch. 2
66. O. Abdel-Hamid, A.-R. Mohamed, J. Hui, and G. Penn, "Applying convolutional neural networks concepts to hybrid NN-HMM model for speech recognition." in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2012, pp. 4277-4280
67. P. Swietojanski, A. Ghoshal, and S. Renals, "Convolutional neural networks for distant speech recognition, " IEEE Signal Process. Lett., vol. 21, no. 9, pp. 1120-1124, Sep. 2014
68. K.Vesely, A.Ghoshal, L. Burget, andD. Povey, "Sequence-discriminative training of deep neural networks, " in Proc. Annu. Conf. Int. Speech Commun. Assoc., Lyon, France, Aug. 2013, pp. 2345-2349
69. D. Povey, et al., "The Kaldi speech recognition toolkit, " in Proc. IEEE Workshop Aut. Speech Recog. Understanding, Dec. 2011
70. S. Renals, N. Morgan, M. Cohen, and H. Franco, "Connectionist probability estimation in the DECIPHER speech recognition system, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 1992, pp. 601-604
71. R. French, "Catastrophic forgetting in connectionist networks: Causes, consequences and solutions, " Trends Cognitive Sci., vol. 3, pp. 128-135, 1999. [Online]. Available: http://citeseerx.ist.psu. edu/viewdoc/summary?.doi=10.1.1.36.3676
72. J. Kaiser, B. Horvat, and Z. Kacic, "A novel loss function for the overall risk criterion based discriminative training of HMMmodels, " in Proc. 6th Int. Conf. Spoken Language Process., 2000, pp. 887-890
73. B. Kingsbury, "Lattice-based optimization of sequence classification criteria for neural-network acoustic modeling, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2009, pp. 3761-3764
74. C.-L. Huang, P. Dixon, S. Matsuda, Y. Wu, X. Lu, M. Saiko, and C. Hori, "The NICT ASR system for IWSLT 2013, " in Proc. Int.Workshop Spoken Language Translation, 2013
75. I. Himawan, P. Motlicek, M. Ferras, and S. Madikeri, "Towards utterancebased neural network adaptation in acoustic modeling, " in Proc. IEEE Workshop Aut. Speech Recog. Understanding, 2015, pp. 289-295
76. L. Samarakoon andK.C. Sim, "On combining i-vectors and discriminative adaptation methods for unsupervised speaker normalization in dnn acoustic models, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2016, pp. 5275-5279
77. M.Wester, Z.Wu, and J. Yamagishi, "Human vs machine spoofing detection on wideband and narrowband data, " in Proc. Annu. Conf. Int. Speech Commun. Assoc., 2015, pp. 2047-2051
78. J. Fiscus, W. Fisher, A. Martin, M. Przybocki, and D. Pallett, "2000 NIST evaluation of conversational speech recognition over the telephone: English and Mandarin performance results, " in Proc. Speech Transcription Workshop, 2000
79. J. Li, J.-T. Huang, and Y. Gong, "Factorized adaptation for deep neural network, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2014, pp. 5537-5541
80. M. Seltzer, D. Yu, and Y.Wang, "An investigation of deep neural networks for noise robust speech recognition, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2013, pp. 7398-7402
81. S. Rennie, V. Goel, and S. Thomas, "Annealed dropout training of deep networks, " in Proc. IEEE Spoken Language Technology Workshop, 2014, pp. 159-164
82. I.J. Goodfellow, D.Warde-Farley, M.Mirza, A. Courville, and Y. Bengio, "Maxout networks, " Proc. Int. Conf. Mach. Learn., JMLR, 2013, pp. 1319-1327
83. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: 2014 A simple way to prevent neural networks from overfitting, " J. Mach. Learn. Res, 15, pp. 1929-1958, . [Online]. Available: http://jmlr.org/papers/v15/srivastava14a.html
84. L. Van der Maaten and G. Hinton, "Visualizing high-dimensional data using t-sne, " J. Mach. Learn. Res., vol. 9, pp. 2579-2605, Nov. 2008