Noisy training for deep neural networks in speech recognition

Eurasip Journal on Audio, Speech, and Music Processing

Volumn 2015, Issue 1, 2015, Pages 1-14

  Yin, Shi ^a,c   Liu, Chao ^a   Zhang, Zhiyong ^a,b   Lin, Yiye ^a,d   Wang, Dong ^a,b   Tejedor, Javier ^e   Zheng, Thomas Fang ^a,b   Li, Yinguo ^c  

^a TSINGHUA UNIVERSITY   (China)

^b TSINGHUA NATIONAL LABORATORY FOR INFORMATION SCIENCE AND TECHNOLOGY   (China)

^c CHONGQING UNIVERSITY OF POSTS AND TELECOMMUNICATIONS   (China)

^d BEIJING INSTITUTE OF TECHNOLOGY   (China)

^e UNIVERSITY OF ALCALÁ   (Spain)

Author keywords

Deep neural network; Noise injection; Speech recognition

Indexed keywords

COMPLEX NETWORKS; SPEECH;

ACOUSTIC CONDITIONS; COMPLEX PATTERN; DEEP NEURAL NETWORKS; NEURAL COMPUTATIONS; NOISE INJECTION; OBJECTIVE FUNCTIONS; PERFORMANCE DEGRADATION; SPEECH SIGNALS;

SPEECH RECOGNITION;

EID: 84922326458     PISSN: 16874714     EISSN: 16874722     Source Type: Journal    
DOI: 10.1186/s13636-014-0047-0     Document Type: Article

References (37)

1. L Deng, D Yu, Deep learning: methods and applications. Foundations Trends Signal Process. 7, 197–387 (2014).
2. H Bourlard, N Morgan, in Adaptive Processing of Sequences and Data Structures, ser. Lecture Notes in Artificial Intelligence (1387),Hybrid HMM/ANN systems for speech recognition: overview and new research directions (USA, 1998), pp. 389–417.
3. H Hermansky, DPW Ellis, S Sharma, in Proc. of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP),Tandem connectionist feature extraction for conventional HMM systems (Istanbul, Turkey, 9 June 2000), pp. 1635–1638.
4. GE Dahl, D Yu, L Deng, A Acero, in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),Large vocabulary continuous speech recognition with context-dependent DBN-HMMs (Prague, Czech Republic, 22 May 2011), pp. 4688–4691.
5. G Hinton, L Deng, D Yu, GE Dahl, A-r Mohamed, N Jaitly, A Senior, V Vanhoucke, P Nguyen, TN Sainath, B Kingsbury, Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 29(6), 82–97 (2012).
6. A Mohamed, G Dahl, G Hinton, in Proc. of Neural Information Processing Systems (NIPS) Workshop Deep Learning for Speech Recognition and Related Applications,Deep belief networks for phone recognition (Vancouver, BC, Canada, 7 December 2009).
7. GE Dahl, D Yu, L Deng, A Acero, Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. IEEE Trans. Audio Speech Lang. Process. 20(1), 30–42 (2012).
8. D Yu, L Deng, G Dahl, in Proc. of NIPS Workshop on Deep Learning and Unsupervised Feature Learning,Roles of pre-training and fine-tuning in context-dependent DBN-HMMs for real-world speech recognition (Vancouver, BC, Canada, 6 December, 2010).
9. N Jaitly, P Nguyen, AW Senior, V Vanhoucke, in Proc. of Interspeech,Application of pretrained deep neural networks to large vocabulary speech recognition (Portland, Oregon, USA, 9–13 September 2012), pp. 2578–2581.
10. TN Sainath, B Kingsbury, B Ramabhadran, P Fousek, P Novak, A-r Mohamed, in Proc. of IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU),Making deep belief networks effective for large vocabulary continuous speech recognition (Hawaii, USA, 11 December 2011), pp. 30–35.
11. TN Sainath, B Kingsbury, H Soltau, B Ramabhadran, Optimization techniques to improve training speed of deep belief networks for large speech tasks. IEEE Trans. Audio Speech Lang. Process. 21(1), 2267–2276 (2013).
12. F Seide, G Li, D Yu, in Proc. of Interspeech,Conversational speech transcription using context-dependent deep neural networks (Florence, Italy, 15 August 2011), pp. 437–440.
13. F Seide, G Li, X Chen, D Yu, in Proc. of IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU),Feature engineering in context-dependent deep neural networks for conversational speech transcription (Waikoloa, HI, USA, 11 December 2011), pp. 24–29.
14. O Vinyals, SV Ravuri, in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),Comparing multilayer perceptron to deep belief network tandem features for robust ASR (Prague, Czech Republic, 22 May 2011), pp. 4596–4599.
15. D Yu, ML Seltzer, in Proc. of Interspeech,Improved bottleneck features using pretrained deep neural networks (Florence, Italy, 15 August 2011), pp. 237–240.
16. P Bell, P Swietojanski, S Renals, in Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),Multi-level adaptive networks in tandem and hybrid ASR systems (Vancouver, BC, Canada, 26 May 2013), pp. 6975–6979.
17. F Grezl, s Fousek P, in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),Optimizing bottle-neck features for LVCSR (Las Vegas, USA, 4 April 2008), pp. 4729–4732.
18. P Lal, S King, Cross-lingual automatic speech recognition using tandem features. IEEE Trans. Audio Speech Lang. Process. 21(12), 2506–2515 (2011).
19. C Plahl, R Schlüter, H Ney, in Proc. of Interspeech,Hierarchical bottle neck features for LVCSR (Makuhari, Japan, 26 September 2010), pp. 1197–1200.
20. TN Sainath, B Kingsbury, B Ramabhadran, in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),Auto-encoder bottleneck features using deep belief networks (Kyoto, Japan, 25 March 2012), pp. 4153–4156.
21. Z Tüske, R Schlüter, H Ney, M Sundermeyer, in Proc. of Interspeech,Context-dependent MLPs for LVCSR: tandem, hybrid or both? (Portland, Oregon, USA, 9 September 2012), pp. 18–21.
22. D Imseng, P Motlicek, PN Garner, H Bourlard, in Proc. of IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU),Impact of deep MLP architecture on different acoustic modeling techniques for under-resourced speech recognition (Olomouc, Czech Republic, 8 December 2013), pp. 332–337.
23. J Qi, D Wang, J Xu, J Tejedor, in Proc. of Interspeech,Bottleneck features based on gammatone frequency cepstral coefficients (Lyon, France, 25 August 2013), pp. 1751–1755.
24. D Yu, ML Seltzer, J Li, J-T Huang, F Seide, in Proc. of International Conference on Learning Representations (ICLR),Feature learning in deep neural networks - a study on speech recognition tasks (Scottsdale, Arizona, USA, 2 May 2013).
25. B Li, KC Sim, in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),Noise adaptive front-end normalization based on vector Taylor series for deep neural networks in robust speech recognition (Vancouver, BC, Canada, 6 May 2013), pp. 7408–7412.
26. B Li, Y Tsao, KC Sim, in Proc. of Interspeech,An investigation of spectral restoration algorithms for deep neural networks based noise robust speech recognition (Lyon, France, 25 August 2013), pp. 3002–3006.
27. ML Seltzer, D Yu, Y Wang, in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),An investigation of deep neural networks for noise robust speech recognition (Vancouver, BC, Canada, 6 May 2013), pp. 7398–7402.
28. P Vincent, H Larochelle, Y Bengio, P-A Manzagol, in Proc. of the 25th International Conference on Machine Learning (ICML),Extracting and composing robust features with denoising autoencoders (Helsinki, Finland, 5 July 2008), pp. 1096–1103.
29. AL Maas, QV Le, O’Neil TM, O Vinyals, P Nguyen, AY Ng, in Proc. of Interspeech,Recurrent neural networks for noise reduction in robust ASR (Portland, Oregon, USA, 9 September 2012), pp. 22–25.
30. X Meng, C Liu, Z Zhang, D Wang, in Proc. of ChinaSIP 2014,Noisy training for deep neural networks (Xi‘an, China, 7 July 2014), pp. 16–20.
31. G An, The effects of adding noise during backpropagation training on a generalization performance. Neural Comput. 8(3), 643–674 (1996).
32. Y Grandvalet, S Canu, Comments on ‘noise injection into inputs in back propagation learning’. IEEE Trans. Syst. Man Cybernet. 25(4), 678–681 (1995).
33. CM Bishop, Training with noise is equivalent to Tikhonov regularization. Neural Comput. 7(1), 108–116 (1995).
34. Y Grandvalet, S Canu, S Boucheron, Noise injection: theoretical prospects. Neural Comput. 9(5), 1093–1108 (1997).
35. J Sietsma, RJF Dow, in Proc. of IEEE International Conference on Neural Networks,Neural net pruning-why and how (San Diego, California, USA, 24 July 1988), pp. 325–333.
36. K Matsuoka, Noise injection into inputs in back-propagation learning. IEEE Trans. Syst. Man Cybernet. 22(3), 436–440 (1992).
37. R Reed, RJ Marks, Seho Oh, Similarities of error regularization, sigmoid gain scaling, target smoothing, and training with jitter. IEEE Trans. Neural Netw. 6(3), 529–538 (1995).