Parameterised sigmoid and ReLU hidden activation functions for DNN acoustic modelling

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Volumn 2015-January, Issue , 2015, Pages 3224-3228

  Zhang, C ^a   Woodland, P C ^a  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

CHEMICAL ACTIVATION; COMPUTER VISION; PARAMETERIZATION; SPEECH COMMUNICATION;

ACOUSTIC MODEL TRAININGS; ACOUSTIC MODELLING; ACTIVATION FUNCTIONS; ADAPTIVE PARAMETERS; CONVERSATIONAL TELEPHONE SPEECH; DEEP NEURAL NETWORKS; SPEAKER ADAPTATION; WORD ERROR RATE;

SPEECH RECOGNITION;

EID: 84959174678     PISSN: 2308457X     EISSN: 19909772     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (29)

1. D. E. Rumelhart, J. L. McClelland, and the PDP ResearchGroup, Parallel Distributed Processing: Explorationsin the Microstructure of Cognition, Volume 1: Foundations. MIT Press, 1986.
2. C. M. Bishop, Neural Networks for Pattern Recognition. Oxford University Press, 1995.
3. X. Glorot and Y. Bengio, "Understand ing the difficultyof training deep feedforward neural networks, " Proc. Internationalconference on artificial intelligence and statistics, 2010.
4. G. E. Hinton and R. Salakhutdinov, "Reducing the dimensionalityof data with neural networks, " Science, Vol. 313, no. 5786, pp. 504-507, Jul 2006.
5. X. Glorot, A. Bordes, and Y. Bengio, "Deep sparse rectifiernetworks, " Proc. AISTATS'11, Ft. Lauderdale, FL, USA, 2011.
6. V. Nair and G. E. Hinton, "Rectified linear units improverestricted boltzmann machines, " Proc. ICML'10, Haifa, Isreal, 2010
7. M. D. Zeiler, M. Ranzato, R. Monga, M. Mao, K. Yang, Q. V. Le, P. Nguyen, A. Senior, V. Vanhoucke, J. Deanet al., "On rectified linear units for speech processing, "Proc. ICASSP'13, Vancouver, Canada, 2013.
8. G. E. Dahl, T. N. Sainath, and G. E. Hinton, "Improvingdeep neural networks for LVCSR using rectified linearunits and dropout, " Proc. ICASSP'13, Vancouver, Canada, 2013.
9. A. L. Maas, A. Y. Hannun, and A. Y. Ng, "Rectifiernonlinearities improve neural network acoustic models, "Proc. ICML'13, Atlanta, GA, USA, 2013.
10. A. Senior and X. Lei, "Fine context, low-rank, softplusdeep neural networks for mobile speech recognition, "Proc. ICASSP'14, Florence, Italy, 2014
11. H. Liao, E. McDermott, and A. Senior, "Large scale deepneural network acoustic modeling with semi-supervisedtraining data for YouTube video transcription, " Proc. ASRU'13, Olomouc, Czech Republic, 2013.
12. S. M. Siniscalchi, J.-Y. Li, and C.-H. Lee, "Hermitianpolynomial for speaker adaptation of connectionistspeech recognition systems, " IEEE Transactions on Audio, Speech, and Language Processing, Vol. 21, No. 10, pp. 2152-2161, 2013.
13. P. Swietojanski and S. Renals, "Learning hidden unit contributionsfor unsupervised speaker adaptation of neuralnetwork acoustic models, " Proc. IWSLT'14, Lake Tahoe, USA, 2014.
14. K. He, X. Zhang, S. Ren, and J. Sun, "Delving deep intorectifiers: Surpassing human-level performance on imagenetclassification, " arXiv preprint arXiv: 1502. 01852, 2015.
15. G. E. Hinton, L. Deng, D. Yu, G. E. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury, "Deep neural networks foracoustic modeling in speech recognition, " IEEE SignalProcessing Magazine, pp. 2-17, Nov. 2012.
16. H. A. Bourlard and N. Morgan, Connectionist SpeechRecognition: A Hybrid Approach. Kluwer Academic Publishers, 1993.
17. B.-H. Juang, W. Chou, and C.-H. Lee, "Minimum classificationerror rate methods for speech recognition, " IEEETransactions on Speech and Audio Processing, Vol. 5, No. 3, pp. 257-265, 1997.
18. J. Barker, L. Josifovski, M. Cooke, and P. D. Green, "Soft decisions in missing data techniques for robust automaticspeech recognition, " Proc. Interspeech'00, Beijing, China, 2000.
19. Y. Bengio, P. Simard, and P. Frasconi, "Learning longtermdependencies with gradient descent is difficult, "IEEE Transactions on Neural Networks, Vol. 5, No. 2, pp. 157-166, 1994.
20. T. Mikolov, "Statistical language models based on neuralnetworks, " Ph. D. dissertation, Ph. D. Thesis, Brno Universityof Technology, 2012.
21. X.-Y. Liu, F. Flego, L.-L. Wang, C. Zhang, M. J. F. Gales, and P. C. Woodland, "The Cambridge University2014 BOLT conversational telephone Mand arin ChineseLVCSR system for speech translation, " Proc. Interspeech'15, Dresden, Germany, 2015.
22. N. Kumar, "Investigation of silicon-auditory models and generalization of linear discriminant analysis for improvedspeech recognition, " Ph. D. dissertation, John HopkinsUniversity, Baltimore, MD, USA, 1997.
23. X.-Y. Liu, M. J. F. Gales, and P. C. Woodland, "Automaticcomplexity control for HLDA systems, " Proc. ICASSP'03, Hong Kong, 2003.
24. P. Ghahremani, B. BabaAli, D. Povey, K. Riedhammer, J. Trmal, and S. Khudanpur, "A pitch extraction algorithmtuned for automatic speech recognition, " Proc. ICASSP'14, Florence, Italy, 2014.
25. D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlcek, Y.-M. Qian, P. Schwarz, J. Silovský, G. Stemmer, and K. Veselý, "The Kaldi speech recognition toolkit, " Proc. ASRU'11, Waikoloa, HI, USA, 2011.
26. S. J. Young, J. J. Odell, and P. C. Woodland, "Tree-basedstate tying for high accuracy acoustic modelling, " Proc. Human Language Technology Workshop. Plainsboro, NJ, USA. Morgan Kaufman Publishers Inc, 1994.
27. C. Zhang and P. C. Woodland, "A general artificial neuralnetwork extension for HTK, " Proc. Interspeech'15, Dresden, Germany, 2015.
28. F. Seide, G. Li, X. Chen, and D. Yu, "Feature engineeringin context-dependent deep neural networks for conversationalspeech transcription, " Proc. ASRU'11, Waikoloa, HI, USA, 2011.
29. S. J. Young, G. Evermann, M. J. F. Gales, T. Hain., D. Kershaw, X.-Y. Liu, G. Moore, J. J. Odell, D. Ollason, D. Povey, V. Valtchev, and P. C. Woodland, The HTKbook (for HTK version 3. 4). Cambridge, UK: CambridgeUniversity Engineering Department, 2006.