Evaluating deep learning architectures for Speech Emotion Recognition

Neural Networks

Volumn 92, Issue , 2017, Pages 60-68

  Fayek, Haytham M ^a   Lech, Margaret ^a   Cavedon, Lawrence ^a  

^a RMIT UNIVERSITY   (Australia)

Author keywords

Affective computing; Deep learning; Emotion recognition; Neural networks; Speech recognition

Indexed keywords

DEEP LEARNING; DEEP NEURAL NETWORKS; NETWORK ARCHITECTURE; NEURAL NETWORKS; RECURRENT NEURAL NETWORKS; SPEECH PROCESSING;

AFFECTIVE COMPUTING; DYNAMIC CLASSIFICATION; EMOTION RECOGNITION; LEARNING ARCHITECTURES; QUANTITATIVE AND QUALITATIVE ASSESSMENTS; SPEAKER INDEPENDENTS; SPEECH EMOTION RECOGNITION; STATE OF THE ART;

SPEECH RECOGNITION;

ARTICLE; ARTIFICIAL NEURAL NETWORK; CONTROLLED STUDY; DEEP NEURAL NETWORK; DISCRIMINANT ANALYSIS; MACHINE LEARNING; MEASUREMENT ACCURACY; PRINCIPAL COMPONENT ANALYSIS; PRIORITY JOURNAL; QUALITATIVE ANALYSIS; QUANTITATIVE ANALYSIS; SPEECH EMOTION RECOGNITION; SUPPORT VECTOR MACHINE; AUTOMATIC SPEECH RECOGNITION; EMOTION;

EMOTIONS; MACHINE LEARNING; NEURAL NETWORKS (COMPUTER); SPEECH RECOGNITION SOFTWARE;

EID: 85017190163     PISSN: 08936080     EISSN: 18792782     Source Type: Journal    
DOI: 10.1016/j.neunet.2017.02.013     Document Type: Article

References (49)

1. Abdel-Hamid, O., Mohamed, A.-R., Jiang, H., Deng, L., Penn, G., Yu, D., Convolutional neural networks for speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing 22 (2014), 1533–1545.
2. Arias, J.P., Busso, C., Yoma, N.B., Energy and F0 contour modeling with functional data analysis for emotional speech detection. Interspeech, 2013, 2871–2875.
3. Ayadi, M.E., Kamel, M.S., Karray, F., Survey on speech emotion recognition: Features, classification schemes, and databases. Pattern Recognition 44 (2011), 572–587.
4. Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., et al. IEMOCAP: interactive emotional dyadic motion capture database. Language Resources and Evaluation 42 (2008), 335–359, 10.1007/s10579-008-9076-6.
5. Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., et al. Emotion recognition in human–computer interaction. IEEE Signal Processing Magazine 18 (2001), 32–80, 10.1109/79.911197.
6. Dauphin, Y., de Vries, H., Bengio, Y., Equilibrated adaptive learning rates for non-convex optimization. Advances in neural information processing systems, 2015, 1504–1512.
7. Eyben, F., Huber, B., Marchi, E., Schuller, D., Schuller, B., Real-time robust recognition of speakers’ emotions and characteristics on mobile platforms. 2015 international conference on affective computing and intelligent interaction (ACII), 2015, 778–780, 10.1109/ACII.2015.7344658.
8. Eyben, F., Scherer, K., Schuller, B., Sundberg, J., Andre, E., Busso, C., et al. The Geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing. IEEE Transactions on Affective Computing, 2015, 190–202, 10.1109/TAFFC.2015.2457417.
9. Fayek, H.M., Lech, M., Cavedon, L., Towards real-time speech emotion recognition using deep neural networks. 2015 9th international conference on signal processing and communication systems (ICSPCS), 2015, 1–5, 10.1109/ICSPCS.2015.7391796.
10. Fayek, H.M., Lech, M., Cavedon, L., Modeling subjectiveness in emotion recognition with deep neural networks: Ensembles vs soft labels. 2016 international joint conference on neural networks (IJCNN), 2016, 566–570, 10.1109/IJCNN.2016.7727250.
11. Fayek, H.M., Lech, M., Cavedon, L., On the correlation and transferability of features between automatic speech recognition and speech emotion recognition. Interspeech 2016, 2016, 3618–3622, 10.21437/Interspeech.2016-868.
12. Fernandez, R., A computational model for the automatic recognition of affect in speech. (Ph.D. thesis), 2004, School of Architecture and Planning, Massachusetts Institute of Technology.
13. Glorot, X., Bordes, A., & Bengio, Y. (2011). Deep sparse rectifier neural networks. In JMLR W&CP: Proceedings of the fourteenth international conference on artificial intelligence and statistics, AISTATS 2011.
14. Goodfellow, I., Bengio, Y., Courville, A., Deep learning. 2016, MIT Press.
15. Graves, A., Supervised sequence labelling with recurrent neural networks. (Ph.D. thesis), 2008, Technische Universitat Munchen.
16. Graves, A., Mohamed, A.-R., Hinton, G., Speech recognition with deep recurrent neural networks. 2013 IEEE international conference on acoustics, speech and signal processing, 2013, 6645–6649, 10.1109/ICASSP.2013.6638947.
17. Han, K., Yu, D., Tashev, I., Speech emotion recognition using deep neural network and extreme learning machine. Interspeech 2014, 2014.
18. He, K., Zhang, X., Ren, S., Sun, J., Deep residual learning for image recognition. 2016 IEEE conference on computer vision and pattern recognition (CVPR), 2016, 770–778, 10.1109/CVPR.2016.90.
19. Hinton, G., Deng, L., Yu, D., Dahl, G.E., Mohamed, A.-R., Jaitly, N., et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine 29 (2012), 82–97, 10.1109/MSP.2012.2205597.
20. Hochreiter, S., Schmidhuber, J., Long short-term memory. Neural Computation 9 (1997), 1735–1780, 10.1162/neco.1997.9.8.1735.
21. Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd international conference on machine learning (pp. 448–456).
22. Kim, Y., Lee, H., Provost, E.M., Deep learning for robust feature generation in audiovisual emotion recognition. 2013 IEEE international conference on acoustics, speech and signal processing, 2013, 3687–3691, 10.1109/ICASSP.2013.6638346.
23. Krizhevsky, A., Sutskever, I., Hinton, G.E., Imagenet classification with deep convolutional neural networks. Neural information processing systems, 2012, 1097–1105.
24. LeCun, Y., Bengio, Y., Hinton, G., Deep learning. Nature 521 (2015), 436–444 Insight.
25. LeCun, Y., Boser, B., Denker, J.S., Howard, R.E., Habbard, W., & Jackel, L.D. et al. (1990). Advances in neural information processing systems 2. chapter Handwritten digit recognition with a back-propagation network. (pp. 396–404).
26. Lee, C.-C., Mower, E., Busso, C., Lee, S., Narayanan, S., Emotion recognition using a hierarchical binary decision tree approach. Speech Communication 53 (2011), 1162–1171.
27. Li, L., Zhao, Y., Jiang, D., Zhang, Y., Wang, F., & Gonzalez, I. et al. (2013). Hybrid deep neural network–hidden markov model (dnn-hmm) based speech emotion recognition. In 2013 humaine association conference on affective computing and intelligent interaction (ACII) (pp. 312–317).
28. Mao, Q., Dong, M., Huang, Z., Zhan, Y., Learning salient features for speech emotion recognition using convolutional neural networks. IEEE Transactions on Multimedia 16 (2014), 2203–2213.
29. Mariooryad, S., Busso, C., Exploring cross-modality affective reactions for audiovisual emotion recognition. IEEE Transactions on Affective Computing 4 (2013), 183–196, 10.1109/T-AFFC.2013.11.
30. Mohamed, A., Dahl, G., Hinton, G., Acoustic modeling using deep belief networks. IEEE Transactions on Audio, Speech, and Language Processing 20 (2012), 14–22, 10.1109/TASL.2011.2109382.
31. Petta, P., Pelachaud, C., Cowie, R., Emotion-oriented systems. The humaine handbook, 2011.
32. Povey, D., Ghoshal, A., Boulianne, G., Burget, L., Glembek, O., Goel, N., et al. The kaldi speech recognition toolkit. IEEE 2011 workshop on automatic speech recognition and understanding, 2011, IEEE Signal Processing Society IEEE Catalog No.: CFP11SRW-USB.
33. Refaeilzadeh, P., Tang, L., Liu, H., Cross-validation. Encyclopedia of database systems, 2009, Springer, 532–538.
34. Rumelhart, D.E., Hinton, G.E., Williams, R.J., Learning representations by back-propagating errors. Nature 323 (1986), 533–536.
35. Schmidhuber, J., Deep learning in neural networks: An overview. Neural Networks 61 (2015), 85–117, 10.1016/j.neunet.2014.09.003.
36. Schuller, B., Steidl, S., Batliner, A., The interspeech 2009 emotion challenge. Interspeech, vol. 2009, 2009, 312–315.
37. Schuller, B., Steidl, S., Batliner, A., Burkhardt, F., Devillers, L., Müller, C.A., et al. The interspeech 2010 paralinguistic challenge. Interspeech, 2010, 2794–2797.
38. Schuller, B., Vlasenko, B., Eyben, F., Rigoll, G., & Wendemuth, A. (2009). Acoustic emotion recognition: A benchmark comparison of performances. In IEEE workshop on automatic speech recognition understanding, 2009 (pp. 552–557).
39. Shah, M., Chakrabarti, C., & Spanias, A. (2014). A multi-modal approach to emotion recognition using undirected topic models. In IEEE international symposium on circuits and systems (ISCAS) (pp. 754–757).
40. Simonyan, K., & Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. In International conference on learning representations (ICLR).
41. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R., Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15 (2014), 1929–1958.
42. Stuhlsatz, A., Meyer, C., Eyben, F., ZieIke, T., Meier, G., & Schuller, B. (2011). Deep neural networks for acoustic emotion recognition: Raising the benchmarks. In 2011 IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 5688–5691).
43. Tahon, M., Devillers, L., Towards a small set of robust acoustic features for emotion recognition: Challenges. IEEE/ACM Transactions on Audio, Speech, and Language Processing 24 (2016), 16–28.
44. Tian, L., Lai, C., & Moore, J. (2015). Recognising emotions in dialogues with disfluencies and non-verbal vocalisations. In R. Lickley (Ed.), The 7th workshop on disfluency in spontaneous speech.
45. Tian, L., Moore, J., & Lai, C. (2015). Emotion recognition in spontaneous and acted dialogues. In 2015 International conference on affective computing and intelligent interaction (ACII) (pp. 698–704).
46. Tieleman, T., Hinton, G., Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 4, 2012.
47. Ververidis, D., Kotropoulos, C., Emotional speech recognition: Resources, features, and methods. Speech Communication 48 (2006), 1162–1181.
48. Vlasenko, B., Schuller, B., Wendemuth, A., Rigoll, G., Frame vs. turn-level: emotion recognition from speech considering static and dynamic processing. Affective computing and intelligent interaction, 2007, Springer, 139–147.
49. Werbos, P.J., Generalization of backpropagation with application to a recurrent gas market model. Neural Networks 1 (1988), 339–356, 10.1016/0893-6080(88)90007-X.