Automatic speech emotion recognition using modulation spectral features

Speech Communication

Volumn 53, Issue 5, 2011, Pages 768-785

  Wu, Siqing ^a   Falk, Tiago H ^b   Chan, Wai Yip ^a  

^a QUEEN S UNIVERSITY   (Canada)

^b UNIVERSITY OF TORONTO   (Canada)

Author keywords

Affective computing; Emotion recognition; Spectro temporal representation; Speech analysis; Speech modulation

Indexed keywords

ACOUSTIC FREQUENCY; AFFECTIVE COMPUTING; AUTOMATIC RECOGNITION; EMOTION RECOGNITION; ESTIMATION PERFORMANCE; HUMAN EVALUATION; HUMAN SPEECH PERCEPTION; MEL-FREQUENCY CEPSTRAL COEFFICIENTS; MODULATION FILTERBANK; PERCEPTUAL LINEAR PREDICTIONS; PROSODIC FEATURES; RECOGNITION PERFORMANCE; RECOGNITION RATES; SPECTRAL FEATURE; SPECTRAL REPRESENTATIONS; SPECTRO-TEMPORAL REPRESENTATION; SPEECH EMOTION RECOGNITION; TEMPORAL MODULATIONS; TEMPORAL REPRESENTATIONS;

FEATURE EXTRACTION; MODULATION; SPEECH ANALYSIS;

SPEECH RECOGNITION;

EID: 79953659944     PISSN: 01676393     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.specom.2010.08.013     Document Type: Article

References (60)

1. Abelin, A., Allwood, J., 2000. Cross linguistic interpretation of emotional prosody. In: Proc. ISCA ITRW on Speech and Emotion, pp. 110-113.
2. A. Aertsen, and P. Johannesma Spectro-temporal receptive fields of auditory neurons in the grass frog. I. Characterization of tonal and natural stimuli Biol. Cybernet. 38 1980 223 234 (Pubitemid 11220294)
3. L. Atlas, and S. Shamma Joint acoustic and modulation frequency EURASIP J. Appl. Signal Process. 2003 668 675
4. Barra, R., Montero, J., Macias-Guarasa, J., D'Haro, L., San-Segundo, R., Cordoba, R., 2006. Prosodic and segmental rubrics in emotion identification. In: Proc. Internat. Conf. on Acoustics, Speech and Signal Processing, Vol. 1, pp. 1085-1088.
5. Batliner, A., Steidl, S., Schuller, B., Seppi, D., Laskowski, K., Vogt, T., Devillers, L., Vidrascu, L., Amir, N., Kessous, L., Aharonson, V., 2006. Combining efforts for improving automatic classification of emotional user states. In: Proc. IS-LTC, pp. 240-245.
6. C. Bishop Pattern Recognition and Machine Learning 2006 Springer New York
7. Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W., Weiss, B., 2005. A database of German emotional speech. In: Proc. Interspeech, pp. 1517-1520. (Pubitemid 43908362)
8. C. Busso, S. Lee, and S. Narayanan Analysis of emotionally salient aspects of fundamental frequency for emotion detection IEEE Trans. Audio Speech Language Process. 17 2009 582 596
9. Chang, C.-C., Lin, C.-J., 2009. LIBSVM: A library for support vector machines. Tech. rep., Department of Computer Science, National Taiwan University. Software available at: .
10. T. Chih, P. Ru, and S. Shamma Multiresolution spectrotemporal analysis of complex sounds J. Acoust. Soc. Amer. 118 2005 887 906 (Pubitemid 41129224)
11. C. Clavel, I. Vasilescu, L. Devillers, G. Richard, and T. Ehrette Fear-type emotion recognition for future audio-based surveillance systems Speech Commun. 50 2008 487 503
12. Cowie, R., Douglas-Cowie, E., 1996. Automatic statistical analysis of the signal and prosodic signs of emotion in speech. In: Proc. Internat. Conf. on Spoken Language Processing, Vol. 3, pp. 1989-1992.
13. R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J. Taylor Emotion recognition in human-computer interaction IEEE Signal Process. Mag. 18 2001 32 80 (Pubitemid 32287669)
14. S. Davis, and P. Mermelstein Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences IEEE Trans. Audio Speech Language Process. 28 1980 357 366 (Pubitemid 11464930)
15. D. Depireux, J. Simon, D. Klein, and S. Shamma Spectro-temporal response field characterization with dynamic ripples in ferret primary auditory cortex J. Neurophysiol. 85 2001 1220 1234 (Pubitemid 32209608)
16. E. Douglas-Cowie, N. Campbell, R. Cowie, and P. Roach Emotional speech: Towards a new generation of databases Speech Commun. 40 2003 33 60
17. P. Ekman Basic Emotions 1999 John Wiley New York pp. 45-60
18. S. Ewert, and T. Dau Characterizing frequency selectivity for envelope fluctuations J. Acoust. Soc. Amer. 108 2000 1181 1196
19. Falk, T.H., Chan, W.-Y., 2008. A non-intrusive quality measure of dereverberated speech. In: Proc. Internat. Workshop for Acoustic Echo and Noise Control.
20. T.H. Falk, and W.-Y. Chan Modulation spectral features for robust far-field speaker identification IEEE Trans. Audio Speech Language Process. 18 2010 90 100
21. T.H. Falk, and W.-Y. Chan Temporal dynamics for blind measurement of room acoustical parameters IEEE Trans. Instrum. Meas. 59 2010 978 989
22. H. Fletcher Auditory patterns Rev. Modern Phys. 12 1940 47 65
23. Giannakopoulos, T., Pikrakis, A., Theodoridis, S., 2009. A dimensional approach to emotion recognition of speech from movies. In: Proc. Internat. Conf. on Acoustics, Speech and Signal Processing, pp. 65-68.
24. B. Glasberg, and B. Moore Derivation of auditory filter shapes from notched-noise data Hearing Res. 47 1990 103 138 (Pubitemid 20244652)
25. M. Grimm, K. Kroschel, E. Mower, and S. Narayanan Primitives-based evaluation and estimation of emotions in speech Speech Commun. 49 2007 787 800 (Pubitemid 47268568)
26. Grimm, M., Kroschel, K., Narayanan, S., 2007b. Support vector regression for automatic recognition of spontaneous emotions in speech. In: Proc. Internat. Conf. on Acoustics, Speech and Signal Processing, Vol. 4, pp. 1085-1088.
27. Grimm, M., Kroschel, K., Narayanan, S., 2008. The Vera am Mittag German audio-visual emotional speech database. In: Proc. Internat. Conf. on Multimedia & Expo, pp. 865-868.
28. H. Gunes, and M. Piccard Bi-modal emotion recognition from expressive face and body gestures J. Network Comput. Appl. 30 2007 1334 1345 (Pubitemid 47263518)
29. H. Hermansky Perceptual linear predictive (PLP) analysis of speech J. Acoust. Soc. Amer. 87 1990 1738 1752 (Pubitemid 20256470)
30. Hsu, C.-C., Chang, C.-C., Lin, C.-J., 2007. A practical guide to support vector classification. Tech. rep., Department of Computer Science, National Taiwan University.
31. Intl. Telecom. Union, 1993. Objective measurement of active speech level. ITU-T P.56, Switzerland.
32. Intl. Telecom. Union, 1996. A silence compression scheme for G.729 optimized for terminals conforming to ITU-T recommendation V.70. ITU-T G.729 Annex B, Switzerland.
33. C. Ishi, H. Ishiguro, and N. Hagita Analysis of the roles and the dynamics of breathy and whispery voice qualities in dialogue speech EURASIP J. Audio Speech Music Process. 2010 article ID 528193, 12 pages
34. Kaiser, J., 1990. On a simple algorithm to calculate the 'energy' of a signal. In: Proc. Internat. Conf. on Acoustics, Speech and Signal Processing, Vol. 1, pp. 381-384.
35. N. Kanederaa, T. Araib, H. Hermanskyc, and M. Pavel On the relative importance of various components of the modulation spectrum for automatic speech recognition Speech Commun. 28 1999 43 55
36. J. Kittler Feature set search algorithms Pattern Recognition Signal Process. 1978 41 60
37. Lee, S., Yildirim, S., Kazemzadeh, A., Narayanan, S., 2005. An articulatory study of emotional speech production. In: Proc. Interspeech, pp. 497-500. (Pubitemid 43908108)
38. Lugger, M., Yang, B., 2008. Cascaded emotion classification via psychological emotion dimensions using a large set of voice quality parameters. In: Proc. Internat. Conf. on Acoustics, Speech and Signal Processing, Vol. 4, pp. 4945-4948.
39. N. Morgan, Q. Zhu, A. Stolcke, K. Sonmez, S. Sivadas, T. Shinozaki, M. Ostendorf, P. Jain, H. Hermansky, D. Ellis, G. Doddington, B. Chen, O. Cetin, H. Bourlard, and M. Athineos Pushing the envelope - aside IEEE Signal Process. Mag. 22 2005 81 88 (Pubitemid 41508702)
40. Mozziconacci, S., 2002. Prosody and emotions. In: Proc. Speech Prosody, pp. 1-9.
41. T. Nwe, S. Foo, and L. De Silva Speech emotion recognition using hidden markov models Speech Commun. 41 2003 603 623
42. R. Picard Affective Computing 1997 The MIT Press
43. L. Rabiner, and B. Juang Fundamentals of Speech Recognition 1993 Prentice-Hall Englewood Cliffs, NJ
44. K. Scherer Vocal communication of emotion: A review of research paradigms Speech Commun. 40 2003 227 256
45. Scherer, S., Schwenker, F., Palm, G., 2007. Classifier fusion for emotion recognition from speech. In: Proc. IET Internat. Conf. on Intelligent Environments, pp. 152-155.
46. Schuller, B., Seppi, D., Batliner, A., Maier, A., Steidl, S., 2006. Emotion recognition in the noise applying large acoustic feature sets. In: Proc. Speech Prosody.
47. Schuller, B., Batliner, A., Seppi, D., Steidl, S., Vogt, T., Wagner, J., Devillers, L., Vidrascu, L., Amir, N., Kessous, L., Aharonson, V., 2007a. The relevance of feature type for the automatic classification of emotional user states: Low level descriptors and functionals. In: Proc. Interspeech, pp. 2253-2256.
48. Schuller, B., Seppi, D., Batliner, A., Maier, A., Steidl, S., 2007b. Towards more reality in the recognition of emotional speech. In: Proc. Internat. Conf. on Acoustics, Speech and Signal Processing, Vol. 4. pp. 941-944.
49. M. Shami, and W. Verhelst An evaluation of the robustness of existing supervised machine learning approaches to the classification of emotions in speech Speech Commun. 49 2007 201 212 (Pubitemid 46413361)
50. S. Shamma On the role of space and time in auditory processing Trends Cogn. Sci. 5 2001 340 348 (Pubitemid 32703803)
51. S. Shamma Encoding sound timbre in the auditory system IETE J. Res. 49 2003 193 205
52. Slaney, M., 1993. An efficient implementation of the Patterson-Holdsworth auditory filterbank. Tech. rep., Apple Computer, Perception Group.
53. Sun, R., E., M., Torres, J., 2009. Investigating glottal parameters for differentiating emotional categories with similar prosodics. In: Proc. Internat. Conf. on Acoustics, Speech and Signal Processing, pp. 4509-4512.
54. Talkin, D., 1995. A robust algorithm for pitch tracking (RAPT). Elsevier Science, Speech Coding and Synthesis (Chapter 14).
55. V. Vapnik The Nature of Statistical Learning Theory 1995 Springer New York
56. D. Ververidis, and C. Kotropoulos Emotional speech recognition: Resources, features, and methods Speech Commun. 48 2006 1162 1181 (Pubitemid 44128615)
57. Vlasenko, B., Schuller, B., Wendemuth, A., Rigoll, G., 2007. Combining frame and turn-level information for robust recognition of emotions within speech. In: Proc. Interspeech, pp. 2225-2228.
58. Wollmer, M., Eyben, F., Reiter, S., Schuller, B., Cox, C., Douglas-Cowie, E., Cowie, R., 2008. Abandoning emotion classes - Towards continuous emotion recognition with modelling of long-range dependencies. In: Proc. Interspeech, pp. 597-600.
59. Wu, S., Falk, T., Chan, W.-Y., 2009. Automatic recognition of speech emotion using long-term spectro-temporal features. In: Proc. Internat. Conf. on Digital Signal Processing, pp. 1-6.
60. G. Zhou, J. Hansen, and J. Kaiser Nonlinear feature based classification of speech under stress IEEE Trans. Audio Speech Language Process. 9 2001 201 216 (Pubitemid 32286594)