Recognizing affect from speech prosody using hierarchical graphical models

Speech Communication

Volumn 53, Issue 9-10, 2011, Pages 1088-1103

  Fernandez, Raul ^a   Picard, Rosalind ^b  

^a IBM T J WATSON RESEARCH CENTER   (United States)

^b MIT MEDIA LAB   (United States)

Author keywords

Affective speech; Graphical models; Paralinguistics; Prosodic modeling

Indexed keywords

DYNAMICAL SYSTEMS; GRAPHIC METHODS; HIERARCHICAL SYSTEMS; LINGUISTICS; SPEECH;

ACOUSTIC PARAMETERS; AFFECTIVE SPEECH; GRAPHICAL MODEL; NATURALLY OCCURRING; PARALINGUISTICS; PERCEPTUAL RECOGNITION; PROSODIC MODELING; SPECIFIC INFORMATION;

SPEECH RECOGNITION;

EID: 79960846934     PISSN: 01676393     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.specom.2011.05.003     Document Type: Article

References (44)

1. Barra-Chicote, R.; Fernandez, R.; Lutfi, S.; Lucas-Cuesta, J.; Macias-Guarasa, J.; Montero, J.; San-Segundo, R.; Pardo, J.; 2009. Acoustic emotion recognition using dynamic Bayesian networks and multi-space distributions. In: Proc. Interspeech. Brighton, UK, pp. 336-339.
2. A. Batliner, D. Seppi, S. Steidl, and B. Schuller Segmenting into adequate units for automatic recognition of emotion-related episodes: a speech-based approach Adv. Human-Comput. Interact. (AHCI) 2010 15
3. M.E. Beckman The parsing of prosody P. Warren, Prosody and Parsing. Special Issue of Language and Cognitive Processes 1996 Psychology Press 17 68 (Pubitemid 126072934)
4. Black, M.; Katsamanis, A.; Lee, C.-C.; Lammert, A.; Baucom, B.; Christense, A.; Georgiu, P.; Narayanan, S.; 2010. Automatic classification of married couples' behavior using audio features. In: Proc. Interspeech. Chiba, Japan, pp. 2030-2033.
5. Burkhardt, F.; Paeschke, A.; Rolfes, M.; Sendlmeier, W.; Weiss, B.; 2005. A database of German emotional speech. In: Proc. Interspeech. Lisbon, Portugal, pp. 1517-1520. (Pubitemid 43908362)
6. Campbell, N.; Mokhtari, P.; 2003. Voice quality: The 4th prosodic dimension. In: Proc. XVth Internat. Congr. Phonetis Sci. Barcelona, Spain, pp. 2417-2420.
7. Chen, D.; Jiang, D.; Ravsye, I.; Sahli, H.; 2009. Audio-Visual emotion recognition based on a DBN model with constrained asynchrony. In: Proc. 5th Internat. Conf. Image and Graphics. pp. 912-916.
8. R. Cowie, and R.R. Cornelius Describing the emotional states that are expressed in speech Speech Commun. 40 1-2 2003 5 32
9. F. Cummins, and R. Port Rhythmic constraints on stress timing in English J. Phonetics 26 1998 145 171 (Pubitemid 128179514)
10. Douglas-Cowie, E.; Devillers, L.; Martin, J.-C.; Cowie, R.; Savvidou, S.; Abrilian, S.; Cox, C.; 2005. Multimodal databases of everyday emotion: facing up to complexity. In: Proc. Interspeech. Lisbon, Portugal, pp. 88-92.
11. Durston, P.J.; Farrell, M.; Attwater, D.; Allen, J.; Kuo, H.-K.J.; Afify, M.; Fosler-Lussier, E.; Lee, C.-H.; 2001. OASIS natural language call steering trial. In: Proc. Eurospeech. Aalborg, Denmark, pp. 1323-1326.
12. F. Eyben, M. Wöllmer, A. Graves, B. Schuller, E. Douglas-Cowie, and R. Cowie On-line emotion recognition in a 3-D activation-valence time continuum using acoustic and linguistic cues J. Multimodal User Interfaces 3 1-2 2010 7 12
13. Fernandez, R.; 2004. A Computational Model for the Automatic Recognition of Affect in Speech. Ph.D. thesis, MIT.
14. Fernandez, R.; Picard, R.; 2005. Classical and novel discriminant features for affect recognition from speech. In: Proc. Interspeech. ISCA, Lisbon, Portugal, pp. 473-476. (Pubitemid 43908102)
15. C. Gobl, and A. Ní Chasaide The role of voice quality in communicating emotion, mood and attitude Speech Commun. 40 1-2 2003 189 212
16. B. Hayes The prosodic hierarchy in meter P. Kiparsky, G. Youmans, Phonetics and Phonology Rythm and Meter 1 1989 Academic Press 201 260
17. Hirschberg, J.; Nakatani, C.; 1998. Using machine learning to identify intonational segments. In: Proc. AAAI Spring Symp. Applying Machine Learning to Discourse Processing. pp. 52-59.
18. D. Koller, and N. Friedman Probabilistic Graphical Models Principles and Techniques 2009 The MIT Press
19. Kompe, R.; Batliner, A.; Kießling, A.; Kilian, U.; Niemman, H.; Nöth, E.; 1994. Automatic classification of prosodically marked phrase boundaries in German. In: Proc. Internat. Conf. Acoustics, Speech, and Signal Processing (ICASSP). Vol. 2. pp. 173-176.
20. D. Ladd Intonational Phonology. No. 79 in Cambridge Studies in Linguisitics 1996 Cambridge University Press
21. J. Laver Principles of Phonetics 1994 Cambridge University Press
22. I. Lehiste Suprasegmentals 1970 MIT Press Cambridge, MA
23. Mao, X.; Chen, L.; Fu, L.; 2009. Multi-level speech emotion recognition based on HMM and ANN. In: Proc. WRI World Congr. on Computer Science and Information Engineering. Los Angeles, USA, pp. 225-229.
24. Murphy, K.; 2007. Bayes Net Toolbox for Matlab. ().
25. E. Nöth, A. Batliner, A. Kießling, R. Kompe, and H. Niemann VERBMOBIL: the use of prosody in the linguistic components of a speech understanding system IEEE Trans. Speech Audio Proc. 8 5 2000 519 532
26. Pereira, C.; 2000. Dimensions of emotional meaning in speech. In: Cowie, R.; Douglas-Cowie, E.; Schröder, M. (Eds.), Proc. ISCA Workshop on Speech and Emotion. Newcastle, Northern Ireland, pp. 25-28.
27. Polzin, T.; 2000. Detecting Verbal and Non-verbal Cues in the Communication of Emotions. Ph.D. thesis, School of Computer Science. Carnegie Mellon University.
28. Scherer, K.; 2005. Proposal for exemplars and work towards them: Theory of emotions. (HUMAINE deliveralbe D3e). .
29. Schröder, M.; Pirker, H.; Lamolle, M.; 2006. First suggestions for an emotion annotation and representation language. In: Proc. LREC '06 Workshop on Corpora for Research on Emotion and Affect. Genoa, Italy, pp. 88-92.
30. Schuller, B.; Rigoll, G.; Lang, M.; 2003. Hidden Markov model-based speech emotion recognition. In: Proc. Internat. Conf. Acoustics, Speech, Signal Proc. (ICASSP). Vol. 2. pp. 1-4.
31. Schuller, B.; Steidl, S.; Batliner, A.; 2009. The INTERSPEECH 2009 Emotion Challenge. In: Proc. Interspeech. Brighton, UK, pp. 312-315.
32. Schuller, B.; Steidl, S.; Batliner, A.; Burkhardt, F.; Devillers, L.; Müller, C.; Narayanan, S.; 2010. The INTERSPEECH 2010 Paralinguistic Challenge. In: Proc. Interspeech. Chiba, Japan, pp. 2794-2797.
33. Sebe, N.; Cohen, I.; Gevers, T.; Huang, T.; 2006. Emotion recognition based on joint visual and audio cues. In: Proc. Internat. Conf. Pattern Recognition (ICPR). pp. 1136-1139.
34. E.O. Selkirk Phonology and Syntax: The Relation Between Sound and Structure 1984 MIT Press Cambridge, MA
35. S. Shattuck-Hufnagel, and A.E. Turk A prosody tutorial for investigators of auditory sentence processing J. Psycholinguistic Res. 25 2 1996 193 247 (Pubitemid 126434983)
36. Vlasenko, B.; Schuller, B.; Mengistu, K.; Rigoll, G.; Wendemuth, A.; 2008. Balancing spoken content adaptation and unit length in the recognition of emotion and interest. In: Proc. Interspeech. ISCA, Brisbane, Australia, pp. 805-808.
37. M. Wang, and J. Hirschberg Automatic classification of intonational phrase boundaries Comput. Speech Lang. 6 1992 175 196
38. Wightman, C.W.; Ostendorf, M.; 1991. Automatic recognition of prosodic phrases. In: Proc. Internat. Conf. Acoustics, Speech, Signal Proc. (ICASSP). Vol. 1. pp. 321-324.
39. Wightman, C.W.; Ostendorf, M.; 1992. Automatic recognition of intonational features. In: Proc. Internat. Conf. Acoustics, Speech, Signal Proc. (ICASSP). Vol. 1. pp. 221-224.
40. C.W. Wightman, and M. Ostendorf Automatic labeling of prosodic patterns IEEE Trans. Speech Audio Process. 2 4 1994 469 481
41. C.W. Wightman, S. Shattuck-Hufnagel, M. Ostendorf, and P.J. Price Segmental durations in the vicinity of prosodic phrase boundaries J. Acoust. Soc. Amer. 91 3 1992 1707 1717
42. Yanushevskaya, I.; Gobl, C.; Níchasaide, A.; 2005. Voice quality and f0 cues for affect expression: implications for synthesis. In: Proc. Interspeech. ISCA, Lisbon, Portugal, pp. 1849-1852. (Pubitemid 43908445)
43. Yanushevskaya, I.; Gobl, C.; Ní Chasaide, A.; 2008. Voice quality and loudness in affect perception. In: Proc. Speech Prosody 2008. ISCA, Campinas, Brazil, pp. 709-712.
44. E. Zwicker, and H. Fastl Psychoacoustics. Facts and models second ed. Springer Series in Information Sciences Vol. 22 1999 Springer-Verlag Berlin