Perceiving emotion: Towards a realistic understanding of the task

Philosophical Transactions of the Royal Society B: Biological Sciences

Volumn 364, Issue 1535, 2009, Pages 3515-3525

  Cowie, Roddy ^a  

^a QUEEN'S UNIVERSITY BELFAST   (United Kingdom)

Author keywords

Emotion; Expression; Face; Naturalistic; Perception; Speech

Indexed keywords

COGNITION; PERCEPTION; PHOTOGRAPH; PSYCHOLOGY;

EID: 73549091284     PISSN: 09628436     EISSN: 14712970     Source Type: Journal    
DOI: 10.1098/rstb.2009.0139     Document Type: Article

References (103)

1. Adolphs, R. 2006 Perception and emotion: how we recognize facial expressions. Curr. Dir. Psychol. Sci. 15 222-226. (doi:10.1111/j.1467-8721.2006. 00440.x) (Pubitemid 44800064)
2. Ambadar, Z. S., Schooler, J. W. & Cohn, J. 2005 Deciphering the enigmatic face: the importance of facial dynamics in interpreting subtle facial expressions. Psychol. Sci. 16, 403-410. (doi:10.1111/j.0956-7976.2005.01548.x)
3. Ang, J., Dhillon, R., Krupski, A., Shriberg, E. & Stolcke, A. 2002 Prosody-based automatic detection of annoyance and frustration in human-computer dialog. In Proc. ICSLP 2002, Denver, Colorado.
4. Bachorowski, J.-A. 1999 Vocal expression and perception of emotion. Curr. Dir. Psychol. Sci. 8, 53-57. (doi:10.1111/ 1467-8721.00013)
5. Banse, R. & Scherer, K. R. 1996 Acoustic profiles in vocal emotion expression. J. Pers. Soc. Psychol. 70, 614-636. (Pubitemid 126420699)
6. Baron-Cohen, S., Golan, O., Wheelwright, S. & Hill, J. J. 2004 Mind reading: the interactive guide to emotions. London, UK: Jessica Kingsley Publishers.
7. Bassili, J. N. 1978 Facial motion in the perception of faces and of emotional expression. J. Exp. Psyschol. 4, 373-379.
8. Bassili, J. N. 1979 Emotion recognition: the role of facial motion and the relative importance of upper and lower areas of the face. J. Pers. Soc. Psychol. 37, 2049-2059.
9. Batliner, A., Fischer, K., Huber, R., Spilker, J. & Noedi, E. 2003 How to find trouble in communication. Speech Commun. 40, 117-143. (doi:10.1016/S0167-6393(02) 00079-1)
10. Batliner, A. et al. 2006 Combining efforts for improving automatic classification of emotional user states. In Proceedings of IS-LTC 2006, Ljubliana, pp. 240-245.
11. Bhuta, T., Patrick, L. & Garnett, J. D. 2004 Perceptual evaluation of voice quality and its correlation with acoustic measurements. J. Voice 18, 299-304. (doi:10.1016/ j.jvoice.2003.12.004) (Pubitemid 39164280)
12. Biederman, I. 1987 Recognition-by-components: a theory of human image understanding. Psychol. Rev. 94, 115-147. (doi: 10.1037/0033-295X.94.2.115)
13. Birdwhistell, R. L. 1970 Kinesics and context. Philadelphia, PA: University of Pennsylvania Press.
14. Burkhardt, F. & Sendlmeier, W. F. 2000 Verification of acoustical correlates of emotional speech using formantsynthesis. In Proc. ISCA Tutorial and Research Workshop (ITRW) on Speech and Emotion, pp. 151-156.
15. Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C. M., Kazemzadeh, A., Lee, S., Neumann, U. & Narayanan, S. 2004 Analysis of emotion recognition using facial expressions, speech and multimodal information. In Proc. Sixth Int. Conf. on Multimodal Interfaces, pp. 205-211. (Pubitemid 40371911)
16. Cahn, J. E. 1990 The generation of affect in synthesized speech. J. Am. Voice I/O Soc. 8, 1-19.
17. Campbell, N. 2004 Speech and expression; the value of a longitudinal corpus. In Proc. LREC 2004, Lisbon, Portugal.
18. Carlson, R., Granström, B. & Lennart, N. 1992 Experiments with emotive speech-acted utterances and synthesized replicas. ICSLP-1992, pp. 671-674.
19. Carroll, J. M. & Russell, J. A. 1997 Facial expressions in Hollywood's portrayal of emotion. J. Pers. Soc. Psychol. 72, 164-176. (Pubitemid 127018160)
20. Castanho, A. P. & Otta, E. 1999 Decoding spontaneous and posed smiles of children who are visually impaired and sighted. J. Vis. Impair. Blind 93, 659-662. (Pubitemid 29480997)
21. Cauldwell, R. T. 2000 Where did the anger go? The role of context in interpreting emotion in speech. In Speech and Emotion: Proceedings of the ISCA workshop, Newcastle, Co. Down, N Ireland, September 2000 (eds R. Cowie, E. Douglas-Cowie & M. Schroeder), pp. 127-131.
22. Chakrabarti, B. & Baron-Cohen, S. 2006 Empathizing: neurocognitive developmental mechanisms and individual differences. Prog. Brain Res. 156, 403-417. (doi:10.1016/ S0079-6123(06)56022-4) (Pubitemid 44479791)
23. Conati, C. 2002 Probabilistic assessment of user's emotions in educational games. J. Appl. Artif. Inull. 16, 555-575. (doi:10.1080/ 08839510290030390) (Pubitemid 35019458)
24. Cowie, R. 2005 What are people doing when they assign everyday emotion terms? Psychol. Inq. 16 11-18.
25. Cowie, R. 2009 Describing the forms of emotional colouring that pervade everyday life. In Oxford handbook of the philosophy of emotion (ed. P. Goldie), pp. 63-94. Oxford, UK: Oxford University Press.
26. Cowie, R. & Cornelius, R. 2003 Describing the emotional states that are expressed in speech. Speech Commun. 40, 5-32. (doi:10.1016/S0167-6393(02) 00071-7)
27. Cowie, R. & Douglas-Cowie, E. 2009 Prosodic and related features that signify emotional colouring in conversational speech. In The role of prosody in affective speech (ed. S. Hancil), pp. 213-240. Berne, Switzerland: Peter Lang.
28. Cowie, R., Douglas-Cowie, E., Apolloni, B., Taylor, J., Romano, A. & Fellenz, W. 1999 What a neural net needs to know about emotion words. In Computational intelligence and applications (ed. N. Mastorakis), pp. 109-114. World Scientific Engineering Society.
29. Cowie, R., Douglas-Cowie, E. & Schroeder, M. (eds) 2000 Speech and Emotion: Proceedings of the ISCA workshop, Newcastle, Co. Down, N. Ireland, 2000.
30. Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W. & Taylor, J. 2001 Emotion recognition in human-computer interaction. IEEE Signal Process. Mag. 18, 32-80. (doi:10.1109/79.911197)
31. Crane, E. & Gross, M. 2007 Motion capture and emotion: affect detection in whole body movement. In Affective computing and intelligent interaction (eds A. Paiva, R. Prada & R. Picard), Proceedings of ACII'2007 Lisbon. Springer Proceedings Series: Lecture Notes in Computer Science, vol.4738.
32. de Gelder, B. & Vroomen, J. 2000 The perception of emotions by ear and by eye. Cogn. Emotion 14, 289-311.
33. Devillers, L., Cowie, R., Martin, J.-C, Douglas-Cowie, E., Abrilian, S. & McRorie, M. 2006 Real life emotions in French and English TV video clips: an integrated annotation protocol combining continuous and discrete approaches. In Proc. Fifth Int. Conf. on Language Resources and Evaluation (LREC), Genoa, Italy.
34. Dimberg, U., Thunberg, M. & Elmehed, K. 2000 Unconscious facial reactions to emotional facial expressions. Psychol. Sci. 11, 86-89. (doi:10.1111/1467-9280.00221)
35. Dittrich, W. H., Troscianko, T., Lea, S. E. G. & Morgan, D. 1996 Perception of emotion from dynamic point-light displays represented in dance. Perception 25, 727-738. (doi:10.1068/p250727) (Pubitemid 126694959)
36. Douglas-Cowie, E., Cowie, R. & Schröder, M. 2000 A new emotion database: considerations, sources and scope. In Speech and Emotion: Proceedings of the ISCA workshop, Newcastle, Co. Down, N Ireland, Sepumber 2000 (eds R. Cowie, E. Douglas-Cowie & M. Schroeder), pp. 9-44.
37. Douglas-Cowie, E., Campbell, N., Cowie, R. & Roach, P. 2003 Emotional speech: towards a new generation of databases. Speech Commun. 40, 33-60. (doi:10.1016/ S0167-6393(02)00070-5)
38. Douglas-Cowie, E., Devillers, L., Martin, J. -C., Cowie, R., Sawidou, S., Abrilian, S. & Cox, C. 2005 Multimodal databases of everyday emotion: facing up to complexity. Interspeech 2005 813-816. (Pubitemid 43908186)
39. Douglas-Cowie, E. K. et al. 2007 The HUMAINE database: addressing the collection and annotation of naturalistic and induced emotional data. In Affective computing and intelligent interaction, pp. 488-500. Berlin, Germany: Springer-Verlag.
40. Ekman, P. 1982 Emotion in the human face. New York, NY: Cambridge University Press.
41. Ekman, P. & Friesen, W. V. 1976a Pictures of facial affect. Palo Alto, CA: Consulting Psychologists Press.
42. Ekman, P. & Friesen, W. V. 1976b Measuring facial movement. J. Nonverbal Behav. 1, 56-75. (doi:10.1007/ BF01115465)
43. Ekman, P. & Friesen, W. V. 1982 Felt, false and miserable smiles. J. Nonverbal Behav. 6, 238-252. (doi:10.1007/ BF00987191)
44. Ekman, P. & O'Sullivan, M. 1991 Who can catch a liar? Am. Psychol. 46, 913-920. (doi:10.1037/0003-066X.46. 9.913)
45. El Kaliouby, R. & Robinson, P. 2005 Real-time inference of complex mental states from facial expressions and head gestures. In Real-time vision for human-compuur inuraction part II (eds B. Kisačanin, V. Pavlović & T. S. Huang), pp. 181-200. Berlin, Germany: Springer.
46. Fernández-Dols, J. M., Wallbott, H. & Sanchez, F. 1991 Emotion category accessibility and the decoding of emotion from facial expression and context. J. Nonverbal Behav. 15, 107-123. (doi:10.1007/BF00998266)
47. Fontaine, J., Scherer, K., Roesch, E. & Ellsworth, P. 2007 The world of emotions is not two-dimensional. Psychol. Sci. 18, 1050-1057. (doi: 10.1111/j. 1467-9280.2007. 02024.x)
48. Frank, M. G., Ekman, P. & Friesen, W V. 1993 Behavioral markers and recognizability of the smile of enjoyment. J. Pers. Soc. Psychol. 64, 83-93.
49. Goldberg, H. 1951 The role of 'cutting' in the perception of motion pictures. J. Appl. Psychol. 35, 70-71. (doi:10. 1037/h0062192)
50. Grammer, K., Kruck, K. B. & Magnusson, M. S. 1998 The courtship dance: patterns of nonverbal synchronization in opposite-sex encounters. J. Nonverbal Behav. 22, 3-29. (doi: 10.1023/A: 1022986608835)
51. Hammarberg, B., Fritzell, B., Gauffin, J., Sundberg, J. & Wedin, L. 1980 Perceptual and acoustic correlates of voice qualities. Acta Otolaryngol. 90, 441-451. (doi: 10. 3109/00016488009131746) (Pubitemid 11251087)
52. Hatfield, E. & Rapson, R. L. 2000 Emotional contagion. In The Corsini encyclopedia of psychology and behavioral science (eds W. E. Craighead & C. B. Nemeroff), pp. 493-495. New York, NY: John Wiley & Sons.
53. Hatfield, E., Cacioppo, J. T. & Rapson, R. L. 1994 Emotional contagion. New York, NY: Cambridge University Press.
54. Hess, U., Banse, R. & Kappas, A. 1995 The intensity of facial expression is determined by underlying affective state and social situation. J. Pers. Soc. Psychol. 69, 280-288.
55. Heylen, D., Bevacqua, E., Tellier, M. & Pelachaud, C. 2007 Searching for prototypical facial feedback signals. Proc. IVA, pp. 147-153. (Pubitemid 350271482)
56. Juslin, P. & Laukka, P. 2003 Communication of emotions in vocal expression and music performance: different channels, same code? Psychol. Bull. 129, 770-814. (doi:10. 1037/0033-2909.129.5.770) (Pubitemid 37394950)
57. Kim, J. & Andre, E. 2008 Emotion recognition based on physiological changes in music listening. IEEE Trans. Pattern Anal. Mach. Intell. 30, 2067-2083.
58. Kwon, O., Chan, K. & Hao, J. 2003 Emotion recognition by speech signals. Proc. Eurospeech 2003, Geneva, pp. 125-128.
59. Lacava, P. G., Golan, O., Baron-Cohen, S. & Myles, B. S. 2007 Using assistive technology to teach emotion recognition to students with asperger syndrome: a pilot study. Remedial Special Educ. 28, 174-181. (doi:10.1177/ 07419325070280030601)
60. Laukka, P. 2004 Vocal expression of emotion. PhD thesis, University of Uppsala, Uppsala, Sweden.
61. Laukkanen, A.-M., Vilkman, E., Alku, P. & Oksanen, H. 1996 Physical variations related to stress and emotional state: a preliminary study. J. Phonetics 24, 313-335. (doi:10.1006/jpho.1996.0017) (Pubitemid 126175306)
62. Laver, J. 1980 The phonetic description of voice quality. Cambridge, UK: Cambridge University Press.
63. Lazarus, R. J. 1999 The cognition-emotion debate: a bit of history. In Handbook of cognition and emotion (eds T. Dalgleish & M. J. Power), pp. 1-19. Chichester, UK: Wiley.
64. Lee, C. & Narayanan, S. 2003 Emotion recognition using a data-driven fuzzy inference system. In Proc. Eurospeech 2003, Geneva, pp. 157-160.
65. Lieberman, P. & Michaels, S. B. 1962 Some aspects of fundamental frequency and envelope amplitude as related to the emotional content of speech. J. Acoust. Soc. Am. 34, 922-927. (doi: 10.1121/1.1918222)
66. Massaro, D. W. 2004 A framework for evaluating multimodal integration by humans and a role for embodied conversational agents. In Proc. Int. Conf. on Multimodal Interfaces 2004, pp. 24-41.
67. McGilloway, S., Cowie, R., Douglas-Cowie, E., Gielen, S., Westerdijk, W & Stroeve, S. 2000 Automatic recognition of emotion from voice: a rough benchmark. In Proc. ISCA ITRWon Speech and Emotion, pp. 207-212.
68. McRorie, M. & Sneddon, I. 2007 Real emotion is dynamic and interactive. In Proc. Affective Computing and Intelligent Interaction 2007, pp. 759-760. Berlin, Germany: Springer.
69. Milner, A. D. & Goodale, M. A. 1995 The visual brain in action. Oxford, UK: Oxford University Press.
70. Moore, R. K. 2007 Spoken language processing: piecing together the puzzle. Speech Commun. 49, 418-435. (doi:10.1016/j.specom.2007.01.011)
71. Mozziconacci, S. 1998 Speech variability and emotion: production and perception. PhD thesis, University of Eindhoven, Eindhoven, The Netherlands.
72. Munn, N. 1940 The effect of knowledge of the situation upon judgment of emotion from facial expressions. J. Abnormal Soc. Psychol. 35, 324-338. (doi:10.1037/ h0063680)
73. Murray, I. R. & Amott, J. L. 1995 Implementation and testing of a system for producing emotion-by-rule in synthetic speech. Speech Commun. 16, 369-390. (doi:10.1016/0167-6393(95)00005-9)
74. Nakatsu, R., Tosa, N. & Nicholson, J. 1999 Emotion recognition and its application to computer agents with spontaneous interactive capabilities. In Proc. IEEE Int. Workshop on Multimedia Signal Processing, pp. 439-444.
75. Niedenthal, P. M., Halberstadt, J. B., Margolin, J. & InnesKer, A. 2000 Emotional state and the detection of change in facial expression of emotion. Eur. J. Soc. Psychol. 30, 211-222. (doi: 10.1002/(SICI) 1099-0992(200003/04)30:2<211::AID-EJSP988>3.0.CO;2-3)
76. Niedenthal, P. M., Barsalou, L. W, Winkielman, P., KrauthGruber, S. & Ric, F. 2005 Embodiment in attitudes, social perception, and emotion. Pers. Soc. Psychol. Rev. 9, 184-211. (doi:10.1207/s15327957pspr0903-1) (Pubitemid 41356682)
77. Oster, A.-M. & Risberg, A. 1986 The identification of the mood of a speaker by hearing impaired listeners. STL-QPSR 4, 79-90.
78. Pandzic, I. & Forscheimer, R. 2002 The origins of the MPEG-4 Facial animation standard. In MPEG-4 facial animation: the standard, implementation and applications (eds I. Pandzic & R. Forscheimer), pp. 3-13. Chichester, UK: Wiley.
79. Picard, R. 1997 Affective computing. Cambridge, MA: MIT Press.
80. Poggi, I. 2006 Le parole del corpo: introduzione alia comunicazione multimodale. Rome, Italy: Carocci.
81. Pollick, F. E., Paterson, H. M., Bruderlin, A. & Sanford, A. J. 2001 Perceiving affect from arm movement. Cognition 82, B51-B61. (doi:10.1016/S0010- 0277(01) 00147-0)
82. Prom-on, S., Xu, Y & Thipakorn, B. 2009 Modeling tone and intonation as target approximation. J. Acoust. Soc. Am. 125, 406-424.
83. Russell, J. A., Bachorowski, J.-A. & Fernandez-Dols, J.-M. 2003 Facial and vocal expressions of emotion. Annu.
84. Rev. Psychol. 54, 329-349 (doi:10.1146/annurev.psych. 54.101601.145102)
85. Scherer, K. R. 2005 What are emotions? And how can they be measured? Social Science Information 44, 695-729. (doi:10.1177/0539018405058216)
86. Scherer, K. R. & Ellgring, H. 2007a Are facial expressions of emotion produced by categorical affect programs or dynamically driven by appraisal? Emotion 7, 113-130. (doi:10. 1037/1528-3542.7.1.113) (Pubitemid 46433169)
87. Scherer, K. R. & Ellgring, H. 2007b Multimodal expression of emotion: affect programs or componential appraisal patterns? Emotion 7, 158-171. (doi: 10.1037/1528-3542. 7.1.158) (Pubitemid 46433172)
88. Schmidt, K. L. & Cohn, J. 2001 Human facial expressions as adaptations: evolutionary questions in facial expression research. Am. J. Phys. Anthropol. 116(Suppl. 33), 3-24. (doi:10.1002/ajpa.20001) (Pubitemid 33610493)
89. Schroeder, M. 2001 Emotional speech synthesis: a review. In Proc. Eurospeech 2001, ISCA, Bonn, Germany, pp. 561-564.
90. Schröder, M. (ed.) 2008 Elements of an EmotionML 1.0 W3C Incubator Group Report. See http://www.w3.org/2005/Incubator/emotion/XGR-emotionml/.
91. Schroeder, M., Cowie, R., Heylen, D., Pantic, M., Pelachaud, C. & Schuller, B. 2008 Towards responsive sensitive artificial listeners. In Fourth Int. Workshop on Human-Computer Conversation, Bellagio, October 2008.
92. Schuller, B., Wöllmer, M., Eyben, F. & Rigoll, G. 2009 Prosodic, spectral or voice quality? Feature type relevance for die discrimination of emotion pairs. In The role of prosody in affective speech (ed. S. Hancil), pp. 285-307. Berne, Switzerland: Peter Lang.
93. Sengers, P., Liesendahl, R., Magar, W, Seibert, C., Müller, B., Joachims, T., Geng, W, Martensson, P. & Höök, K. 2002 The enigmatics of affect. In Proc. DIS 2002, pp. 87-98. New York, NY: ACM Press
94. Stocker, M. & Hegman, E. 1992 Valuing emotions. Cambridge, UK: Cambridge University Press.
95. Terzopoulos, D. & Waters, K. 1993 Analysis and synthesis of facial image sequences using physical and anatomical models. IEEE PAMI 15, 569-579. (Pubitemid 23675487)
96. Tian, Y.-I., Kanade, T. & Cohn, J. 2005 Facial expression analysis. In Handbook of face recognition (eds S. Z. Li & A. K. Jain), pp. 247-266. Berlin, Germany: Springer.
97. Valstar, M., Gunes, H. & Pantic, M. 2007 How to distinguish posed from spontaneous smiles using geometric features. In Proc. ICMI'07, Nagoya Aichi, Japan, pp. 38-45.
98. Wehrle, T., Kaiser, S., Schmidt, S. & Scherer, K. R. 2000 Studying the dynamics of emotional expression using synthesized facial muscle. Mov. J. Pers. Soc. Psychol. 78, 105-119. (doi:10.1037/0022-3514.78.1.105)
99. Wilhelm, P. & Schoebi, D. 2007 Assessing mood in daily life European. J. Psychol. Assess. 23, 258-267. (doi: 10.1027/ 1015-5759.23.4.258)
100. Wollmer, M., Eyben, F., Reiter, S., Schuller, B., Cox, C., Douglas-Cowie, E. & Cowie, R. 2008 Abandoning emotion classes-towards continuous emotion recognition with modelling of long-range dependencies. In Proc. Interspeech 2008, Brisbane, Australia.
101. Yacoub, S., Simske, S., Lin, X. & Burns, J. 2003 Recognition of emotions in interactive voice response systems. In Proc. Eurospeech 2003, Geneva, pp. 729-732.
102. Young, A. W., Rowland, D., Calder, A. J., Etcoff, N. L., Seth, A. & Perrett, D. I. 1997 Facial expression megamix: tests of dimensional and category accounts of emotion recognition. Cognition 63, 271-313. (doi:10.1016/ S0010-0277(97)00003-6) (Pubitemid 127432166)
103. Zhou, G., Hansen, J. & Kaiser, J. 1999 Methods for stress classification: nonlinear TEO and linear speech based features. In Proc. IEEE Int. Conf. on Acoustics, Speech, and Signal Processing, vol. 4, pp. 2087-2090.