Crossmodal and incremental perception of audiovisual cues to emotional speech

Language and Speech

Volumn 53, Issue 1, 2010, Pages 3-30

  Barkhuysen, Pashiera ^a   Krahmer, Emiel ^a   Swerts, Marc ^a  

^a TILBURG UNIVERSITY   (Netherlands)

Author keywords

Audiovisual speech; Emotional speech; Multimodality; Perception; Timing

Indexed keywords

EID: 67949108119     PISSN: 00238309     EISSN: None     Source Type: Journal    
DOI: 10.1177/0023830909348993     Document Type: Article

References (47)

1. Adolphs, R. (2002). Recognizing emotion from facial expressions: Psychological and neurological mechanisms. Behavioral and Cognitive Neuroscience Review, 1(1), 21- 61.
2. Attardo, S., Eisterhold, J., Hay, J., & Poggi, I. (2003). Multimodal markers of irony and sarcasm. Humor: International Journal of Humor Research, 16(2), 243-260.
3. Aubergé, V., & Cathiard, M.-A. (2003). Can we hear the prosody of smile? Speech Communication, 40(1-2), 87-97.
4. Bachorowski, J.-A. (1999). Vocal expression and perception of emotion. Current Directions in Psychological Science, 8(2), 53-57.
5. Banse, R., & Scherer, K.R. (1996). Acoustic profiles in vocal emotion expression. Journal of Personality and Social Psychology, 70(3), 614-636.
6. Barkhuysen, P., Krahmer, E., & Swerts, M. (2004). Problem detection in human-machine interactions based on facial expressions of users. Speech Communication, 45(3), 343-359.
7. Barkhuysen, P., Krahmer, E., & Swerts, M. (2008). The interplay between the auditory and visual modality for end-of-utterance detection. The Journal of the Acoustical Society of America, 123(1), 354-365.
8. Calvert, G.A. (2001). Crossmodal processing in the human brain: Insights from functional neuroimaging studies. Cerebral Cortex, 11, 1110-1123.
9. Calvert, G.A., Brammer, M.J., & Iversen, S.D. (1998). Crossmodal identification. Trends in Cognitive Sciences, 2(7), 247-253.
10. Carroll, J.M., & Russell, J.A. (1996) Do facial expressions signal specific emotions? Judging emotions from the face in context. Journal of Personality and Social Psychology, 70(2), 205-218.
11. Cohn, J.F., & Katz, G.S. (1998, September). Bimodal expression of emotion by face and voice. Paper presented at the Workshop on Face/Gesture Recognition and Their Applications, The Sixth ACM International Multimedia Conference, Bristol, UK.
12. Cohn, J.F., & Schmidt K.L. (2004). The timing of facial motion in posed and spontaneous smiles. International Journal of Wavelets, Multiresolution and Information Processing, 2, 1-12.
13. de Gelder, B., Bocker, K., Tuomainen, J., Hensen, M., & Vroomen, J. (1999). The combined perception of emotion from voice and face: Early interaction revealed by human electric brain responses. Neuroscience Letters, 260, 133-136.
14. de Gelder, B., & Vroomen, J. (2000). The perception of emotions by ear and by eye. Cognition and Emotion, 14(3), 289-311.
15. Ekman, P. (1993). Facial expression and emotion. American Psychologist, 48, 384-392.
16. Ekman, P. (2004). Emotions revealed: Recognizing faces and feelings to improve communication and emotional life. New York: Owl Books.
17. Ekman, P., Davidson, R.J., & Friesen, W.V. (1990). The Duchenne smile: Emotional expression and brain physiology II. Journal of Personality and Social Psychology, 58(2), 342-353.
18. Ekman, P., & Friesen, W.V. (1978). Facial Action Coding System: A technique for the measurement of facial movement. Palo Alto, CA: Psychologists Press.
19. Ekman, P., Friesen, W.V., & Ellsworth, P. (1972). Emotion in the human face: Guidelines for research and an integration of findings. New York: Pergamon Press.
20. Ekman, P., Friesen, W.V., O'Sullivan, M., Chan, A., Diacoyanni-Tarlatis, I., Heider, K.,et al.(1987). Universals and cultural differences in the judgments of facial expressions of emotion. Journal of Personality and Social Psychology, 53(4), 712-717.
21. Elfenbein, H., & Ambady, N. (2003). When familiarity breeds accuracy: Cultural exposure and facial emotion recognition. Journal of Personality and Social Psychology, 85(2), 276-290.
22. Fox, E., Lester, V., Russo, R., Bowles, R., Pichler, A., & Dutton, K. (2000). Facial expressions of emotion: Are angry faces detected more efficiently? Cognition and Emotion, 14(1), 61-92.
23. Gazzaniga, M.S., & Smylie, C.S. (1990). Hemispheric mechanisms controlling voluntary and spontaneous facial expressions. Journal of Cognitive Neuroscience, 2(3), 239-245.
24. Ghazanfar, A.A., Maier, J.X., Hoffman, K.L., & Logothetis, N.K. (2005). Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. The Journal of Neuroscience, 25(20), 5004-5012.
25. Graf, H.P., Cosatto, E., Ström, V., & Huang, F.J. (2002, May). Visual prosody: Facial movements accompanying speech. Paper presented at the Fifth IEEE International Conference on Automatic Face and Gesture Recognition, Washington, DC, USA.
26. Grosjean, F. (1996). Gating. Language and Cognitive Processes, 11(6), 597-604.
27. Grossmann, T., Striano, T., & Friederici, A. (2006). Crossmodal integration of emotional information from face and voice in the infant brain. Developmental Science, 9(3), 309-315.
28. Hietanen, J.K., Manninen, P., Sams, M., & Rusakka, V. (2001). Does audiovisual speech perception use information about facial configuration? European Journal of Cognitive Psychology, 13(3), 395-407.
29. Hirschberg, J., Litman, D., & Swerts, M. (2004). Prosodic and other cues to speech recognition failures. Speech Communication, 43, 155-175.
30. Horstmann, G. (2002). Facial expressions of emotion: Does the prototype represent central tendency, frequency of instantiation, or an ideal? Emotion, 2(3), 297-305.
31. Humphreys, G.W., Donnelly, N., & Riddoch, M.J. (1993). Expression is computed separately form facial identity, and it is computed separately for moving and static faces: Neuropsychological evidence. Neuropsychologia, 31(2), 173-181.
32. Krahmer, E.J., Van Dorst, J., & Ummelen, N. (2004). Mood, persuasion and information presentation: The influence of mood on the effectiveness of persuasive digital documents. Information Design Journal + Document Design, 12(3), 40-52.
33. Leppänen, J., & Hietanen, J.K. (2004). Positive facial expressions are recognized faster than negative facial expressions, but why? Psychological Research/Psychologische Forschung, 69, 22-29.
34. Mackie, D., & Worth, L. (1989). Processing deficits and the mediation of positive affect in persuasion. Journal of Personality and Social Psychology, 57, 27-40.
35. McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746-748.
36. Pourtois, G., de Gelder, B., Vroomen, J., Rossion, B., & Crommelinck, M. (2000). The time-course of intermodal binding between seeing and hearing affective information. Cognitive Neuroscience, 11(6), 1329-1333.
37. Rinn, W.E. (1984). The neuropsychology of facial expressions: A review of the neurological and psychological mechanisms for producing facial expressions. Psychological Bulletin, 95(1), 52-77.
38. Rinn, W.E. (1991) Neuropsychology of facial expression. In R. S. Feldman & B. Rimé (Eds.), Fundamentals of nonverbal behavior (pp. 3-30). Cambridge: Cambridge University Press.
39. Russell, J.A., Bachorowski, J., & Fernández-Dols, J. (2003). Facial and vocal expressions of emotion. Annual Review of Psychology, 54, 329-49.
40. Scherer, K.R. (2003). Vocal communication of emotion: A review of research paradigms. Speech Communication, 40, 227-256.
41. Schmidt, K., & Cohn, J. (2001). Human facial expressions as adaptations: Evolutionary questions in facial expression research. Yearbook of Physical Anthropology, 44, 3-24.
42. Sekiyama, K., Kanno, I., Miura, S., & Sugita, Y. (2003). Auditory-visual speech perception examined by fMRI and PET. Neuroscience Research, 47, 277-287.
43. Valstar, M., Pantic, M., Ambadar, Z., & Cohn, J. (2006, November). Spontaneous vs. posed facial behavior: Automatic analysis of brow actions. Paper presented at ICMI 2006, Banff, Canada.
44. Velten, E. (1968). A laboratory task for induction of mood states. Behavior Research Therapy, 6, 473-482.
45. Vuilleumier, P., & Pourtois, G. (2007). Distributed and interactive brain mechanisms during emotion face perception: Evidence from functional neuroimaging. Neuropsychologia, 45, 174-194.
46. Westermann, R., Spies, K., Stahl, G., & Hesse, F.W. (1996). Relative effectiveness and validity of mood induction procedures: A meta analysis. European Journal of Social Psychology, 26, 557-580.
47. Wilting, J., Krahmer, E., & Swerts, M. (2006). Congruent vs. incongruent emotional speech. Interspeech 2006, Pittsburgh PA, USA.