Seeing emotion with your ears: Emotional prosody implicitly guides visual attention to faces

PLoS ONE

Volumn 7, Issue 1, 2012, Pages

  Rigoulot, Simon ^a,b   Pell, Marc D ^a,b  

^a MCGILL UNIVERSITY   (Canada)

^b MCGILL UNIVERSITY   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ANGER; ARTICLE; ATTENTION; AUDITORY STIMULATION; CONTROLLED STUDY; EMOTION; EYE MOVEMENT; FACIAL EXPRESSION; FEAR; FEMALE; GAZE; HAPPINESS; HEARING; HUMAN; HUMAN EXPERIMENT; MALE; NORMAL HUMAN; RECALL; SACCADIC EYE MOVEMENT; SPEECH; STIMULUS RESPONSE; VISION; VISUAL STIMULATION; BEHAVIOR; EAR; EYE FIXATION; FACE; PHYSIOLOGY; STIMULATION; TIME;

ATTENTION; BEHAVIOR; EAR; EMOTIONS; FACE; FEMALE; FIXATION, OCULAR; HUMANS; MALE; MENTAL RECALL; PHYSICAL STIMULATION; SACCADES; TIME FACTORS; VISUAL PERCEPTION; YOUNG ADULT;

EID: 84856386083     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0030740     Document Type: Article

References (105)

1. Brosch T, Grandjean D, Sander D, Scherer KR, (2008) Behold the voice of wrath: cross-modal modulation of visual attention by anger prosody. Cognition 106: 1497-1503.
2. Calvert GA, Brammer MJ, Iversen SD, (1998) Crossmodal identification. Trends in Cognitive Sciences 2: 247-253.
3. Summerfield Q, (1979) Use of visual information for phonetic perception. Phonetica 36: 314-331.
4. Gleitman LR, January D, Nappa R, Trueswell JC, (2007) On the give and take between event apprehension and utterance formulation. Journal of Memory and Language 57: 544-569.
5. McGurk H, MacDonald J, (1976) Hearing lips and seeing voices. Nature 264: 746-748.
6. Dolan RJ, Morris JS, de Gelder B, (2001) Crossmodal binding of fear in voice and face. Proc Natl Acad Sci U S A 98: 10006-10010.
7. Pourtois G, Debatisse D, Despland PA, de Gelder B, (2002) Facial expressions modulate the time course of long latency auditory brain potentials. Brain Res Cogn Brain Res 14: 99-105.
8. Vroomen J, Driver J, de Gelder B, (2001) Is cross-modal integration of emotional expressions independent of attentional resources? Cogn Affect Behav Neurosci 1: 382-387.
9. Juslin PN, Laukka P, (2003) Communication of Emotions in Vocal Expression and Music Performance: Different Channels, Same Code? Psychological Bulletin 129 (5): 770-814.
10. Aue T, Cuny C, Sander D, Grandjean D, (2011) Peripheral responses to attended and unattended angry prosody: A dichotic listening paradigm. Psychophysiology 48 (3): 385-92.
11. Schirmer A, Kotz SA, (2006) Beyond the right hemisphere: Brain mechanisms mediating vocal emotional processing. Trends in Cognitive Sciences 10 (1): 24-30.
12. Collignon O, Girard S, Gosselin F, Roy S, Saint-Amour D, et al. (2008) Audio-visual integration of emotion expression. Brain Res 1242: 126-135.
13. Paulmann S, Pell M, (2011) Is there an advantage for recognizing multi-modal emotional stimuli? Motivation and Emotion 35 (2): 192-201.
14. Calvert GA, Campbell R, Brammer MJ, (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10: 649-657.
15. De Gelder B, Böcker KBE, Tuomainen J, Hensen M, Vroomen J, (1999) The combined perception of emotion from voice and face: Early interaction revealed by human electric brain responses. Neuroscience Letters 260: 133-136.
16. Ethofer T, Pourtois G, Wildgruber D, (2006) Investigating audiovisual integration of emotional signals in the human brain. Prog Brain Res 156: 345-361.
17. Pourtois G, de Gelder B, Bol A, Crommelinck M, (2005) Perception of facial expressions and voices and of their combination in the human brain. Cortex 41: 49-59.
18. Pourtois G, de Gelder B, Vroomen J, Rossion B, Crommelinck M, (2000) The time-course of intermodal binding between seeing and hearing affective information. Neuroreport 11: 1329-1333.
19. Brosch T, Grandjean D, Sander D, Scherer KR, (2009) Cross-modal emotional attention: emotional voices modulate early stages of visual processing. J Cogn Neurosci 21: 1670-1679.
20. De Gelder B, Vroomen J, (2000) The perception of emotions by ear and by eye. Cognition and Emotion 14: 289-311.
21. Hietanen JK, Leppänen JM, Illi M, Surakka V, (2004) Evidence for the integration of audiovisual emotional information at the perceptual level of processing. European Journal of Cognitive Psychology 16: 769-790.
22. Massaro DW, Egan PB, (1996) Perceiving affect from the voice and the face. Psychonomic Bulletin and Review 3: 215-221.
23. Pell MD, (2005) Nonverbal emotion priming: Evidence from the 'Facial Affect Decision Task'. Journal of Nonverbal Behavior 29: 45-73.
24. Pell MD, (2005) Prosody-face interactions in emotional processing as revealed by the facial affect decision task. Journal of Nonverbal Behavior 29: 193-215.
25. Paulmann S, Pell MD, (2010) Contextual influences of emotional speech prosody on face processing: how much is enough? Cognitive, Affective and Behavioral Neuroscience 10: 230-242.
26. Vroomen J, de Gelder B, (2000) Sound enhances visual perception: cross-modal effects of auditory organization on vision. J Exp Psychol Hum Percept Perform 26: 1583-1590.
27. Bostanov V, Kotchoubey B, (2004) Recognition of affective prosody: continuous wavelet measures of event-related brain potentials to emotional exclamations. Psychophysiology 41: 259-268.
28. Nygaard LC, Queen JS, (2008) Communicating emotion: linking affective prosody and word meaning. J Exp Psychol Hum Percept Perform 34: 1017-1030.
29. Schirmer A, Kotz SA, Friederici AD, (2002) Sex differentiates the role of emotional prosody during word processing. Brain Res Cogn Brain Res 14: 228-233.
30. Seubert J, Kellermann T, Loughead J, Boers F, Brensinger C, et al. (2010) Processing of disgusted faces is facilitated by odor primes: a functional MRI study. Neuroimage 53: 746-756.
31. Spruyt A, Hermans D, De Houwer J, Vandromme H, Eelen P, (2007) On the nature of the affective priming effect: effects of stimulus onset asynchrony and congruency proportion in naming and evaluative categorization. Mem Cognit 35: 95-106.
32. Zhang Q, Li X, Gold BT, Jiang Y, (2010) Neural correlates of cross-domain affective priming. Brain Res 1329: 142-151.
33. Pell MD, Jaywant A, Monetta L, Kotz SA, (2011) Emotional speech processing: Disentangling the effects of prosody and semantic cues. Cognition and Emotion 25: 834-853.
34. Borod JC, Pick LH, Hall S, Sliwinski M, Madigan N, et al. (2000) Relationships among facial, prosodic, and lexical channels of emotional perceptual processing. Cognition and Emotion 14: 193-211.
35. Niedenthal PM, (2007) Embodying emotion. Science 316: 1002-1005.
36. Niedenthal PM, Halberstadt JB, Innes-Ker AH, (1999) Emotional response categorization. Psychological Review 106: 337-361.
37. Paulmann S, Jessen S, Kotz SA, (2009) Investigating the multimodal nature of human communication: Insights from ERPs. Journal of Psychophysiology 23: 63-76.
38. Ethofer T, Anders S, Erb M, Droll C, Royen L, et al. (2006) Impact of voice on emotional judgment of faces: an event-related fMRI study. Hum Brain Mapp 27: 707-714.
39. Ethofer T, Anders S, Erb M, Herbert C, Wiethoff S, et al. (2006) Cerebral pathways in processing of affective prosody: A dynamic causal modeling study. Neuroimage 30: 580-587.
40. Bach DR, Grandjean D, Sander D, Herdener M, Strik WK, et al. (2008) The effect of appraisal level on processing of emotional prosody in meaningless speech. Neuroimage 42: 919-927.
41. Grandjean D, Sander D, Pourtois G, Schwartz S, Seghier ML, et al. (2005) The voices of wrath: brain responses to angry prosody in meaningless speech. Nat Neurosci 8: 145-146.
42. Paulmann S, Kotz SA, (2008) Early emotional prosody perception based on different speaker voices. Neuroreport 19: 209-213.
43. Pell MD, Skorup V, (2008) Implicit processing of emotional prosody in a foreign versus native language. Speech Communication 50: 519-530.
44. Sauter DA, Eimer M, (2010) Rapid detection of emotion from human vocalizations. Journal of Cognitive Neuroscience 22: 474-481.
45. Schirmer A, (2010) Mark my words: Tone of voice changes affective word representations in memory. PLoS One 5 (2): e9080.
46. Calder AJ, Young AW, (2005) Understanding the recognition of facial identity and facial expression. Nat Rev Neurosci 6: 641-651.
47. Palermo R, Rhodes G, (2007) Are you always on my mind? A review of how face perception and attention interact. Neuropsychologia 45: 75-92.
48. Vuilleumier P, Pourtois G, (2007) Distributed and interactive brain mechanisms during emotion face perception: evidence from functional neuroimaging. Neuropsychologia 45: 174-194.
49. Eastwood JD, Smilek D, Merikle PM, (2001) Differential attentional guidance by unattended faces expressing positive and negative emotion. Percept Psychophys 63: 1004-1013.
50. Frischen A, Eastwood JD, Smilek D, (2008) Visual Search for Faces With Emotional Expressions. Psychological Bulletin 134: 662-676.
51. Ohman A, Lundqvist D, Esteves F, (2001) The face in the crowd revisited: a threat advantage with schematic stimuli. J Pers Soc Psychol 80: 381-396.
52. Tipples J, Atkinson AP, Young AW, (2002) The eyebrow frown: a salient social signal. Emotion 2: 288-296.
53. Bradley BP, Mogg K, Falla SJ, Hamilton LR, (1998) Attentional Bias for Threatening Facial Expressions in Anxiety: Manipulation of Stimulus Duration. Cognition and Emotion 12: 737-753.
54. Calvo MG, Avero P, Lundqvist D, (2006) Facilitated detection of angry faces: Initial orienting and processing efficiency. Cognition and Emotion 20: 785-811.
55. Calvo MG, Nummenmaa L, (2009) Eye-movement assessment of the time course in facial expression recognition: Neurophysiological implications. Cogn Affect Behav Neurosci 9: 398-411.
56. Fox E, Damjanovic L, (2006) The eyes are sufficient to produce a threat superiority effect. Emotion 6: 534-539.
57. Fox E, Lester V, Russo R, Bowles RJ, Pichler A, et al. (2000) Facial Expressions of Emotion: Are Angry Faces Detected More Efficiently? Cognition and Emotion 14: 61-92.
58. Williams MA, Moss SA, Bradshaw JL, Mattingley JB, (2005) Look at me, I'm smiling: Visual search for threatening and nonthreatening facial expressions. Visual Cognition 12: 29-50.
59. Juth P, Lundqvist D, Karlsson A, Ohman A, (2005) Looking for foes and friends: perceptual and emotional factors when finding a face in the crowd. Emotion 5: 379-395.
60. Leppänen JM, Hietanen JK, (2004) Positive facial expressions are recognized faster than negative facial expressions, but why? Psychological Research 69: 22-29.
61. Brosch T, Sander D, Scherer KR, (2007) That baby caught my eye... attention capture by infant faces. Emotion 7: 685-689.
62. Tanenhaus MK, Spivey-Knowlton MJ, Eberhard KM, Sedivy JC, (1995) Integration of visual and linguistic information in spoken language comprehension. Science 268: 1632-1634.
63. Huettig F, Altmann GTM, (2005) Word meaning and the control of eye fixation: Semantic competitor effects and the visual world paradigm. Cognition 96: B23-B32.
64. Paulmann S, Titone D, Pell MD, (2012) How emotional prosody guides your way: evidence from eye movements. Speech Communication 54: 92-107.
65. Spielberger CD, Gorsuch RL, Lushene PR, Vagg PR, Jacobs AG, (1983) Manual for the State-Trait Anxiety Inventory (Form Y). Consulting Psychologists Press, Inc.: Palo Alto.
66. Pell MD, Paulmann S, Dara C, Alasseri A, Kotz SA, (2009) Factors in the recognition of vocally expressed emotions: A comparison of four languages. Journal of Phonetics 37: 417-435.
67. Pell MD, Baum SR, (1997) The ability to perceive and comprehend intonation in linguistic and affective contexts by brain-damaged adults. Brain and Language 57: 80-99.
68. Scherer KR, Banse R, Wallbott HG, Goldbeck T, (1991) Vocal cues in emotion encoding and decoding. Motivation and Emotion 15: 123-148.
69. Boersma P, Weenink D, (2010) Praat (Version 5.2.03) [Computer software]. Retrieved November 19, 2010, fromhttp://www.praat.org/.
70. Calvo MG, Lang PJ, (2005) Parafoveal semantic processing of emotional visual scenes. J Exp Psychol Hum Percept Perform 31: 502-519.
71. Paulmann S, Pell MD, (2009) Facial expression decoding as a function of emotional meaning status: ERP evidence. Neuroreport 20: 1603-1608.
72. Vasey MW, Thayer JF, (1987) The continuing problem of false positives in repeated measures ANOVA in psychophysiology: A multivariate solution. Psychophysiology 24: 479-486.
73. Rosenthal R, (1991) Meta-analytic procedures for social research. Appl Soc Res Methods 6: 19.
74. Pourtois G, Grandjean D, Sander D, Vuilleumier P, (2004) Electrophysiological correlates of rapid spatial orienting towards fearful faces. Cereb Cortex 14: 619-633.
75. Pourtois G, Schwartz S, Seghier ML, Lazeyras F, Vuilleumier P, (2006) Neural systems for orienting attention to the location of threat signals: an event-related fMRI study. Neuroimage 31: 920-933.
76. Reynolds MG, Eastwood JD, Partanen M, Frischen A, Smilek D, (2009) Monitoring eye movements while searching for affective faces. Visual Cognition 17: 318-333.
77. Reinders AA, Glascher J, de Jong JR, Willemsen AT, den Boer JA, Buchel C, (2006) Detecting fearful and neutral faces: BOLD latency differences in amygdala-hippocampal junction. Neuroimage 33 (2): 805-814.
78. Becker MW, Detweiler-Bedell B, (2009) Early detection and avoidance of threatening faces during passive viewing. Q J Exp Psychol (Colchester) 62: 1257-1264.
79. Whalen PJ, Shin LM, McInerney SC, Fischer H, Wright CI, et al. (2001) A functional MRI study of human amygdala responses to facial expressions of fear versus anger. Emotion 1: 70-83.
80. Peelen MV, Atkinson AP, Vuilleumier P, (2010) Supramodal representations of perceived emotions in the human brain. J Neurosci 30: 10127-10134.
81. Campanella S, Belin P, (2007) Integrating face and voice in person perception. Trends Cogn Sci 11: 535-543.
82. Seyfarth RM, Cheney DL, (2009) Seeing who we hear and hearing who we see. Proc Nat Acad Sci USA 106: 669-670.
83. Pell MD, Kotz SA, (2011) On the time course of vocal emotion recognition. PLoS ONE 6 (11): e27256.
84. Ekman P, (1992) Are there basic emotions? Psychol Rev 99: 550-553.
85. Zuckerman M, Lipets M, Hall Koivumaki J, Rosenthal R, (1975) Encoding and decoding nonverbal cues of emotion. Journal of Personality and Social Psychology 32 (6): 1068-1076.
86. Thönnessen H, Boers F, Dammers J, Chen YH, Norra C, et al. (2010) Early sensory encoding of affective prosody: Neuromagnetic tomography of emotional category changes. Neuroimage 50: 250-259.
87. Compton RJ, (2003) The interface between emotion and attention: a review of evidence from psychology and neuroscience. Behav Cogn Neurosci Rev 2: 115-129.
88. Bechara A, Damasio H, Damasio AR, (2000) Emotion, decision making and the orbitofrontal cortex. Cerebral Cortex 10: 295-307.
89. Buchanan TW, (2007) Retrieval of Emotional Memories. Psychological Bulletin 133: 761-779.
90. Kensinger EA, (2009) Remembering the details: Effects of emotion. Emotion Review 1: 99-113.
91. Weymar M, Löw A, Melzig CA, Hamm AO, (2009) Enhanced long-term recollection for emotional pictures: Evidence from high-density ERPs. Psychophysiology 46: 1200-1207.
92. Dara C, Monetta L, Pell MD, (2008) Vocal emotion processing in Parkinson's disease: reduced sensitivity to negative emotions. Brain Res 1188: 100-111.
93. Ethofer T, Van De Ville D, Scherer K, Vuilleumier P, (2009) Decoding of Emotional Information in Voice-Sensitive Cortices. Current Biology 19: 1028-1033.
94. Spreckelmeyer KN, Kutas M, Urbach T, Altenmuller E, Munte TF, (2009) Neural processing of vocal emotion and identity. Brain Cogn 69: 121-126.
95. Wittfoth M, Schroder C, Schardt DM, Dengler R, Heinze HJ, et al. (2010) On emotional conflict: interference resolution of happy and angry prosody reveals valence-specific effects. Cereb Cortex 20: 383-392.
96. Batty M, Taylor MJ, (2003) Early processing of the six basic facial emotional expressions. Brain Res Cogn Brain Res 17: 613-620.
97. Esslen M, Pascual-Marqui RD, Hell D, Kochi K, Lehmann D, (2004) Brain areas and time course of emotional processing. Neuroimage 21: 1189-1203.
98. Park JY, Gu BM, Kang DH, Shin YW, Choi CH, et al. (2010) Integration of cross-modal emotional information in the human brain: an fMRI study. Cortex 46: 161-169.
99. Lang PJ, Greenwald MK, Bradley MM, Hamm AO, (1993) Looking at pictures: affective, facial, visceral, and behavioral reactions. Psychophysiology 30: 261-273.
100. Kiss M, Van Velzen J, Eimer M, (2008) The N2pc component and its links to attention shifts and spatially selective visual processing. Psychophysiology 45: 240-249.
101. Russell JA, Bachorowski JA, Fernández-Dols JM, (2003) Facial and Vocal Expressions of Emotion. Ann Rev Psych 54: 329-349.
102. Swerts M, (2009) The relevance of visual prosody for studies in language and speech-language pathology. International Journal of Speech-Language Pathology 11: 282-286.
103. Noppeney U, Ostwald D, Werner S, (2010) Perceptual decisions formed by accumulation of audiovisual evidence in prefrontal cortex. Journal of Neuroscience 30: 7434-7446.
104. Kamachi M, Hill H, Lander K, Vatikiotis-Bateson E, (2003) 'Putting the Face to the Voice': Matching Identity across Modality. Current Biology 13: 1709-1714.
105. Green MJ, Williams LM, Davidson D, (2003) In the face of danger: Specific viewing strategies for facial expressions of threat? Cognition and Emotion 17 (5): 779-786.