Emotion in the voice influences the way we scan emotional faces

Speech Communication

Volumn 65, Issue , 2014, Pages 36-49

  Rigoulot, Simon ^a,b   Pell, Marc D ^a,b  

^a MCGILL UNIVERSITY   (Canada)

^b MCGILL UNIVERSITY   (Canada)

Author keywords

Cross modal; Emotion; Eye tracking; Face; Prosody; Speech

Indexed keywords

BEHAVIORAL RESEARCH; EYE MOVEMENTS; SPEECH;

CROSS-MODAL; EMOTION; EYE-TRACKING; FACE; PROSODY;

FACE RECOGNITION;

EID: 84903150064     PISSN: 01676393     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.specom.2014.05.006     Document Type: Article

References (77)

1. R. Adolphs, F. Gosselin, T.W. Buchanan, D. Tranel, P. Schyns, and A.R. Damasio A mechanism for impaired fear recognition after amygdala damage Nature 433 7021 2005 68 72 (Pubitemid 40101144)
2. R. Banse, and K.R. Scherer Acoustic profiles in vocal emotion expression J. Pers. Soc. Psychol. 70 3 1996 614 636 (Pubitemid 126420699)
3. J.N. Bassili Emotion recognition: the role of facial movement and the relative importance of upper and lower areas of the face J. Pers. Soc. Psychol. 37 11 1979 2049 2058
4. S. Bate, C. Haslam, and T.L. Hodgson Angry faces are special too: evidence from the visual scanpath Neuropsychology 23 5 2009 658 667
5. D.J. Bayle, M.-A. Henaff, and P. Krolak-Salmon Unconsciously perceived fear in peripheral vision alerts the limbic system: a MEG study PLoS ONE 4 12 2009 e8207
6. O. Beaudry, A. Roy-Charland, M. Perron, I. Cormier, and R. Tapp Featural processing in recognition of emotional facial expressions Cogn. Emot. 28 3 2014 416 432
7. M.W. Becker, and B. Detweiler-Bedell Early detection and avoidance of threatening faces during passive viewing Quart. J. Exp. Psychol. 62 7 2009 1257 1264
8. C. Blais, C. Roy, D. Fiset, M. Arguin, and F. Gosselin The eyes are not the window to basic emotions Neuropsychologia 50 2012 2830 2838
9. T. Brosch, D. Grandjean, D. Sander, and K.R. Scherer Cross-modal emotional attention: emotional voices modulate early stages of visual processing J. Cogn. Neurosci. 21 9 2009 1670 1679
10. V. Bruce, A.M. Burton, and I. Craw Modelling face recognition Philos. Trans. Roy. Soc. Lond. B Biol. Sci. 335 1273 1992 121 127 (Discussion 127)
11. A.J. Calder, and A.W. Young Understanding the recognition of facial identity and facial expression Nat. Rev. Neurosci. 6 2005 641 651 (Pubitemid 41059338)
12. A.J. Calder, A.W. Young, J. Keane, and M. Dean Configural information in facial expression perception J. Exp. Psychol. Hum. Percept. Perform. 26 2 2000 527 551
13. M.G. Calvo, and A. Fernández-Martín Can the eyes reveal a person's emotions? Biasing role of the mouth expression Motiv. Emot. 37 1 2013 202 211
14. M.G. Calvo, and P.J. Lang Parafoveal semantic processing of emotional visual scenes JEP: Hum. Percept. Perform. 31 3 2005 502 519 (Pubitemid 41017473)
15. M.G. Calvo, and H. Marrero Visual search of emotional faces: the role of affective content and featural distinctiveness Cogn. Emot. 23 4 2009 782 806
16. M.G. Calvo, and L. Nummenmaa Detection of emotional faces: salient physical features guide effective visual search J. Exp. Psycho.: Gen. 137 3 2008 471 494
17. M.G. Calvo, and L. Nummenmaa Time course of discrimination between emotional facial expressions: the role of visual saliency Vis. Res. 51 15 2011 1751 1759
18. S. Campanella, and P. Belin Integrating face and voice in person perception Trends Cogn. Sci. 11 12 2007 535 543 (Pubitemid 350181565)
19. M. Charash, and D. McKay Attention bias for disgust J. Anxiety Disord. 16 5 2002 529 541
20. J.M. Cisler, B.O. Olatunji, J.M. Lohr, and N.L. Williams Attentional bias differences between fear and disgust: implications for the role of disgust in disgust-related anxiety disorders Cogn. Emot. 23 4 2009 675 687
21. O. Collignon, S. Girard, F. Gosselin, S. Roy, D. Saint-Amour, and M. Lassonde et al. Audio-visual integration of emotion expression Brain Res. 1242 2008 126 135
22. R.M. Cooper The control of eye fixation by the meaning of spoken language. A new methodology for the real-time investigation of speech perception, memory, and language processing Cogn. Psychol. 6 1974 84 107
23. E. Cvejic, J. Kim, and C. Davis Prosody off the top of the head: prosodic contrasts can be discriminated by head motion Speech Commun. 52 6 2010 555 564
24. D. Dahan, J.S. Magnuson, and M.K. Tanenhaus Time course of frequency effects in spoken-word recognition: evidence from eye movements Cogn. Psychol. 42 4 2001 317 367 (Pubitemid 33647297)
25. C. Darwin The Expression of the Emotions in Man and Animals third ed. 1872 Philosophical Library New York (London: Harper Collins, New York: Oxford University Press, 1998)
26. B. De Gelder, and J. Vroomen The perception of emotions by ear and by eye Cogn. Emot. 14 3 2000 289 311 (Pubitemid 30367746)
27. R.J. Dolan, J.S. Morris, and B. de Gelder Crossmodal binding of fear in voice and face Proc. Natl. Acad. Sci. USA 98 17 2001 10006 10010 (Pubitemid 32769418)
28. H. Eisenbarth, and G.W. Alpers Happy mouth and sad eyes: scanning emotional facial expressions Emotion 11 4 2011 860 865
29. P. Ekman, and W.V. Friesen Pictures of Facial Affect 1976 Consulting Psychologists Press Palo Alto, CA
30. P. Ekman, R.J. Davidson, and W. Friesen The Duchenne smile: emotional expression and brain physiology: II J. Pers. Soc. Psychol. 58 1990 343 353
31. Ekman, P., Friesen, W.V., Hager, J., 2002. The Facial Action Coding System. London, UK.
32. M.S. Gordon, and M. Hibberts Audiovisual speech from emotionally expressive and lateralized faces Quart. J. Exp. Psychol. 64 4 2011 730 750
33. F. Gosselin, and P.G. Schyns Bubbles: a technique to reveal the use of information in recognition tasks Vis. Res. 41 17 2001 2261 2271 (Pubitemid 32635177)
34. M.J. Green, L.M. Williams, and D. Davidson In the face of danger: specific viewing strategies for facial expressions of threat? Cogn. Emot. 17 5 2003 779 786 (Pubitemid 36926454)
35. J.K. Hall, S.B. Hutton, and M.J. Morgan Sex differences in scanning faces: does attention to the eyes explain female superiority in facial expression recognition? Cogn. Emot. 24 4 2010 629 637
36. F. Huettig, and G.T.M. Altmann Word meaning and the control of eye fixation: semantic competitor effects and the visual world paradigm Cognition 96 1 2005 B23 B32
37. S. Hunnius, T.C.J. de Wit, S. Vrins, and C. von Hofsten Facing threat: infants' and adults' visual scanning of faces with neutral, happy, sad, angry, and fearful emotional expressions Cogn. Emot. 25 2 2011 193 205
38. R.E. Jack, C. Blais, C. Scheepers, P.G. Schyns, and R. Caldara Cultural confusions show that facial expressions are not universal Curr. Biol. 19 2009 1543 1548
39. A. Jaywant, and M.D. Pell Categorical processing of negative emotions from speech prosody Speech Commun. 54 1 2012 1 10
40. P.N. Juslin, and P. Laukka Communication of emotions in vocal expression and music performance: different channels, same code? Psychol. Bull. 129 5 2003 770 814 (Pubitemid 37394950)
41. G.L. Malcolm, L.J. Lanyon, A.J.B. Fugard, and J.J.S. Barton Scan patterns during the processing of facial expression versus identity: an exploration of task-driven and stimulus-driven effects J. Vis. 8 8 2008 (Art. 2)
42. D. Matsumoto Face, culture, and judgments of anger and fear: do the eyes have it? J. Nonverbal Behav. 13 3 1989 171 188
43. D.S. Messinger, W.I. Mattson, M.H. Mahoor, and J.F. Cohn The eyes have it: making positive expressions more positive and negative expressions more negative Emotion 12 3 2012 430 436
44. K. Neath, and R.J. Itier Facial expression discrimination varies with presentation time but not with fixation on features: a backward masking study using eye-tracking Cogn. Emot. 28 1 2014 115 131
45. P.M. Niedenthal Embodying emotion Science 316 5827 2007 1002 1005 (Pubitemid 46799482)
46. R. Palermo, and G. Rhodes Are you always on my mind? A review of how face perception and attention interact Neuropsychologia 45 1 2007 75 92 (Pubitemid 44821232)
47. S. Paulmann, and M.D. Pell Contextual influences of emotional speech prosody on face processing: how much is enough? Cogn., Affect. Behav. Neurosci. 10 2 2010 230 242
48. S. Paulmann, and M.D. Pell Is there an advantage for recognizing multi-modal emotional stimuli? Motiv. Emot. 35 2011 192 201
49. S. Paulmann, D. Titone, and M.D. Pell How emotional prosody guides your way: evidence from eye movements Speech Commun. 54 2012 92 107
50. M.D. Pell The temporal organization of affective and non-affective speech in patients with right-hemisphere infarcts Cortex 35 4 1999 455 477 (Pubitemid 29490767)
51. M.D. Pell Fundamental frequency encoding of linguistic and emotional prosody by right hemisphere-damaged speakers Brain Lang. 69 2 1999 161 192 (Pubitemid 29408976)
52. M.D. Pell Prosody-face interactions in emotional processing as revealed by the facial affect decision task J. Nonverbal Behav. 29 4 2005 193 215 (Pubitemid 41807466)
53. M.D. Pell, and S.R. Baum Unilateral brain damage, prosodic comprehension deficits, and the acoustic cues to prosody Brain Lang. 57 2 1997 195 214 (Pubitemid 27187882)
54. M.D. Pell, and S.A. Kotz On the time course of vocal emotion recognition PLoS ONE 6 11 2011
55. M.D. Pell, S. Paulmann, C. Dara, A. Alasseri, and S.A. Kotz Factors in the recognition of vocally expressed emotions: a comparison of four languages J. Phonet. 37 4 2009 417 435
56. M.D. Pell, A. Jaywant, L. Monetta, and S.A. Kotz Emotional speech processing: disentangling the effects of prosody and semantic cues Cogn. Emot. 25 5 2011 834 853
57. G. Pourtois, B. de Gelder, A. Bol, and M. Crommelinck Perception of facial expressions and voices and of their combination in the human brain Cortex 41 1 2005 49 59 (Pubitemid 39600225)
58. S. Rigoulot, and M.D. Pell Seeing emotion with your ears: emotional prosody implicitly guides visual attention to faces PLoS ONE 7 1 2012 e30740
59. S. Rigoulot, F. D'Hondt, S. Defoort-Dhellemmes, P. Despretz, J. Honoré, and H. Sequeira Fearful faces impact in peripheral vision: behavioral and neural evidence Neuropsychologia 49 2011 2013 2021
60. S. Rigoulot, F. D'Hondt, J. Honoré, and H. Sequeira Implicit emotional processing in peripheral vision: behavioral and neural evidence Neuropsychologia 50 2012 2887 2896
61. Rigoulot, S., Fish, K., Pell, M.D., in press. Neural correlates of inferring speaker sincerity from white lies: an event-related potential source localization study. Brain Res.
62. G. Rousselet, O. Joubert, and M. Fabre-Thorpe How long to get to the "gist" of real-world natural scenes? Vis. Cogn. 12 6 2005 852 877 (Pubitemid 41283168)
63. K.R. Scherer, R. Banse, H.G. Wallbott, and T. Goldbeck Vocal cues in emotion encoding and decoding Motiv. Emot. 15 2 1991 123 148
64. P.G. Schyns, L. Bonnar, and F. Gosselin Show me the features! Understanding recognition from the use of visual information Psychol. Sci. 13 5 2002 402 409 (Pubitemid 41692260)
65. J.F. Stins, K. Roelofs, J. Villan, K. Koojiman, M.A. Hagenaars, and P.J. Beek Walk to me when I smile, step back when I'm angry: emotional faces modulate whole-body approach-avoidance behaviors Exp. Brain Res. 212 2011 603 611
66. M. Swerts, and E. Krahmer Facial expression and prosodic prominence: effects of modality and facial area J. Phonet. 36 2008 219 238
67. A. Tanaka, A. Koizumi, H. Imai, S. Hiramatsu, E. Hiramoto, and B. de Gelder I feel your voice: cultural differences in the multisensory perception of emotion Psychol. Sci. 21 9 2010 1259 1262
68. M.K. Tanenhaus, M.J. Spivey-Knowlton, K.M. Eberhard, and J.C. Sedivy Integration of visual and linguistic information in spoken language comprehension Science 268 1995 1632 1634
69. L.A. Thompson, D.M. Malloy, and K.L. LeBlanc Lateralization of visuospatial attention across face regions varies with emotional prosody Brain Cogn. 69 1 2009 108 115
70. N. Tottenham, J.W. Tanaka, A.C. Leon, T. McCarry, M. Nurse, and T.A. Hare et al. The NimStim set of facial expressions: judgments from untrained research participants Psychiat. Res. 168 3 2009 242 249
71. M.W. Vasey, and J.F. Thayer The continuing problem of false positives in repeated measures ANOVA in psychophysiology: a multivariate solution Psychophysiology 24 4 1987 479 486 (Pubitemid 17114214)
72. S. Vassallo, S.L. Cooper, and J.M. Douglas Visual scanning in the recognition of facial affect: is there an observer sex difference? J. Vis. 9 3 2009 (Art. 11)
73. B. Wong, A. Cronin-Golomb, and S. Neargarder Patterns of visual scanning as predictors of emotion identification in normal aging Neuropsychology 19 6 2005 739 749 (Pubitemid 41803520)
74. A. Yarbus Eye Movements and Vision 1967 Plenum Press New York
75. E. Yee, and J.C. Sedivy Eye movements to pictures reveal transient semantic activation during spoken word recognition J. Exp. Psychol. Learn. Mem. Cogn. 32 1 2006 1 14 (Pubitemid 43283742)
76. E. Yee, E. Overton, and S.L. Thompson-Schill Looking for meaning: eye movements are sensitive to overlapping semantic features, not association Psychon. Bull. Rev. 16 5 2009 869 874
77. M. Yuki, W.W. Maddux, and T. Masuda Are the windows to the soul the same in the east and west? Cultural differences in using the eyes and mouth as cues to recognize emotions in Japan and the United States J. Exp. Soc. Psychol. 43 2007 303 311