Integrating face and voice in person perception

Trends in Cognitive Sciences

Volumn 11, Issue 12, 2007, Pages 535-543

  Campanella, Salvatore ^a   Belin, Pascal ^b  

^a BRUGMANN UNIVERSITY HOSPITAL   (Belgium)

^b UNIVERSITY OF GLASGOW   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

CROSSTALK; FEEDBACK; IMAGE PROCESSING; INFORMATION ANALYSIS;

AUDIOVISUAL SPEECH PERCEPTION; PERSON PERCEPTION; SOCIAL INTERACTIONS;

SPEECH RECOGNITION;

AMYGDALOID NUCLEUS; AUDITORY CORTEX; BEHAVIOR CONTROL; BEHAVIOR MODIFICATION; BRAIN REGION; ELECTROPHYSIOLOGY; EMOTION; EVOKED CORTICAL RESPONSE; FACE; FACE PROFILE; FACIAL EXPRESSION; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; IDENTITY; INFORMATION PROCESSING; NERVE CELL; NERVE CONDUCTION; NEUROIMAGING; PERCEPTIVE DISCRIMINATION; PHONATION; POSITRON EMISSION TOMOGRAPHY; RECOGNITION; REVIEW; SOCIAL INTERACTION; SPEECH DISCRIMINATION; STIMULUS RESPONSE; VISUAL CORTEX; VOICE;

ANIMALS; AUDITORY PERCEPTION; BRAIN MAPPING; EMOTIONS; FACE; HIGHER NERVOUS ACTIVITY; HUMANS; PATTERN RECOGNITION, VISUAL; SOCIAL PERCEPTION; VOICE;

EID: 36549046939     PISSN: 13646613     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.tics.2007.10.001     Document Type: Review

References (90)

1. Ernst M.O., and Bülthoff H.H. Merging the senses into a robust percept. Trends Cogn. Sci. 8 (2004) 162-169
2. Ellis A.W. Neuro-cognitive processing of faces and voices. In: Young A.W., and Ellis H.D. (Eds). Handbook of Research on Face Processing (1989), Elsevier 207-215
3. Belin P., et al. Thinking the voice: neural correlates of voice perception. Trends Cogn. Sci. 8 (2004) 129-135
4. Calvert G.A., et al. Crossmodal identification. Trends Cogn. Sci. 2 (1998) 247-253
5. Campbell R. The processing of audio-visual speech: empirical and neural bases. Philos. Trans. R. Soc. Lond. B Biol. Sci (2007) 10.1098/rstb.2007.2155
6. Miller L.M., and D'Esposito M. Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J. Neurosci. 25 (2005) 5884-5893
7. McGurk H., and MacDonald J.D. Hearing lips and seeing voices. Nature 264 (1976) 746-748
8. Ethofer T., et al. Impact of voice on emotional judgment of faces: an event-related fMRI study. Hum. Brain Mapp. 27 (2006) 707-714
9. Beauchamp M.S., et al. Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41 (2004) 809-823
10. Callan D.E., et al. Multisensory integration sites identified by perception of spatial wavelet filtered visual speech gesture information. J. Cogn. Neurosci. 16 (2004) 805-816
11. Calvert G.A., et al. Activation of auditory cortex during silent lipreading. Science 276 (1997) 593-596
12. Pekkola J., et al. Primary auditory cortex activation by visual speech: an fMRI study at 3T. Neuroreport 16 (2005) 125-128
13. Besle J., et al. Bimodal speech: early suppressive visual effects in human auditory cortex. Eur. J. Neurosci. 20 (2004) 2225-2234
14. Klucharev V., et al. Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Brain Res. Cogn. Brain Res. 18 (2003) 65-75
15. Ghazanfar A.A., et al. Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. J. Neurosci. 25 (2005) 5004-5012
16. Falchier A., et al. Anatomical evidence of multimodal integration in primate striate cortex. J. Neurosci. 22 (2002) 5749-5759
17. Posamentier M.T., and Abdi H. Processing faces and facial expressions. Neuropsychol. Rev. 13 (2003) 113-143
18. Adolphs R., et al. A mechanism for impaired fear recognition after amygdala damage. Nature 433 (2005) 68-72
19. Morris J.S., et al. A differential neural response in the human amygdala to fearful and happy facial expressions. Nature 383 (1996) 812-815
20. George M.S., et al. Understanding emotional prosody activates right hemisphere region. Arch. Neurol. 53 (1996) 665-670
21. Morris J.S., et al. Saying it with feeling: neural responses to emotional vocalizations. Neuropsychologia 37 (1999) 1155-1163
22. Buchanan T.W., et al. Recognition of emotional prosody and verbal components of spoken language: an fMRI study. Brain Res. Cogn. Brain Res. 9 (2000) 227-238
23. Sander D., et al. Emotion and attention interactions in social cognition: brain regions involved in processing anger prosody. NeuroImage 28 (2005) 848-854
24. Grandjean D., et al. The voices of wrath: brain responses to angry prosody in meaningless speech. Nat. Neurosci. 8 (2005) 145-146
25. Schirmer A., and Kotz S.A. Beyond the right hemisphere: brain mechanisms mediating vocal emotional processing. Trends Cogn. Sci. 10 (2006) 24-30
26. Wildgruber D., et al. Cerebral processing of linguistic and emotional prosody: fMRI studies. Prog. Brain Res. 156 (2006) 249-268
27. Sander K., and Scheich H. Left auditory cortex and amygdala, but right insula dominance for human laughing and crying. J. Cogn. Neurosci. 17 (2005) 1519-1531
28. Fecteau S., et al. Amygdala responses to nonlinguistic emotional vocalizations. Neuroimage 36 (2007) 480-487
29. de Gelder B., and Vroomen J. The perception of emotions by ear and by eye. Cogn. Emotion 14 (2000) 289-311
30. de Gelder B., et al. Fear recognition in the voice is modulated by unconsciously recognized facial expressions but not by unconsciously recognized affective pictures. Proc. Natl Acad. Sci. USA 99 (2002) 4121-4126
31. de Gelder B., et al. The combined perception of emotion from voice and face: early interaction revealed by human electric brain responses. Neurosci. Lett. 260 (1999) 133-136
32. Pourtois G., et al. The time-course of intermodal binding between seeing and hearing affective information. Neuroreport 11 (2000) 1329-1333
33. Pourtois G., et al. Facial expressions modulate the time course of long latency auditory brain potentials. Brain Res. Cogn. Brain Res. 14 (2002) 99-105
34. Foxe J.J., and Schroeder C.E. The case for feedforward multisensory convergence during early cortical processing. Neuroreport 16 (2005) 419-423
35. Dolan R.J., et al. Crossmodal binding of fear in voice and face. Proc. Natl Acad. Sci. USA 98 (2001) 10006-10010
36. Pourtois G., et al. Perception of facial expressions and voices and of their combination in the human brain. Cortex 41 (2005) 49-59
37. Kreifelts B., et al. Audiovisual integration of emotional signals in voice and face: An event-related fMRI study. NeuroImage 37 (2007) 1445-1456
38. Morris J.S., et al. A neuromodulatory role for the human amygdala in processing emotional facial expressions. Brain 121 (1998) 47-57
39. Puce A., et al. Face-sensitive regions in human extrastriate cortex studied by functional MRI. J. Neurophysiol. 74 (1995) 1192-1199
40. Kanwisher N., et al. The fusiform face area: a module in human extrastriate cortex specialized for face perception. J. Neurosci. 17 (1997) 4302-4311
41. Haxby J.V., et al. The distributed human neural system for face perception. Trends Cogn. Sci. 4 (2000) 223-233
42. De Renzi E., et al. Prosopagnosia can be associated with damage confined to the right hemisphere-an MRI and PET study and a review of the literature. Neuropsychologia 32 (1994) 893-902
43. Campanella S., et al. Right N170 modulation in a face discrimination task: An account for categorical perception of familiar faces. Psychophysiology 37 (2000) 796-806
44. Bentin S., et al. Controlling interstimulus perceptual variance does not abolish N170 face sensitivity. Nat. Neurosci. 10 (2007) 802-803
45. Kreiman J. Listening to voices: theory and practice in voice perception research. In: Johnson K., and Mullenix J. (Eds). Talker Variability in Speech Research (1997), Academic Press 85-108
46. Sokhi D.S., et al. Male and female voices activate distinct regions in the male brain. Neuroimage 27 (2005) 572-578
47. Lattner S., et al. Voice perception: sex, pitch, and the right hemisphere. Hum. Brain Mapp. 24 (2005) 11-20
48. von Kriegstein K., et al. Neural representation of auditory size in the human voice and in sounds from other resonant sources. Curr. Biol. 17 (2007) 1123-1128
49. Fitch W.T. Vocal tract length and formant frequency dispersion correlate with body size in rhesus macaques. J. Acoust. Soc. Am. 102 (1997) 1213-1222
50. Nakamura K., et al. Neural substrates for recognition of familiar voices: a PET study. Neuropsychologia 39 (2001) 1047-1054
51. Belin P., et al. Human temporal-lobe response to vocal sounds. Brain Res. Cogn. Brain Res. 13 (2002) 17-26
52. Belin P., and Zatorre R.J. Adaptation to speaker's voice in right anterior temporal-lobe. Neuroreport 14 (2003) 2105-2109
53. von Kriegstein K., et al. Modulation of neural responses to speech by directing attention to voices or verbal content. Brain Res. Cogn. Brain Res. 17 (2003) 48-55
54. Belin P., et al. Voice-selective areas in human auditory cortex. Nature 403 (2000) 309-312
55. Beauchemin M., et al. Electrophysiological markers of voice familiarity. Eur. J. Neurosci. 23 (2006) 3081-3086
56. Ellis H.D., et al. Intra- and inter-modal repetition priming of familiar faces and voices. Br. J. Psychol. 88 (1997) 143-156
57. Schweinberger S.R., et al. Auditory long-term memory: repetition priming of voice recognition. Quart. J. Exp. Psychol. 50A (1997) 498-517
58. Lachs L., and Pisoni D.B. Crossmodal source identification in speech perception. Ecol. Psychol. 16 (2004) 159-187
59. Kamachi M., et al. Putting the face to the voice: matching identity across modality. Curr. Biol. 13 (2003) 1709-1714
60. Sheffert S.M., and Olson E. Audiovisual speech facilitates voice learning. Percept. Psychophys. 66 (2004) 352-362
61. Joassin F., et al. When audition alters vision: an event-related potential study of the cross-modal interactions between faces and voices. Neurosci. Lett. 369 (2004) 132-137
62. Schweinberger S.R., et al. Hearing facial identities. Quart. J. Exp. Psychol. 60 (2007) 1446-1456
63. von Kriegstein K., et al. Interaction of face and voice areas during speaker recognition. J. Cogn. Neurosci. 17 (2005) 367-376
64. Shah N.J., et al. The neural correlates of person familiarity. A functional magnetic resonance imaging study with clinical implications. Brain 124 (2001) 804-815
65. von Kriegstein K., and Giraud A.L. Implicit multisensory associations influence voice recognition. PLoS Biol. 4 (2006) e326
66. Valente G., et al. Comparison of methods for fMRI brain reading: a study on generalization and interpretability. In: Büchel C.E.-A. (Ed). Human Brain Mapping (2007), Elsevier S103
67. Ghazanfar A.A., and Schroeder C.E. Is neocortex essentially multisensory?. Trends Cogn. Sci. 10 (2006) 278-285
68. Cappe C., and Barone P. Heteromodal connections supporting multisensory integration at low levels of cortical processing in the monkey. Eur. J. Neurosci. 22 (2005) 2886-2902
69. Bushara K.O., et al. Neural correlates of cross-modal binding. Nat. Neurosci. 6 (2003) 190-195
70. De Gelder B., and Bertelson P. Multisensory integration, perception and ecological validity. Trends Cogn. Sci. 7 (2003) 460-467
71. Sugihara T., et al. Integration of auditory and visual communication information in the primate ventrolateral prefrontal cortex. J. Neurosci. 26 (2006) 11138-11147
72. Bahrick L.E., et al. Intermodal perception of adult and child faces and voices by infants. Child Dev. 69 (1998) 1263-1275
73. Bahrick L.E., et al. The development of infant learning about specific face-voice relations. Dev. Psychol. 41 (2005) 541-552
74. Lewkowicz D.J., and Ghazanfar A.A. The decline of cross-species intersensory perception in human infants. Proc. Natl Acad. Sci. USA 103 (2006) 6771-6774
75. Walker A.S. Intermodal perception of expressive behaviors by human infants. J. Exp. Child Psychol. 33 (1982) 514-535
76. Flom R., and Bahrick L.E. The development of infant discrimination of affect in multimodal and unimodal stimulation: The role of intersensory redundancy. Dev. Psychol. 43 (2007) 238-252
77. Grossmann T., et al. Crossmodal integration of emotional information from face and voice in the infant brain. Dev. Sci. 9 (2006) 309-315
78. Walker-Andrews A.S. Infants' perception of expressive behaviors: differentiation of multimodal information. Psychol. Bull. 121 (1997) 437-456
79. Bahrick L.E., and Lickliter R. Intersensory redundancy guides attentional selectivity and perceptual learning in infancy. Dev. Psychol. 36 (2000) 190-201
80. Izumi A., and Kojima S. Matching vocalizations to vocalizing faces in a chimpanzee (Pan troglodytes). Anim. Cogn. 7 (2004) 179-184
81. Parr L.A. Perceptual biases for multimodal cues in chimpanzee (Pan troglodytes) affect recognition. Anim. Cogn. 7 (2004) 171-178
82. Ghazanfar A.A., and Logothetis N.K. Neuroperception: facial expressions linked to monkey calls. Nature 423 (2003) 937-938
83. Surguladze S.A., et al. Audio-visual speech perception in schizophrenia: an fMRI study. Psychiatry Res. Neuroim 106 (2001) 1-14
84. Mandal M.K., et al. Facial expressions of emotions and schizophrenia: A review. Schizophr. Bull. 24 (1998) 399-412
85. Ross E.D., et al. Affective-prosodic deficits in schizophrenia: comparison to patients with brain damage and relation to schizophrenic symptoms. J. Neurol. Neurosurg. Psychiatry 70 (2001) 597-604
86. de Gelder B., et al. Multisensory integration of emotional faces and voices in schizophrenics. Schizophr. Res. 72 (2005) 195-203
87. Surguladze S., et al. A reversal of the normal pattern of parahippocampal response to neutral and fearful faces is associated with reality distortion in schizophrenia. Biol. Psychiatry 60 (2006) 423-431
88. Rubinow D.R., and Post R.M. Impaired recognition of affect in facial expression in depressed patients. Biol. Psychiatry 31 (1992) 947-953
89. Maurage P., et al. Is the P300 deficit in alcoholism associated with early visual impairments (P100, N170)? An oddball paradigm. Clin. Neurophysiol. 118 (2007) 633-644
90. Maurage, P. et al. The auditory-visual integration of anger stimuli is impaired in alcoholism: An ERP study. J. Psychiatry Neurosci.(in press)