Audiovisual integration of emotional information from voice and face

Integrating Face and Voice in Person Perception

Volumn , Issue , 2013, Pages 225-251

  Kreifelts, Benjamin ^a   Wildgruber, Dirk ^a   Ethofer, Thomas ^a  

^a UNIVERSITY OF TÜBINGEN   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84886413557     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1007/978-1-4614-3585-3_12     Document Type: Chapter

References (111)

1. Adolphs R, Tranel D, Damasio H, Damasio A (1994) Impaired recognition of emotion in facial expressions following bilateral damage to the human amygdala. Nature 372(6507):669-672.
2. Barraclough NE, Xiao D, Baker CI, Oram MW, Perrett DI (2005) Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions. Journal of Cognitive Neuroscience 17(3):377-391.
3. Beauchamp MS, Argall BD, Bodurka J, Duyn JH, Martin A (2004a) Unraveling multisensory integration: Patchy organization within human STS multisensory cortex. Nature Neuroscience 7 (11): 1190-1192.
4. Beauchamp MS, Lee KE, Argall BD, Martin A (2004b) Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41(5):809-823.
5. Belin P, Zatorre RJ, Lafaille P, Ahad P, Pike B (2000) Voice-selective areas in human auditory cortex. Nature 403(6767):309-312.
6. Breiter HC, Etcoff NL, Whalen PJ, Kennedy WA, Rauch SL, Buckner RL et al (1996) Response and habituation of the human amygdala during visual processing of facial expression. Neuron 17(5):875-887.
7. Bruce C, Desimone R, Gross CG (1981) Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque. Journal of Neurophysiology 46(2):369-384.
8. Calvert GA (2001) Crossmodal processing in the human brain: Insights from functional neuroimaging studies. Cerebral Cortex 11(12):1110-1123.
9. Calvert GA, Brammer MJ, Bullmore ET, Campbell R, Iversen SD, David AS (1999) Response amplification in sensory-specific cortices during crossmodal binding. NeuroReport 10(12):2619-2623.
10. Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Current Biology 10(11):649-657.
11. Calvert GA, Spence C, Stein BE (eds) (2004) The handbook of multisensory processes. MIT Press, Cambridge, MA.
12. Calvert GA, Thesen T (2004) Multisensory integration: Methodological approaches and emerging principles in the human brain. Journal of Physiology 98(1-3):191-205.
13. Chandrasekaran C, Ghazanfar AA (2009) Different neural frequency bands integrate faces and voices differently in the superior temporal sulcus. Journal of Neurophysiology 101(2):773-788.
14. Chavis DA, Pandya DN (1976) Further observations on corticofrontal connections in the rhesus monkey. Brain Research 117(3):369-386.
15. Chen YH, Edgar JC, Holroyd T, Dammers J, Thonnessen H, Roberts TP et al (2010) Neuromagnetic oscillations to emotional faces and prosody. European Journal of Neuroscience 31(10): 1818-1827.
16. Collignon O, Girard S, Gosselin F, Roy S, Saint-Amour D, Lassonde M et al (2008) Audio-visual integration of emotion expression. Brain Research 1242:126-135.
17. Collignon O, Girard S, Gosselin F, Saint-Amour D, Lepore F, Lassonde M (2010) Women process multisensory emotion expressions more efficiently than men. Neuropsychologia 48(1):220-225.
18. Dahl CD, Logothetis NK, Kayser C (2010) Modulation of visual responses in the superior temporal sulcus by audio-visual congruency. Frontiers in Integrative Neuroscience 4:10.
19. Damasio AR (1989) Time-locked multiregional retroactivation: A systems-level proposal for the neural substrates of recall and recognition. Cognition 33(1-2):25-62.
20. de Gelder B, Bocker KB, Tuomainen J, Hensen M, Vroomen J (1999) The combined perception of emotion from voice and face: Early interaction revealed by human electric brain responses. Neuroscience Letters 260(2):133-136.
21. De Gelder B, Vroomen J (2000) The perception of emotions by ear and by eye. Cognition and Emotion 14(3):289-311.
22. de Gelder B, Vroomen J, de Jong SJ, Masthoff ED, Trompenaars FJ, Hodiamont P (2005) Multisensory integration of emotional faces and voices in schizophrenics. Schizophrenia Research 72(2-3):195-203.
23. de Jong JJ, Hodiamont PP, de Gelder B (2010) Modality-specific attention and multisensory integration of emotions in schizophrenia: Reduced regulatory effects. Schizophrenia Research 122(1-3): 136-143.
24. de Jong JJ, Hodiamont PP, Van den Stock J, de Gelder B (2009) Audiovisual emotion recognition in schizophrenia: Reduced integration of facial and vocal affect. Schizophrenia Research 107(2-3):286-293.
25. Dolan RJ, Morris JS, de Gelder B (2001) Crossmodal binding of fear in voice and face. Proceedings of the National Academy of Sciences of the United States of America 98(17):10006-10010.
26. Driver J, Spence C (2000) Multisensory perception: Beyond modularity and convergence. Current Biology 10(20):R731-R735.
27. Epstein R, Harris A, Stanley D, Kanwisher N (1999) The parahippocampal place area: Recognition, navigation, or encoding? Neuron 23(1):115-125.
28. Ethofer T, Anders S, Erb M, Droll C, Royen L, Saur R et al (2006a) Impact of voice on emotional judgment of faces: An event-related fMRI study. Human Brain Mapping 27(9):707-714.
29. Ethofer T, Pourtois G, Wildgruber D (2006b) Investigating audiovisual integration of emotional signals in the human brain. Progress in Brain Research 156:345-361.
30. Fallon JH, Benevento LA, Loe PR (1978) Frequency-dependent inhibition to tones in neurons of cat insular cortex (AIV). Brain Research 145(1):161-167.
31. Fox MD, Snyder AZ, Vincent JL, Corbetta M, Van Essen DC, Raichle ME (2005) The human brain is intrinsically organized into dynamic, anticorrelated functional networks. Proceedings of the National Academy of Sciences of the United States of America 102(27):9673-9678.
32. Foxe JJ, Schroeder CE (2005) The case for feedforward multisensory convergence during early cortical processing. NeuroReport 16(5):419-423.
33. Fries W (1984) Cortical projections to the superior colliculus in the macaque monkey: A retrograde study using horseradish peroxidase. The Journal of Comparative Neurology 230(1):55-76.
34. Friston KJ, Buechel C, Fink GR, Morris J, Rolls E, Dolan RJ (1997) Psychophysiological and modulatory interactions in neuroimaging. NeuroImage 6(3):218-229.
35. Friston KJ, Harrison L, Penny W (2003) Dynamic causal modelling. NeuroImage 19(4):1273-1302.
36. Friston KJ, Holmes AP, Price CJ, Buchel C, Worsley KJ (1999) Multisubject fMRI studies and conjunction analyses. NeuroImage 10(4):385-396.
37. Friston KJ, Josephs O, Rees G, Turner R (1998) Nonlinear event-related responses in fMRI. Magnetic Resonance in Medicine 39(1):41-52.
38. Friston KJ, Mechelli A, Turner R, Price CJ (2000) Nonlinear responses in fMRI: The Balloon model, Volterra kernels, and other hemodynamics. NeuroImage 12(4):466-477.
39. Friston KJ, Penny WD, Glaser DE (2005) Conjunction revisited. NeuroImage 25(3):661-667.
40. Ghazanfar AA, Chandrasekaran C, Logothetis NK (2008) Interactions between the superior temporal sulcus and auditory cortex mediate dynamic face/voice integration in rhesus monkeys. Journal of Neuroscience 28(17):4457-4469.
41. Ghazanfar AA, Maier JX, Hoffman KL, Logothetis NK (2005) Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. Journal of Neuroscience 25(20):5004-5012.
42. Giard MH, Peronnet F (1999) Auditory-visual integration during multimodal object recognition in humans: A behavioral and electrophysiological study. Journal of Cognitive Neuroscience 11(5):473-490.
43. Gitelman DR, Penny WD, Ashburner J, Friston KJ (2003) Modeling regional and psychophysiologic interactions in fMRI: The importance of hemodynamic deconvolution. NeuroImage 19(1):200-207.
44. Goldman-Rakic PS (1995) Architecture of the prefrontal cortex and the central executive. Annals of the New York Academy of Sciences 769:71-83.
45. Gordon B (1973) Receptive fields in deep layers of cat superior colliculus. Journal of Neurophysiology 36(2):157-178.
46. Grossmann T, Striano T, Friederici AD (2006) Crossmodal integration of emotional information from face and voice in the infant brain. Developmental Science 9(3):309-315.
47. Hagan CC, Woods W, Johnson S, Calder AJ, Green GG, Young AW (2009) MEG demonstrates a supra-additive response to facial and vocal emotion in the right superior temporal sulcus. Proceedings of the National Academy of Sciences of the United States of America 106(47): 20010-20015.
48. Hikosaka K, Iwai E, Saito H, Tanaka K (1988) Polysensory properties of neurons in the anterior bank of the caudal superior temporal sulcus of the macaque monkey. Journal of Neurophysiology 60(5):1615-1637.
49. John ER (2002) The neurophysics of consciousness. Brain Research Brain Research Reviews 39(1): 1-28.
50. Jones EG, Powell TP (1970) An anatomical study of converging sensory pathways within the cerebral cortex of the monkey. Brain 93(4):793-820.
51. Kanwisher N, McDermott J, Chun MM (1997) The fusiform face area: A module in human extrastriate cortex specialized for face perception. Journal of Neuroscience 17(11):4302-4311.
52. Kayser C, Logothetis NK (2009) Directed interactions between auditory and superior temporal cortices and their role in sensory integration. Frontiers in Integrative Neuroscience 3:7.
53. Kayser C, Logothetis NK, Panzeri S (2010) Visual enhancement of the information representation in auditory cortex. Current Biology 20(1):19-24.
54. Kayser C, Petkov CI, Augath M, Logothetis NK (2005) Integration of touch and sound in auditory cortex. Neuron 48(2):373-384.
55. Kayser C, Petkov CI, Augath M, Logothetis NK (2007) Functional imaging reveals visual modulation of specific fields in auditory cortex. Journal of Neuroscience 27(8):1824-1835.
56. Kreifelts B, Ethofer T, Grodd W, Erb M, Wildgruber D (2007) Audiovisual integration of emotional signals in voice and face: An event-related fMRI study. NeuroImage 37(4):1445-1456.
57. Kreifelts B, Ethofer T, Huberle E, Grodd W, Wildgruber D (2010) Association of trait emotional intelligence and individual fMRI-activation patterns during the perception of social signals from voice and face. Human Brain Mapping 31(7):979-991.
58. Kreifelts B, Ethofer T, Shiozawa T, Grodd W, Wildgruber D (2009) Cerebral representation of nonverbal emotional perception: fMRI reveals audiovisual integration area between voice- and face-sensitive regions in the superior temporal sulcus. Neuropsychologia 47(14):3059-3066.
59. Lambrecht L, Kreifelts B, Wildgruber D (2012) Age-related decrease in recognition of emotional facial and prosodic expressions. Emotion. Epub ahead of print.
60. Laurienti PJ, Perrault TJ, Stanford TR, Wallace MT, Stein BE (2005) On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies. Experimental Brain Research 166(3-4):289-297.
61. Lewis JW, Beauchamp MS, DeYoe EA (2000) A comparison of visual and auditory motion processing in human cerebral cortex. Cerebral Cortex 10(9):873-888.
62. Llinas R, Ribary U (2001) Consciousness and the brain. The thalamocortical dialogue in health and disease. Annals of the New York Academy of Science 929:166-175.
63. Loe PR, Benevento LA (1969) Auditory-visual interaction in single units in the orbito-insular cortex of the cat. Electroencephalography and Clinical Neurophysiology 26(4):395-398.
64. Macaluso E, Frith C, Driver J (2000) Selective spatial attention in vision and touch: Unimodal and multimodal mechanisms revealed by PET. Journal of Neurophysiology 83(5):3062-3075.
65. Massaro DW, Egan PB (1996) Perceiving affect from the voice and the face. Psychonomic Bulletin and Review 3(2):215-221.
66. McDonald AJ (1998) Cortical pathways to the mammalian amygdala. Progress in Neurobiology 55(3):257-332.
67. Mechelli A, Price CJ, Friston KJ (2001) Nonlinear coupling between evoked rCBF and BOLD signals: A simulation study of hemodynamic responses. NeuroImage 14(4):862-872.
68. Meredith MA, Stein BE (1983) Interactions among converging sensory inputs in the superior colliculus. Science 221(4608):389-391.
69. Mesulam MM (1998) From sensation to cognition. Brain 121(Pt 6):1013-1052.
70. Mesulam MM, Mufson EJ (1982) Insula of the old world monkey. III: Efferent cortical output and comments on function. The journal of Comparative Neurology 212(1):38-52.
71. Miller J (1982) Divided attention: Evidence for coactivation with redundant signals. Cognitive Psychology 14(2):247-279.
72. Morris JS, Frith CD, Perrett DI, Rowland D, Young AW, Calder AJ et al (1996) A differential neural response in the human amygdala to fearful and happy facial expressions. Nature 383(6603): 812-815.
73. Mufson EJ, Mesulam MM (1984) Thalamic connections of the insula in the rhesus monkey and comments on the paralimbic connectivity of the medial pulvinar nucleus. The Journal of Comparative Neurology 227(1):109-120.
74. Müller VI, Habel U, Derntl B, Schneider F, Zilles K, Turetsky BI et al (2011) Incongruence effects in crossmodal emotional integration. NeuroImage 54(3):2257-2266.
75. Murray EA, Mishkin M (1985) Amygdalectomy impairs crossmodal association in monkeys. Science 228(4699):604-606.
76. Nichols T, Brett M, Andersson J, Wager T, Poline JB (2005) Valid conjunction inference with the minimum statistic. NeuroImage 25(3):653-660.
77. Park JY, Gu BM, Kang DH, Shin YW, Choi CH, Lee JM et al (2010) Integration of cross-modal emotional information in the human brain: An fMRI study. Cortex 46(2):161-169.
78. Pearson RC, Brodal P, Gatter KC, Powell TP (1982) The organization of the connections between the cortex and the claustrum in the monkey. Brain Research 234(2):435-441.
79. Peck CK (1987) Visual-auditory interactions in cat superior colliculus: Their role in the control of gaze. Brain Research 420(1):162-166.
80. Phillips ML, Young AW, Scott SK, Calder AJ, Andrew C, Giampietro V et al (1998) Neural responses to facial and vocal expressions of fear and disgust. Proceedings of the Biological Sciences 265(1408):1809-1817.
81. Pitkânen A (2000) Connectivity of the rat amygdaloid complex. Oxford University Press, In The Amygdala. A functional analysis. New York.
82. Pourtois G, de Gelder B, Bol A, Crommelinck M (2005) Perception of facial expressions and voices and of their combination in the human brain. Cortex 41(1):49-59.
83. Pourtois G, de Gelder B, Vroomen J, Rossion B, Crommelinck M (2000) The time-course of intermodal binding between seeing and hearing affective information. NeuroReport 11(6): 1329-1333.
84. Pourtois G, Debatisse D, Despland PA, de Gelder B (2002) Facial expressions modulate the time course of long latency auditory brain potentials. Brain Research Cognitive Brain Research 14(1):99-105.
85. Price CJ, Friston KJ (1997) Cognitive conjunction: A new approach to brain activation experiments. NeuroImage 5(4 Pt 1):261-270.
86. Raichle ME, MacLeod AM, Snyder AZ, Powers WJ, Gusnard DA, Shulman GL (2001) A default mode of brain function. Proceedings of the National Academy of Sciences of the United States of America 98(2):676-682.
87. Robins DL, Hunyadi E, Schultz RT (2009) Superior temporal activation in response to dynamic audio-visual emotional cues. Brain and Cognition 69(2):269-278.
88. Schroger E, Widmann A (1998) Speeded responses to audiovisual signal changes result from bimodal integration. Psychophysiology 35(6):755-759.
89. Schutte N, Malouff J, Hall L, Haggerty D, Cooper J, Golden C et al (1998) Development and validation of a measure of emotional intelligence. Personality and Individual Differences 25:167-177.
90. Scott SK, Young AW, Calder AJ, Hellawell DJ, Aggleton JP, Johnson M (1997) Impaired auditory recognition of fear and anger following bilateral amygdala lesions. Nature 385(6613):254-257.
91. Seltzer B, Pandya DN (1978) Afferent cortical connections and architectonics of the superior temporal sulcus and surrounding cortex in the rhesus monkey. Brain Research 149(1):1-24.
92. Stanford TR, Stein BE (2007) Superadditivity in multisensory integration: Putting the computation in context. NeuroReport 18(8):787-792.
93. Stein BE, London N, Wilkinson LK, Price DD (1996) Enhancement of perceived visual intensity by auditory stimuli: A psychophysical analysis. Journal of Cognitive Neuroscience 8:497-506.
94. Stein BE, Meredith MA (1993) Merging of senses. MIT Press, Cambridge.
95. Stein BE, Wallace MT (1996) Comparisons of cross-modality integration in midbrain and cortex. Progress in Brain Research 112:289-299.
96. Stevenson RA, Geoghegan ML, James TW (2007) Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects. Experimental Brain Research 179(1):85-95.
97. Stevenson RA, James TW (2009) Audiovisual integration in human superior temporal sulcus: Inverse effectiveness and the neural processing of speech and object recognition. Neuroimage 44(3): 1210-1223.
98. Szycik GR, Munte TF, Dillo W, Mohammadi B, Samii A, Emrich HM et al (2009) Audiovisual integration of speech is disturbed in schizophrenia: An fMRI study. Schizophrenia Research 110(1-3): 111-118.
99. van Atteveldt N, Formisano E, Goebel R, Blomert L (2004) Integration of letters and speech sounds in the human brain. Neuron 43(2):271-282.
100. von Kriegstein K, Giraud AL (2006) Implicit multisensory associations influence voice recognition. PLoS Biology 4(10):e326.
101. Vroomen J, Driver J, de Gelder B (2001) Is cross-modal integration of emotional expressions independent of attentional resources? Cognitive, Affective, & Behavioral Neuroscience 1(4):382-387.
102. Wallace MT, Meredith MA, Stein BE (1992) Integration of multiple sensory modalities in cat cortex. Experimental Brain Research 91(3):484-488.
103. Wallace MT, Meredith MA, Stein BE (1993) Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior colliculus. Journal of Neurophysiology 69(6):1797-1809.
104. Wallace MT, Wilkinson LK, Stein BE (1996) Representation and integration of multiple sensory inputs in primate superior colliculus. Journal of Neurophysiology 76(2):1246-1266.
105. Wang Y, Celebrini S, Trotter Y, Barone P (2008) Visuo-auditory interactions in the primary visual cortex of the behaving monkey: Electrophysiological evidence. BMC Neuroscience 9:79.
106. Werner S, Noppeney U (2010a) Distinct functional contributions of primary sensory and association areas to audiovisual integration in object categorization. Journal of Neuroscience 30(7):2662-2675.
107. Werner S, Noppeney U (2010b) Superadditive responses in superior temporal sulcus predict audiovisual benefits in object categorization. Cerebral Cortex 20(8):1829-1842.
108. White M (1999) Representation of facial expressions of emotion. The American Journal of Psychology 112(3):371-381.
109. Williams LE, Light GA, Braff DL, Ramachandran VS (2010) Reduced multisensory integration in patients with schizophrenia on a target detection task. Neuropsychologia 48(10):3128-3136.
110. Worsley K, Marrett S, Neelin P, Vandal AC, Friston KJ, Evans A (1996) A unified statistical approach for determining significant signals in images of cerebral activation. Human Brain Mapping 4(1):74-90.
111. Wright TM, Pelphrey KA, Allison T, McKeown MJ, McCarthy G (2003) Polysensory interactions along lateral temporal regions evoked by audiovisual speech. Cerebral Cortex 13(10): 1034-1043.