Multisensory emotions: Perception, combination and underlying neural processes

Reviews in the Neurosciences

Volumn 23, Issue 4, 2012, Pages 381-392

  Klasen, Martin ^a,b   Chen, Yu Han ^c   Mathiak, Klaus ^a,b,d  

^a RWTH AACHEN UNIVERSITY   (Germany)

^b JARA Translational Brain Medicine   (Germany)

^c UNIVERSITY OF NEW MEXICO   (United States)

^d FORSCHUNGSZENTRUM JÜLICH   (Germany)

Author keywords

Amygdala; EEG; Emotion; FMRI; MEG; Multisensory integration

Indexed keywords

AFFECT; AMYGDALOID NUCLEUS; ASSOCIATION; AUDITORY STIMULATION; BEHAVIOR; ELECTROENCEPHALOGRAPHY; ELECTROPHYSIOLOGY; EMOTION; EMOTIONALITY; FACIAL EXPRESSION; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; NEUROIMAGING; PERCEPTION; POSITRON EMISSION TOMOGRAPHY; PRIORITY JOURNAL; REVIEW; SENSORIMOTOR INTEGRATION; VISUAL STIMULATION; VOICE ANALYSIS; ANIMAL; BRAIN; BRAIN MAPPING; PHYSIOLOGY;

ANIMALS; BRAIN; BRAIN MAPPING; EMOTIONS; HUMANS; NEUROIMAGING; PERCEPTION;

EID: 84868636262     PISSN: 03341763     EISSN: None     Source Type: Journal    
DOI: 10.1515/revneuro-2012-0040     Document Type: Review

References (94)

1. Barraclough, N.E., Xiao, D., Baker, C.I., Oram, M.W., and Perrett, D.I. (2005). Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions. J. Cogn. Neurosci. 17, 377-391. (Pubitemid 40364527)
2. Beauchamp, M.S., Lee, K.E., Argall, B.D., and Martin, A. (2004). Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41, 809-823. (Pubitemid 38326842)
3. Bolognini, N., Frassinetti, F., Serino, A., and Ladavas, E. (2005).'Acoustical vision'of below threshold stimuli: interaction among spatially converging audiovisual inputs. Exp. Brain Res. 160, 273-282. (Pubitemid 40169419)
4. Brosch, T., Sander, D., and Scherer, K.R. (2007). That baby caught my eye ... Attention capture by infant faces. Emotion 7, 685-689. (Pubitemid 47445369)
5. Brosch, T., Grandjean, D., Sander, D., and Scherer, K.R. (2009). Cross-modal emotional attention: emotional voices modulate early stages of visual processing. J. Cogn. Neurosci. 21, 1670-1679.
6. Calvert, G.A., Bullmore, E.T., Brammer, M.J., Campbell, R., Williams, S.C., McGuire, P.K., Woodruff, P.W., Iversen, S.D., and David, A.S. (1997). Activation of auditory cortex during silent lipreading. Science 276, 593-596. (Pubitemid 27193683)
7. Calvert, G.A. (2001). Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb. Cortex 11, 1110-1123. (Pubitemid 33108238)
8. Calvert, G.A. and Thesen, T. (2004). Multisensory integration: methodological approaches and emerging principles in the human brain. J. Physiol. 98, 191-205. (Pubitemid 39345371)
9. Campanella, S. and Belin, P. (2007). Integrating face and voice in person perception. Trends. Cogn. Sci. 11, 535-543. (Pubitemid 350181565)
10. Chen, Y.H., Edgar, J.C., Holroyd, T., Dammers, J., Thönnessen, H., Roberts, T.P., and Mathiak, K. (2010). Neuromagnetic oscillations to emotional faces and prosody. Eur. J. Neurosci. 31, 1818- 1827.
11. Colin, C., Radeau, M., and Deltenre, P. (2011). Top-down and bottom-up modulation of audiovisual integration in speech. Eur. J. Cogn. Psychol. 17, 541-560.
12. Collignon, O., Girard, S., Gosselin, F., Roy, S., Saint-Amour, D., Lassonde, M., and Lepore, F. (2008). Audio-visual integration of emotion expression. Brain Res. 1242, 126-135.
13. Crick, F.C. and Koch, C. (2005). What is the function of the claustrum? Philos. Trans. R. Soc. Lond. B Biol. Sci. 360, 1271-1279. (Pubitemid 43142415)
14. de Gelder, B. and Vroomen, J. (2000). The perception of emotions by ear and by eye. Cogn. Emot. 14, 289-311. (Pubitemid 30367746)
15. de Gelder, B., Böcker, K.B., Tuomainen, J., Hensen, M., and Vroomen, J. (1999). The combined perception of emotion from voice and face: early interaction revealed by human electric brain responses. Neurosci. Lett. 260, 133-136. (Pubitemid 29066883)
16. de Gelder, B., Pourtois, G., and Weiskrantz. L. (2002). Fear recognition in the voice is modulated by unconsciously recognized facial expressions but not by unconsciously recognized affective pictures. Proc. Nat. Acad. Sci. USA 99, 4121-4126. (Pubitemid 34252239)
17. de Gelder, B., Vroomen, J., and Pourtois, G. (2004). Multisensory perception of affect, its time course and its neural basis. Handbook of Multisensory Processes, G. Calvert, C. Spence and B.E. Stein, eds. (Cambridge, MA: MIT), pp. 581-596.
18. Diederich, A. and Colonius, H. (2004). Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction time. Percept. Psychophys. 66, 1388-1404. (Pubitemid 40778227)
19. Dolan, R.J., Morris, J.S., and de Gelder, B. (2001). Crossmodal binding of fear in voice and face. Proc. Nat. Aca. Sci. USA 98, 10006-10010. (Pubitemid 32769418)
20. Dolan, R.J. and Vuilleumier, P. (2003). Amygdala automaticity in emotional processing. Ann. New York Acad. Sci. 985, 348-355. (Pubitemid 36521126)
21. Eimer, M. and Holmes, A. (2007). Event-related brain potential correlates of emotional face processing. Neuropsychologia 45, 15-31. (Pubitemid 44821228)
22. Ernst, M.O. and Bülthoff, H.H. (2004). Merging the senses into a robust percept. Trends Cogn. Sci. 8, 162-169. (Pubitemid 38429926)
23. Ethofer, T., Anders, S., Erb, M., Droll, C., Royen, L., Saur, R., Reiterer, S., Grodd, W., and Wildgruber, D. (2006a). Impact of voice on emotional judgment of faces: an event-related fMRI study. Hum. Brain Mapp. 27, 707-714. (Pubitemid 44412046)
24. Ethofer, T., Pourtois, G., and Wildgruber, D. (2006b). Investigating audiovisual integration of emotional signals in the human brain. Prog. Brain Res. 156, 345-361. (Pubitemid 44466055)
25. Föcker, J., Gondan, M., and Röder, B. (2011). Preattentive processing of audio-visual emotional signals. Acta Psychol (Amst.). 137, 36-47.
26. Friston, K., Penny, W.D., and Glaser, D.E. (2005). Conjunction revisited. Neuroimage 25, 661-667. (Pubitemid 40461705)
27. Fusar-Poli, P., Placentino, A., Carletti, F., Landi, P., Allen, P., Surguladze, S., Benedetti, F., Abbamonte, M., Gasparotti, R., Barale, F. et al. (2009). Functional atlas of emotional faces processing: a voxel-based meta-analysis of 105 functional magnetic resonance imaging studies. J. Psychiatry Neurosci. 3 4, 418- 4 32.
28. Garrido, M.I., Kilner, J.M., Stephan, K.E., and Friston, K.J. (2009). The mismatch negativity: a review of underlying mechanisms. Clin. Neurophysiol. 120, 453-463.
29. Green, A.M. and Angelaki, D.E. (2010). Multisensory integration: resolving sensory ambiguities to build novel representations. Curr. Opin. Neurobiol. 20, 353 -3 60.
30. Hagan, C.C., Woods, W., Johnson, S., Calder, A.J., Green, G.G., and Young, A.W. (2009). MEG demonstrates a supra-additive response to facial and vocal emotion in the right superior temporal sulcus. Proc. Nat. Acad. Sci. USA 106, 20010-20015.
31. Hagmann, P., Cammoun, L., Gigandet, X., Meuli, R., Honey, C.J., Wedeen, V.J., and Sporns, O. (2008). Mapping the structural core of human cerebral cortex. PLoS Biol. 6, e159.
32. Hess, U., Kappas, A., and Scherer, K. (1988). Multichannel communication of emotion: Synthetic signal production. Facets of Emotion: Recent Research, K. Scherer, ed. (Hillsdale, NJ: Erlbaum). pp. 161- 182.
33. Jessen, S. and Kotz, S.A. (2011). The temporal dynamics of processing emotions from vocal, facial, and bodily expressions. Neuroimage 58, 665-674.
34. Kanwisher, N. and Yovel, G. (2006). The fusiform face area: a cortical region specialized for the perception of faces. Philos. Trans. Roy Soc. Lond B Biol. Sci. 361, 2109-2128. (Pubitemid 46025883)
35. Killgore, W.D. and Yurgelun-Todd, D.A. (2004). Activation of the amygdala and anterior cingulate during nonconscious processing of sad versus happy faces. Neuroimage 21, 1215-1223. (Pubitemid 38446552)
36. King, A.J. and Palmer, A.R. (1985). Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Exp. Brain Res. 60, 492-500. (Pubitemid 16191793)
37. Klasen, M., Kenworthy, C.A., Mathiak, K.A., Kircher, T.T., and Mathiak, K. (2011). Supramodal representation of emotions. J. Neurosci. 31, 13635-13643.
38. Klimesch, W., Sauseng, P., and Hanslmayr, S. (2007). EEG alpha oscillations: the inhibition-timing hypothesis. Brain Res. Rev. 53, 63-68.
39. Kober, H., Barrett, L.F., Joseph, J., Bliss-Moreau, E., Lindquist, K., and Wager, T.D. (2008). Functional grouping and corticalsubcortical interactions in emotion: a meta-analysis of neuroimaging studies. Neuroimage 42, 998-1031.
40. Koelewijn, T., Bronkhorst, A., and Theeuwes, J. (2010). Attention and the multiple stages of multisensory integration: a review of audiovisual studies. Acta Psychol. (Amst) 134, 372-384.
41. Kreifelts, B., Ethofer, T., Grodd, W., Erb, M., and Wildgruber, D. (2007). Audiovisual integration of emotional signals in voice and face: an event-related fMRI study. Neuroimage 37, 1445-1456. (Pubitemid 47385290)
42. Kreifelts, B., Ethofer, T., Shiozawa, T., Grodd, W., and Wildgruber, D. (2009). Cerebral representation of non-verbal emotional perception: fMRI reveals audiovisual integration area between voice- and face-sensitive regions in the superior temporal sulcus. Neuropsychologia 47, 3059-3066.
43. Kreifelts, B., Ethofer, T., Huberle, E., Grodd, W., and Wildgruber, D. (2010). Association of trait emotional intelligence and individual fMRI-activation patterns during the perception of social signals from voice and face. Hum. Brain Mapp. 31, 979-991.
44. Lippert, M., Logothetis, N.K., and Kayser, C. (2007). Improvement of visual contrast detection by a simultaneous sound. Brain Res. 1173, 102-109. (Pubitemid 47445452)
45. Mack, A. and Rock, I. (1998). Inattentional Blindness. (Cambridge, MA: MIT Press).
46. Maddock, R.J. (1999). The retrosplenial cortex and emotion: new insights from functional neuroimaging of the human brain. Trends. Neurosci. 22, 310-316. (Pubitemid 29307808)
47. Mar, R.A. (2011). The neural bases of social cognition and story comprehension. Ann Rev Psychol. 10, 103-134.
48. Massaro, D.W. and Egan, P.B. (1996). Perceiving affect from the voice and the face. Psychonomic Bull Rev. 3, 215-221. (Pubitemid 126421515)
49. McGurk, H. and MacDonald, J. (1976). Hearing lips and seeing voices. Nature 264, 746-748.
50. Meeren, H.K., Hadjikhani, N., Ahlfors, S.P., Häm äläinen, M.S., and de Gelder, B. (2008). Early category-specific cortical activation revealed by visual stimulus inversion. PLoS One 3, e3503.
51. Miller L.M. and D'Esposito, M. (2005). Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J. Neurosci. 25, 5884-5893. (Pubitemid 40911419)
52. Mitchell, R.L. (2007a). fMRI delineation of working memory for emotional prosody in the brain: commonalities with the lexico-semantic emotion network. Neuroimage 36, 1015-1025. (Pubitemid 46907637)
53. Mitchell, D.G., Nakic, M., Fridberg, D., Kamel, N., Pine, D.S., and Blair, R.J. (2007b). The impact of processing load on emotion. Neuroimage 34, 1299-1309. (Pubitemid 46084854)
54. Molholm, S., Ritter, W., Murray, M.M., Javitt, D.C., Schroeder, C.E., and Foxe, J.J. (2002). Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Brain Res. Cogn. Brain Res. 14, 115-128. (Pubitemid 34626802)
55. Morris, J.S., Friston, K.J., Büchel, C., Frith, C.D., Young, A.W., Calder, A.J., and Dolan R.J. (1998). A neuromodulatory role for the human amygdala in processing emotional facial expressions. Brain 121, 47-57. (Pubitemid 28045505)
56. Müller, V.I., Habel, U., Derntl, B., Schneider, F., Zilles, K., Turetsky, B.I., and Eickhoff, S.B. (2011). Incongruence effects in crossmodal emotional integration. Neuroimage 54, 2257-2266.
57. Nichols, T., Brett, M., Andersson, J., Wager, T., and Poline, J.B. (2005). Valid conjunction inference with the minimum statistic. Neuroimage 25, 653-660. (Pubitemid 40461704)
58. Noesselt, T., Bergmann, D., Hake, M., Heinze, H.J., and Fendrich, R. (2008). Sound increases the saliency of visual events. Brain Res. 1220, 157-163.
59. Ochsner, K.N., Hughes, B., Robertson, E.R., Cooper, J.C., and Gabrieli, J.D. (2009). Neural systems supporting the control of affective and cognitive conflicts. J. Cogn. Neurosci. 21, 1842-1855.
60. Olson, I.R., Gatenby, J.C., and Gore, J.C. (2002). A comparison of bound and unbound audio-visual information processing in the human cerebral cortex. Brain Res. Cogn. Brain Res. 14, 129-138. (Pubitemid 34626803)
61. Park, J.Y., Gu, B.M., Kang, D.H., Shin, Y.W., Choi, C.H., Lee, J.M., and Kwon, J.S. (2008). Integration of cross-modal emotional information in the human brain: an fMRI study. Cortex 46, 161-169.
62. Paulmann, S. and Kotz, S.A. (2008). Early emotional prosody perception based on different speaker voices. Neuroreport 19, 209-213.
63. Paulmann, S., Jessen, S., and Kotz, S.A. (2009). Investigating the multimodal nature of human communication: insights from ERPS. J. Psychophysiol. 23, 63-76.
64. Pessoa, L., Padmala, S., and Morland, T. (2005). Fate of unattended fearful faces in the amygdala is determined by both attentional resources and cognitive modulation. Neuroimage 28, 249-255. (Pubitemid 41393904)
65. Phan, K.L., Wager, T., Taylor, S.F., and Liberzon, I. (2002). Functional neuroanatomy of emotion: a meta-analysis of emotion activation studies in PET and fMRI. Neuroimage 16, 331-348. (Pubitemid 34852450)
66. Pourtois, G., de Gelder, B., Vroomen, J., Rossion, B., and Crommelinck, M. (2000). The time-course of intermodal binding between seeing and hearing affective information. Neuroreport 11, 1329-1333. (Pubitemid 30262982)
67. Pourtois, G., Debatisse, D., Despland, P.A., and de Gelder, B. (2002). Facial expressions modulate the time course of long latency auditory brain potentials. Brain Res. Cogn. Brain Res. 14, 99-105. (Pubitemid 34626800)
68. Pourtois, G., de Gelder, B., Bol, A., and Crommelinck, M. (2005). Perception of facial expression and voices and of their combination in the human brain. Cortex 41, 49-59. (Pubitemid 39600225)
69. Rach, S., Diederich, A., and Colonius, H. (2011). On quantifying multisensory interaction effects in reaction time and detection rate. Psychol. Res. 75, 77-94.
70. Robins, D.L., Hunyadi, E., and Schultz, R.T. (2008). Superior temporal activation in response to dynamic audiovisual cues. Brain Cogn. 69, 269-278.
71. Rosenblum, L.D. and Saldaña, H.M. (1996). An audiovisual test of kinematic primitives for visual speech perception. J. Exp. Psychol. Hum. 22, 318-331. (Pubitemid 126418397)
72. Sabatinelli, D., Lang, P.J., Bradley, M.M., Costa, V.D., and Keil, A. (2009). The timing of emotional discrimination in human amygdala and ventral visual cortex. J. Neurosci. 29, 14864-14868.
73. Sabatinelli, D., Fortune, E.E., Li, Q., Siddiqui, A., Krafft, C., Oliver, W.T., Beck, S., and Jeffries, J. (2011). Emotional perception: meta-analyses of face and natural scene processing. Neuroimage 54, 2524-2533.
74. Schröger, E. and Widmann, A. (1998). Speeded responses to audiovisual signal changes result from bimodal integration. Psychophysiology 35, 755-759. (Pubitemid 28533608)
75. Seltzer, B. and Pandya, D.N. (1978). Afferent cortical connections and architectonics of the superior temporal sulcus and surrounding cortex in the rhesus monkey. Brain Res. 149, 1-24. (Pubitemid 8350020)
76. Senkowski, D., Schneider, T.R., Foxe, J.J., and Engel, A.K. (2008). Crossmodal binding by neural coherence: Implications for multisensory processing. Trends Neurosci. 31, 401-409.
77. Sergerie, K., Chochol, C., and Armony, J.L. (2008). The role of the amygdala in emotional processing: a quantitative meta-analysis of functional neuroimaging studies. Neurosci. Biobehav. Rev. 32, 811-830.
78. Shams, L., Kamitani, Y., and Shimojo, S. (2000). What you see is what you hear. Nature 408, 788.
79. Shimojo, S., Miyauchi, S., and Hikosaka, O. (1997). Visual motion sensation yielded by non-visually driven attention. Vision Res. 37, 1575-1580. (Pubitemid 27277611)
80. Shomstein, S. and Yantis, S. (2004). Control of attention shifts between vision and audition in human cortex. J. Neurosci. 24, 10702-10706. (Pubitemid 39557644)
81. Stein, B.E. and Meredith, M.A. (1993). Merging of the senses. (Cambridge, MA: MIT Press).
82. Stein, B.E. and Wallace, M.T. (1996). Comparisons of cross-modality integration in midbrain and cortex. Prog. Brain Res. 112, 289-299. (Pubitemid 26400858)
83. Steriade, M. (2006). Grouping of brain rhythms in corticothalamic systems. Neuroscience 137, 1087-1106. (Pubitemid 43133136)
84. Talsma, D., Senkowski, D., Soto-Faraco, S., and Woldorff, M.G. (2010). The multifaceted interplay between attention and multisensory integration. Trends Cogn. Sci. 14, 400-410.
85. Tamietto, M. and de Gelder, B. (2010). Neural bases of the non-conscious perception of emotional signals. Nat. Rev. Neurosci. 11, 697-709.
86. Vogt, B.A., Vogt, L., and Laureys, S. (2006). Cytology and functionally correlated circuits of human posterior cingulate areas. Neuroimage 29, 452-466. (Pubitemid 43063302)
87. Vroomen, J. and de Gelder, B. (2004). Perceptual effects of cross-modal stimulation: the cases of ventriloquism and the freezing phenomenon. Handbook of Multisensory Processes, G. Calvert, C. Spence and B.E. Stein, eds. (Cambridge, MA: MIT), pp. 141-150.
88. Vroomen, J., Driver, J., and de Gelder, B. (2001). Is cross-modal integration of emotional expressions independent of attentional resources? Cogn. Affect Behav. Neurosci. 1, 382-387. (Pubitemid 38391259)
89. Vuilleumier, P., Armony, J.L., Driver, J., and Dolan, R.J. (2001). Effects of attention and emotion on face processing in the human brain: an event-related fMRI study. Neuron 30, 829-841. (Pubitemid 32607346)
90. Walker-Andrews, A. and Grolnick, W. (1983). Discrimination of vocal expressions by young infants. Infant Behav. Dev. 6, 491-498.
91. Weissman, D.H., Warner, L.M., and Woldorff, M. (2004). The neural mechanisms for minimizing cross-modal distraction. J. Neurosci. 24, 10941-10949. (Pubitemid 39602405)
92. Wittfoth, M., Schröder, C., Schardt, D.M., Dengler, R., Heinze, H.J., and Kotz, S.A. (2010). On emotional conflict: interference resolution of happy and angry prosody reveals valence-specific effects. Cereb. Cortex 20, 383-392.
93. Wright, T.M., Pelphrey, K.A., Allison, T., McKeown, M.J., and McCarthy, G. (2003). Polysensory interactions along lateral temporal regions evoked by audiovisual speech. Cereb. Cortex 13, 1034-1043. (Pubitemid 37163202)
94. Zvyagintsev, M., Nikolaev, A.R., Sachs, O., and Mathiak, K. (2011). Early attention modulates perceptual interpretation of multisensory stimuli. Neuro Report 22, 586-591.