Multimodal human communication - Targeting facial expressions, speech content and prosody

NeuroImage

Volumn 60, Issue 4, 2012, Pages 2346-2356

  Regenbogen, Christina ^a,b   Schneider, Daniel A ^a,b   Gur, Raquel E ^c   Schneider, Frank ^a,b,c   Habel, Ute ^a,b   Kellermann, Thilo ^a,b  

^a RWTH AACHEN UNIVERSITY   (Germany)

^b JARA TRANSLATIONAL BRAIN MEDICINE   (Germany)

^c UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ANGULAR GYRUS; ARTICLE; AUDITORY CORTEX; CONTROLLED STUDY; EMOTION; EMOTIONALITY; EMPATHY; FACIAL EXPRESSION; FEMALE; FRONTAL CORTEX; FUNCTIONAL MAGNETIC RESONANCE IMAGING; FUSIFORM GYRUS; HUMAN; HUMAN EXPERIMENT; INFERIOR FRONTAL CORTEX; INTERPERSONAL COMMUNICATION; LANGUAGE PROCESSING; MALE; MEDIAL PREFRONTAL CORTEX; MIDDLE TEMPORAL GYRUS; NERVE CELL NETWORK; NEUROIMAGING; NORMAL HUMAN; PRIORITY JOURNAL; PROSODY; SEMANTICS; SPEECH; SPEECH CONTENT; TASK PERFORMANCE; THEORY OF MIND;

ADULT; BRAIN; BRAIN MAPPING; EMOTIONS; FACIAL EXPRESSION; FEMALE; HUMANS; IMAGE INTERPRETATION, COMPUTER-ASSISTED; MAGNETIC RESONANCE IMAGING; MALE; MIDDLE AGED; SOCIAL BEHAVIOR; SPEECH; YOUNG ADULT;

EID: 84858731630     PISSN: 10538119     EISSN: 10959572     Source Type: Journal    
DOI: 10.1016/j.neuroimage.2012.02.043     Document Type: Article

References (87)

1. Adolphs R. Recognizing emotion from facial expressions: psychological and neurological mechanisms. Behav. Cogn. Neurosci. Rev. 2002, 1:21-61.
2. Adolphs R. The social brain: neural basis of social knowledge. Annu. Rev. Psychol. 2009, 60:693-716.
3. Akshoomoff N.A., Courchesne E., Townsend J. Attention coordination and anticipatory control. Int. Rev. Neurobiol. 1997, 41:575-598.
4. Ambadar Z., Schooler J.W., Cohn J.F. Deciphering the enigmatic face: the importance of facial dynamics in interpreting subtle facial expressions. Psychol. Sci. 2005, 16:403-410.
5. Ashburner J., Friston K.J. Unified segmentation. Neuroimage 2005, 26:839-851.
6. Batson C.D. These things called empathy. The Social Neuroscience of Empathy 2009, 3-16. MIT Press, Cambridge, Ma. J. Decety, W. Ickes (Eds.).
7. Beaucousin V., Lacheret A., Turbelin M.R., Morel M., Mazoyer B., Tzourio-Mazoyer N. FMRI study of emotional speech comprehension. Cereb. Cortex 2007, 17:339-352.
8. Berthoz S., Armony J.L., Blair R.J., Dolan R.J. An fMRI study of intentional and unintentional (embarrassing) violations of social norms. Brain 2002, 125:1696-1708.
9. Binder J.R., Desai R.H., Graves W.W., Conant L.L. Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies. Cereb. Cortex 2009, 19:2767-2796.
10. Campanella S., Belin P. Integrating face and voice in person perception. Trends Cogn. Sci. 2007, 11:535-543.
11. Cato M.A., Crosson B., Gokcay D., Soltysik D., Wierenga C., Gopinath K., Himes N., Belanger H., Bauer R.M., Fischler I.S., Gonzalez-Rothi L., Briggs R.W. Processing words with emotional connotation: an FMRI study of time course and laterality in rostral frontal and retrosplenial cortices. J. Cogn. Neurosci. 2004, 16:167-177.
12. Cavanna A.E., Trimble M.R. The precuneus: a review of its functional anatomy and behavioural correlates. Brain 2006, 129:564-583.
13. Celsis P., Boulanouar K., Doyon B., Ranjeva J.P., Berry I., Nespoulous J.L., Chollet F. Differential fMRI responses in the left posterior superior temporal gyrus and left supramarginal gyrus to habituation and change detection in syllables and tones. Neuroimage 1999, 9:135-144.
14. Collignon O., Girard S., Gosselin F., Roy S., Saint-Amour D., Lassonde M., Lepore F. Audio-visual integration of emotion expression. Brain Res. 2008, 1242:126-135.
15. Darwin C. The expression of the emotions in man and animals 1872, John Murray, London.
16. de Gelder B. Towards the neurobiology of emotional body language. Nat. Rev. Neurosci. 2006, 7:242-249.
17. de Gelder B., Vroomen J. The perception of emotions by ear and by eye. Cogn. Emotion 2000, 14:289-311.
18. de Waal F.B.M. Putting the Altruism Back into Altruism: The Evolution of Empathy. Annu. Rev. Psychol. 2008, 59:279-300.
19. Decety J., Jackson P.L. The Functional Architecture of Human Empathy. Behav. Cogn. Neurosci. Rev. 2004, 3(2):71-100.
20. Derntl B., Kryspin-Exner I., Fernbach E., Moser E., Habel U. Emotion recognition accuracy in healthy young females is associated with cycle phase. Horm. Behav. 2008, 53:90-95.
21. Ekman P., Friesen W.V. Pictures of Facial Affect, Palo Alto, CA 1976.
22. Ekman P., Friesen W.V. Facial action coding system: A technique for the measurement of facial movement 1978, Consulting Psychologists Press, Palo Alto, California.
23. Ethofer T., Anders S., Erb M., Droll C., Royen L., Saur R., Reiterer S., Grodd W., Wildgruber D. Impact of voice on emotional judgment of faces: an event-related fMRI study. Hum. Brain Mapp. 2006, 27:707-714.
24. Ethofer T., Pourtois G., Wildgruber D. Investigating audiovisual integration of emotional signals in the human brain. Prog. Brain Res. 2006, 156:345-361.
25. Ethofer T., Van De Ville D., Scherer K., Vuilleumier P. Decoding of emotional information in voice-sensitive cortices. Curr. Biol. 2009, 19:1028-1033.
26. Farrow T.F., Zheng Y., Wilkinson I.D., Spence S.A., Deakin J.F., Tarrier N., Griffiths P.D., Woodruff P.W. Investigating the functional anatomy of empathy and forgiveness. Neuroreport 2001, 12:2433-2438.
27. Ferstl E.C., Neumann J., Bogler C., von Cramon D.Y. The extended language network: a meta-analysis of neuroimaging studies on text comprehension. Hum. Brain Mapp. 2008, 29:581-593.
28. Fletcher P.C., Happe F., Frith U., Baker S.C., Dolan R.J., Frackowiak R.S., Frith C.D. Other minds in the brain: a functional imaging study of "theory of mind" in story comprehension. Cognition 1995, 57:109-128.
29. Grandjean D., Sander D., Pourtois G., Schwartz S., Seghier M.L., Scherer K.R., Vuilleumier P. The voices of wrath: brain responses to angry prosody in meaningless speech. Nat. Neurosci. 2005, 8:145-146.
30. Hammerschmidt K., Jürgens U. Acoustical correlates of affective prosody. J. Voice 2007, 21:531-540.
31. Heberlein A.S., Padon A.A., Gillihan S.J., Farah M.J., Fellows L.K. Ventromedial frontal lobe plays a critical role in facial emotion recognition. J. Cogn. Neurosci. 2008, 20:721-733.
32. Hooker C.I., Verosky S.C., Germine L.T., Knight R.T., D'Esposito M. Neural activity during social signal perception correlates with self-reported empathy. Brain Res. 2010, 1308:100-113.
33. Ickes W. Empathic Accuracy. J. Pers. 1993, 61:587-610.
34. Itier R.J., Batty M. Neural bases of eye and gaze processing: The core of social cognition. Neurosci. Biobehav. Rev. 2009, 33:843-863.
35. Ivry R. Cerebellar timing systems. Int. Rev. Neurobiol. 1997, 41:555-573.
36. Kelly S.D., Kravitz C., Hopkins M. Neural correlates of bimodal speech and gesture comprehension. Brain Lang. 2004, 89:253-260.
37. Kilts C.D., Egan G., Gideon D.A., Ely T.D., Hoffman J.M. Dissociable neural pathways are involved in the recognition of emotion in static and dynamic facial expressions. Neuroimage 2003, 18:156-168.
38. Koelewijn T., Bronkhorst A., Theeuwes J. Attention and the multiple stages of multisensory integration: A review of audiovisual studies. Acta Psychol. (Amst) 2010, 134:372-384.
39. Kotz S.A., Meyer M., Alter K., Besson M., von Cramon D.Y., Friederici A.D. On the lateralization of emotional prosody: an event-related functional MR investigation. Brain Lang. 2003, 86:366-376.
40. Kotz S.A., Meyer M., Paulmann S. Lateralization of emotional prosody in the brain: an overview and synopsis on the impact of study design. Prog. Brain Res. 2006, 156:285-294.
41. Krämer U.M., Mohammadi B., Doñamayor N., Samii A., Münte T.F. Emotional and cognitive aspects of empathy and their relation to social cognition-an fMRI-study. Brain Res. 2010, 1311:110-120.
42. Kreifelts B., Ethofer T., Grodd W., Erb M., Wildgruber D. Audiovisual integration of emotional signals in voice and face: an event-related fMRI study. Neuroimage 2007, 37:1445-1456.
43. Kuchinke L., Jacobs A.M., Grubich C., Vo M.L., Conrad M., Herrmann M. Incidental effects of emotional valence in single word processing: an fMRI study. Neuroimage 2005, 28:1022-1032.
44. LaBar K.S., Crupain M.J., Voyvodic J.T., McCarthy G. Dynamic perception of facial affect and identity in the human brain. Cereb. Cortex 2003, 13:1023-1033.
45. Lamm C., Meltzoff A.N., Decety J. How Do We Empathize with Someone Who Is Not Like Us? A Functional Magnetic Resonance Imaging Study. J. Cogn. Neurosci. 2009, 22(2):362-376.
46. Langner R., Kellermann T., Eickhoff S.B., Boers F., Chatterjee A., Willmes K., Sturm W. Staying responsive to the world: modality-specific and -nonspecific contributions to speeded auditory, tactile, and visual stimulus detection. Hum. Brain Mapp. 2012, 33:398-418.
47. Lieberman M.D. Social cognitive neuroscience: a review of core processes. Annu. Rev. Psychol. 2007, 58:259-289.
48. Lundstrom B.N., Petersson K.M., Andersson J., Johansson M., Fransson P., Ingvar M. Isolating the retrieval of imagined pictures during episodic memory: activation of the left precuneus and left prefrontal cortex. Neuroimage 2003, 20:1934-1943.
49. Lundstrom B.N., Ingvar M., Petersson K.M. The role of precuneus and left inferior frontal cortex during source memory episodic retrieval. Neuroimage 2005, 27:824-834.
50. Lyon D.C., Nassi J.J., Callaway E.M. A disynaptic relay from superior colliculus to dorsal stream visual cortex in macaque monkey. Neuron 2010, 65:270-279.
51. May P.J. The mammalian superior colliculus: laminar structure and connections. Prog. Brain Res. 2006, 151:321-378.
52. Mitchell R.L. How does the brain mediate interpretation of incongruent auditory emotions? The neural response to prosody in the presence of conflicting lexico-semantic cues. Eur. J. Neurosci. 2006, 24:3611-3618.
53. Mitchell R.L., Elliott R., Barry M., Cruttenden A., Woodruff P.W. The neural response to emotional prosody, as revealed by functional magnetic resonance imaging. Neuropsychologia 2003, 41:1410-1421.
54. Nichols T., Brett M., Andersson J., Wager T., Poline J.B. Valid conjunction inference with the minimum statistic. Neuroimage 2005, 25:653-660.
55. Oldfield R.C. The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia 1971, 9:97-113.
56. O'Reilly J.X., Mesulam M.M., Nobre A.C. The cerebellum predicts the timing of perceptual events. J. Neurosci. 2008, 28:2252-2260.
57. Parvizi J., Anderson S.W., Martin C.O., Damasio H., Damasio A.R. Pathological laughter and crying: a link to the cerebellum. Brain 2001, 124:1708-1719.
58. Patton P., Belkacem-Boussaid K., Anastasio T.J. Multimodality in the superior colliculus: an information theoretic analysis. Brain Res. Cogn. Brain Res. 2002, 14:10-19.
59. Pichon S., de Gelder B., Grezes J. Two different faces of threat. Comparing the neural systems for recognizing fear and anger in dynamic body expressions. Neuroimage 2009, 47:1873-1883.
60. Pourtois G., de Gelder B., Bol A., Crommelinck M. Perception of facial expressions and voices and of their combination in the human brain. Cortex 2005, 41:49-59.
61. Price C.J. The anatomy of language: a review of 100 fMRI studies published in 2009. Ann. N. Y. Acad. Sci. 2010, 1191:62-88.
62. Regenbogen C., Schneider D.A., Finkelmeyer A., Kohn N., Derntl B., Kellermann T., Gur R.E., Schneider F., Habel U. The differential contribution of facial expressions, prosody, and speech content to empathy. Cogn. Emotion 2012, 10.1080/02699931.2011.631296.
63. Rinne T., Stecker G.C., Kang X., Yund E.W., Herron T.J., Woods D.L. Attention modulates sound processing in human auditory cortex but not the inferior colliculus. Neuroreport 2007, 18:1311-1314.
64. Robins D.L., Hunyadi E., Schultz R.T. Superior temporal activation in response to dynamic audio-visual emotional cues. Brain Cogn. 2009, 69:269-278.
65. Ruby P., Decety J. How would you feel versus how do you think she would feel? A neuroimaging study of perspective-taking with social emotions. J. Cogn. Neurosci. 2004, 16:988-999.
66. Said C.P., Moore C.D., Engell A.D., Todorov A., Haxby J.V. Distributed representations of dynamic facial expressions in the superior temporal sulcus. J. Vis. 2010, 10.
67. Sato W.S., Kochiyama T., Yoshikawa S., Naito E., Matsumura M. Enhanced neural activity in response to dynamic facial expressions of emotion: an fMRI study Cognit. Brain Res. 2004, 20:81-91.
68. Schilbach L., Wohlschlaeger A.M., Kraemer N.C., Newen A., Shah N.J., Fink G.R., Vogeley K. Being with virtual others: Neural correlates of social interaction. Neuropsychologia 2006, 44:718-730.
69. Schneider K.A., Kastner S. Visual responses of the human superior colliculus: a high-resolution functional magnetic resonance imaging study. J. Neurophysiol. 2005, 94:2491-2503.
70. Seubert J., Kellermann T., Loughead J., Boers F., Brensinger C., Schneider F., Habel U. Processing of disgusted faces is facilitated by odor primes: a functional MRI study. Neuroimage 2010, 53:746-756.
71. Shamay-Tsoory S.G., Aharon-Peretz J., Perry D. Two systems for empathy: a double dissociation between emotional and cognitive empathy in inferior frontal gyrus versus ventromedial prefrontal lesions. Brain 2009, 132:617-627.
72. Simons R.F., Detender B.H., Roedema T.M., Reiss J.E. Emotion processing in three systems: The medium and the message. Psychophysiology 1999, 36:619-627.
73. Singer T. The neuronal basis and ontogeny of empathy and mind reading: review of literature and implications for future research. Neurosci. Biobehav. Rev. 2006, 30:855-863.
74. Singer T., Lamm C. The social neuroscience of empathy. Ann. N. Y. Acad. Sci. 2009, 1156:81-96.
75. Stoodley C.J., Schmahmann J.D. Functional topography in the human cerebellum: a meta-analysis of neuroimaging studies. Neuroimage 2009, 44:489-501.
76. Trautmann S.A., Fehr T., Herrmann M. Emotions in motion: dynamic compared to static facial expressions of disgust and happiness reveal more widespread emotion-specific activations. Brain Res. 2009, 1284:100-115.
77. van Atteveldt N., Formisano E., Goebel R., Blomert L. Integration of Letters and Speech Sounds in the Human Brain. Neuron 2004, 43:271-282.
78. Van Overwalle F. Social cognition and the brain: a meta-analysis. Hum. Brain Mapp. 2009, 30:829-858.
79. Vogeley K., Fink G.R. Neural correlates of the first-person-perspective. Trends Cogn. Sci. 2003, 7:38-42.
80. Weyers P., Mühlberger A., Hefele C., Pauli P. Electromyographic responses to static and dynamic avatar emotional facial expressions. Psychophysiology 2006, 43:450-453.
81. Wiethoff S., Wildgruber D., Kreifelts B., Becker H., Herbert C., Grodd W., Ethofer T. Cerebral processing of emotional prosody - influence of acoustic parameters and arousal. Neuroimage 2008, 39:885-893.
82. Wildgruber D., Riecker A., Hertrich I., Erb M., Grodd W., Ethofer T., Ackermann H. Identification of emotional intonation evaluated by fMRI. Neuroimage 2005, 24:1233-1241.
83. Wildgruber D., Ackermann H., Kreifelts B., Ethofer T. Cerebral processing of linguistic and emotional prosody: fMRI studies. Prog. Brain Res. 2006, 156:249-268.
84. Wittchen H.U., Zaudig M., Fydrich T. Strukturiertes klinisches Interview für DSM-IV: SKID [Structural Clinical Interview for DSMIV: SCID] 1997, Hogrefe, Göttingen.
85. Woodruff P.W., Benson R.R., Bandettini P.A., Kwong K.K., Howard R.J., Talavage T., Belliveau J., Rosen B.R. Modulation of auditory and visual cortex by selective attention is modality-dependent. Neuroreport 1996, 7:1909-1913.
86. Zangenehpour S., Chaudhuri A. Neural activity profiles of the neocortex and superior colliculus after bimodal sensory stimulation. Cereb. Cortex 2001, 11:924-935.
87. Zeelenberg R., Bocanegra B.R. Auditory emotional cues enhance visual perception. Cognition 2010, 115:202-206.