A network analysis of audiovisual affective speech perception

Neuroscience

Volumn 256, Issue , 2014, Pages 230-241

  Jansma, H ^a,b   Roebroeck, A ^b   Munte T F ^a  

^a UNIVERSITY OF LÜBECK   (Germany)

^b MAASTRICHT UNIVERSITY   (Netherlands)

Author keywords

Amygdala; Audiovisual speech; Emotion; Facial affect perception; Granger causality

Indexed keywords

ADULT; AMYGDALOID NUCLEUS; ARTICLE; AUDIOVISUAL EQUIPMENT; BOLD SIGNAL; CLINICAL ARTICLE; CONNECTOME; CONTROLLED STUDY; EMOTIONALITY; FACIAL EXPRESSION; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; FUSIFORM GYRUS; HUMAN; INFERIOR OCCIPITAL GYRUS; MALE; MIDDLE OCCIPITAL GYRUS; MIDDLE TEMPORAL GYRUS; NERVE CELL NETWORK; OCCIPITAL GYRUS; PRIORITY JOURNAL; SPEECH ANALYSIS; SPEECH PERCEPTION; BLOOD; BRAIN CORTEX; BRAIN MAPPING; EMOTION; HEMISPHERIC DOMINANCE; IMAGE PROCESSING; NERVE TRACT; NUCLEAR MAGNETIC RESONANCE IMAGING; PHOTOSTIMULATION; PHYSIOLOGY; VASCULARIZATION;

OXYGEN;

AMYGDALA; BRAIN MAPPING; CEREBRAL CORTEX; EMOTIONS; FEMALE; FUNCTIONAL LATERALITY; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; MAGNETIC RESONANCE IMAGING; MALE; NEURAL PATHWAYS; OXYGEN; PHOTIC STIMULATION; SPEECH PERCEPTION;

EID: 84888099439     PISSN: 03064522     EISSN: 18737544     Source Type: Journal    
DOI: 10.1016/j.neuroscience.2013.10.047     Document Type: Article

References (77)

1. Adolphs R. Recognizing emotion from facial expressions: psychological and neurological mechanisms. Behav Cogn Neurosci Rev 2002, 1:21-62.
2. Adolphs R. Neural systems for recognizing emotion. Curr Opin Neurobiol 2002, 12:169-177.
3. Adolphs R., Damasio H., Tranel D. Neural systems for recognizing emotional prosody. Emotion 2002, 2:23-51.
4. Allison T., Ginter H., McCarthy G., Nobre A.C., Puce A., Luby M., Spencer D.D. Face recognition in human extrastriate cortex. J Neurophysiol 1994, 71:821-825.
5. Barraclough N.E., Xiao D., Baker C.I., Oram M.W., Perrett D.I. Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions. J Cogn Neurosci 2005, 17:377-391.
6. Baumgartner T., Lutz K., Schmidt C.F., Jancke L. The emotional power of music: how music enhances the feeling of affective pictures. Brain Res 2006, 1075:151-164.
7. Belin P., Zatorre R.J., Ahad P. Human temporal-lobe response to vocal sounds. Cogn Brain Res 2002, 13:17-26.
8. Bestelmeyer P.E., Latinus M., Bruckert L., Rouger J., Crabbe F., Belin P. Implicitly perceived vocal attractiveness modulates prefrontal cortex activity. Cereb Cortex 2012, 22:1263-1270.
9. Blair R.J.R., Morris J.S., Frith C.D., Perrett D.I., Dolan R.J. Dissociable neural responses to facial expressions of sadness and anger. Brain 1999, 122:883-893.
10. Blank H., Anwander A., von Kriegstein K. Direct structural connections between voice- and face-recognition areas. J Neurosci 2011, 31:12906-12915.
11. Breiter H.C., Etcoff N.L., Whalen P.J., Kennedy W.A., Rauch S.L., Buckner R.L., Strauss M.M., Hyman S.E., Rosen B.R. Response and habituation of the human amygdala during visual processing of facial expression. Neuron 1996, 17:875-887.
12. Bressler S.L., Tang W., Sylvester C.M., Shulman G.L., Corbetta M. Top-down control of human visual cortex by frontal and parietal cortex in anticipatory visual spatial attention. J Neurosci 2008, 28:10056-10061.
13. Bruce V., Young A.W. Understanding face recognition. Brit J Psychol 1986, 77:305-327.
14. Buchanan T.W., Lutz K., Mirzazade S., Specht K., Shah N.J., Zilles K., Jäncke L. Recognition of emotional prosody and verbal components of spoken language: an fMRI study. Cogn Brain Res 2000, 9:227-238.
15. Calder A.J., Young A.W. Understanding the recognition of facial identity and facial expression. Nat Rev Neurosci 2005, 6:641-651.
16. Calvert G.A., Campbell R. Reading speech from still and moving faces: the neural substrates of visible speech. J Cogn Neurosci 2003, 15:57-70.
17. Calvert G.A., Campbell R., Brammer M.J. Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 2000, 10:649-657.
18. Campanella S., Belin P. Integrating face and voice in person perception. Trends Cogn Sci 2007, 11:535-543.
19. Collignon O., Girard S., Gosselin F., Roy S., Saint-Amour D., Lassonde M., Lepore F. Audio-visual integration of emotion expression. Brain Res 2008, 1242:126-135.
20. Critchley H.D., Daly E., Phillips M., Brammer M., Bullmore E., Williams S., van Amelsvoort T., Robertson D., David A., Murphy D. Explicit and implicit neural mechanisms for processing of social information from facial expressions: a functional magnetic imaging study. Hum Brain Mapp 2000, 9:93-105.
21. de Gelder B., Vroomen J. The perception of emotions by ear and by eye. Cognition & Emotion 2000, 14:289-311.
22. de Gelder B., Bocker K.B., Tuomainen J., Hensen M., Vroomen J. The combined perception of emotion from voice and face: early interaction revealed by human electric brain responses. Neurosci Lett 1999, 260:133-136.
23. de Gelder B., Pourtois G., Weisskrantz L. Fear recognition in the voice is modulated by unconsciously recognized facial expressions but not by unconsciously recognized affective pictures. Proc Natl Acad Sci USA 2002, 99:4121-4126.
24. de Gelder B., Frissen I., Barton J., Hadjikhani N. A modulatory role for facial expressions in prosopagnosia. Proc Natl Acad Sci USA 2003, 100:13105-13110.
25. Dolan R.J., Morris J.S., de Gelder B. Crossmodal binding of fear in voice and face. Proc Natl Acad Sci USA 2001, 98:10006-10010.
26. Ekman P. Strong evidence for universals in facial expressions: a reply to Russell_s mistaken critique. Psychol Bull 1994, 115:268-287.
27. Engell A.D., Haxby J.V. Facial expression and gaze-direction in human superior temporal sulcus. Neuropsychologia 2007, 45:3234-3241.
28. Ethofer T., Anders S., Erb M., Droll C., Royen L., Saur R., Reiterer S., Grodd W., Wildgruber D. Impact of voice on emotional judgment of faces: an event-related fMRI study. Hum Brain Mapp 2006, 27:707-714.
29. Ethofer T., Pourtois G., Wildgruber D. Investigating audiovisual integration of emotional signals in the human brain. Prog Brain Res 2006, 156:345-360.
30. Ethofer T., Van De Ville D., Scherer K., Vuilleumier P. Decoding of emotional information in voice-sensitive cortices. Curr Biol 2009, 19:1028-1033.
31. Fairhall S.L., Ishai A. Effective connectivity within the distributed cortical network for face perception. Cereb Cortex 2007, 17:2400-2406.
32. Forman S.D., Cohen J.D., Fitzgerald M., Eddy W.F., Mintun M.A., Noll D.C. Improved assessment of significant activation in functional magnetic resonance imaging (fMRI): use of a cluster-size threshold. Magn Reson Med 1995, 33:636-647.
33. Formisano E., Linden D.E., Di Salle F., Trojano L., Esposito F., Sack A.T., Grossi D., Zanella F.E., Goebel R. Tracking the mind's image in the brain I: timeresolved fMRI during visuospatial mental imagery. Neuron 2002, 35:185-194.
34. Ganel T., Valyear K.F., Goshen-Gottstein Y., Goodale M.A. The involvement of the "fusiform face area" in processing facial expression. Neuropsychologia 2005, 43:1645-1654.
35. Gelder B.D., Teunisse J.P., Benson P.J. Categorical perception of facial expressions: categories and their internal structure. Cognition & Emotion 1997, 11:1-23.
36. Goebel R., Roebroeck A., Kim D.S., Formisano E. Investigating directed cortical interactions in time-resolved fMRI data using vector autoregressive modeling and Granger causality mapping. Magn Reson Imaging 2003, 21:1251-1261.
37. Goebel R., Esposito F., Formisano E. Analysis of functional image analysis contest (FIAC) data with brainvoyager QX: from single-subject to cortically aligned group general linear model analysis and self-organizing group independent component analysis. Hum Brain Mapp 2006, 27:392-401.
38. Haxby J.V., Hoffman E.A., Gobbini M.I. The distributed human neural system for face perception. Trends Cogn Sci 2000, 4:223-233.
39. Hein G., Knight R.T. Superior temporal sulcus - it's my area: or is it?. J Cogn Neurosci 2008, 20:2125-2136.
40. Hocking J., Price C.J. The role of the posterior superior temporal sulcus in audiovisual processing. Cereb Cortex 2008, 18:2439-2449.
41. Ishai A. Let's face it: it's a cortical network. Neuroimage 2008, 40:415-419.
42. Kanwisher N., Yovel G. The fusiform face area: a cortical region specialized for the perception of faces. Philos Trans R Soc Lond B Biol Sci 2006, 361:2109-2128.
43. Klasen M., Kenworthy C.A., Mathiak K.A., Kircher T.T., Mathiak K. Supramodal representation of emotions. J Neurosci 2011, 31:13635-13643.
44. Kreifelts B., Ethofer T., Grodd W., Erb M., Wildgruber D. Audiovisual integration of emotional signals in voice and face: an event-related fMRI study. Neuroimage 2007, 37:1445-1456.
45. Kreifelts B., Ethofer T., Shiozawa T., Grodd W., Wildgruber D. Cerebral representation of non-verbal emotional perception: fMRI reveals audiovisual integration area between voice- and face-sensitive regions in the superior temporal sulcus. Neuropsychologia 2009, 47:3059-3066.
46. McGurk H., MacDonald J. Hearing lips and seeing voices. Nature 1976, 264:746-748.
47. Morris J.S., Frith C.D., Perrett D.I., Rowland D., Young A.W., Calder A.J., Dolan R.J. A differential neural response in the human amygdala to fearful and happy facial expressions. Nature 1996, 383:812-815.
48. Morris J.S., Scott S.K., Dolan R.J. Saying it with feeling: neural responses to emotional vocalizations. Neuropsychologia 1999, 37:1155-1163.
49. Noesselt T., Driver J., Heinze H.J., Dolan R. Asymmetrical activation in the human brain during processing of fearful faces. Curr Biol 2005, 15:424-429.
50. Phillips M.L., Young A.W., Scott S.K., Calder A.J., Andrew C., Giampietro V., Williams S.C.R., Bullmore E.T., Brammer M., Gray J.A. Neural responses to facial and vocal expressions of fear and disgust. Proc R Soc London Ser B 1998, 265:1809-1817.
51. Pourtois G., de Gelder B., Vroomen J., Rossion B., Crommelinck M. The time-course of intermodal binding between seeing and hearing affective information. Neuroreport 2000, 11:1329-1333.
52. Pourtois G., de Gelder B., Bol A., Crommelinck M. Perception of facial expressions and voices and of their combination in the human brain. Cortex 2005, 41:49-59.
53. Roebroeck A., Formisano E., Goebel R. Mapping directed influence over the brain using Granger causality and fMRI. Neuroimage 2005, 25:230-242.
54. Ross L.A., Saint-Amour D., Leavitt V.M., Javitt D.C., Foxe J.J. Do you see what i am saying? Exploring visual enhancement of speech comprehension in noisy environments. Cereb Cortex 2007, 17:1147-1153.
55. Rossion B., Joyce C.A., Cottrell G.W., Tarr M.J. Early lateralization and orientation tuning for face, word, and object processing in the visual cortex. Neuroimage 2003, 20:1609-1624.
56. Scherer K.R. Vocal communication of emotion: a review of research paradigms. Speech Comm 2003, 40:227-256.
57. Scherer K.R., Banse R., Wallbott H.G. Emotion inferences from vocal expression correlate across languages and cultures. J Cross-Cult Psychol 2001, 32:76-92.
58. Schwartz J.L., Berthommier F., Savariaux C. Seeing to hear better: evidence for early audio-visual interactions in speech identification. Cognition 2004, 93:B69-B78.
59. Sekiyama K., Kanno I., Miura S., Sugita Y. Auditory-visual speech perception examined by fMRI and PET. Neurosci Res 2003, 47:277-287.
60. Skipper J.I., van Wassenhove V., Nusbaum H.C., Small S.L. Hearing lips and seeing voices: how cortical areas supporting speech production mediate audiovisual speech perception. Cereb Cortex 2007, 17:2387-2399.
61. Sridharan D., Levitin D.J., Menon V. A critical role for the right fronto-insular cortex in switching between central-executive and default-mode networks. Proc Natl Acad Sci USA 2008, 105:12569-12574.
62. Stephan K.E., Roebroeck A. A short history of causal modeling of fMRI data. Neuroimage 2012, 62:856-863.
63. Sumby W.H., Pollack I. Visual contribution to speech intelligibility in noise. J Acoust Soc Am 1954, 26:212-215.
64. Szycik G.R., Tausche P., Münte T.F. A novel approach to study audiovisual integration in speech perception: localizer fMRI and sparse sampling. Brain Res 2008, 1220:142-149.
65. Szycik G.R., Jansma H., Münte T.F. Audiovisual integration during speech comprehension: an fMRI study comparing ROI-based and whole brain analyses. Hum Brain Mapp 2009, 30:1990-1999.
66. Szycik G.R., Stadler J., Tempelmann C., Münte T.F. Examining the McGurk illusion using high-field 7 Tesla functional MRI. Front Hum Neurosci 2012, 6:95.
67. Talairach J., Tournoux P. Co-planar stereotaxic atlas of the human brain 1988, Thieme Medical Publisher Inc., New York.
68. Valdes-Sosa P.A., Roebroeck A., Daunizeau J., Friston K. Effective connectivity: influence, causality and biophysical modeling. Neuroimage 2011, 58:339-361.
69. Vo M.L., Jacobs A.M., Conrad M. Cross-validating the Berlin affective word list. Behav Res Methods 2006, 38:606-609.
70. von Kriegstein K., Kleinschmidt A., Sterzer P., Giraud A.L. Interaction of face and voice areas during speaker recognition. J Cogn Neurosci 2005, 17:367-376.
71. Vuilleumier P. How brains beware: neural mechanisms of emotional attention. Trends Cogn Sci 2005, 9:585-594.
72. Vuilleumier P., Pourtois G. Distributed and interactive brain mechanisms during emotion face perception: evidence from functional neuroimaging. Neuropsychologia 2007, 45:174-194.
73. Vuilleumier P., Armony J.L., Driver J., Dolan R.J. Effects of attention and emotion on face processing in the human brain: an event-related fMRI study. Neuron 2001, 30:829-841.
74. Wager T.D., Phan K.L., Liberzon I., Taylor S.F. Valence, gender, and lateralization of functional brain anatomy in emotion: a meta-analysis of findings from neuroimaging. Neuroimage 2003, 19:513-531.
75. Whalen P.J., Shin L.M., McInerney S.C., Fischer H., Wright C.I., Rauch S.L. A functional MRI study of human amygdala responses to facial expressions of fear versus anger. Emotion 2001, 1:70-83.
76. Williams A.C. Facial expression of pain: an evolutionary account. Behav Brain Sci 2002, 25:439-455.
77. Wright T.M., Pelphrey K.A., Allison T., McKeown M.J., McCarthy G. Polysensory interactions along lateral temporal regions evoked by audiovisual speech. Cereb Cortex 2003, 13:1034-1043.