The temporal dynamics of processing emotions from vocal, facial, and bodily expressions

NeuroImage

Volumn 58, Issue 2, 2011, Pages 665-674

  Jessen, S ^a,b   Kotz, S A ^a  

^a MAX PLANCK INSTITUTE FOR HUMAN COGNITIVE AND BRAIN SCIENCES   (Germany)

^b FREIE UNIVERSITÄT BERLIN   (Germany)

Author keywords

Alpha; Beta; EEG; Emotion; ERP; Multisensory

Indexed keywords

ADULT; ALPHA RHYTHM; ANGER; ARTICLE; AUDITORY RESPONSE; AUDITORY STIMULATION; BETA RHYTHM; BODY MOVEMENT; CONCEPT FORMATION; CONTROLLED STUDY; ELECTROENCEPHALOGRAM; EMOTION; EVENT RELATED POTENTIAL; FACIAL EXPRESSION; FEAR; FEMALE; HUMAN; MALE; PRIORITY JOURNAL; STIMULUS RESPONSE; TASK PERFORMANCE; VISION; VISUAL STIMULATION; VOCALIZATION;

ACOUSTIC STIMULATION; ADULT; ALPHA RHYTHM; ANGER; AROUSAL; BETA RHYTHM; DATA INTERPRETATION, STATISTICAL; ELECTROENCEPHALOGRAPHY; EMOTIONS; EVOKED POTENTIALS; FACIAL EXPRESSION; FEAR; FEMALE; HUMANS; KINESICS; MALE; PHOTIC STIMULATION; PSYCHOMOTOR PERFORMANCE; SOCIAL PERCEPTION; VOICE; YOUNG ADULT;

EID: 80051800582     PISSN: 10538119     EISSN: 10959572     Source Type: Journal    
DOI: 10.1016/j.neuroimage.2011.06.035     Document Type: Article

References (79)

1. Alfimova M.V., Uvarova L.G. Changes in EEG spectral power on perception of neutral and emotional words in patients with schizophrenia, their relatives, and healthy subjects from the general population. Neurosci. Behav. Physiol. 2008, 38:533-540.
2. Ashley V., Vuilleumier P., Swick D. Time course and specificity of event-related potentials to emotional expressions. Neuroreport 2004, 15:211-216.
3. Beauchamp M.S. Statistical criteria in fMRI studies of multisensory integration. Neuroinformatics 2005, 3:93-113.
4. Belin P., Fillion-Bilodeau S., Gosselin F. The montreal affective voices: a validated set of nonverbal affect bursts for research on auditory affective processing. Behav. Res. Methods 2008, 40:531-539.
5. Blake R., Shiffrar M. Perception of human motion. Annu. Rev. Psychol. 2007, 58:47-73.
6. Bostanov V., Kotchoubey B. Recognition of affective prosody: continuous wavelet measures of event-related brain potentials to emotional exclamations. Psychophysiology 2004, 41:259-268.
7. Brosch T., Grandjean D., Sander D., Scherer K.R. Cross-modal emotional attention: emotional voices modulate early stages of visual processing. J. Cogn. Neurosci. 2009, 21:1670-1679.
8. Calvert G.A., Hansen P.C., Iversen S.D., Brammer M.J. Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage 2001, 14:427-438.
9. Carretié L., Hinojosa J.A., Lépez-Martín S., Albert J., Tapia M., Pozo M.A. Danger is worse when it moves: neural and behavioral indices of enhanced attentional capture by dynamic threatening stimuli. Neuropsychologia 2009, 47:364-369.
10. Cochin S., Barthelemy C., Lejeune B., Roux S., Martineau J. Perception of motion and qEEG activity in human adults. Electroencephalogr. Clin. Neurophysiol. 1998, 107:287-295.
11. Cuthbert B.N., Schupp H.T., Bradley M.M., Birbaumer N., Lang P.J. Brain potentials in affective pictures processing: covariation with autonomic arousal and affective report. Biol. Psychol. 2000, 52:95-111.
12. de Gelder B. Why bodies? Twelve reasons for including bodily expressions in affective neuroscience. Philos. Trans. R. Soc. Lond. B Biol. Sci. 2009, 364:3475-3484.
13. de Gelder B., Böcker K.B., Tumainen J., Hensen M., Vroomen J. The combined perception of emotion from voice and face: early interaction revealed by human electric brain responses. Neurosci. Lett. 1999, 260:133-136.
14. Dekio-Hotta S., Kojima T., Karino S., Yamasoba T., Dekio I., Ito Y.M., Satake H., Kaga K. N1 component reflects difference of terminal chords in three-chord sequences. Neuroreport 2009, 20:251-256.
15. Dietrich S., Hertrich I., Alter K., Ischebeck A., Ackermann H. Understanding the emotional expression of verbal interjections: a functional MRI study. Neuroreport 2008, 19:1751-1755.
16. Doallo S., Cadaveira F., Holguín S.R. Time course of attentional modulations on automatic emotional processing. Neurosci. Lett. 2007, 418:111-116.
17. Dolan R.J., Morris J.S., de Gelder B. Crossmodal binding of fear in voice and face. Proc. Natl. Acad. Sci. U. S. A. 2001, 98:10006-10010.
18. Engel A.K., Fries P. Beta-band oscillations-signalling the status quo?. Curr. Opin. Neurobiol. 2010, 20:156-165.
19. Ethofer T., Anders S., Erb M., Droll C., Royen L., Saur R., Reiterer S., Grodd W., Wildgruber D. Impact of voice on emotional judgment of faces: an event-related fMRI study. Hum. Brain Mapp. 2006, 27:707-714.
20. Ethofer T., Pourtois G., Wildgruber D. Investigating audiovisual integration of emotional signals in the human brain. Prog. Brain Res. 2006, 156:345-361.
21. Ethofer T., Kreifelts B., Wiethoff S., Wolf J., Grodd W., Vuilleumier P., Wildgruber D. Differential influences of emotion, task, and novelty on brain regions underlying the processing of speech melody. J. Cogn. Neurosci. 2009, 21:1255-1268.
22. Ethofer T., Ville D.V.D., Scherer K., Vuilleumier P. Decoding of emotional information in voice-sensitive cortices. Curr. Biol. 2009, 19:1028-1033.
23. Fawcett I.P., Hillebrand A., Singh K.D. The temporal sequence of evoked and induced cortical responses to implied-motion processing in human motion area V5/MT+. Eur. J. Neurosci. 2007, 26:775-783.
24. Fecteau S., Belin P., Joanette Y., Armony J.L. Amygdala responses to nonlinguistic emotional vocalizations. Neuroimage 2007, 36:480-487.
25. Foti D., Olvet D.M., Klein D.N., Hajcak G. Recduced electrocortical response to threatening faces in major depressive disorder. Depress. Anxiety 2010, 27:813-820.
26. Frassinetti F., Bolognini N., Làdavas E. Enhancement of visual perception by crossmodal visuo-auditory interaction. Exp. Brain Res. 2002, 147:332-343.
27. Giard M., Peronnet F. Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J. Cogn. Neurosci. 1999, 11(5):473-490.
28. Glauser E.S.D., Scherer K.R. Neuronal processes involved in subjective feeling emergence: oscillatory activity during an emotional monitoring task. Brain Topogr. 2008, 20:224-231.
29. Grandjean D., Sander D., Pourtois G., Schwartz S., Seghier M.L., Scherer K.R., Vuilleumier P. The voices of wrath: brain responses to angry prosody in meaningless speech. Nat. Neurosci. 2005, 8:145-146.
30. Greenhouse S., Geisser S. On methods in the analysis of profile data. Psychometrika 1959, 24:95-112.
31. Grèzes J., Pichon S., de Gelder B. Perceiving fear in dynamic body expressions. Neuroimage 2007, 35:959-967.
32. Hajcak G., Nieuwenhuis S. Reappraisal modulates the electrocortical response to unpleasant pictures. Cogn. Affect. Behav. Neurosci. 2006, 6(4):291-297.
33. Holle H., Obleser J., Rueschemeyer S.A., Gunter T.C. Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions. Neuroimage 2010, 49:875-884.
34. Holm S. A simple sequentially rejective multiple test procedure. Scand. J. Stat. 1979, 6(2):65-70.
35. Huang Y.X., Luo Y.J. Temporal course of emotional negativity bias: an ERP study. Neurosci. Lett. 2006, 398:91-96.
36. Hurtado E., Haye A., González R., Manes F., Ibáñez A. Contextual blending of ingroup/outgroup face stimuli and word valence: LPP modulation and convergence of measures. BMC Neurosci. 2009, 10:69.
37. Kanske P., Kotz S.A. Concreteness in emotional words: ERP evidence from a hemifield study. Brain Res. 2007, 1148:138-148.
38. Kilts C.D., Egan G., Gideon D.A., Ely T.D., Hoffman J.M. Dissociable neural pathways are involved in the recognition of emotion in static and dynamic facial expressions. Neuroimage 2003, 18:156-168.
39. Kissler J., Herbert C., Winkler I., Junghofer M. Emotion and attention in visual word processing: an ERP study. Biol. Psychol. 2009, 80:75-83.
40. Kreifelts B., Ethofer T., Grodd W., Erb M., Wildgruber D. Audiovisual integration of emotional signals in voice and face: an event-related fMRI study. Neuroimage 2007, 37:1445-1456.
41. Kreifelts B., Ethofer T., Shiozawa T., Grodd W., Wildgruber D. Cerebral representation of non-verbal emotional perception: fMRI reveals audiovisual integration area between voice- and face-sensitive regions in the superior temporal sulcus. Neuropsychologia 2009, 47:3059-3066.
42. Kucharska-Pietura K., Phillips M.L., Gernand W., David A.S. Perception of emotions from faces and voices following unilateral brain damage. Neuropsychologia 2003, 41:1082-1090.
43. Lang J. Technology in mental health care delivery systems. Chapter Behavioral Treatment and Bio-behavioral Assessment: Computer Applications 1980, 119-137. Ablex, Norwood, NJ.
44. Laurienti P.J., Perrault T.J., Stanford T.R., Wallace M.T., Stein B.E. On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies. Exp. Brain Res. 2005, 166:289-297.
45. Maris E., Oostenveld R. Nonparametric statistical testing of EEG- and MEG-data. J. Neurosci. Methods 2007, 164:177-190.
46. Marrufo M.V., Vaquero E., Cardoso M.J., Gómez C.M. Temporal evolution of alpha and beta bands during visual spatial attention. Brain Res. Cogn. Brain Res. 2001, 12:315-320.
47. Meeren H.K.M., van Heijnsbergen C.C.R.J., de Gelder B. Rapid perceptual integration of facial expression and emotional body language. Proc. Natl. Acad. Sci. U. S. A. 2005, 102:16518-16523.
48. Moser J., Hajcak G., Bukay E., Simons R.F. Intentional modulation of emotional responding to unpleasant pictures: an ERP study. Psychophysiology 2006, 43:92-296.
49. Muthukumaraswamy S.D., Singh K.D. Modulation of the human mirror neuron system during cognitive activity. Psychophysiology 2008, 45:896-905.
50. Nakagawa K., Aokage Y., Fukuri T., Kawahara Y., Hashizume A., Kurisu K., Yuge L. Neuromagnetic beta oscillation changes during motor imagery and motor execution of skilled movements. Neuroreport 2011, 22:217-222.
51. Neuper C., Scherer R., Wriessnegger S., Pfurtscheller G. Motor imagery and action observation: modulation of sensorimotor brain rhythms during mental control of a brain-computer interface. Clin. Neurophysiol. 2009, 120:239-247.
52. Oostenveld R., Fries P., Maris E., Schoffelen J.M. FieldTrip: open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data. Comput. Intell. Neurosci. 2011, 2011:156869.
53. Paulmann S., Jessen S., Kotz S.A. Investigating the multimodal nature of human communication: an ERP study. J. Psychophysiol. 2009, 23(2):63-76.
54. Perry A., Bentin S., Shalev I., Israel S., Uzefovsky F., Bar-On D., Ebstein R.P. Intranasal oxytocin modulates EEG mu/alpha and beta rhythms during perception of biological motion. Psychoneuroendocrinology 2010, 35:1446-1453.
55. Perry A., Troje N.F., Bentin S. Exploring motor system contributions to the perception of social information: evidence from EEG activity in the mu/alpha frequency range. Soc. Neurosci. 2010, 5(3):272-284.
56. Pichon S., de Gelder B., Grèzes J. Emotional modulation of visual and motor areas by dynamic body expressions of anger. Soc. Neurosci. 2008, 3:199-212.
57. Pichon S., de Gelder B., Grèzes J. Two different faces of threat. Comparing the neural systems for recognizing fear and anger in dynamic body expressions. Neuroimage 2009, 47:1873-1883.
58. Pourtois G., de Gelder B., Vroomen J., Rossion B., Crommelinck M. The time-course of intermodal binding between seeing and hearing affective information. Neuroreport 2000, 11:1329-1333.
59. Pourtois G., Debatisse D., Despland P.A., de Gelder B. Facial expressions modulate the time course of long latency auditory brain potentials. Brain Res. Cogn. Brain Res. 2002, 14:99-105.
60. Robins D.L., Hunyadi E., Schultz R.T. Superior temporal activation in response to dynamic audio-visual emotional cues. Brain Cogn. 2009, 69:269-278.
61. Sander D., Grandjean D., Pourtois G., Schwartz S., Seghier M.L., Scherer K.R., Vuilleumier P. Emotion and attention interactions in social cognition: brain regions involved in processing anger prosody. Neuroimage 2005, 28:848-858.
62. Sato W., Kochiyama T., Yoshikawa S., Naito E., Matsumura M. Enhanced neural activity in response to dynamic facial expressions of emotion: an fMRI study. Brain Res. Cogn. Brain Res. 2004, 20:81-91.
63. Sauter D.A., Eimer M. Rapid detection of emotion from human vocalizations. J. Cogn. Neurosci. 2010, 22:474-481.
64. Schirmer A., Kotz S.A. Beyond the right hemisphere: brain mechanisms mediating vocal emotional processing. Trends Cogn. Sci. 2006, 10:24-30.
65. Scott G.G., O'Donnell P.J., Leuthold H., Sereno S.C. Early emotion word processing: evidence from event-related potentials. Biol. Psychol. 2009, 80:95-104.
66. Senkowski D., Gomez-Ramirez M., Lakatos P., Wylie G.R., Molholm S., Schroeder C.E., Fox J.J. Multisensory processing and oscillatory activity: analyzing non-linear electrophysiological measures in humans and simians. Exp. Brain Res. 2007, 177:184-195.
67. Sharbrough F., Chatrian G., Lesser R., Lüders H., Nuwer M., Picton T. Guidelines for standard electrode position nomenclature. J. Clin. Neurophysiol. 1991, 8:200-202.
68. Stekelenburg J.J., Vroomen J. Neural correlates of multisensory integration of ecologically valid audiovisual events. J. Cogn. Neurosci. 2007, 19:1964-1973.
69. Stevenson R.A., James T.W. Audiovisual integration in human superior temporal sulcus: inverse effectiveness and the neural processing of speech and object recognition. Neuroimage 2009, 44:1210-1223.
70. Suied C., Bonneel N., Viaud-Delmon I. Integration of auditory and visual information in the recognition of realistic objects. Exp. Brain Res. 2009, 194:91-102.
71. Tallon-Baudry C., Bertrand O. Oscillatory gamma activity in humans and its role in object representation. Trends Cogn. Sci. 1999, 3(4):151-162.
72. Tzagarakis C., Ince N.F., Leuthold A.C., Pellizzer G. Beta-band activity during motor planning reflects response uncertainty. J. Neurosci. 2010, 30:11270-11277.
73. van den Stock J., Righart R., de Gelder B. Body expressions influence recognition of emotions in the face and voice. Emotion 2007, 7:487-494.
74. van den Stock J., Grèzes J., de Gelder B. Human and animal sounds influence recognition of body language. Brain Res. 2008, 1242:185-190.
75. van Heijnsbergen C.C.R.J., Meeren H.K.M., Grèzes J., de Gelder B. Rapid detection of fear in body expressions, an ERP study. Brain Res. 2007, 1186:233-241.
76. Vroomen J., Stekelenburg J.J. Visual anticipatory information modulates multisensory interactions of artificial audiovisual stimuli. J. Cogn. Neurosci. 2010, 22:1583-1596.
77. Vroomen J., Driver J., de Gelder B. Is cross-modal integration of emotional expressions independent of attentional resources?. Cogn. Affect. Behav. Neurosci. 2001, 1:382-387.
78. Ward L.M. Synchronous neural oscillations and cognitive processes. Trends Cogn. Sci. 2003, 7:553-559.
79. Wiethoff S., Wildgruber D., Grodd W., Ethofer T. Response and habituation of the amygdala during processing of emotional prosody. Neuroreport 2009, 20:1356-1360.