How bodies and voices interact in early emotion perception

PLoS ONE

Volumn 7, Issue 4, 2012, Pages

  Jessen, Sarah ^a,b   Obleser, Jonas ^a   Kotz, Sonja A ^a  

^a MAX PLANCK INSTITUTE FOR HUMAN COGNITIVE AND BRAIN SCIENCES   (Germany)

^b FREIE UNIVERSITÄT BERLIN   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ARTICLE; AUDITORY STIMULATION; BETA RHYTHM; BODY EXPRESSION; COGNITION; ELECTROENCEPHALOGRAPHY; EMOTION; FEMALE; HEARING; HUMAN; HUMAN EXPERIMENT; MALE; NOISE; NONVERBAL COMMUNICATION; NORMAL HUMAN; PERCEPTION; STIMULUS RESPONSE; VISION; VISUAL STIMULATION; VOCALIZATION; VOICE; ART; BEHAVIOR; EVOKED RESPONSE; PHYSIOLOGY; PSYCHOSEXUAL DEVELOPMENT; TIME;

ACOUSTIC STIMULATION; ADULT; BEHAVIOR; EMOTIONS; EVOKED POTENTIALS; FEMALE; HUMAN BODY; HUMANS; LATENCY PERIOD (PSYCHOLOGY); MALE; PERCEPTION; TIME FACTORS; VOICE;

EID: 84860491311     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0036070     Document Type: Article

References (66)

1. Pollick FE, Kay JW, Heim K, Stringer R, (2005) Gender recognition from point-light walkers. J Exp Psychol Hum Percept Perform 31: 1247-1265.
2. Atkinson AP, Dittrich WH, Gemmell AJ, Young AW, (2004) Emotion perception from dynamic and static body expressions in point-light and full-light displays. Perception 33: 717-746.
3. Paulmann S, Kotz SA, (2008) Early emotional prosody perception based on different speaker voices. Neuroreport 19: 209-213.
4. Loula F, Prasad S, Harber K, Shiffrar M, (2005) Recognizing people from their movement. J Exp Psychol Hum Percept Perform 31: 210-220.
5. Latinus M, Belin P, (2011) Human voice perception. Curr Biol 21: R143-R145.
6. Van den Stock J, Grèzes J, de Gelder B, (2008) Human and animal sounds inuence recognition of body language. Brain Res 1242: 185-190.
7. Jessen S, Kotz SA, (2011) The temporal dynamics of processing emotions from vocal, facial, and bodily expressions. Neuroimage 58: 665-674.
8. Petrini K, McAleer P, Pollick F, (2010) Audiovisual integration of emotional signals from music improvisation does not depend on temporal correspondence. Brain Res 1323: 139-148.
9. Petrini K, Crabbe F, Sheridan C, Pollick FE, (2011) The music of your emotions: neural substrates involved in detection of emotional correspondence between auditory and visual music actions. PLoS One 6: e19165.
10. Stein BE, Meredith M, (1993) The merging of the senses. MIT Press, Cambridge, Massachusetts.
11. Ethofer T, Pourtois G, Wildgruber D, (2006) Investigating audiovisual integration of emotional signals in the human brain. Prog Brain Res 156: 345-361.
12. Campanella S, Belin P, (2007) Integrating face and voice in person perception. Trends Cogn Sci 11: 535-543.
13. Calvert GA, Hansen PC, Iversen SD, Brammer MJ, (2001) Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage 14: 427-438.
14. Senkowski D, Gomez-Ramirez M, Lakatos P, Wylie GR, Molholm S, et al. (2007) Multisensory processing and oscillatory activity: analyzing non-linear electrophysiological measures in humans and simians. Exp Brain Res 177: 184-195.
15. Beauchamp MS, (2005) Statistical criteria in FMRI studies of multisensory integration. Neuroinformatics 3: 93-113.
16. Laurienti PJ, Perrault TJ, Stanford TR, Wallace MT, Stein BE, (2005) On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies. Exp Brain Res 166: 289-297.
17. Ghazanfar AA, Maier JX, Hoffman KL, Logothetis NK, (2005) Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. J Neurosci 25: 5004-5012.
18. Lakatos P, Chen CM, O'Connell MN, Mills A, Schroeder CE, (2007) Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53: 279-292.
19. Holle H, Obleser J, Rueschemeyer SA, Gunter TC, (2010) Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions. Neuroimage 49: 875-884.
20. Werner S, Noppeney U, (2010) Superadditive responses in superior temporal sulcus predict audiovisual benefits in object categorization. Cereb Cortex 20: 1829-1842.
21. Senkowski D, Saint-Amour D, Höfle M, Foxe JJ, (2011) Multisensory interactions in early evoked brain activity follow the principle of inverse effectiveness. Neuroimage 56: 2200-2208.
22. Collignon O, Girard S, Gosselin F, Roy S, Saint-Amour D, et al. (2008) Audio-visual integration of emotion expression. Brain Res 1242: 126-135.
23. Pourtois G, Debatisse D, Despland PA, de Gelder B, (2002) Facial expressions modulate the time course of long latency auditory brain potentials. Brain Res Cogn Brain Res 14: 99-105.
24. Meeren HKM, van Heijnsbergen CCRJ, de Gelder B, (2005) Rapid perceptual integration of facial expression and emotional body language. Proc Natl Acad Sci U S A 102: 16518-16523.
25. Van den Stock J, Righart R, de Gelder B, (2007) Body expressions inuence recognition of emotions in the face and voice. Emotion 7: 487-494.
26. Paulmann S, Jessen S, Kotz SA, (2009) Investigating the multimodal nature of human communication: an ERP study. Journal of Psychophysiology 23 (2): 63-76.
27. Thomas JP, Shiffrar M, (2010) I can see you better if I can hear you coming: Action-consistent sounds facilitate the visual detection of human gait. Journal of Vision 10(12):14: 1-11.
28. de Gelder B, Vroomen J, (2000) The perception of emotions by ear and by eye. Cognition and Emotion 14 (3): 289-311.
29. Kreifelts B, Ethofer T, Grodd W, Erb M, Wildgruber D, (2007) Audiovisual integration of emotional signals in voice and face: an event-related fMRI study. Neuroimage 37: 1445-1456.
30. Stekelenburg JJ, Vroomen J, (2007) Neural correlates of multisensory integration of ecologically valid audiovisual events. J Cogn Neurosci 19: 1964-1973.
31. van Wassenhove V, Grant KW, Poeppel D, (2005) Visual speech speeds up the neural processing of auditory speech. Proc Natl Acad Sci U S A 102: 1181-1186.
32. Ulloa ER, Pineda JA, (2007) Recognition of point-light biological motion: mu rhythms and mirror neuron activity. Behav Brain Res 183: 188-194.
33. Perry A, Troje NF, Bentin S, (2010) Exploring motor system contributions to the perception of social information: Evidence from EEG activity in the mu/alpha frequency range. Social Neuroscience 5 (3): 272-284.
34. Dietrich S, Hertrich I, Alter K, Ischebeck A, Ackermann H, (2008) Understanding the emotional expression of verbal interjections: a functional MRI study. Neuroreport 19: 1751-1755.
35. Pichon S, de Gelder B, Grèzes J, (2008) Emotional modulation of visual and motor areas by dynamic body expressions of anger. Soc Neurosci 3: 199-212.
36. Sharbrough F, Chatrian G, Lesser R, Lüders H, Nuwer M, et al. (1991) Guidelines for standard electrode position nomenclature. Journal of Clinical Neurophysiology 8: 200-202.
37. MacMillan N, Creelman C, eds. (2005) Detection Theory: A User's Guide (2nd edition). Lawrence Erlbaum Associates Publishers, Mahwah, New Jersey.
38. Greenhouse S, Geisser S, (1959) On methods in the analysis of profile data. Psychometrika 24: 95-112.
39. Holm S, (1979) A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6 (2): 65-70.
40. Rosenthal R, (1991) Meta-analytic procedures for social research (revised) Newbury Park, CA Sage.
41. Oostenveld R, Fries P, Maris E, Schoffelen JM, (2011) FieldTrip: Open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data. Comput Intell Neurosci 2011: 156869.
42. Tallon-Baudry C, Bertrand O, (1999) Oscillatory gamma activity in humans and its role in object representation. Trends Cogn Sci 3 (4): 151-162.
43. Lachaux JP, Rodriguez E, Martinerie J, Varela FJ, (1999) Measuring phase synchrony in brain signals. Human Brain Mapping 8: 194-208.
44. Gross J, Kujala J, Hamalainen M, Timmermann L, Schnitzler A, et al. (2001) Dynamic imaging of coherent sources: Studying neural interactions in the human brain. Proc Natl Acad Sci U S A 98: 694-699.
45. Medendorp WP, Kramer GFI, Jensen O, Oostenveld R, Schoffelen JM, et al. (2007) Oscillatory activity in human parietal and occipital cortex shows hemispheric lateralization and memory effects in a delayed double-step saccade task. Cereb Cortex 17: 2364-2374.
46. Haegens S, Osipova D, Oostenveld R, Jensen O, (2010) Somatosensory working memory performance in humans depends on both engagement and disengagement of regions in a distributed network. Hum Brain Mapp 31: 26-35.
47. Kiesel A, Miller J, Jolicoeur P, Brisson B, (2008) Measurement of ERP latency differences: a comparison of single-participant and jackknife-based scoring methods. Psychophysiology 45: 250-274.
48. Shahin AJ, Picton TW, Miller LM, (2009) Brain oscillations during semantic evaluation of speech. Brain Cogn 70: 259-266.
49. Sauter DA, Eimer M, (2010) Rapid detection of emotion from human vocalizations. J Cogn Neurosci 22: 474-481.
50. Callan DE, Jones JA, Munhall K, Callan AM, Kroos C, et al. (2003) Neural processes underlying perceptual enhancement by visual speech gestures. Neuroreport 14: 2213-2218.
51. Maier JX, Neuhoff JG, Logothetis NK, Ghazanfar AA, (2004) Multisensory integration of looming signals by rhesus monkeys. Neuron 43: 177-181.
52. Maier JX, Chandrasekaran C, Ghazanfar AA, (2008) Integration of bimodal looming signals through neuronal coherence in the temporal lobe. Curr Biol 18: 963-968.
53. Cappe C, Thut G, Romei V, Murray MM, (2009) Selective integration of auditory-visual looming cues by humans. Neuropsychologia 47: 1045-1052.
54. Romei V, Murray MM, Cappe C, Thut G, (2009) Preperceptual and stimulus-selective enhancement of low-level human visual cortex excitability by sounds. Curr Biol 19: 1799-1805.
55. de Gelder B, Böcker KB, Tumainen J, Hensen M, Vroomen J, (1999) The combined perception of emotion from voice and face: early interaction revealed by human electric brain responses. Neuroscience Letters 260: 133-136.
56. Stevenson RA, James TW, (2009) Audiovisual integration in human superior temporal sulcus: Inverse effectiveness and the neural processing of speech and object recognition. Neuroimage 44: 1210-1223.
57. Stein BE, Stanford TR, Ramachandran R, Perrault TJ, Rowland BA, (2009) Challenges in quantifying multisensory integration: alternative criteria, models, and inverse effectiveness. Exp Brain Res 198: 113-126.
58. Cochin S, Barthelemy C, Lejeune B, Roux S, Martineau J, (1998) Perception of motion and qEEG activity in human adults. Electroencephalogr Clin Neurophysiol 107: 287-295.
59. Pineda JA, (2005) The functional significance of mu rhythms: translating "seeing" and "hearing" into "doing". Brain Res Brain Res Rev 50: 57-68.
60. Oberman L, McCleery J, Ramachandran V, Pineda JA, (2007) EEG evidence for mirror neuron activity during the observation of human and robot actions: Towards an analysis of the human qualities of interactive robots. Neurocomputing 70: 2194-2203.
61. Muthukumaraswamy SD, Johnson BW, Gaetz WC, Cheyne DO, (2006) Neural processing of observed oro-facial movements reects multiple action encoding strategies in the human brain. Brain Res 1071: 105-112.
62. Muthukumaraswamy SD, Singh KD, (2008) Modulation of the human mirror neuron system during cognitive activity. Psychophysiology 45: 896-905.
63. Grèzes J, Decety J, (2001) Functional anatomy of execution, mental simulation, observation, and verb generation of actions: A meta-analysis. Hum Brain Mapp 12: 1-19.
64. Saygin AP, Wilson SM, Hagle DJ Jr, Bates E, Sereno MI, (2004) Point-light biological motion perception activates human premotor cortex. The Journal of Neuroscience 24 (27): 6181-6188.
65. Wheaton KJ, Thompson JC, Syngeniotis A, Abbott DF, Puce A, (2004) Viewing the motion of human body parts activates different regions of premotor, temporal, and parietal cortex. Neuroimage 22: 277-288.
66. de Gelder B, (2006) Towards the neurobiology of emotional body language. Nat Rev Neurosci 7: 242-249.