Investigating the multimodal nature of human communication: Insights from ERPs

Journal of Psychophysiology

Volumn 23, Issue 2, 2009, Pages 63-76

  Paulmann, Silke ^a,c   Jessen, Sarah ^b   Kotz, Sonja A ^b  

^a MCGILL UNIVERSITY   (Canada)

^b MAX PLANCK INSTITUTE FOR HUMAN COGNITIVE AND BRAIN SCIENCES   (Germany)

^c UNIVERSITY OF ESSEX   (United Kingdom)

Author keywords

ERPs; Facilitation; N300; P200

Indexed keywords

ADULT; ARTICLE; ASSOCIATION; CONTROLLED STUDY; EMOTION; EVENT RELATED POTENTIAL; FACIAL EXPRESSION; FACILITATION; FEMALE; HUMAN; HUMAN EXPERIMENT; INTERPERSONAL COMMUNICATION; LANGUAGE PROCESSING; MALE; NORMAL HUMAN; SEMANTICS; STIMULUS RESPONSE; TASK PERFORMANCE; VISUAL INFORMATION;

EID: 70149093662     PISSN: 02698803     EISSN: None     Source Type: Journal    
DOI: 10.1027/0269-8803.23.2.63     Document Type: Article

References (59)

1. Adolphs, R. (2002). Neural systems for recognizing emotion. Current Opinion in Neurobiology, 12, 69-177.
2. Anderson, A.K., & Phelps, E.A. (2001). Lesions of the human amygdala impair enhanced perception of emotionally salient events. Nature, 411, 305-309.
3. American Electroencephalographic Society. (1991). Guidelines for standard electrode position nomenclature. Journal of Clinical Neurophysiology, 8, 200-202.
4. Ashley, V., Vuilleumier, P., & Swick, D. (2004). Time course and specificity of event related potentials to emotional expressions. Neuroreport, 15, 211-216.
5. Balconi, M., & Pozzoli, U. (2003). Face-selective processing and the effect of pleasant and unpleasant emotional expressions on ERP correlates. International Journal of Psychophysiology, 49, 67-74.
6. Batty, M., & Taylor, M.J. (2003). Early processing of the six basic facial emotional expressions. Brain Research. Cognitive Brain Research, 17, 613-620.
7. Borod, J.C., Pick, L.H., Hall, S., Sliwinski, M., Madigan, N., Obler, L.K. et al. (2000). Relationships among facial, prosodic, and lexical channels of emotional perceptual processing. Cognition and Emotion, 14, 193-211.
8. Bostanov, V., & Kotchoubey, B. (2004). Recognition of affective prosody: Continuous wavelet measures of event-related brain potentials to emotional exclamations. Psychophysiology, 41, 259-268.
9. Calvert, G.A., Brammer, M.J., & Iversen, S.D. (1998). Crossmodal identification. Trends in Cognitive Science, 2, 247-253.
10. Campanella, S., Gaspard, C., Bruyer, R., Debatisse, D., Crommelinck, M., & Guerit, J.M. (2002). Discrimination of emotional facial expressions in a visual oddball task: An ERP study. Biological Psychiatry, 59, 171-186.
11. Campbell, R. (2008). The processing of audio-visual speech: Empirical and neural bases. Philosophical Transactions of the Royal Society London B Biological Science, 363, 1001-1010.
12. Carretié, L., & Iglesias, J. (1995). An ERP study on the specificity of facial expression processing. International Journal of Psychophysiology, 19, 183-192.
13. Carretié, L., Iglesias, J., & García, T. (1997). A study on the emotional-processing of visual stimuli through event-related potentials. Brain and Cognition, 34, 207-217.
14. Carretié, L., Iglesias, J., García, T., & Ballesteros, M. (1997). N300, P300 and the emotional processing of visual stimuli. Electroencephalography and Clinical Neurophysiology, 103, 298-303.
15. Carretié, L., Mercado, F., Tapia, M., & Hinojosa, J.A. (2001). Emotion, attention, and the "negativity bias," studied through event-related potentials. International Journal of Psychophysiology, 41(1), 75-85.
16. de Gelder, B., & Bertelson, P. (2003). Multisensory integration, perception and ecological validity. Trends in Cognitive Science, 7, 460-467.
17. de Gelder, B., & Vroomen, J. (2000). The perception of emotions by ear and by eye. Cognition and Emotion, 14, 289-311.
18. de Gelder, B., Böcker, K.B.E., Tuomainen, J., Hensen, M., & Vroomen, J. (1999). The combined perception of emotion from voice and face: Early interaction revealed by human electric brain responses. Neuroscience Letters, 260, 133-136.
19. de Gelder, B., Vrommen, J., & Pourtois, G. (1999). Seeing cries and hearing smiles: Crossmodal perception of emotional expressions. In G. Aschersleben, T. Bachmann, & J. Müsseler (Eds.), Cognitive contributions to the perception of spatial and temporal events. Advances in Psychology, 129 (pp. 425-438). Amsterdam: North-Holland/Elsevier.
20. Eimer, M., & Holmes, A. (2002). An ERP study on the time course of emotional face processing. Neuroreport, 13, 427-431.
21. Eimer, M., Holmes, A., & McGlone, F.P. (2003). The role of spatial attention in the processing of facial expression: An ERP study of rapid brain responses to six basic emotions. Cognitive, Affective, and Behavioral Neuroscience, 3, 97-110.
22. Ekman, P. (1992). An argument for basic emotions. Cognition and Emotion, 6, 169-200.
23. Elfenbein, H.A., & Ambady, N. (2002). On the universality and cultural specificity of emotion recognition: A meta-analysis. Psychological Bulletin, 128, 203-235.
24. Fox, E., Lester, V., Russo, R., Bowles, R.J., Pichler, A., & Dutton, K. (2000). Facial expressions of emotion: Are angry faces detected more efficiently? Cognition and Emotion, 14, 61-92.
25. Federmeier, K.D., & Kutas, M. (2002). Picture the difference: Electrophysiological investigations of picture processing in the two cerebral hemispheres. Neuropsychologia, 40, 730-747.
26. Giard, M.H., & Peronnet, F. (1999). Auditory-visual integration during multimodal object recognition in humans: A behavioral and electrophysiological study. Journal of Cognitive Neuroscience, 11, 473-490.
27. Greenhouse, S., & Geisser, S. (1959). On methods in the analysis of profile data. Psychometrika, 24, 95-112.
28. Hamm, J.P., Johnson, B.W., & Kirk, I.J. (2002). Comparison of the N300 and N400 ERPs to picture stimuli in congruent and incongruent contexts. Clinical Neurophysiology, 113, 1339-1350.
29. Keppel, G., 1991. Design and analysis: A researcher's handbook. Englewood Cliffs, NJ: Prentice Hall.
30. Kotz, S.A., & Paulmann, S. (2007). When emotional prosody and semantics dance cheek to cheek: ERP evidence. Brain Research, 1151, 107-118.
31. Kreifelts, B., Ethofer, T., Grodd, W., Erb, M., & Wildgruber, D. (2007). Audiovisual integration of emotional signals in voice and face: An event-related fMRI study. Neuroimage, 37, 1445-1456.
32. Lang, P.J. (1993). The network model of emotion: Motivational concerns. In R.S. Wyer & T.K. Scrull (Eds.), Advances in social cognition (pp. 109-133). Hillsdale, NJ: Erlbaum.
33. Lang, P.J., & Davis, M. (2006). Emotion, motivation, and the brain: Reflex foundations in animal and human research. Progress in Brain Research, 156, 3-29.
34. Lebib, R., Papo, D., Douiri, A., de Bode, S., Gillon Dowens, M., & Baudonniere, P.M. (2004). Modulations of late event-related brain potentials in human by dynamic audiovisual speech stimuli. Neuroscience Letters, 372(1-2), 74-79.
35. LeDoux, J.E. (2000). Emotion circuits in the brain. Annual Review of Neuroscience, 23, 155-184.
36. Luck, S.J., & Hillyard, S.A. (1994). Electrophysiological correlates of feature analysis during visual search. Psychophysiology, 31, 291-308.
37. Miller, L.M., & D'Esposito, M. (2005). Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. The Journal of Neuroscience, 25, 5884-5893.
38. Niedenthal, P.M. (2007). Embodying Emotion. Science, 316, 1002-1005.
39. Pantev, C., Elbert, T., Ross, B., Eulitz, C., & Terhardt, E. (1996). Binaural fusion and the representation of virtual pitch in the human auditory cortex. Hearing Research, 100, 164-170.
40. Paulmann, S. (2006). Electrophysiological evidence on the processing of emotional prosody: Insights from healthy and patient populations (Max Planck Series in Human Cognitive and Brain Sciences,Vol. 71). Leipzig, Germany: Max Planck Institute for Human Cognitive and Brain Sciences.
41. Paulmann, S.,& Kotz, S.A. (2006, August). Valence, arousal, and task effects on the P200 in emotional prosody processing. Poster presented at the 12th Annual Conference on Architectures and Mechanisms for Language Processing 2006 (AMLaP), Nijmegen, The Netherlands.
42. Paulmann, S., & Kotz, S.A. (2008a). Early emotional prosody perception based on different speaker voices. Neuroreport, 19, 209-213.
43. Paulmann, S., & Kotz, S.A. (2008b). An ERP investigation on the temporal dynamics of emotional prosody and emotional semantics in pseudo- and lexical-sentence context. Brain and Language, 105, 59-69.
44. Paulmann, S., & Pell, M.D. (2009, April). Rapid processing of vocal emotion expressions as revealed by ERPs. Poster presented at the Evoked Potentials International Conference XV, Indiana University, IN, USA.
45. Paulmann, S., Pell, M.D., & Kotz, S.A. (2008). How aging affects the recognition of emotional speech. Brain and Language, 104, 262-269.
46. Paulmann, S., Schmidt, P., Pell, M.D., & Kotz, S.A. (2008, May). Rapid processing of emotional and voice information as evidenced by ERPs. In Speech Prosody 2008, Fourth Conference on Speech Prosody, Campinas, Brazil.
47. Pell, M.D., Monetta, L., Paulmann, S., & Kotz, S.A. (in press). Recognizing emotions in a foreign language. Journal of Nonverbal Behavior.
48. Phillips, M.L., Drevets, W.C., Rauch, S.L., & Lane, R. (2003). Neurobiology of emotion perception I: The neural basis of normal emotion perception. Biological Psychiatry, 54, 504-514.
49. Picton, T.W., Woods, D.L., Barribeau-Braun, J., & Healy, T.M.G. (1977). Evoked potential audiometry. Journal of Otolaryngology, 6, 90-119.
50. Pizzagalli, D., Regard, M., Lehmann, D. (1999). Rapid emotional face processing in the human right and left brain hemispheres: An ERP study. Neuroreport, 10, 2691-2698.
51. Pourtois, G., de Gelder, B., Vroomen, J., Rossion, B., & Crommelinck, M. (2000). The time-course of intermodal binding between seeing and hearing affective information. Neuroreport, 11, 1329-1333.
52. Pourtois, G., Debatisse, D., Despland, P.-A., & de Gelder, B. (2002). Facial expressions modulate the time course of long latency auditory brain potentials. Brain Research. Cognitive Brain Research, 14, 99-105.
53. Schirmer, A., & Kotz, S.A. (2006). Beyond the right hemisphere: Brain mechanisms mediating vocal emotion processing. Trends in Cognitive Science, 10, 24-30.
54. Schutter, D.J.L.G., de Haan, E.H.F., & van Honk, J. (2004). Functionally dissociated aspects in anterior and posterior electrocortical processing of facial threat. International Journal of Psychophysiology, 53, 29-36.
55. Stekelenburg, J.J., & Vroomen, J. (2007). Neural correlates of multisensory integration of ecologically valid audiovisual events. Journal of Cognitive Neuroscience, 19, 1964-1973.
56. Steinhauer, K., Alter, K.,&Friederici, A.D. (1999). Brain responses indicate immediate use of prosodic cues in natural speech processing. Nature Neuroscience, 2, 191-196.
57. van Wassenhove, V., Grant, K.W., & Poeppel, D. (2005). Visual speech speeds up the neural processing of auditory speech. Proceedings of the National Academy of Sciences, 102, 1181-1186.
58. Vogel, E.K., & Luck, S.J. (2000). The visual N1 component as an index of a discrimination process. Psychophysiology, 37, 190-203.
59. Wu, Y.C., & Coulson, S. (2007). How iconic gestures enhance communication: An ERP study. Brain & Language, 101, 234-245.