Emotional cues during simultaneous face and voice processing: Electrophysiological insights

PLoS ONE

Volumn 7, Issue 2, 2012, Pages

  Liu, Taosheng ^a,b   Pinheiro, Ana ^b,c   Zhao, Zhongxin ^d   Nestor, Paul G ^b,e   McCarley, Robert W ^b   Niznikiewicz, Margaret A ^b  

^a SECOND MILITARY MEDICAL UNIVERSITY   (China)

^b HARVARD MEDICAL SCHOOL   (United States)

^c UNIVERSITY OF MINHO   (Portugal)

^d SECOND MILITARY MEDICAL UNIVERSITY   (China)

^e UNIVERSITY OF MASSACHUSETTS BOSTON   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; AMPLITUDE MODULATION; ARTICLE; ASSOCIATION; AUDITORY STIMULATION; BEHAVIOR; BRAIN ELECTROPHYSIOLOGY; ELECTRODE; EMOTION; EVENT RELATED POTENTIAL; FACE; FEMALE; FRONTAL CORTEX; HUMAN; HUMAN EXPERIMENT; LATENT PERIOD; MALE; NORMAL HUMAN; OCCIPITAL CORTEX; PARIETAL CORTEX; VISUAL STIMULATION; VOICE; ANALYSIS OF VARIANCE; BRAIN; ELECTROENCEPHALOGRAPHY; ELECTROPHYSIOLOGY; EVOKED RESPONSE; FACIAL EXPRESSION; FRONTAL LOBE; METHODOLOGY; MIDDLE AGED; PATHOLOGY; PHYSIOLOGY; RECOGNITION; REPRODUCIBILITY; TIME;

ADULT; ANALYSIS OF VARIANCE; BRAIN; CUES; ELECTRODES; ELECTROENCEPHALOGRAPHY; ELECTROPHYSIOLOGY; EMOTIONS; EVOKED POTENTIALS; FACE; FACIAL EXPRESSION; FEMALE; FRONTAL LOBE; HUMANS; MALE; MIDDLE AGED; RECOGNITION (PSYCHOLOGY); REPRODUCIBILITY OF RESULTS; TIME FACTORS; VOICE;

EID: 84857459994     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0031001     Document Type: Article

References (75)

1. Eimer M, Holmes A, (2007) Event-related brain potential correlates of emotional face processing. Neuropsychologia 45: 15-31.
2. Liu T, Pinheiro AP, Deng G, Nestor PG, McCarley RM, et al. (2012) Electrophysiological insights into processing nonverbal emotional vocalizations. NeuroReport 23: 108-12.
3. Paulmann S, Kotz SA, (2008) Early emotional prosody perception based on different speaker voices. Neuroreport 19: 209-213.
4. Posamentier MT, Abdi H, (2003) Processing faces and facial expressions. Neuropsychol Rev 13: 113-43.
5. Wildgruber D, Ackermann H, Kreifelts B, Ethofer T, (2006) Cerebral processing of linguistic and emotional prosody: fMRI studies. Prog Brain Res 156: 249-268.
6. Kotz SA, Meyer M, Paulmann S, (2006) Lateralization of emotional prosody in the brain: an overview and synopsis on the impact of study design. Prog Brain Res 156: 285-294.
7. Paulmann S, Pell MD, (2009) Facial expression decoding as a function of emotional meaning status: ERP evidence. Neuroreport 20: 1603-8.
8. Focker J, Gondan M, Roder B, (2011) Preattentive processing of audio-visual emotional signals. Acta Psychol (Amst) 137: 36-47.
9. Muller VI, Habel U, Derntl B, Schneider F, Zilles K, et al. (2011) Incongruence effects in crossmodal emotional integration. Neuroimage 54: 2257-2266.
10. Batty M, Taylor MJ, (2003) Early processing of the six basic facial emotional expressions. Brain Res Cogn Brain Res 17: 613-620.
11. Bentin S, Allison T, Puce A, Perez A, McCarthy G, (1996) Electrophysiological studies of face perception in humans. J Cogn Neurosci 8: 551-565.
12. Eimer M, Holmes A, (2002) An ERP study on the time course of emotional face processing. Neuroreport 13: 427-431.
13. Ashley V, Vuilleumier P, Swick D, (2004) Time-course and specificity of event- related potentials to emotional expressions. NeuroReport 15: 211-215.
14. Streit M, Ioannides AA, Liu L, Wolwer W, Dammers J, et al. (1999) Neurophysiological correlates of the recognition of facial expressions of emotion as revealed by magnetoencephalography. Brain Res Cogn Brain Res 7: 481-491.
15. Streit M, Wolwer W, Brinkmeyer J, Ihl R, Gaebel W, (2001) EEG-correlates of facial affect recognition and categorisation of blurred faces in schizophrenic patients and healthy volunteers. Schizophr Res 49: 145-155.
16. Eimer M, McCarthy RA, (1999) Prosopagnosia and structural encoding of faces: evidence from event-related potentials. Neuroreport 10: 255-259.
17. Liu J, Harris A, Kanwisher N, (2002) Stages of processing in face perception: an MEG study. Nat Neurosci 5: 910-916.
18. An SK, Lee SJ, Lee CH, Cho HS, Lee PG, et al. (2003) Reduced P3 amplitudes by negative facial emotional photographs in schizophrenia. Schizophr Res 64: 125-135.
19. Lang PJ, Bradley MM, Cuthbert BN, (1990) Emotion, attention, and the startle reflex. Psychological Review 97: 377-398.
20. Turetsky BI, Kohler CG, Indersmitten T, Bhati MT, Charbonnier D, et al. (2007) Facial emotion recognition in schizophrenia: when and why does it go awry? Schizophr Res 94: 253-263.
21. Paulmann S, Seifert S, Kotz SA, (2010) Orbito-frontal lesions cause impairment during late but not early emotional prosodic processing. Soc Neurosci 5: 59-75.
22. Spreckelmeyer KN, Kutas M, Urbach TP, Altenmuller E, Munte TF, (2006) Combined perception of emotion in pictures and musical sounds. Brain Res 1070: 160-170.
23. Thierry G, Roberts MV, (2007) Event-related potential study of attention capture by affective sounds. Neuroreport 18: 245-248.
24. Sauter DA, Eimer M, (2010) Rapid detection of emotion from human vocalizations. J Cogn Neurosci 22: 474-481.
25. Everhart DE, Demaree HA, Shipley AJ, (2006) Perception of emotional prosody: moving toward a model that incorporates sex-related differences. Behav Cogn Neurosci Rev 5: 92-102.
26. Delplanque S, Silvert L, Hot P, Sequeira H, (2005) Event-related P3a and P3b in response to unpredictable emotional stimuli. Biol Psychol 68: 107-120.
27. Ethofer T, Anders S, Erb M, Droll C, Royen L, et al. (2006) Impact of voice on emotional judgment of faces: an event-related fMRI study. Hum Brain Mapp 27: 707-714.
28. Kreifelts B, Ethofer T, Grodd W, Erb M, Wildgruber D, (2007) Audiovisual integration of emotional signals in voice and face: an event-related fMRI study. Neuroimage 37: 1445-1456.
29. Massaro DW, Egan PB, (1996) Perceiving affect from the voice and the face. Psychon Bull Rev 3: 215-221.
30. de Gelder B, Vroomen J, (2000) The perception of emotions by ear and by eye. Cogn Emot 14: 289-311.
31. de Gelder B, Bocker KB, Tuomainen J, Hensen M, Vroomen J, (1999) The combined perception of emotion from voice and face: early interaction revealed by human electric brain responses. Neurosci Lett 260: 133-136.
32. Pourtois G, de Gelder B, Vroomen J, Rossion B, Crommelinck M, (2000) The time-course of intermodal binding between seeing and hearing affective information. Neuroreport 11: 1329-1333.
33. Pourtois G, Debatisse D, Despland PA, de Gelder B, (2002) Facial expressions modulate the time course of long latency auditory brain potentials. Brain Res Cogn Brain Res 14: 99-105.
34. Frassinetti F, Bolognini N, Làdavas E, (2002) Enhancement of visual perception by crossmodal visuo-auditory interaction. Exp Brain Res 147: 332-343.
35. Bolognini N, Frassinetti F, Serino A, Làdavas E, (2005) "Acoustical vision" of below threshold stimuli: interaction among spatially converging audiovisual inputs. Exp Brain Res 160: 273-282.
36. McDonald JJ, Teder-Sälejärvi WA, Hillyard SA, (2000) Involuntary orienting to sound improves visual perception. Nature 407: 906-908.
37. Giard MH, Peronnet F, (1999) Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J Cogn Neurosci 11: 473-490.
38. Brefczynski-Lewis J, Lowitszch S, Parsons M, Lemieux S, Puce A, (2009) Audiovisual non-verbal dynamic faces elicit converging fMRI and ERP responses. Brain Topogr 21: 193-206.
39. Lang PJ, Bradley MM, Cuthbert BN, (2008) International affective picture system (IAPS): Affective ratings of pictures and instruction manual. Technical Report A-8 University of Florida, Gainesville, FL.
40. Puce A, Epling JA, Thompson JC, Carrick OK, (2007) Neural responses elicited to face motion and vocalization pairings. Neuropsychologia 45: 93-106.
41. Critchley HD, Daly EM, Phillips M, Brammer M, Bullmore E, et al. (2000) Explicit and implicit neural mechanisms for processing of social information from facial expressions: A functional magnetic resonance imaging study. Hum Brain Mapp 9: 93-105.
42. Eimer M, Holmes A, McGlone F, (2003) The role of spatial attention in the processing of facial expression: An ERP study of rapid brain responses to six basic emotions. Cogn Affect Behav Neurosci 3: 97-110.
43. Pourtois G, Grandjean D, Sander D, Vuilleumier P, (2004) Electrophysiological correlates of rapid spatial orienting towards fearful faces. Cereb Cortex 14: 619-33.
44. Oldfield RC, (1971) The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia 9: 97-113.
45. Langner O, Dotsch R, Bijlstra G, Wigboldus DHJ, Hawk ST, et al. (2010) Presentation and validation of the Radboud Faces Database. Cogn Emot 24: 1377-1388.
46. Bradley MM, Lang PJ, (2007) The International Affective Digitized Sounds (2nd Edition; IADS-2): Affective ratings of sounds and instruction manual. Technical report B-3 University of Florida, Gainesville, Fl.
47. Gratton G, Coles MG, Donchin E, (1983) A new method for off-line removal of ocular artifact. Electroencephalogr Clin Neurophysiol 55: 468-484.
48. Vuilleumier P, Armony JL, Driver J, Dolan RJ, (2001) Effects of attention and emotion on face processing in the human brain: an event-related fMRI study. Neuron 30: 829-841.
49. Mishra J, Hillyard SA, (2009) Endogenous attention selection during binocular rivalry at early stages of visual processing. Vision Res 49: 1073-80.
50. Woods DL, (1995) The component structure of the N1 wave of the human auditory evoked potential. Electroencephalogr Clin Neurophysiol Suppl 44 pp. 102-9.
51. Luo W, Feng W, He W, Wang NY, Luo YJ, (2010) Three stages of facial expression processing: ERP study with rapid serial visual presentation. Neuroimage 49: 1857-67.
52. Crowley KE, Colrain IM, (2004) A review of the evidence for P2 being an independent component process: age, sleep and modality. Clin Neurophysiol 115: 732-744.
53. Garcia-Larrea L, Lukaszewicz AC, Mauguiere F, (1992) Revisiting the oddball paradigm. Non-target vs neutral stimuli and the evaluation of ERP attentional effects. Neuropsychologia 30: 723-741.
54. Lenz D, Schadow J, Thaerig S, Busch NA, Herrmann CS, (2007) What's that sound? Matches with auditory long-term memory induce gamma activity in human EEG. Int J Psychophysiol 64: 31-38.
55. Novak G, Ritter W, Vaughan HG, (1992) The chronometry of attention-modulated processing and automatic mismatch detection. Psychophysiology 29: 412-430.
56. Schutter DJ, de Haan EH, van Honk J, (2004) Functionally dissociated aspects in anterior and posterior electrocortical processing of facial threat. Int J Psychophysiol 53: 29-36.
57. Pizzagalli DA, Lehmann D, Hendrick AM, Regord M, Pascual-Marqui RD, et al. (2002) Affective judgments of faces modulate early activity (approximately 160 ms) within the fusiform gyri. NeuroImage 16: 663-677.
58. Paulmann S, Kotz SA, (2008) An ERP investigation on the temporal dynamics of emotional prosody and emotional semantics in pseudo- and lexical-sentence context. Brain Lang 105: 59-69.
59. Streit M, Wolwer W, Brinkmeyer J, Ihl R, Gaebel W, (2000) Electrophysiological correlates of emotional and structural face processing in humans. Neurosci Lett 278: 13-16.
60. Wynn JK, Lee J, Horan WP, Green MF, (2008) Using event related potentials to explore stages of facial affect recognition deficits in schizophrenia. Schizophr Bull 34: 679-687.
61. Lebib R, Papo D, Douiri A, de Bode S, Gillon Dowens M, et al. (2004) Modulations of 'late' event-related brain potentials in humans by dynamic audiovisual speech stimuli. Neurosci Lett 372: 74-79.
62. Munte TF, Brack M, Grootheer O, Wieringa BM, Matzke M, et al. (1998) Brain potentials reveal the timing of face identity and expression judgments. Neurosci Res 30: 25-34.
63. Williams LM, Palmer D, Liddell BJ, Song L, Gordon E, (2006) The 'when' and 'where' of perceiving signals of threat versus non-threat. Neuroimage 31: 458-67.
64. Curran T, Tanaka JW, Weiskopf DM, (2002) An electrophysiological comparison of visual categorization and recognition memory. Cogn Affect Behav Neurosci 2: 1-18.
65. Polich J, Pitzer A, (1999) P300 and Alzheimer's disease: oddball task difficulty and modality effects. Electroencephalogr Clin Neurophysiol Suppl 50 pp. 281-287.
66. Carretie L, Iglesias J, Garcia T, (1997) A study on the emotional processing of visual stimuli through event-related potentials. Brain Cogn 34: 207-217.
67. Carretie L, Iglesias J, Garcia T, Ballesteros M, (1997) N300, P300 and the emotional processing of visual stimuli. Electroencephalogr Clin Neurophysiol 103: 298-303.
68. Johnston VS, Miller DR, Burleson MH, (1986) Multiple P3s to emotional stimuli and their theoretical significance. Psychophysiology 23: 684-694.
69. Dolcos F, Cabeza R, (2002) Event-related potentials of emotional memory: encoding pleasant, unpleasant, and neutral pictures. Cogn Affect Behav Neurosci 2: 252-263.
70. Weinstein AM, (1995) Visual ERPs evidence for enhanced processing of threatening information in anxious university students. Biol Psychiatry 37: 847-58.
71. Olofsson JK, Nordin S, Sequeira H, Polich J, (2008) Affective picture processing: an integrative review of ERP findings. Biol Psychol 77: 247-265.
72. Dominguez-Borras J, Garcia-Garcia M, Escera C, (2008) Emotional context enhances auditory novelty processing: behavioural and electrophysiological evidence. Eur J Neurosci 28: 1199-1206.
73. Saint-Amour D, De Sanctis P, Molholm S, Ritter W, Foxe JJ, (2007) Seeing voices: High-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion. Neuropsychologia 45: 587-597.
74. Kayser C, Logothetis NK, Panzeri S, (2010) Visual enhancement of the information representation in auditory cortex. Curr Biol 20: 19-24.
75. Werner S, Noppeney U, (2010) Distinct functional contributions of primary sensory and association areas to audiovisual integration in object categorization. J Neurosci 30: 2662-75.