Preattentive processing of audio-visual emotional signals

Acta Psychologica

Volumn 137, Issue 1, 2011, Pages 36-47

  Föcker, Julia ^a,b   Gondan, Matthias ^c,d   Röder, Brigitte ^a  

^a UNIVERSITY OF HAMBURG   (Germany)

^b UNIVERSITY OF ROCHESTER   (United States)

^c UNIVERSITY OF REGENSBURG   (Germany)

^d UNIVERSITY OF HEIDELBERG   (Germany)

Author keywords

Attention; Emotion; Facial expression; Multisensory; Response conflict; Vocal expression

Indexed keywords

ADULT; ANALYSIS OF VARIANCE; ARTICLE; ATTENTION; EMOTION; FACIAL EXPRESSION; FEMALE; HUMAN; MALE; PERCEPTION; PHYSIOLOGY; REACTION TIME; VISION;

ADULT; ANALYSIS OF VARIANCE; ATTENTION; EMOTIONS; FACIAL EXPRESSION; FEMALE; HUMANS; MALE; REACTION TIME; SOCIAL PERCEPTION; VISUAL PERCEPTION;

EID: 79954665435     PISSN: 00016918     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.actpsy.2011.02.004     Document Type: Article

References (53)

1. Belin P., Fillion-Bilodeau S., Gosselin F. The Montreal Affective Voices: A validated set of nonverbal affect bursts for research on auditory affective processing. Behavior Research Methods 2008, 40:531-539.
2. Bertelson P., Vroomen J., de Gelder B., Driver J. The ventriloquist effect does not depend on the direction of deliberate visual attention. Perception & Psychophysics 2000, 62:321-332.
3. Collignon O., Girard S., Gosselin F., Roy S., Saint-Amour D., Lassonde M., Lepore F. Audio-visual integration of emotion expression. Brain Research 2008, 25:126-135.
4. Darwin C. The expressions of the emotions in man and animals 1872, John Murray, London. first ed.
5. de Gelder B., Böcker K.B., Tuomainen J., Hensen M., Vroomen J. The combined perception of emotion from voice and face: early interaction revealed by human electric brain responses. Neuroscience Letters 1999, 260:133-136.
6. de Gelder B., Morris J.S., Dolan R.J. Unconscious fear influences emotional awareness of faces and voices. Proceedings of the National Academy of Sciences of the United States of America 2005, 102:18682-18687.
7. de Gelder B., Pourtois G., Weiskrantz L. Fear recognition in the voice is modulated by unconsciously recognized facial expressions but not by unconsciously recognized affective pictures. Proceedings of the National Academy of Sciences of the United States of America 2002, 99:4121-4126.
8. de Gelder B., Vroomen J. The perception of emotions by ear and by eye. Cognition and Emotion 2000, 14:289-311.
9. de Jong J.J., Hodiamont P.P., Van den Stock J., de Gelder B. Audiovisual emotion recognition in schizophrenia: reduced integration of facial and vocal affect. Schizophrenia Research 2009, 107:286-293.
10. de Jong J.J., Hodiamont P.P., Van den Stock J., de Gelder B. Modality-specific attention and multisensory integration of emotions in schizophrenia: reduced regulatory effects. Schizophrenia Research 2010, 122:136-143.
11. Doehrmann O., Naumer M.J. Semantics and the multisensory brain: How meaning modulates processes of audio-visual integration. Brain Research 2008, 1242:136-150.
12. Dolan R.J., Morris J.S., de Gelder B. Crossmodal binding of fear in voice and face. Proceedings of the National Academy of Sciences of the United States of America 2001, 98:10006-10010.
13. Driver J. Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading. Nature 1996, 381:66-68.
14. Ekman P., Friesen W.V. Pictures of facial affect 1976, Consulting Psychologists Press, Palo Alto, CA.
15. Ethofer T., Anders S., Erb M., Droll C., Royen L., Saur R., Reiterer S., Grodd W., Wildgruber D. Impact of voice on emotional judgment of faces: An event-related fMRI study. Human Brain Mapping 2006, 27:707-714.
16. Fairhall S.L., Macaluso E. Spatial attention can modulate audiovisual integration at multiple cortical and subcortical sites. The European Journal of Neuroscience 2009, 29:1247-1257.
17. Flom R., Bahrick L.E. The development of infant discrimination of affect in multimodal and unimodal stimulation: The role of intersensory redundancy. Developmental Psychology 2007, 43:238-252.
18. Ghazanfar A.A., Logothetis N.K. Neuroperception: Facial expressions linked to monkey calls. Nature 2003, 423:937-938.
19. Grossmann T., Striano T., Friederici A.D. Crossmodal integration of emotional information from face and voice in the infant brain. Developmental Science 2006, 9:309-315.
20. Hagan C.C., Woods W., Johnson S., Calder A.J., Green G.G., Young A.W. MEG demonstrates a supra-additive response to facial and vocal emotion in the right superior temporal sulcus. Proceedings of the National Academy of Sciences of the United States of America 2009, 106:20010-20015.
21. Hanley J.R., Smith S.T., Hadfield J. I recognise you but I can't place you: An investigation of familiar-only experience during tests of voice and face recognition. The Quarterly Journal of Experimental Psychology 1998, 51:179-195.
22. Hsu J. Multiple comparisons. Theory and Methods 1996, Chapman & Hall.
23. Huynh H., Feldt L.S. Estimation of the box correction for degrees of freedom from sample data in randomized block and split-plot designs. Journal of Educational Statistics 1976, 1:69-82.
24. Joassin F., Maurage P., Bruyer R., Crommelinck M., Campanella S. When audition alters vision: an event related-potential study of the cross-modal interactions between faces and voices. Neuroscience Letters 2004, 369:132-137.
25. Kreifelts B., Ethofer T., Grodd W., Erb M., Wildgruber D. Audiovisual integration of emotional signals in voice and face: An event-related fMRI study. Neuroimage 2007, 37:1445-1456.
26. Kreifelts B., Ethofer T., Huberle E., Grodd W., Wildgruber D. Association of trait emotional intelligence and individual fMRI-activation patterns during the perception of social signals from voice and face. Human Brain Mapping 2010, 31:979-991.
27. Kreifelts B., Ethofer T., Shiozawa T., Grodd W., Wildgruber D. Cerebral representation of non-verbal emotional perception: fMRI reveals audiovisual integration area between voice- and face-sensitive regions in the superior temporal sulcus. Neuropsychologia 2009, 47:3059-3066.
28. Laurienti P.J., Kraft R.A., Maldjian J.A., Burdette J.H., Wallace M.T. Semantic congruence is a critical factor in multisensory behavioral performance. Experimental Brain Research 2004, 158:405-414.
29. LeDoux J. The emotional brain: The mysterious underpinnings of emotional life 1996, Simon & Schuster, New York.
30. Magnée M.J., de Gelder B., van Engeland H., Kemner C. Atypical processing of fearful face-voice pairs in Pervasive Developmental Disorder: An ERP study. Clinical Neurophysiology 2008, 119:2004-2010.
31. Massaro D.W. Perceiving talking faces: From speech perception to a behavioral principle 1998, MIT, Cambridge, MA.
32. Massaro D.W., Egan P.B. Perceiving affect from the voice and the face. Psychonomic Bulletin & Review 1996, 3:215-221.
33. Öhman A., Flykt A., Esteves F. Emotion drives attention: Detecting the snake in the grass. Journal of Experimental Psychology: General 2001, 130:466-478.
34. Park J.Y., Gu B.M., Kang D.H., Shin Y.W., Choi C.H., Lee J.M., Kwon J.S. Integration of cross-modal emotional information in the human brain: An fMRI study. Cortex 2010, 46:161-169.
35. Pourtois G., de Gelder B., Bol A., Crommelinck M. Perception of facial expressions and voices and of their combination in the human brain. Cortex 2005, 41:49-59.
36. Pourtois G., de Gelder B., Vroomen J., Rossion B., Crommelinck M. The time-course of intermodal binding between seeing and hearing affective information. NeuroReport 2000, 11:1329-1333.
37. Pourtois G., Debatisse D., Despland P.A., de Gelder B. Facial expressions modulate the time course of long latency auditory brain potentials. Cognitive Brain Research 2002, 14:99-105.
38. Robins D.L., Hunyadi E., Schultz R.T. Superior temporal activation in response to dynamic audio-visual emotional cues. Brain and Cognition 2009, 69:269-278.
39. Schirmer A., Kotz S.A. ERP evidence for a sex-specific Stroop effect in emotional speech. Journal of Cognitive Neuroscience 2003, 15:1135-1148.
40. Sheffert S.M., Olson E. Audiovisual speech facilitates voice learning. Perception & Psychophysics 2004, 66:352-362.
41. Simon D., Craig K.D., Gosselin F., Belin P., Rainville P. Recognition and discrimination of prototypical dynamic expressions of pain and emotions. Pain 2008, 135:55-64.
42. Spreckelmeyer K.N., Kutas M., Urbach T.P., Altenmüller E., Münte T.F. Combined perception of emotion in pictures and musical sounds. Brain Research 2006, 1070:160-170.
43. Stein B.E., Stanford T.R. Multisensory integration: Current issues from the perspective of the single neuron. Nature Reviews. Neuroscience 2008, 9:255-267.
44. Talsma D., Doty T.J., Woldorff M.G. Selective attention and audiovisual integration: Is attending to both modalities a prerequisite for early integration?. Cerebral Cortex 2007, 17:679-690.
45. Talsma D., Senkowski D., Soto-Faraco S., Woldorff M.G. The multifaceted interplay between attention and multisensory integration. Trends in Cognitive Sciences 2010, 14:400-410.
46. Tanaka A., Koizumi A., Imai H., Hiramatsu S., Hiramoto E., de Gelder B. I feel your voice: Cultural differences in the multisensory perception of emotion. Psychological Science 2010, 21:1259-1262.
47. Thönnessen H., Boers F., Dammers J., Chen Y.H., Norra C., Mathiak K. Early sensory encoding of affective prosody: Neuromagnetic tomography of emotional category changes. Neuroimage 2010, 50:250-259.
48. Townsend J.T., Ashby F.G. Methods of modeling capacity in simple processing systems. Cognitive theory 1978, Vol. 3:199-239. Erlbaum, Hillsdale, NJ.
49. Van den Stock J.B., Grèzes J., de Gelder B. Human and animal sounds influence recognition of body language. Brain Research 2008, 1242:185-190.
50. Van den Stock J., Righart R., de Gelder B. Body expressions influence recognition of emotions in the face and voice. Emotion 2007, 7:487-494.
51. von Kriegstein K., Giraud A.L. Implicit multisensory associations influence voice recognition. PLoS Biology 2006, 4:1809-1820.
52. Vroomen J., Driver J., de Gelder B. Is cross-modal integration of emotional expressions independent of attentional resources?. Cognitive, Affective & Behavioral Neuroscience 2001, 4:382-387.
53. Vuilleumier P., Armony J.L., Driver J., Dolan R.J. Effects of attention and emotion on face processing in the human brain: An event-related fMRI study. Neuron 2001, 30:829-841.