On the role of crossmodal prediction in audiovisual emotion perception

Frontiers in Human Neuroscience

Volumn , Issue JUN, 2013, Pages

  Jessen, Sarah ^a   Kotz, Sonja A ^a,b  

^a MAX PLANCK INSTITUTE FOR HUMAN COGNITIVE AND BRAIN SCIENCES   (Germany)

^b UNIVERSITY OF MANCHESTER   (United Kingdom)

Author keywords

Audiovisual; Crossmodal Prediction; EEG; Emotion; Multisensory

Indexed keywords

ARTICLE; AUDITORY INFORMATION; AUDITORY STIMULATION; BRAIN CORTEX; BRAIN FUNCTION; CROSSMODAL PREDICTION; ELECTROENCEPHALOGRAPHY; EMOTION; FACIAL EXPRESSION; HUMAN; INFORMATION; MAGNETOENCEPHALOGRAPHY; MULTISENSORY EMOTION PERCEPTION; NEUROMODULATION; PERCEPTION; PREDICTION; SOMATOSENSORY STIMULATION; SPEECH; SPEECH PERCEPTION; SUPERIOR TEMPORAL SULCUS; VISUAL INFORMATION; VISUAL STIMULATION;

EID: 84933678589     PISSN: 16625161     EISSN: 16625161     Source Type: Journal    
DOI: 10.3389/fnhum.2013.00369     Document Type: Article

References (44)

1. Arnal, L.H., Morillon, B., Kell, C.A., and Giraud, A.-L. (2009). Dual neural routing of visual facilitation in speech processing. J Neurosci 29, 13445-13453.
2. Arnal, L.H., Wyart, V., and Giraud, A.L. (2011). Transitions in neural oscillations reflect prediction errors generated in audiovisual speech. Nat Neurosci 14, 797-801.
3. Bernstein, L.E., Auer, E.T., Jr., Wagner, M., and Ponton, C.W. (2008). Spatiotemporal dynamics of audiovisual speech processing. Neuroimage 39, 423-435.
4. Besle, J., Bertrand, O., and Giard, M.H. (2009). Electrophysiological (EEG, sEEG, MEG) evidence for multiple audiovisual interactions in the human auditory cortex. Hear Res 258, 143-151.
5. Besle, J., Fort, A., Delpuech, C., and Giard, M. (2004). Bimodal speech: Early suppressive visual effects in human auditory cortex. Eur J Neurosci 20, 2225-2234.
6. Buzsaki, G., and Draguhn, A. (2004). Neuronal oscillations in cortical networks. Science 304, 1926-1929.
7. Callan, D.E., Jones, J.A., Munhall, K., Callan, A.M., Kroos, C., and Vatikiotis-Bateson, E. (2003). Neural processes underlying perceptual enhancement by visual speech gestures. Neuroreport 14, 2213-2218.
8. Calvert, G.A., Brammer, M.J., Bullmore, E.T., Campbell, R., Iversen, S.D., and David, A.S. (1999). Response amplification in sensory-specific cortices during crossmodal binding. Neuroreport 10, 2619-2623.
9. Calvert, G.A., Brammer, M.J., and Iversen, S.D. (1998). Crossmodal identification. Trends Cogn Sci 2, 247-253.
10. Calvert, G.A., Campbell, R., and Brammer, M.J. (2000). Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10, 649-657.
11. Carroll, N.C., and Young, A.W. (2005). Priming of emotion recognition. Q J Exp Psychol A 58, 1173-1197.
12. Chandrasekaran, C., Trubanova, A., Stillittano, S., Caplier, A., and Ghazanfar, A.A. (2009). The natural statistics of audiovisual speech. PLoS Comput Biol 5, e1000436.
13. Chen, Y.H., Edgar, J.C., Holroyd, T., Dammers, J., Thonnessen, H., Roberts, T.P., and Mathiak, K. (2010). Neuromagnetic oscillations to emotional faces and prosody. Eur J Neurosci 31, 1818-1827.
14. Davis, C., Kislyuk, D., Kim, J., and Sams, M. (2008). The effect of viewing speech on auditory speech processing is different in the left and right hemispheres. Brain Res 1242, 151-161.
15. De Gelder, B., Böcker, K.B.E., Tuomainen, J., Hensen, M., and Vroomen, J. (1999). The combined perception of emotion from voice and face: Early interaction revealed by human electric brain responses. Neurosci Lett 260, 133-136.
16. Doesburg, S.M., Emberson, L.L., Rahi, A., Cameron, D., and Ward, L.M. (2008). Asynchrony from synchrony: long-range gamma-band neural synchrony accompanies perception of audiovisual speech asynchrony. Exp Brain Res 185, 11-20.
17. Ghazanfar, A.A., and Schroeder, C.E. (2006). Is neocortex essentially multisensory? Trends Cogn Sci 10, 278-285.
18. Giard, M.H., and Peronnet, F. (1999). Auditory-visual integration during multimodal object recognition in humans: A behavioral and electrophysiological study. Journal of Cognitive Neuroscience 11(5), 473-490.
19. Jessen, S., and Kotz, S.A. (2011). The temporal dynamics of processing emotions from vocal, facial, and bodily expressions. Neuroimage 58, 665-674.
20. Jessen, S., Obleser, J., and Kotz, S.A. (2012). How bodies and voices interact in early emotion perception. PLoS One 7, e36070.
21. Kayser, C., Petkov, C.I., and Logothetis, N.K. (2008). Visual modulation of neurons ons in auditory cortex. Cereb Cortex 18, 1560-1574.
22. Klasen, M., Chen, Y.H., and Mathiak, K. (2012). Multisensory emotions: perception, combination and underlying neural processes. Rev Neurosci 23, 381-392.
23. Klucharev, V., Mottonen, R., and Sams, M. (2003). Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Brain Res Cogn Brain Res 18, 65-75.
24. Kreifelts, B., Ethofer, T., Grodd, W., Erb, M., and Wildgruber, D. (2007). Audiovisual integration of emotional signals in voice and face: An event-related fMRI study. Neuroimage 37, 1445-1456.
25. Lakatos, P., O'connell, M.N., Barczak, A., Mills, A., Javitt, D.C., and Schroeder, C.E. (2009). The leading sense: supramodal control of neurophysiological context by attention. Neuron 64, 419-430.
26. Laurienti, P.J., Wallace, M.T., Maldjian, J.A., Susi, C.M., Stein, B., and Burdette, J.H. (2003). Cross-modal sensory processing in the anterior cingulate and medial prefrontal cortices. Hum Brain Mapp 19, 213-223.
27. Meeren, H.K.M., Van Heijnsbergen, C.C.R.J., and De Gelder, B. (2005). Rapid perceptual integration of facial expression and emotional body language. Proc Natl Acad Sci U A 102, 16518-16523.
28. Paulmann, S., Jessen, S., and Kotz, S.A. (2009). Investigating the Multimodal Nature of Human Communication. Insights from ERPs. Journal of Psychophysiology 23(2), 63-76.
29. Pichon, S., De Gelder, B., and Grèzes, J. (2009). Two different faces of threat. Comparing the neural systems for recognizing fear and anger in dynamic body expressions. Neuroimage 47, 1873-1883.
30. Pourtois, G., De Gelder, B., Vroomen, J., Rossion, B., and Crommelinck, M. (2000). The time-course of intermodal binding between seeing and hearing affective information. Neuroreport 11, 1329-1333.
31. Pourtois, G., Debatisse, D., Despland, P.-A., and De Gelder, B. (2002). Facial expressions modulate the time course of long latency auditory brain potentials. Brain Res Cogn Brain Res 14, 99-105.
32. Robins, D.L., Hunyadi, E., and Schultz, R.T. (2009). Superior temporal activation in response to dynamic audio-visual emotional cues. Brain Cogn 69, 269-278.
33. Schirmer, A., Kotz, S.A., and Friederici, A.D. (2002). Sex differentiates the role of emotional prosody during word processing. Cognitive Brain Research 14, 228-233.
34. Schroeder, C.E., Lakatos, P., Kajikawa, Y., Partan, S., and Puce, A. (2008). Neuronal oscillations and visual amplification of speech. Trends Cogn Sci 12, 106-113.
35. Senkowski, D., Schneider, T.R., Foxe, J.J., and Engel, A.K. (2008). Crossmodal binding through neural coherence: Implications for multisensory processing. Trends Neurosci 31, 401-409.
36. Sperdin, H.F., Cappe, C., Foxe, J.J., and Murray, M.M. (2009). Early, low-level auditory42 somatosensory multisensory interactions impact reaction time speed. Front Integr Neurosci 3, 2.
37. Stekelenburg, J.J., and De Gelder, B. (2004). The neural correlates of perceiving human bodies: An ERP study on the body-inversion effect. Neuroreport 15, 777-780.
38. Stekelenburg, J.J., and Vroomen, J. (2007). Neural correlates of multisensory integration of ecologically valid audiovisual events. J Cogn Neurosci 19, 1964-1973.
39. Stekelenburg, J.J., and Vroomen, J. (2012). Electrophysiological correlates of predictive coding of auditory location in the perception of natural audiovisual events. Front Integr Neurosci 6, 26.
40. Summerfield, C., and Egner, T. (2009). Expectation (and attention) in visual cognit1 ion. Trends Cogn Sci 13, 403-409.
41. Tallon-Baudry, C., and Bertrand, O. (1999). Oscillatory gamma activity in humans and its role in object representation. Trends Cogn Sci 3(4), 151-162.
42. Thorne, J.D., De Vos, M., Viola, F.C., and Debener, S. (2011). Cross-modal phase reset predicts auditory task performance in humans. J Neurosci 31, 3853-3861.
43. Van Wassenhove, V., Grant, K.W., and Poeppel, D. (2005). Visual speech speeds up the neural processing of auditory speech. Proc Natl Acad Sci U S A 102, 1181-1186.
44. Werheid, K., Alpay, G., Jentzsch, I., and Sommer, W. (2005). Priming emotional facial expressions as evidenced by event-related brain potentials. International Journal of Psychophysiology 55, 209-219.