The representation of audiovisual regularities in the human brain

Journal of Cognitive Neuroscience

Volumn 25, Issue 3, 2013, Pages 365-373

  Besle, Julien ^a,b,c   Hussain, Zahra ^c   Giard, Marie Hélène ^a,b   Bertrand, Olivier ^a,b  

^a LYON NEUROSCIENCE RESEARCH CENTER   (France)

^b UNIVERSITÉ LYON 1   (France)

^c UNIVERSITY OF NOTTINGHAM   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ARTICLE; AUDITORY CORTEX; AUDITORY STIMULATION; BRAIN ELECTROPHYSIOLOGY; FEMALE; HEARING; HUMAN; HUMAN EXPERIMENT; MALE; NORMAL HUMAN; PRIORITY JOURNAL; SOUND; VISION; VISUAL STIMULATION;

EID: 84873110150     PISSN: 0898929X     EISSN: 15308898     Source Type: Journal    
DOI: 10.1162/jocn_a_00334     Document Type: Article

References (57)

1. Aoyama, A., Endo, H., Honda, S., & Takeda, T. (2006). Modulation of early auditory processing by visually based sound prediction. Brain Research, 1068, 194-204.
2. Arnal, L. H., Morillon, B., Kell, C. A., & Giraud, A. L. (2009). Dual neural routing of visual facilitation in speech processing. Journal of Neuroscience, 29, 13445-13453.
3. Baldeweg, T. (2007). ERP repetition effects and mismatch negativity generation: A predictive coding perspective. Journal of Psychophysiology, 21, 204-213.
4. Besle, J., Caclin, A., Mayet, R., Bauchet, F., Delpuech, C., Giard, M. H., et al. (2007). Audiovisual events in sensory memory. Journal of Psychophysiology, 21, 231-238.
5. Besle, J., Fischer, C., Bidet-Caulet, A., Lecaignard, F., Bertrand, O., & Giard, M. H. (2008). Visual activation and audiovisual interactions in the auditory cortex during speech perception: Intracranial recordings in humans. The Journal of Neuroscience, 28, 14301-14310.
6. Besle, J., Fort, A., Delpuech, C., & Giard, M. H. (2004). Bimodal speech: Early suppressive visual effects in the human auditory cortex. European Journal of Neuroscience, 20, 2225-2234.
7. Besle, J., Fort, A., & Giard, M. H. (2005). Is the auditory sensory memory sensitive to visual information? Experimental Brain Research, 166, 337-344.
8. Bulkin, D. A., & Groh, J. M. (2006). Seeing sounds: Visual and auditory interactions in the brain. Current Opinion in Neurobiology, 16, 415-419.
9. Colin, C., Radeau, M., Soquet, A., Dachy, B., & Deltenre, P. (2002). Electrophysiology of spatial scene analysis: The mismatch negativity (MMN) is sensitive to the ventriloquism illusion. Clinical Neurophysiology, 113, 507-518.
10. Colin, C., Radeau, M., Soquet, A., & Deltenre, P. (2004). Generalization of the generation of an MMN by illusory McGurk percepts: Voiceless consonants. Clinical Neurophysiology, 115, 1989-2000.
11. Colin, C., Radeau, M., Soquet, A., Demolin, D., Colin, F., & Deltenre, P. (2002). Mismatch negativity evoked by the McGurk-MacDonald effect: A phonetic representation within short-term memory. Clinical Neurophysiology, 113, 495-506.
12. Czigler, I. (2007). Visual mismatch negativity: Violation of nonattended environmental regularities. Journal of Psychophysiology, 21, 224-230.
13. Dittmann-Balcar, A., Thienel, R., & Schall, U. (1999). Attentiondependent allocation of auditory processing resources as measured by mismatch negativity. NeuroReport, 10, 3749-3753.
14. Fort, A., Delpuech, C., Pernier, J., & Giard, M. H. (2002a). Dynamics of cortico-subcortical crossmodal operations involved in audio-visual object detection in humans. Cerebral Cortex, 12, 1031-1039.
15. Fort, A., Delpuech, C., Pernier, J., & Giard, M. H. (2002b). Early auditory-visual interactions in human cortex during nonredundant target identification. Cognitive Brain Research, 14, 20-30.
16. Froyen, D., Van Atteveldt, N., Bonte, M., & Blomert, L. (2008). Cross-modal enhancement of the MMN to speech-sounds indicates early and automatic integration of letters and speech-sounds. Neuroscience Letters, 430, 23-28.
17. Ghazanfar, A. A., & Schroeder, C. E. (2006). Is neocortex essentially multisensory? Trends in Cognitive Sciences, 10, 278-285.
18. Giard, M. H., & Peronnet, F. (1999). Auditory-visual integration during multimodal object recognition in humans: A behavioral and electrophysiological study. Journal of Cognitive Neuroscience, 11, 473-490.
19. Giard, M. H., Perrin, F., Pernier, J., & Bouchet, P. (1990). Brain generators implicated in the processing of auditory stimulus deviance: A topographic event-related potential study. Psychophysiology, 27, 627-640.
20. Gomes, H., Bernstein, R., Ritter, W., Vaughan, H. G., & Miller, J. (1997). Storage of feature conjunctions in transient auditory memory. Psychophysiology, 34, 712-716.
21. Hertrich, I., Mathiak, K., Lutzenberger, W., Menning, H., & Ackermann, H. (2007). Sequential audiovisual interactions during speech perception: A whole-head MEG study. Neuropsychologia, 45, 1342-1354.
22. Jacobsen, T., & Schröger, E. (2001). Is there pre-attentive memorybased comparison of pitch? Psychophysiology, 38, 723-727.
23. Jacobsen, T., & Schröger, E. (2003). Measuring duration mismatch negativity. Clinical Neurophysiology, 114, 1133-1143.
24. Korzyukov, O. A., Winkler, I., Gumenyuk, V. I., & Alho, K. (2003). Processing abstract auditory features in the human auditory cortex. Neuroimage, 20, 2245-2258.
25. Kropotov, J. D., Näätänen, R., Sevostianov, A. V., Alho, K., Reinikainen, K., & Kropotova, O. V. (1995). Mismatch negativity to auditory stimulus change recorded directly from the human temporal cortex. Psychophysiology, 32, 418-422.
26. Maxwell, S. E., & Delaney, H. D. (2004). Designing experiments and analyzing data: A model comparison perspective. Mahwah, NJ: Lawrence Erlbaum Associates.
27. May, P. J., & Tiitinen, H. (2009). Mismatch negativity (MMN), the deviance-elicited auditory deflection, explained. Psychophysiology, 47, 66-122.
28. McGurk, H., & McDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746-748.
29. Mittag, M., Takegata, R., & Kujala, T. (2011). The effects of visual material and temporal synchrony on the processing of letters and speech sounds. Experimental Brain Research, 211, 287-298.
30. Möttönen, R., Krause, C. M., Tiippana, K., & Sams, M. (2002). Processing of changes in visual speech in the human auditory cortex. Cognitive Brain Research, 13, 417-425.
31. Muller-Gass, A., Stelmack, R. M., & Campbell, K. B. (2006). The effect of visual task difficulty and attentional direction on the detection of acoustic change as indexed by the mismatch negativity. Brain Research, 1078, 112-130.
32. Näätänen, R. (1992). Attention and brain function. Hillsdale, MI: LEA, Inc.
33. Näätänen, R., Paavilainen, P., Rinne, T., & Alho, K. (2007). The mismatch negativity (MMN) in basic research of central auditory processing: A review. Clinical Neurophysiology, 118, 2544-2590.
34. Näätänen, R., Tervaniemi, M., Sussman, E., Paavilainen, P., & Winkler, I. (2001). "Primitive intelligence" in the auditory cortex. Trends in Neurosciences, 24, 283-288.
35. Otten, L. J., Alain, C., & Picton, T. W. (2000). Effects of visual attentional load on auditory processing. NeuroReport, 11, 875-880.
36. Paavilainen, P., Arajarvi, P., & Takegata, R. (2007). Preattentive detection of nonsalient contingencies between auditory features. NeuroReport, 18, 159-163.
37. Perrin, F., Pernier, J., Bertrand, O., & Echallier, J. F. (1989). Spherical splines for scalp potential and current density mapping. Electroencephalography and Clinical Neurophysiology, 72, 184-187.
38. Perrin, F., Pernier, J., Bertrand, O., & Giard, M. H. (1987). Mapping of scalp potentials by surface spline interpolation. Electroencephalography and Clinical Neurophysiology, 66, 75-81.
39. Saarinen, J., Paavilainen, P., Schoger, E., Tervaniemi, M., & Näätänen, R. (1992). Representation of abstract attributes of auditory stimuli in the human brain. NeuroReport, 3, 1149-1151.
40. Saint-Amour, D., De Sanctis, P., Molholm, S., Ritter, W., & Foxe, J. J. (2007). Seeing voices: High-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion. Neuropsychologia, 45, 587-597.
41. Sams, M., Aulanko, R., Hamalainen, H., Hari, R., Lounasmaa, O. V., Lu, S. T., et al. (1991). Seeing speech: Visual information from lip movements modifies activity in the human auditory cortex. Neuroscience Letters, 127, 141-145.
42. Schroeder, C. E., & Foxe, J. (2005). Multisensory contributions to low-level, "unisensory" processing. Current Opinion in Neurobiology, 15, 454-458.
43. Schröger, E., & Wolff, C. (1996). Mismatch response of the human brain to changes in sound location. NeuroReport, 7, 3005-3008.
44. Stekelenburg, J. J., Vroomen, J., & de Gelder, B. (2004). Illusory sound shifts induced by the ventriloquist illusion evoke the mismatch negativity. Neuroscience Letters, 357, 163-166.
45. Sussman, E. (2007). A new view on the MMN and attention debate. Journal of Psychophysiology, 21, 164-175.
46. Sussman, E., Winkler, I., Huotilainen, M., Ritter, W., & Näätänen, R. (2002). Top-down effects can modify the initially stimulus-driven auditory organization. Cognitive Brain Research, 13, 393-405.
47. Takegata, R., Brattico, E., Tervaniemi, M., Varyagina, O., Näätänen, R., & Winkler, I. (2005). Preattentive representation of feature conjunctions for concurrent spatially distributed auditory objects. Brain Research, Cognitive Brain Research, 25, 169-179.
48. Thomas, G. J. (1941). Experimental study of the influence of vision on sound localization. Journal of Experimental Psychology, 28, 163-177.
49. Ullsperger, P., Erdmann, U., Freude, G., & Dehoff, W. (2006). When sound and picture do not fit: Mismatch negativity and sensory interaction. International Journal of Psychophysiology, 59, 3-7.
50. Valtonen, J., May, P., Makinen, V., & Tiitinen, H. (2003). Visual short-term memory load affects sensory processing of irrelevant sounds in human auditory cortex. Brain Research, Cognitive Brain Research, 17, 358-367.
51. van Atteveldt, N. M., Blau, V. C., Blomert, L., & Goebel, R. (2010). fMR-adaptation indicates selectivity to audiovisual content congruency in distributed clusters in human superior temporal cortex. BMC Neuroscience, 11, 11.
52. Wacongne, C., Changeux, J. P., & Dehaene, S. (2012). A neuronal model of predictive coding accounting for the mismatch negativity. Journal of Neuroscience, 32, 3665-3678.
53. Widmann, A., Kujala, T., Tervaniemi, M., Kujala, A., & Schröger, E. (2004). From symbols to sounds: Visual symbolic information activates sound representations. Psychophysiology, 41, 709-715.
54. Winkler, I. (2007). Interpreting the mismatch negativity. Journal of Psychophysiology, 21, 147-163.
55. Winkler, I., Horvath, J., Weisz, J., & Trejo, L. J. (2009). Deviance detection in congruent audiovisual speech: Evidence for implicit integrated audiovisual memory representations. Biological Psychology, 82, 281-292.
56. Winkler, I., Paavilainen, P., & Näätänen, R. (1992). Can echoic memory store two traces simultaneously? A study of event-related brain potentials. Psychophysiology, 29, 337-349.
57. Yumoto, M., Uno, A., Itoh, K., Karino, S., Saitoh, O., Kaneko, Y., et al. (2005). Audiovisual phonological mismatch produces early negativity in auditory cortex. NeuroReport, 16, 803-806.