Seeing voices: High-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion

Neuropsychologia

Volumn 45, Issue 3, 2007, Pages 587-597

  Saint Amour, Dave ^a   De Sanctis, Pierfilippo ^a   Molholm, Sophie ^a,b   Ritter, Walter ^a,b   Foxe, John J ^a,b  

^a NATHAN S KLINE INSTITUTE FOR PSYCHIATRIC RESEARCH   (United States)

^b CITY COLLEGE OF NEW YORK   (United States)

Author keywords

Audio visual speech; McGurk illusion; Mismatch negativity; Multisensory integration; Preattentive; Topography

Indexed keywords

ACOUSTICS; ADULT; ARTICLE; AUDITORY CORTEX; AUDITORY STIMULATION; BRAIN MAPPING; BRAIN REGION; CONTROLLED STUDY; ELECTROENCEPHALOGRAM; EVENT RELATED POTENTIAL; FEMALE; GESTURE; HEARING; HEMISPHERIC DOMINANCE; HUMAN; HUMAN EXPERIMENT; ILLUSION; LEFT HEMISPHERE; MALE; MCGURK ILLUSION; MISMATCH NEGATIVITY; PHONETICS; SCALP; SENSE ORGAN; SPEECH ARTICULATION; SPEECH PERCEPTION; SUPERIOR TEMPORAL GYRUS; TEMPORAL CORTEX; TEMPORAL LOBE; VISUAL STIMULATION; WAVEFORM;

ACOUSTIC STIMULATION; ADULT; ANALYSIS OF VARIANCE; AUDITORY PERCEPTION; BRAIN MAPPING; CONTINGENT NEGATIVE VARIATION; ELECTROENCEPHALOGRAPHY; FEMALE; HUMANS; ILLUSIONS; IMAGE PROCESSING, COMPUTER-ASSISTED; MALE; PHOTIC STIMULATION; REACTION TIME; SPECTRUM ANALYSIS; VISION; VOICE;

EID: 33845453618     PISSN: 00283932     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neuropsychologia.2006.03.036     Document Type: Article

References (60)

1. Bernstein L.E., Auer Jr. E.T., Moore J.K., Ponton C.W., Don M., and Singh M. Visual speech perception without primary auditory cortex activation. Neuroreport 13 (2002) 311-315
2. Besle J., Fort A., and Giard M.H. Is the auditory sensory memory sensitive to visual information?. Experimental Brain Research (2005)
3. Callan D.E., Callan A.M., Kroos C., and Vatikiotis-Bateson E. Multimodal contribution to speech perception revealed by independent component analysis: A single-sweep EEG case study. Brain Research Cognitive Brain Research 10 (2001) 349-353
4. Calvert G.A., Brammer M.J., Bullmore E.T., Campbell R., Iversen S.D., and David A.S. Response amplification in sensory-specific cortices during crossmodal binding. Neuroreport 10 (1999) 2619-2623
5. Calvert G.A., Bullmore E.T., Brammer M.J., Campbell R., Williams S.C., McGuire P.K., et al. Activation of auditory cortex during silent lipreading. Science 276 (1997) 593-596
6. Colin C., Radeau M., Soquet A., and Deltenre P. Generalization of the generation of an MMN by illusory McGurk percepts: Voiceless consonants. Clinical Neurophysiology 115 (2004) 1989-2000
7. Colin C., Radeau M., Soquet A., Demolin D., Colin F., and Deltenre P. Mismatch negativity evoked by the McGurk-MacDonald effect: A phonetic representation within short-term memory. Clinical Neurophysiology 113 (2002) 495-506
8. Csèpe V., Karmos G., and Molnar M. Evoked potential correlates of stimulus deviance during wakefulness and sleep in cat-Animal model of mismatch negativity. Electroencephalographic Clinical Neurophysiology 66 (1987) 571-578
9. Dehaene-Lambertz G. Electrophysiological correlates of categorical phoneme perception in adults. Neuroreport 8 (1997) 919-924
10. Falchier A., Clavagnier S., Barone P., and Kennedy H. Anatomical evidence of multimodal integration in primate striate cortex. Journal of Neuroscience 22 (2002) 5749-5759
11. Foxe J.J., Morocz I.A., Murray M.M., Higgins B.A., Javitt D.C., and Schroeder C.E. Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping. Brain Research Cognitive Brain Research 10 (2000) 77-83
12. Foxe J.J., and Schroeder C.E. The case for feedforward multisensory convergence during early cortical processing. Neuroreport 16 (2005) 419-423
13. Foxe J.J., and Simpson G.V. Flow of activation from V1 to frontal cortex in humans. A framework for defining "early" visual processing. Experimental Brain Research 142 (2002) 139-150
14. Foxe J.J., Wylie G.R., Martinez A., Schroeder C.E., Javitt D.C., Guilfoyle D., et al. Auditory-somatosensory multisensory processing in auditory association cortex: An fMRI study. Journal of Neurophysiology 88 (2002) 540-543
15. Giard M.H., and Peronnet F. Auditory-visual integration during multimodal object recognition in humans: A behavioral and electrophysiological study. Journal of Cognitive Neuroscience 11 (1999) 473-490
16. Giard M.H., Perrin F., Pernier J., and Bouchet P. Brain generators implicated in the processing of auditory stimulus deviance: A topographic event-related potential study. Psychophysiology 27 (1990) 627-640
17. Guthrie D., and Buchwald J.S. Significance testing of difference potentials. Psychophysiology 28 (1991) 240-244
18. Hari R., Hämäläinen M., Ilmoniemi R., Kaukoranta E., Reinikainen K., Salminen J., et al. Responses of the primary auditory cortex to pitch changes in a sequence of tone pips: Neuromagnetic recordings in man. Neuroscience Letters 50 (1984) 127-132
19. Heslenfeld D.J. Visual mismatch negativity. In: Polich J. (Ed). Detection of change: Event-related potential and fMRI findings (2003), Kluwer Academic Publishers, Dordrecht 41-60
20. Javitt D.C., Steinschneider M., Schroeder C.E., Vaughan Jr. H.G., and Arezzo J.C. Detection of stimulus deviance within primate primary auditory cortex: Intracortical mechanisms of mismatch negativity (MMN) generation. Brain Research 667 (1994) 192-200
21. Jones J.A., and Callan D.E. Brain activity during audiovisual speech perception: An fMRI study of the McGurk effect. Neuroreport 14 (2003) 1129-1133
22. Liberman A.M., and Mattingly I.G. The motor theory of speech perception revised. Cognition 21 (1985) 1-36
23. MacSweeney M., Amaro E., Calvert G.A., Campbell R., David A.S., McGuire P., et al. Silent speechreading in the absence of scanner noise: An event-related fMRI study. Neuroreport 11 (2000) 1729-1733
24. Marco-Pallares J., Grau C., and Ruffini G. Combined ICA-LORETA analysis of mismatch negativity. Neuroimage 25 (2005) 471-477
25. McGurk H., and MacDonald J. Hearing lips and seeing voices. Nature 264 (1976) 746-748
26. Molholm S., and Foxe J.J. Look 'hear', primary auditory cortex is active during lip-reading. Neuroreport 16 (2005) 123-124
27. Molholm S., Martinez A., Ritter W., Javitt D.C., and Foxe J.J. The neural circuitry of pre-attentive auditory change-detection: An fMRI study of pitch and duration mismatch negativity generators. Cerebral Cortex 15 (2005) 545-551
28. Molholm S., Ritter W., Javitt D.C., and Foxe J.J. Multisensory visual-auditory object recognition in humans: A high-density electrical mapping study. Cerebral Cortex 14 (2004) 452-465
29. Molholm S., Ritter W., Murray M.M., Javitt D.C., Schroeder C.E., and Foxe J.J. Multisensory auditory-visual interactions during early sensory processing in humans: A high-density electrical mapping study. Brain Research Cognitive Brain Research 14 (2002) 115-128
30. Möttönen R., Krause C.M., Tiippana K., and Sams M. Processing of changes in visual speech in the human auditory cortex. Brain Research Cognitive Brain Research 13 (2002) 417-425
31. Murray M.M., Molholm S., Michel C.M., Heslenfeld D.J., Ritter W., Javitt D.C., et al. Grabbing your ear: Rapid auditory-somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. Cerebral Cortex 15 (2005) 963-974
32. Murray M.M., Wylie G.R., Higgins B.A., Javitt D.C., Schroeder C.E., and Foxe J.J. The spatiotemporal dynamics of illusory contour processing: Combined high-density electrical mapping, source analysis, and functional magnetic resonance imaging. Journal of Neuroscience 22 (2002) 5055-5073
33. Musacchia G., Sams M., Nicol T., and Kraus N. Seeing speech affects acoustic information processing in the human brainstem. Experimental Brain Research 168 (2006) 1-10
34. Näätänen R. The perception of speech sounds by the human brain as reflected by the mismatch negativity (MMN) and its magnetic equivalent (MMNm). Psychophysiology 38 (2001) 1-21
35. Näätänen R., and Alho K. Mismatch negativity-A unique measure of sensory processing in audition. International Journal of Neuroscience 80 (1995) 317-337
36. Näätänen R., Lehtokoski A., Lennes M., Cheour M., Huotilainen M., Iivonen A., et al. Language-specific phoneme representations revealed by electric and magnetic brain responses. Nature 385 (1997) 432-434
37. Novak G., Ritter W., and Vaughan Jr. H.G. Mismatch detection and the latency of temporal judgements. Psychophysiology 29 (1992) 398-411
38. Pazo-Alvarez P., Cadaveira F., and Amenedo E. MMN in the visual modality: A review. Biology Psychology 63 (2003) 199-236
39. Pekkola J., Ojanen V., Autti T., Jääskeläinen I.P., Möttönen R., Tarkiainen A., et al. Primary auditory cortex activation by visual speech: An fMRI study at 3T. Neuroreport 16 (2005) 125-128
40. Picton T.W., Alain C., Otten L., Ritter W., and Achim A. Mismatch negativity: Different water in the same river. Audiology Neurootology 5 (2000) 111-139
41. Rinne T., Alho K., Alku P., Holi M., Sinkkonen J., Virtanen J., et al. Analysis of speech sounds is left-hemisphere predominant at 100-150 ms after sound onset. Neuroreport 10 (1999) 1113-1117
42. Ritter W., Deacon D., Gomes H., Javitt D.C., and Vaughan Jr. H.G. The mismatch negativity of event-related potentials as a probe of transient auditory memory: A review. Ear Hearing 16 (1995) 52-67
43. Ritter W., Sussman E., Molholm S., and Foxe J.J. Memory reactivation or reinstatement and the mismatch negativity. Psychophysiology 39 (2002) 158-165
44. Rosburg T., Trautner P., Dietl T., Korzyukov O.A., Boutros N.N., Schaller C., et al. Subdural recordings of the mismatch negativity (MMN) in patients with focal epilepsy. Brain 128 (2005) 819-828
45. Rosenblum L.D., and Saldana H.M. Discrimination tests of visually influenced syllables. Perceptive Psychophysics 52 (1992) 461-473
46. Ross, L., Saint-Amour, D., Leavitt, V., Javitt, D., & Foxe, J. J. (in press). Do you see what I'm saying? Optimal visual enhancement of speech comprehension in noisy environments. Cerebral Cortex.
47. Saint-Amour D., Molholm S., Sehatpour P., Ross L., Mehta A., Schwartz T., et al. Multisensory mismatch negativity processes in temporal cortex: A human intracranial investigation of the McGurk illusion (2005), Cognitive Neuroscience Society, New York
48. Sams M., Aulanko R., Hämäläinen M., Hari R., Lounasmaa O.V., Lu S.T., et al. Seeing speech: Visual information from lip movements modifies activity in the human auditory cortex. Neuroscience Letters 127 (1991) 141-145
49. Sams M., Hämäläinen M., Antervo A., Kaukoranta E., Reinikainen K., and Hari R. Cerebral neuromagnetic responses evoked by short auditory stimuli. Electroencephalographic Clinical Neurophysiology 61 (1985) 254-266
50. Scherg M., and Berg P. Use of prior knowledge in brain electromagnetic source analysis. Brain Topography 4 (1991) 143-150
51. Schroeder C.E., and Foxe J.J. The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. Brain Research Cognitive Brain Research 14 (2002) 187-198
52. Schroeder C.E., and Foxe J.J. Multisensory contributions to low-level 'unisensory' processing. Current Opinion on Neurobiology 15 (2005) 454-458
53. Sharma A., and Dorman M.F. Cortical auditory evoked potential correlates of categorical perception of voice-onset time. The Journal of the Acoustical Society of America 106 (1999) 1078-1083
54. Simson R., Vaughan Jr. H.G., and Ritter W. The scalp topography of potentials in auditory and visual Go/NoGo tasks. Electroencephalographic Clinical Neurophysiology 43 (1977) 864-875
55. Stekelenburg J.J., Vroomen J., and de Gelder B. Illusory sound shifts induced by the ventriloquist illusion evoke the mismatch negativity. Neuroscience Letters 357 (2004) 163-166
56. Sumby W.H., and Pollack L. Visual contribution to speech intelligibility in noise. The Journal of the Acoustical Society of America (1954) 212-215
57. Summerfield Q., and McGrath M. Detection and resolution of audio-visual incompatibility in the perception of vowels. Quarterly Journal of Experimental Psychology Section A 36 (1984) 51-74
58. Sussman E., Ritter W., and Vaughan Jr. H.G. Attention affects the organization of auditory input associated with the mismatch negativity system. Brain Research 789 (1998) 130-138
59. Tervaniemi M., Schröger E., and Näätänen R. Pre-attentive processing of spectrally complex sounds with asynchronous onsets: An event-related potential study with human subjects. Neuroscience Letters 227 (1997) 197-200
60. Winkler I., Lehtokoski A., Alku P., Vainio M., Czigler I., Csèpe V., et al. Pre-attentive detection of vowel contrasts utilizes both phonetic and auditory memory representations. Brain Research Cognitive Brain Research 7 (1999) 357-369