The visual mismatch negativity elicited with visual speech stimuli

Frontiers in Human Neuroscience

Volumn , Issue JUN, 2013, Pages

  Files, Benjamin T ^a   Auer, Edward T ^b   Bernstein, Lynne E ^a,b  

^a UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^b GEORGE WASHINGTON UNIVERSITY   (United States)

Author keywords

Hemispheric laterazation for speech; Lipreading; Mismatch negativity (MMN); Scalp electrophysiology; Speech perception; Visual perception

Indexed keywords

ADULT; ARTICLE; AUDITORY CORTEX; BRAIN ASYMMETRY; BRAIN DEPTH STIMULATION; BRAIN FUNCTION; CONSONANT; CONTROLLED STUDY; DIPOLE; ELECTROENCEPHALOGRAPHY; EVENT RELATED POTENTIAL; FEMALE; GLOBAL FIELD POWER; HUMAN; MALE; MISMATCH NEGATIVITY; PHONEME; PHONETICS; POSTERIOR TEMPORAL CORTEX; RIGHT HANDEDNESS; SPEECH; TEMPORAL CORTEX; VISION; VISUAL DISCRIMINATION; VISUAL MISMATCH NEGATIVITY; VISUAL SPEECH STIMULATION; VISUAL STIMULATION; VOWEL;

EID: 85172042478     PISSN: None     EISSN: 16625161     Source Type: Journal    
DOI: 10.3389/fnhum.2013.00371     Document Type: Article

References (106)

1. Ahissar, M., Nahum, M., Nelken, I., and Hochstein, S. (2008). Reverse hierarchies and sensory learning. Philosophical Transactions of the Royal Society B 364, 285-299.
2. Auer, E.T., Jr., and Bernstein, L.E. (1997). Speechreading and the structure of the lexicon: computationally modeling the effects of reduced phonetic distinctiveness on lexical uniqueness. Journal of The Acoustical Society Of America 102, 3704-3710.
3. Auer, E.T., Jr., and Bernstein, L.E. (2007). Enhanced visual speech perception in individuals with early-onset hearing impairment. Journal of Speech, Language, and Hearing Research 50, 1157-1165.
4. Baas, J.M., Kenemans, J.L., and Mangun, G.R. (2002). Selective attention to spatial frequency: an ERP and source localization analysis. Clinical neurophysiology: official journal of the International Federation of Clinical Neurophysiology 113, 1840-1854.
5. Baillet, S., Mosher, J.C., and Leahy, R.M. (2001). Electromagnetic brain mapping. Signal Processing Magazine, IEEE 18, 14-30.
6. Bernstein, L.E. (2012). "Visual speech perception, " in AudioVisual Speech Processing, eds. E. Vatikiotis-Bateson, G. Bailly & P. Perrier. (Cambridge: Cambridge University), 21-39.
7. Bernstein, L.E., Auer, E.T., Jr., Moore, J.K., Ponton, C.W., Don, M., and Singh, M. (2002). Visual speech perception without primary auditory cortex activation. Neuroreport 13, 311-315.
8. Bernstein, L.E., Auer, E.T., Jr., Wagner, M., and Ponton, C.W. (2008). Spatiotemporal dynamics of audiovisual speech processing. NeuroImage 39, 423-435.
9. Bernstein, L.E., Demorest, M.E., and Tucker, P.E. (2000). Speech perception without hearing. Perception & Psychophysics 62, 233-252.
10. Bernstein, L.E., Jiang, J., Pantazis, D., Lu, Z.L., and Joshi, A. (2011). Visual phonetic processing localized using speech and nonspeech face gestures in video and point-light displays. Human Brain Mapping 32, 1660-1676.
11. Besle, J., Fort, A., Delpuech, C., and Giard, M.-H. (2004). Bimodal speech: early suppressive visual effects in human auditory cortex. European Journal of Neuroscience 20, 2225-2234.
12. Binder, J.R., Frost, J.A., Hammeke, T.A., Bellgowan, P.S., Springer, J.A., Kaufman, J.N., and Possing, E.T. (2000). Human temporal lobe activation by speech and nonspeech sounds. Cerebral Cortex 10, 512-528.
13. Blair, R.C., and Karniski, W. (1993). An alternative method for significance testing of waveform difference potentials. Psychophysiology 30, 518-524.
14. Calvert, G.A. (2001). Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cerebral Cortex 11, 1110-1123.
15. Calvert, G.A., Bullmore, E.T., Brammer, M.J., Campbell, R., Williams, S.C., Mcguire, P.K., Woodruff, P.W., Iversen, S.D., and David, A.S. (1997). Activation of auditory cortex during silent lipreading. Science 276, 593-596.
16. Calvert, G.A., and Campbell, R. (2003). Reading speech from still and moving faces: the neural substrates of visible speech. Journal of Cognitive Neuroscience 15, 57-70.
17. Campbell, R. (1986). The lateralization of lip-read sounds: a first look. Brain and Cognition 5, 1-21.
18. Campbell, R. (2011). Speechreading and the Bruce-Young model of face recognition: early findings and recent developments. British Journal of Psychology 102, 704-710.
19. Campbell, R., Landis, T., and Regard, M. (1986). Face recognition and lipreading. A neurological dissociation. Brain 109 (Pt 3), 509-521.
20. Campbell, R., Macsweeney, M., Surguladze, S., Calvert, G., Mcguire, P., Suckling, J., Brammer, M.J., and David, A.S. (2001). Cortical substrates for the perception of face actions: an fMRI study of the specificity of activation for seen speech and for meaningless lowerface acts (gurning). Cognitive Brain Research 12, 233-243.
21. Capek, C.M., Macsweeney, M., Woll, B., Waters, D., Mcguire, P.K., David, A.S., Brammer, M.J., and Campbell, R. (2008). Cortical circuits for silent speechreading in deaf and hearing people. Neuropsychologia 46, 1233-1241.
22. Colin, C., Radeau, M., Soquet, A., and Deltenre, P. (2004). Generalization of the generation of an MMN by illusory McGurk percepts: voiceless consonants. Clinical Neurophysiology 115, 1989-2000.
23. Colin, C., Radeau, M., Soquet, A., Demolin, D., Colin, F., and Deltenre, P. (2002). Mismatch negativity evoked by the McGurk-MacDonald effect: a phonetic representation within short-term memory. Clinical Neurophysiology 113, 495-506.
24. Czigler, I. (2007). Visual mismatch negativity: Violation of nonattended environmental regularities. Journal of Psychophysiology 21, 224-230.
25. Czigler, I., Balazs, L., and Pato, L.G. (2004). Visual change detection: event-related potentials are dependent on stimulus location in humans. Neuroscience Letters 364, 149-153.
26. Czigler, I., Balazs, L., and Winkler, I. (2002). Memory-based detection of task-irrelevant visual changes. Psychophysiology 39, 869-873.
27. Dahaene-Lambertz, G. (1997). Electrophysiological correlates of categorical phoneme perception in adults. Neuroreport 8, 919-924.
28. Darvas, F., Ermer, J.J., Mosher, J.C., and Leahy, R.M. (2006). Generic head models for atlasbased EEG source analysis. Human Brain Mapping 27, 129-143.
29. Delorme, A., and Makeig, S. (2004). EEGLAB: an open source toolbox for analysis of singletrial EEG dynamics including independent component analysis. Journal of Neuroscience Methods 134, 9-21.
30. Edgington, E.S., and Onghena, P. (2007). Randomization tests. Boca Raton, FL: Chapman & Hall/CRC.
31. Eggermont, J.J. (2001). Between sound and perception: reviewing the search for a neural code. Hearing Research 157, 1-42.
32. Ethofer, T., Gschwind, M., and Vuilleumier, P. (2011). Processing social aspects of human gaze: a combined fMRI-DTI study. NeuroImage 55, 411-419.
33. Folstein, J.R., and Petten, C.V. (2008). Influence of cognitive control and mismatch on the N2 component of the ERP: a review. Psychophysiology 45, 152-170.
34. Fuchs, M., Wagner, M., Köhler, T., and Wischmann, H.A. (1999). Linear and nonlinear current density reconstructions. Journal of Clinical Neurophysiology 16, 267-295.
35. Goldstone, R.L. (1994). Influences of categorization on perceptual discrimination. Journal of Experimental Psychology: Human Perceptual Performance 123, 178-200.
36. Gramfort, A., Papadopoulo, T., Olivi, E., and Clerc, M. (2010). OpenMEEG: opensource software for quasistatic bioelectromagnetics. BioMedical Engineering OnLine 9, 45.
37. Green, D.M., and Swets, J.A. (1966). Signal Detection Theory and Psychophysics. New York: Wiley.
38. Grill-Spector, K., Henson, R., and Martin, A. (2006). Repetition and the brain: neural models of stimulus-specific effects. Trends in Cognitive Sciences 10, 14-23.
39. Grill-Spector, K., Kourtzi, Z., and Kanwisher, N. (2001). The lateral occipital complex and its role in object recognition. Vision Research 41, 1409-1422.
40. Groppe, D.M., Urbach, T.P., and Kutas, M. (2011). Mass univariate analysis of event-related brain potentials/fields I: a critical tutorial review. Psychophysiology 48, 1711-1725.
41. Guthrie, D., and Buchwald, J.S. (1991). Significance testing of difference potentials. Psychophysiology 28, 240-244.
42. Hickok, G., and Poeppel, D. (2007). The cortical organization of speech processing. Nature Reviews: Neuroscience 8, 393-402.
43. Jaaskelainen, I.P., Ahveninen, J., Bonmassar, G., Dale, A.M., Ilmoniemi, R.J., Levanen, S., Lin, F.-H., May, P., Melcher, J., Stufflebeam, S., Tiitinen, H., and Belliveau, J.W. (2004). Human posterior auditory cortex gates novel sounds to consciousness. Proceedings of the National Academy of Sciences, USA, 101, 6809-6814.
44. Jesse, A., and Massaro, D.W. (2010). The temporal distribution of information in audiovisual spoken-word identification. Attention, Perception, & Psychophysics 72, 209-225.
45. Jiang, J., Alwan, A., Keating, P., Auer, E.T., Jr., and Bernstein, L.E. (2002). On the relationship between face movements, tongue movements, and speech acoustics. EURASIP Journal on Applied Signal Processing: Special issue on Joint Audio-Visual Speech Processing 2002, 1174-1188.
46. Jiang, J., Auer, E.T., Jr., Alwan, A., Keating, P.A., and Bernstein, L.E. (2007a). Similarity structure in visual speech perception and optical phonetic signals. Perception & Psychophysics 69, 1070-1083.
47. Jiang, X., Bradley, E., Rini, R.A., Zeffiro, T., Vanmeter, J., and Riesenhuber, M. (2007b). Categorization training results in shape-and category-selective human neural plasticity. Neuron 53, 891-903.
48. Kanwisher, N., Mcdermott, J., and Chun, M.M. (1997). The fusiform face area: a module in human extrastriate cortex specialized for face perception. Journal of Neuroscience 17, 4302-4311.
49. Kecskes-Kovacs, K., Sulykos, I., and Czigler, I. (2013). Visual mismatch negativity is sensitive to symmetry as a perceptual category. The European journal of neuroscience 37, 662-667.
50. Kimura, M., Katayama, J., Ohira, H., and Schroger, E. (2009). Visual mismatch negativity: New evidence from the equiprobable paradigm. Psychophysiology 46, 402-409.
51. Kimura, M., Schroger, E., and Czigler, I. (2011). Visual mismatch negativity and its importance in visual cognitive sciences. Neuroreport 22, 669-673.
52. Kriegeskorte, N., Simmons, W.K., Bellgowan, P.S., and Baker, C.I. (2009). Circular analysis in systems neuroscience: the dangers of double dipping. Nat Neurosci 12, 535-540.
53. Kujala, T., Tervaniemi, M., and Schroger, E. (2007). The mismatch negativity in cognitive and clinical neuroscience: theoretical and methodological considerations. Biological psychology 74, 1-19.
54. Lehmann, D., and Skrandies, W. (1980). Reference-free identification of components of checkerboard-evoked multichannel potential fields. Electroencephalography and Clinical Neurophysiology 48, 609-621.
55. Leitman, D.I., Sehatpour, P., Shpaner, M., Foxe, J.J., and Javitt, D.C. (2009). Mismatch negativity to tonal contours suggests preattentive perception of prosodic content. Brain imaging and behavior 3, 284-291.
56. Li, X., Lu, Y., Sun, G., Gao, L., and Zhao, L. (2012). Visual mismatch negativity elicited by facial expressions: new evidence from the equiprobable paradigm. Behavioral and brain functions: BBF 8, 7.
57. Macmillan, N.A., and Creelman, C.D. (1991). Detection theory: a user's guide. Cambridge England; New York: Cambridge University Press.
58. May, P.J.C., and Tiitinen, H. (2010). Mismatch negativity (MMN), the deviance-elicited auditory deflection, explained. Psychophysiology 47, 66-122.
59. Michel, C.M., Murray, M.M., Lantz, G., Gonzalez, S., Spinelli, L., and De Peralta, R.G. (2004). EEG source imaging. Clinical Neurophysiology 115, 2195-2222.
60. Miki, K., Watanabe, S., Kakigi, R., and Puce, A. (2004). Magnetoencephalographic study of occipitotemporal activity elicited by viewing mouth movements. Clinical neurophysiology: official journal of the International Federation of Clinical Neurophysiology 115, 1559-1574.
61. Möttönen, R., Krause, C.M., Tiippana, K., and Sams, M. (2002). Processing of changes in visual speech in the human auditory cortex. Cognitive Brain Research 13, 417-425.
62. Näätänen, R., Gaillard, A.W.K., and Mäntysalo, S. (1978). Early selective-attention effect on evoked potential reinterpreted. Acta Psychologica 42, 313-329.
63. Näätänen, R., Jacobsen, T., and Winkler, I. (2005). Memory-based or afferent processes in mismatch negativity (MMN): a review of the evidence. Psychophysiology 42, 25-32.
64. Näätänen, R., Kujala, T., and Winkler, I. (2011). Auditory processing that leads to conscious perception: A unique window to central auditory processing opened by the mismatch negativity and related responses. Psychophysiology 48, 4-22.
65. Näätänen, R., Paavilainen, P., Rinne, T., and Alho, K. (2007). The mismatch negativity (MMN) in basic research of central auditory processing: a review. Clin Neurophysiol 118, 2544-2590.
66. Nath, A.R., and Beauchamp, M.S. (2011). Dynamic changes in superior temporal sulcus connectivity during perception of noisy audiovisual speech. Journal of Neuroscience 31, 1704-1714.
67. Nath, A.R., and Beauchamp, M.S. (2012). A neural basis for interindividual differences in the McGurk effect, a multisensory speech illusion. NeuroImage 59, 781-787.
68. Nichols, T.E., and Holmes, A.P. (2002). Nonparametric permutation tests for functional neuroimaging: a primer with examples. Human Brain Mapping 15, 1-25.
69. Nousak, J.M., Deacon, D., Ritter, W., and Vaughan, H.G., Jr. (1996). Storage of information in transient auditory memory. Brain research. Cognitive brain research 4, 305-317.
70. Obleser, J., and Eisner, F. (2009). Pre-lexical abstraction of speech in the auditory cortex. Trends in Cognitive Sciences 31, 14-19.
71. Oldfield, R.C. (1971). The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia 9, 97-113.
72. Paulesu, E., Perani, D., Blasi, V., Silani, G., Borghese, N.A., De Giovanni, U., Sensolo, S., and Fazio, F. (2003). A functional-anatomical model for lipreading. Journal of Neurophysiology 90, 2005-2013.
73. Pazo-Alvarez, P., Amenedo, E., and Cadaveira, F. (2004). Automatic detection of motion direction changes in the human brain. The European journal of neuroscience 19, 1978-1986.
74. Pazo-Alvarez, P., Cadaveira, F., and Amenedo, E. (2003). MMN in the visual modality: a review. Biological Psychology 63, 199-236.
75. Pekkola, J., Ojanen, V., Autti, T., Jaaskelainen, I.P., Mottonen, R., and Sams, M. (2006). Attention to visual speech gestures enhances hemodynamic activity in the left planum temporale. Human Brain Mapping 27, 471-477.
76. Pekkola, J., Ojanen, V., Autti, T., Jaaskelainen, I.P., Mottonen, R., Tarkiainen, A., and Sams, M. (2005). Primary auditory cortex activation by visual speech: an fMRI study at 3 T. Neuroreport 16, 125-128.
77. Picton, T.W., Bentin, S., Berg, P., Donchin, E., Hillyard, S.A., Johnson, R., Miller, G.A., Ritter, W., Ruchkin, D.S., Rugg, M.D., and Taylor, M.J. (2000). Guidelines for using human event-related potentials to study cognition: recording standards and publication criteria. Psychophysiology 37, 127-152.
78. Ponton, C.W., Bernstein, L.E., and Auer, E.T., Jr. (2009). Mismatch negativity with visual-only and audiovisual speech. Brain Topography 21, 207-215.
79. Ponton, C.W., Don, M., Eggermont, J.J., and Kwong, B. (1997). Integrated mismatch negativity (MMNi): a noise-free representation of evoked responses allowing single-point distribution-free statistical tests. Electroencephalography and Clinical Neurophysiology 104, 143-150.
80. Proverbio, A.M., Del Zotto, M., Crotti, N., and Zani, A. (2009). A no-go related prefrontal negativity larger to irrelevant stimuli that are difficult to suppress. Behavioral and brain functions: BBF 5, 25.
81. Puce, A., Allison, T., Asgari, M., Gore, J.C., and Mccarthy, G. (1996). Differential sensitivity of human visual cortex to faces, letterstrings, and textures: a functional magnetic resonance imaging study. The Journal of Neuroscience 16, 5205-5215.
82. Puce, A., Allison, T., Bentin, S., Gore, J.C., and Mccarthy, G. (1998). Temporal cortex activation in humans viewing eye and mouth movements. The Journal of Neuroscience 18, 2188-2199.
83. Puce, A., Epling, J.A., Thompson, J.C., and Carrick, O.K. (2007). Neural responses elicited to face motion and vocalization pairings. Neuropsychologia 45, 93-106.
84. Puce, A., Smith, A., and Allison, T. (2000). Erps evoked by viewing facial movements. Cognitive Neuropsychology 17, 221-239.
85. Puce, A., Syngeniotis, A., Thompson, J.C., Abbott, D.F., Wheaton, K.J., and Castiello, U. (2003). The human temporal lobe integrates facial form and motion: evidence from fMRI and ERP studies. NeuroImage 19, 861-869.
86. Saint-Amour, D., Sanctis, P.D., Molholm, S., Ritter, W., and Foxe, J.J. (2007). Seeing voices: High-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion. Neuropsychologia 45, 587-597.
87. Sams, M., Alho, K., and Naatanen, R. (1984). Short-term habituation and dishabituation of the mismatch negativity of the ERP. Psychophysiology 21, 434-441.
88. Sams, M., Aulanko, R., Hämäläinen, M., Hari, R., Lounasmaa, O.V., Lu, S.T., and Simola, J. (1991). Seeing speech: visual information from lip movements modifies activity in the human auditory cortex. Neuroscience Letters 127, 141-145.
89. Schroger, E., and Wolff, C. (1996). Mismatch response of the human brain to changes in sound location. Neuroreport 7, 3005-3008.
90. Schroger, E., and Wolff, C. (1997). Fast preattentive processing of location: a functional basis for selective listening in humans. Neuroscience Letters 232, 5-8.
91. Scott, S.K. (2005). Auditory processing--speech, space and auditory objects. Current Opinion in Neurobiology 15, 197-201.
92. Scott, S.K., Blank, C.C., Rosen, S., and Wise, R.J. (2000). Identification of a pathway for intelligible speech in the left temporal lobe. Brain 123 Pt 12, 2400-2406.
93. Scott, S.K., and Johnsrude, I.S. (2003). The neuroanatomical and functional organization of speech perception. Trends In Neurosciences 26, 100-107.
94. Semlitsch, H.V., Anderer, P., Schuster, P., and Presslich, O. (1986). A Solution for Reliable and Valid Reduction of Ocular Artifacts, Applied to the P300 ERP. Psychophysiology 23, 695-703.
95. Skipper, J.I., Nusbaum, H.C., and Small, S.L. (2005). Listening to talking faces: motor cortical activation during speech perception. Neuroimage 25, 76-89.
96. Skrandies, W. (1990). Global field power and topographic similarity. Brain Topography 3, 137-141.
97. Spitsyna, G., Warren, J.E., Scott, S.K., Turkheimer, F.E., and Wise, R.J. (2006). Converging language streams in the human temporal lobe. Journal of Neuroscience 26, 7328-7336.
98. Stefanics, G., Csukly, G., Komlosi, S., Czobor, P., and Czigler, I. (2012). Processing of unattended facial emotions: a visual mismatch negativity study. NeuroImage 59, 3042-3049.
99. Stefanics, G., Kimura, M., and Czigler, I. (2011). Visual mismatch negativity reveals automatic detection of sequential regularity violation. Frontiers in human neuroscience 5, 46.
100. Tadel, F., Baillet, S., Mosher, J.C., Pantazis, D., and Leahy, R.M. (2011). Brainstorm: a userfriendly application for MEG/EEG analysis. Computational Intelligence and Neuroscience 2011, 879716.
101. Thompson, J.C., Hardee, J.E., Panayiotou, A., Crewther, D., and Puce, A. (2007). Common and distinct brain activation to viewing dynamic sequences of face and hand movements. NeuroImage 37, 966-973.
102. Wheaton, K.J., Thompson, J.C., Syngeniotis, A., Abbott, D.F., and Puce, A. (2004). Viewing the motion of human body parts activates different regions of premotor, temporal, and parietal cortex. NeuroImage 22, 277-288.
103. Winkler, I., and Czigler, I. (2012). Evidence from auditory and visual event-related potential (ERP) studies of deviance detection (MMN and vMMN) linking predictive coding theories and perceptual object representations. International Journal of Psychophysiology 83, 132-143.
104. Winkler, I., Czigler, I., Sussman, E., Horváth, J., and Balazs, L. (2005). Preattentive binding of auditory and visual stimulus features. Journal of Cognitive Neuroscience 17, 320-339.
105. Winkler, I., Horváth, J., Weisz, J., and Trejo, L.J. (2009). Deviance detection in congruent audiovisual speech: evidence for implicit integrated audiovisual memory representations. Biological Psychology 82, 281-292.
106. Yehia, H., Rubin, P., and Vatikiotis-Bateson, E. (1998). Quantitative association of vocal-tract and facial behavior. Speech Communication 26, 23-43.