Deviance detection in congruent audiovisual speech: Evidence for implicit integrated audiovisual memory representations

Biological Psychology

Volumn 82, Issue 3, 2009, Pages 281-292

  Winkler, István ^a,b   Horváth, János ^a   Weisz, Júlia ^a   Trejo, Leonard J ^c  

^a INSTITUTE OF EXPERIMENTAL MEDICINE   (Hungary)

^b UNIVERSITY OF SZEGED   (Hungary)

^c Pacific Development and Technology LLC   (United States)

Author keywords

Audiovisual integration; Deviance detection; Event related brain potentials (ERP); Implicit memory; Mismatch negativity (MMN); Speech perception

Indexed keywords

ADULT; ARTICLE; ATTENTION; AUDITORY DISCRIMINATION; AUDITORY MEMORY; AUDITORY STIMULATION; BEHAVIOR; CONTROLLED STUDY; EVENT RELATED POTENTIAL; FEMALE; HUMAN; HUMAN EXPERIMENT; IMPLICIT MEMORY; MALE; MEMORY CONSOLIDATION; MISMATCH NEGATIVITY; NORMAL HUMAN; PRIORITY JOURNAL; SPEECH DISCRIMINATION; SPEECH PERCEPTION; TASK PERFORMANCE; VISUAL INFORMATION; VISUAL MEMORY; VISUAL STIMULATION;

ACOUSTIC STIMULATION; ADULT; ANALYSIS OF VARIANCE; ATTENTION; BRAIN; BRAIN MAPPING; ELECTROENCEPHALOGRAPHY; EVOKED POTENTIALS, AUDITORY; EVOKED POTENTIALS, VISUAL; FEMALE; HUMANS; MALE; MENTAL RECALL; PERCEPTUAL MASKING; PHOTIC STIMULATION; PSYCHOMOTOR PERFORMANCE; REACTION TIME; RECOGNITION (PSYCHOLOGY); SIGNAL PROCESSING, COMPUTER-ASSISTED; SPEECH; SPEECH PERCEPTION; VISUAL PERCEPTION;

EID: 70350126934     PISSN: 03010511     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.biopsycho.2009.08.011     Document Type: Article

References (89)

1. Alsius A., Navarra J., Campbell R., and Soto-Faraco S. Audiovisual integration of speech falters under high attention demands. Current Biology 15 9 (2005) 839-843
2. Assmann P.F., and Summerfield A.Q. The perception of speech under adverse conditions. In: Greenberg S., Ainsworth W.A., Popper A.N., and Fay R.R. (Eds). Speech Processing in the Auditory System. Springer Handbook of Auditory Research vol. 14 (2004), Springer Verlag, New York pp. 231-308
3. Baldeweg T. Repetition effects to sounds: evidence for predictive coding in the auditory system. Trends in Cognitive Sciences 10 3 (2006) 93-94
4. Bernstein L.E., Auer Jr. E.T., and Takayanagi S. Auditory speech detection in noise enhanced by lipreading. Speech Communication 44 1-4 (2004) 5-18
5. Bertelson P., Vroomen J., and de Gelder B. Visual recalibration of auditory speech identification: a McGurk aftereffect. Psychological Science 14 6 (2003) 592-597
6. Besle J., Fort A., Delpuech C., and Giard M.-H. Bimodal speech: early suppressive visual effects in human auditory cortex. European Journal of Neuroscience 20 8 (2004) 2225-2234
7. Besle J., Fort A., and Giard M.-H. Is the auditory sensory memory sensitive to visual information?. Experimental Brain Research 166 3-4 (2005) 337-344
8. Bregman A.S. Auditory Scene Analysis: The Perceptual Organization of Sound (1990), MIT Press, Cambridge, MA
9. Broadbent D.E. Perception and Communication (1958), Pergamon Press, New York
10. Bushara K.O., Hanakawa T., Immisch I., Toma K., Kansaku K., and Hallett M. Neural correlates of cross-modal binding. Nature Neuroscience 6 2 (2003) 190-195
11. Callan D.E., Jones J.A., Munhall K., Callan A.M., Kroos C., and Vatikiotis-Bateson E. Neural processes underlying perceptual enhancement by visual speech gestures. Neuroreport 14 17 (2003) 2213-2218
12. Callan D.E., Jones J.A., Munhall K., Kroos C., Callan A.M., and Vatikiotis-Bateson E. Multisensory integration sites identified by perception of spatial wavelet filtered visual speech gesture information. Journal of Cognitive Neuroscience 16 5 (2004) 805-816
13. Calvert G.A., and Campbell R. Reading speech from still and moving faces: the neural substrates of visible speech. Journal of Cognitive Neuroscience 15 1 (2003) 57-70
14. Colin C., Radeau M., Soquet A., Demolin D., Colin F., and Deltenre P. Mismatch negativity evoked by the McGurk-MacDonald effect: a phonetic representation within short-term memory. Clinical Neurophysiology 113 4 (2002) 495-506
15. Colin C., Radeau M., Soquet A., and Deltenre P. Generalization of the generation of an MMN by illusory McGurk percepts: voiceless consonants. Clinical Neurophysiology 115 9 (2004) 1989-2000
16. Coltheart M. Sensory memory-a tutorial review. In: Bouman H., and Bouwhuis D.G. (Eds). Attention & Performance X: Control of Language Processes (1984), Erlbaum, Hillsdale, NJ pp. 259-285
17. Czigler I. Visual mismatch negativity: violation of non-attended environmental regulations. Journal of Psychophysiology 21 3-4 (2007) 224-230
18. Davis C., and Kim J. Audio-visual interactions with intact clearly audible speech. Quarterly Journal of Experimental Psychology A 57 6 (2004) 1103-1121
19. Denham S.L., and Winkler I. The role of predictive models in the formation of auditory streams. Journal of Neurophysiology-Paris 100 1-3 (2006) 154-170
20. Dixon N.F., and Spitz L. The detection of auditory visual desynchrony. Perception 9 6 (1980) 719-721
21. Erber N.P. Auditory-visual perception of speech. Journal of Speech and Hearing Disorders 40 4 (1975) 481-492
22. Escera C., Alho K., Schröger E., and Winkler I. Involuntary attention and distractibility as evaluated with event related brain potentials. Audiology and Neuro-Otology 5 3-4 (2000) 151-166
23. Fernandez-Duque D., and Thornton I.M. Explicit mechanisms do not account for implicit localization and identification of change: an empirical reply to Mitroff et al. (2002). Journal of Experimental Psychology: Human Perception and Performance 29 5 (2003) 846-858
24. Fingelkurts A.A., Fingelkurts A.A., Krause C.M., Möttönen R., and Sams M. Cortical operational synchrony during audio-visual speech integration. Brain and Language 85 2 (2003) 297-312
25. Fodor J.A. Modularity of Mind: An Essay on Faculty Psychology (1983), MIT Press, Cambridge, MA
26. Fort A., and Giard M.-H. Multiple electrophysiological mechanisms of audio-visual integration in human perception. In: Calvert G., Spence C., and Stein B. (Eds). The Handbook of Multisensory Processes (2004), MIT Press, Cambridge, MA 503-514
27. Friedman D., Cycowicz Y.M., and Gaeta H. The novelty P3: an event-related brain potential (ERP) sign of the brain's evaluation of novelty. Neuroscience and Biobehavioral Reviews 25 4 (2001) 355-373
28. Fritz J., Shamma S., Elhilali M., and Klein D. Rapid task-related plasticity of spectrotemporal receptive fields in primary auditory cortex. Nature Neuroscience 6 11 (2003) 1216-1223
29. Fu K.M., Johnston T.A., Shah A.S., Arnold L., Smiley J., Hackett T.A., Garraghty P.E., and Schroeder C.E. Auditory cortical neurons respond to somatosensory stimulation. The Journal of Neuroscience 23 20 (2003) 7510-7515
30. Garrido M., Kilner J.M., Stephan K.E., and Friston K.J. The mismatch negativity: a review of underlying mechanisms. Clinical Neurophysiology 120 3 (2009) 453-463
31. Grant K.W. The effect of speechreading on masked detection thresholds for filtered speech. Journal of the Acoustical Society of America 109 5 Pt 1 (2001) 2272-2275
32. Grant K.W., and Seitz P. The use of visible speech cues for improving auditory detection of spoken sentences. Journal of the Acoustical Society of America 108 3 Pt 1 (2000) 1197-1208
33. Haenschel C., Vernon D.J., Dwivedi P., Gruzelier J.H., and Baldeweg T. Event-related brain potential correlates of human auditory sensory memory-trace formation. The Journal of Neuroscience 25 45 (2005) 10494-10501
34. Helfer K.S., and Freyman R.L. The role of visual speech cues in reducing energetic and informational masking. Journal of the Acoustical Society of America 117 2 (2005) 842-849
35. Hillyard S.A., and Picton T.W. Event-related potentials and selective information processing in man. In: Desmedt J.E. (Ed). Progress in Clinical Neurophysiology, vol. 6. Cognitive Components in Cerebral Event-related Potentials and Selective Attention (1979), Karger, Basel pp. 1-52
36. Jääskeläinen I.P., Ojanen V., Ahveninen J., Auranen T., Levänen S., Möttönen R., Tarnanen I., and Sams M. Adaptation of neuromagnetic N1 responses to phonetic stimuli by visual speech in humans. Neuroreport 15 18 (2004) 2741-2744
37. Jones J.A., and Callan D.E. Brain activity during audiovisual speech perception: an fMRI study of the McGurk effect. Neuroreport 14 8 (2003) 1129-1133
38. Jones J.A., and Jarick M. Multisensory integration of speech signals: the relationship between space and time. Experimental Brain Research 174 3 (2006) 588-594
39. Kaiser J., Hertrich I., Ackermann H., Mathiak K., and Lutzenberger W. Hearing lips: gamma-band activity during audiovisual speech perception. Cerebral Cortex 15 5 (2005) 646-653
40. Kang E., Lee D.S., Kang H., Hwang C.H., Oh S.H., Kim C.S., Chung J.K., and Lee M.C. The neural correlates of cross-modal interaction in speech perception during a semantic decision task on sentences: a PET study. Neuroimage 32 1 (2006) 423-431
41. Kim J., and Davis C. Hearing foreign voices: does knowing what is said affect masked visual speech detection?. Perception 32 1 (2003) 111-120
42. Kim J., and Davis C. Testing the cuing hypothesis for the AV speech detection. Proceedings of AVSP'2003. St Jorioz, France (2003) 9-12
43. Klucharev V., Möttönen R., and Sams M. Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Cognitive Brain Research 18 1 (2003) 65-75
44. Korzyukov O., Pflieger M.E., Wagner M., Bowyer S.M., Rosburg T., Sundaresan K., Elger C.E., and Boutros N.N. Generators of the intracranial P50 response in auditory sensory gating. Neuroimage 35 2 (2007) 814-826
45. Kujala T., Tervaniemi M., and Schröger E. The mismatch negativity in cognitive and clinical neuroscience: theoretical and methodological considerations. Biological Psychology 74 1 (2007) 1-19
46. Laurienti P.J., Burdette J.H., Wallace M.T., Yen Y.-F., Field A.S., and Stein B.E. Deactivation of sensory-specific cortex by cross-modal stimuli. Journal of Cognitive Neuroscience 14 3 (2002) 420-429
47. Lebib R., Papo D., de Bode S., and Baudonniere P.-M. Evidence of a visual-to-auditory cross-modal sensory gating phenomenon as reflected by the human P50 event-related brain potential modulation. Neuroscience Letters 341 3 (2003) 185-188
48. Lebib R., Papo D., Douiri A., de Bode S., Gillon Dowens M., and Baudonniere P.-M. Modulations of 'late' event-related brain potentials in humans by dynamic audiovisual speech stimuli. Neuroscience Letters 372 1-2 (2004) 74-79
49. Macaluso E., George N., Dolan R., Spence C., and Driver J. Spatial and temporal factors during processing of audiovisual speech: a PET study. Neuroimage 21 2 (2004) 725-732
50. Macmillan N.A., and Creelman C.D. Detection Theory: A User's Guide (1991), Cambridge University Press, Cambridge
51. Martin B.A., Kurtzberg D., and Stapells D.R. The effects of decreased audibility produced by high-pass noise masking on N1 and the mismatch negativity to speech sounds/ba/and/da/. Journal of Speech Language & Hearing Research 42 2 (1999) 271-286
52. McArthur G.M., Bishop D.V., and Proudfoot M. Do video sounds interfere with auditory event-related potentials?. Behavior Research Methods, Instruments, & Computers 35 2 (2003) 329-333
53. McCarthy G., and Wood C.C. Scalp distributions of event-related potentials: an ambiguity associated with analysis of variance models. Electroencephalography and Clinical Neurophysiology 62 3 (1985) 203-208
54. McGurk H., and MacDonald J. Hearing lips and seeing voices. Nature 264 5588 (1976) 746-748
55. Merikle P.M., Smilek D., and Eastwood J.D. Perception without awareness: perspectives from cognitive psychology. Cognition 79 1-2 (2001) 115-134
56. Miller L.M., and D'Esposito M. Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. Journal of Neuroscience 25 25 (2005) 5884-5893
57. Mitterer H. On the causes of compensation for coarticulation: evidence for phonological mediation. Perception & Psychophysics 68 7 (2006) 1227-1240
58. Möttönen R., Krause C.M., Tiippana K., and Sams M. Processing of changes in visual speech in the human auditory cortex. Cognitive Brain Research 13 3 (2002) 417-425
59. Möttönen R., Schürmann M., and Sams M. Time course of multisensory interactions during audiovisual speech perception in humans: a magnetoencephalographic study. Neuroscience Letters 363 2 (2004) 112-115
60. Muller-Gass A., Marcoux A., Logan J., and Campbell K.B. The intensity of masking noise affects the mismatch negativity to speech sounds in human subjects. Neuroscience Letters 299 3 (2001) 197-200
61. Näätänen R. The role of attention in auditory information-processing as revealed by event-related potentials and other brain measures of cognitive function. Behavioral and Brain Sciences 13 2 (1990) 201-288
62. Näätänen R., Gaillard A.W.K., and Mäntysalo S. Early selective attention effect on evoked potential reinterpreted. Acta Psychologica 42 4 (1978) 313-329
63. Navarra J., Vatakis A., Zampini M., Soto-Faraco S., Humphreys W., and Spence C. Exposure to asynchronous audiovisual speech extends the temporal window for audiovisual integration. Cognitive Brain Research 25 2 (2005) 499-507
64. Neisser U. Cognitive Psychology (1967), Appleton Century Crofts, New York
65. Novitski N., Maess B., and Tervaniemi M. Frequency specific impairment of automatic pitch change detection by fMRI acoustic noise: an MEG study. Journal of Neuroscience Methods 155 1 (2006) 149-159
66. Ojanen V., Möttönen R., Pekkola J., Jääskeläinen I.P., Joensuu R., Autti T., and Sams M. Processing of audiovisual speech in Broca's area. Neuroimage 25 2 (2005) 333-338
67. Olson I.R., Gatenby J.C., and Gore J.C. A comparison of bound and unbound audio-visual information processing in the human cerebral cortex. Cognitive Brain Research 14 1 (2002) 129-138
68. Polich J., and Criado J.R. Neuropsychology and neuropharmacology of the P3a and P3b. International Journal of Psychophysiology 60 2 (2006) 172-185
69. Pourtois G., de Gelder B., Vroomen J., Rossion B., and Crommelinck M. The time-course of intermodal binding between seeing and hearing affective information. Neuroreport 11 6 (2000) 1329-1333
70. Reale R.A., Calvert G.A., Thesen T., Jenison R.L., Kawasaki H., Oya H., Howard M.A., and Brugge J.F. Auditory-visual processing represented in the human superior temporal gyrus. Neuroscience 145 1 (2007) 162-184
71. Saint-Amour D., De Sanctis P., Molholm S., Ritter W., and Foxe J.J. Seeing voices: high-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion. Neuropsychologia 45 3 (2007) 587-597
72. Sams M., Aulanko R., Hämäläinen M., Hari R., Lounasmaa O.V., Lu S.T., and Simola J. Seeing speech: visual information from lip movements modifies activity in the human auditory cortex. Neuroscience Letters 127 1 (1991) 141-145
73. Sams M., Möttönen R., and Sihvonen T. Seeing and hearing others and oneself talk. Cognitive Brain Research 23 2-3 (2005) 429-435
74. Schröger E., and Winkler I. Presentation rate and magnitude of stimulus deviance effects on pre-attentive change detection. Neuroscience Letters 193 3 (1995) 185-188
75. Schwartz J.-L., Berthommier F., and Savariaux C. Seeing to hear better: evidence for early audio-visual interactions in speech identification. Cognition 93 2 (2004) B69-B78
76. Sekiyama K. McGurk effect and incompatibility: a cross-language study on auditory-visual speech perception. Studies and Essays in Behavioral Sciences and Philosophy vol. 14 (1991), Kanazawa University pp. 29-62
77. Skipper J.I., Nusbaum H.C., and Small S.L. Listening to talking faces: motor cortical activation during speech perception. Neuroimage 25 1 (2005) 76-89
78. Skipper J.I., van Wassenhove V., Nusbaum H.C., and Small S.L. Hearing lips and seeing voices: how cortical areas supporting speech production mediate audiovisual speech perception. Cerebral Cortex 17 10 (2007) 2387-2399
79. Sommers M.S., Tye-Murray N., and Spehar B. Auditory-visual speech perception and auditory-visual enhancement in normal-hearing younger and older adults. Ear & Hearing 26 3 (2005) 263-275
80. Soto-Faraco S., Navarra J., and Alsius A. Assessing automaticity in audiovisual speech integration: evidence from the speeded classification task. Cognition 92 3 (2004) B13-B23
81. Sumby W.H., and Pollack I. Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America 26 2 (1954) 212-215
82. Summerfield Q. Use of visual information for phonetic perception. Phonetica 36 4-5 (1979) 314-331
83. Trejo L.J., Johnson T., and Hyatt A. Visual-auditory interactions and mismatch negativity: a study of the McGurk effect. Second International Congress on Mismatch Negativity and its Clinical Applications. Barcelona, Spain, June 15-18 (2000)
84. van Wassenhove V., Grant K.W., and Poeppel D. Visual speech speeds up the neural processing of auditory speech. Proceedings of the National Academy of Sciences of the United States of America 102 4 (2005) 1181-1186
85. van Wassenhove V., Grant K.W., and Poeppel D. Temporal window of integration in auditory-visual speech perception. Neuropsychologia 45 3 (2007) 598-607
86. Widmann A., Kujala T., Tervaniemi M., Kujala A., and Schröger E. From symbols to sounds: visual symbolic information activates sound representations. Psychophysiology 41 5 (2004) 709-715
87. Winkler I. Change detection in complex auditory environment: beyond the oddball paradigm. In: Polich J. (Ed). Detection of Change: Event-Related Potential and fMRI Findings (2003), Kluwer Academic Publishers, Boston pp. 61-81
88. Winkler I. Interpreting the mismatch negativity (MMN). Journal of Psychophysiology 21 3-4 (2007) 147-163
89. Winkler I., Karmos G., and Näätänen R. Adaptive modeling of the unattended acoustic environment reflected in the mismatch negativity event-related potential. Brain Research 742 1-2 (1996) 239-252