Electrophysiological evidence for speech-specific audiovisual integration

Neuropsychologia

Volumn 53, Issue 1, 2014, Pages 115-121

  Baart, Martijn ^a,b   Stekelenburg, Jeroen J ^b   Vroomen, Jean ^b  

^a BASQUE CENTER ON COGNITION   (Spain)

^b TILBURG UNIVERSITY   (Netherlands)

Author keywords

Audiovisual integration; Audiovisual speech; N1; P2; Sine wave speech

Indexed keywords

ADULT; AMPLITUDE MODULATION; ARTICLE; AUDIOVISUAL INTEGRATION; AUDIOVISUAL SPEECH; AUDITORY SYSTEM; BRAIN ELECTROPHYSIOLOGY; CONTROLLED STUDY; ELECTROENCEPHALOGRAM; ELECTROENCEPHALOGRAPHY; EVOKED AUDITORY RESPONSE; FEMALE; HUMAN; LIP READING; MALE; PHONETICS; SINE WAVE SPEECH; SOUND; SPEECH; VISUAL STIMULATION; YOUNG ADULT;

AUDIOVISUAL INTEGRATION; AUDIOVISUAL SPEECH; N1; P2; SINE-WAVE SPEECH;

ACOUSTIC STIMULATION; ADOLESCENT; ADULT; AUDITORY PERCEPTION; BRAIN; ELECTROENCEPHALOGRAPHY; EVOKED POTENTIALS; EVOKED POTENTIALS, AUDITORY; FEMALE; HUMANS; LIPREADING; MALE; PHONETICS; PHOTIC STIMULATION; SPEECH; SPEECH PERCEPTION; TIME FACTORS; VISUAL PERCEPTION; YOUNG ADULT;

EID: 84889673713     PISSN: 00283932     EISSN: 18733514     Source Type: Journal    
DOI: 10.1016/j.neuropsychologia.2013.11.011     Document Type: Article

References (53)

1. Arnal L.H., Morillon B., Kell C.A., Giraud A.L. Dual neural routing of visual facilitation in speech processing. Journal of Neuroscience 2009, 29(43):13445-13453. 10.1523/JNEUROSCI.3194-09.2009.
2. Benson R.R., Richardson M., Whalen D.H., Lai S. Phonetic processing areas revealed by sinewave speech and acoustically similar non-speech. Neuroimage 2006, 31(1):342-353. 10.1016/j.neuroimage.2005.11.029.
3. Benson R.R., Whalen D.H., Richardson M., Swainson B., Clark V.P., Lai S., et al. Parametrically dissociating speech and nonspeech perception in the brain using fMRI. Brain and Language 2001, 78:364-396. 10.1006/brln.2001.2484.
4. Besle J., Fort A., Delpuech C., Giard M.H. Bimodal speech: Early suppressive visual effects in human auditory cortex. European Journalof Neuroscience 2004, 20:2225-2234. 10.1111/j.1460-9568.2004.03670.x.
5. Boersma, P., & Weenink, K. (2005). Praat: Doing phonetics by computer. Retrieved from http://www.fon.hum.uva.nl/praat.
6. Chandrasekaran C., Trubanova A., Stillittano S., Caplier A., Ghazanfar A.A. The natural statistics of audiovisual speech. PLoS Computational Biology 2009, 5(7):e1000436. 10.1371/journal.pcbi.1000436.
7. Colin C., Radeau M., Soquet A., Deltenre P. Generalization of the generation of an MMN by illusory McGurk percepts: Voiceless consonants. Clinical Neurophysiology 2004, 115(9):1989-2000. 10.1016/j.clinph.2004.03.027.
8. Colin C., Radeau M., Soquet A., Demolin D., Colin F., Deltenre P. Mismatch negativity evoked by the McGurk-MacDonald effect: A phonetic representation within short-term memory. Clinical Neurophysiology 2002, 113(4):495-506. 10.1016/S1388-2457(02)00024-X.
9. Crowley K.E., Colrain I.M. A review of the evidence for P2 being an independent component process: Age, sleep and modality. Clinical Neurophysiology 2004, 115(4):732-744. 10.1016/j.clinph.2003.11.021.
10. Dehaene-Lambertz G., Pallier C., Serniclaes W., Sprenger-Charolles L., Jobert A., Dehaene S. Neural correlates of switching from auditory to speech perception. Neuroimage 2005, 24(1):21-33. 10.1016/j.neuroimage.2004.09.039.
11. Eskelund K., Tuomainen J., Andersen T.S. Multistage audiovisual integration of speech: Dissociating identification and detection. Experimental Brain Research 2011, 208(3):447-457. 10.1007/s00221-010-2495-9.
12. Fort A., Delpuech C., Pernier J., Giard M.H. Early auditory-visual interactions in human cortex during nonredundant target identification. Cognitive Brain Research 2002, 14:20-30. 10.1016/S0926-6410(02)00058-7.
13. Giard M.H., Peronnet F. Auditory-visual integration during multimodal object recognition in humans: A behavioral and electrophysiological study. Journal of Cognitive Neuroscience 1999, 11(5):473-490. 10.1162/089892999563544.
14. Gratton G., Coles M.G., Donchin E. A new method for off-line removal of ocular artifact. Electroencephalography and Clinical Neurophysiology 1983, 55(4):468-484. 10.1016/0013-4694(83)90135-9.
15. Guthrie D., Buchwald J.S. Significance testing of difference potentials. Psychophysiology 1991, 28(2):240-244. 10.1111/j.1469-8986.1991.tb00417.x.
16. Hickok G., Poeppel D. The cortical organization of speech processing. Nature Reviews Neuroscience 2007, 8:393-402. 10.1038/nrn2113.
17. Jaramillo M., Ilvonen T., Kujala T., Alku P., Tervaniemi M., Alho K. Are different kinds of acoustic features processed differently for speech and non-speech sounds?. Cognitive Brain Research 2001, 12:459-466. 10.1016/S0926-6410(01)00081-7.
18. Jessen S., Kotz S.A. The temporal dynamics of processing emotions from vocal, facial, and bodily expressions. Neuroimage 2011, 58(2):665-674. 10.1016/j.neuroimage.2011.06.035.
19. Kislyuk D.S., Möttönen R., Sams M. Visual processing affects the neural basis of auditory discrimination. Journal of Cognitive Neuroscience 2008, 20:1-10. 10.1162/jocn.2008.20152.
20. Klucharev V., Mottonen R., Sams M. Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Cognitive Brain Research 2003, 18(1):65-75. 10.1016/j.cogbrainres.2003.09.004.
21. Kuhl P.K., Williams K.A., Meltzoff A.N. Cross-modal speech perception in adults and infants using nonspeech auditory stimuli. Journal of Experimental Psychology: Human Perception and Performance 1991, 17(3):829-840. 10.1037/0096-1523.17.3.829.
22. Lee H., Noppeney U. Physical and perceptual factors shape the neural mechanisms that integrate audiovisual signals in speech comprehension. Journal of Neuroscience 2011, 31(31):11338-11350. 10.1523/Jneurosci.6510-10.2011.
23. Liberman A.M., Mattingly I.G. A specialization for speech perception. Science 1989, 243(4890):489-494. 10.1126/science.2643163.
24. Magnée M.J., de Gelder B., van Engeland H., Kemner C. Audiovisual speech integration in pervasive developmental disorder: evidence from event-related potentials. Journal of Child Psychology and Psychiatry 2008, 49(9):995-1000. 10.1111/j.1469-7610.2008.01902.x.
25. Massaro D.W. Perceiving talking faces: From speech perception to a behavioral principle 1998, The MIT Press, Cambridge.
26. McCarthy G., Wood C.C. Scalp distributions of event-related potentials: An ambiguity associated with analysis of variance models. Electroencephalography and Clinical Neurophysiology 1985, 62(3):203-208. 10.1016/0168-5597(85)90015-2.
27. McGurk H., MacDonald J. Hearing lips and seeing voices. Nature 1976, 264:746-748. 10.1038/264746a0.
28. Metting van Rijn A.C., Peper A., Grimbergen C.A. High-quality recording of bioelectric events. Medical and Biological Engineering and Computing 1990, 28(5):389-397. 10.1007/BF02441961.
29. Molholm S., Ritter W., Murray M.M., Javitt D.C., Schroeder C.E., Foxe J.J. Multisensory auditory-visual interactions during early sensory processing in humans: A high-density electrical mapping study. Cognitive Brain Research 2002, 14(1):115-128. 10.1016/S0926-6410(02)00066-6.
30. Möttönen R., Calvert G.A., Jääskeläinen I.P., Matthews P.M., Thesen T., Tuomainen J., et al. Perceiving identical sounds as speech or non-speech modulates activity in the left posterior superior temporal sulcus. Neuroimage 2006, 30(2):563-569. 10.1016/j.neuroimage.2005.10.002.
31. Näätänen R., Gaillard A.W.K., Mäntysalo S. Early selective-attention effect in evoked potential reinterpreted. Acta Psychologica 1978, 42:313-329. 10.1016/0001-6918(78)90006-9.
32. Näätänen R., Kujala T., Winkler I. Auditory processing that leads to conscious perception: A unique window to central auditory processing opened by the mismatch negativity and related responses. Psychophysiology 2011, 48(1):4-22. 10.1111/j.1469-8986.2010.01114.x.
33. Remez R.E., Rubin P.E., Pisoni D.B., Carrell T.D. Speech perception without traditional speech cues. Science 1981, 212:947-949. 10.1126/science.7233191.
34. Repp B.H. Phonetic trading relations and context effects: New experimental evidence for a speech mode of perception. Psychological Bulletin 1982, 92(1):81-110. 10.1037/0033-2909.92.1.81.
35. Saint-Amour D., De Sanctis P., Molholm S., Ritter W., Foxe J.J. Seeing voices: High-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion. Neuropsychologia 2007, 45(3):587-597. 10.1016/j.neuropsychologia.2006.03.036.
36. Sams M., Aulanko R., Hämäläinen M., Hari R., Lounasmaa O.V., Lu S.T., et al. Seeing speech: Visual information from lip movements modifies activity in the human auditory cortex. Neuroscience Letters 1991, 127(1):141-145. 10.1016/0304-3940(91)90914-F.
37. Schwartz J.L., Berthommier F., Savariaux C. Seeing to hear better: Evidence for early audio-visual interactions in speech identification. Cognition 2004, 93(2):B69-78. 10.1016/j.cognition.2004.01.006.
38. Soto-Faraco S., Alsius A. Deconstructing the McGurk-MacDonald illusion. Journal of Experimental Psychology: Human Perception and Performance 2009, 35(2):580-587. 10.1037/a0013483.
39. Stekelenburg J.J., Maes J.P., van Gool A.R., Sitskoorn M., Vroomen J. Deficient multisensory integration in schizophrenia: An event-related potential study. Schizophrenia Research 2013, 147:253-261. 10.1016/j.schres.2013.04.038.
40. Stekelenburg J.J., Vroomen J. Neural correlates of multisensory integration of ecologically valid audiovisual events. Journal of Cognitive Neuroscience 2007, 19(12):1964-1973. 10.1162/jocn.2007.19.12.1964.
41. Stekelenburg J.J., Vroomen J. Electrophysiological correlates of predictive coding of auditory location in the perception of natural audiovisual events. Frontiers in Integrative Neuroscience 2012, 6. 10.3389/fnint.2012.00026.
42. Stekelenburg J.J., Vroomen J. Electrophysiological evidence for a multisensory speech-specific mode of perception. Neuropsychologia 2012, 50:1425-1431. 10.1016/j.neuropsychologia.2012.02.027.
43. Sumby W.H., Pollack I. Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America 1954, 26:212-215. 10.1121/1.1907309.
44. Talsma D., Doty T.J., Woldorff M.G. Selective attention and audiovisual integration: Is attending to both modalities a prerequisite for early integration?. Cerebral Cortex 2007, 17(3):679-690. 10.1093/cercor/bhk016.
45. Talsma D., Woldorff M.G. Selective attention and multisensory integration: Multiple phases of effects on the evoked brain activity. Journal of Cognitive Neuroscience 2005, 17(7):1098-1114. 10.1162/0898929054475172.
46. Teder-Sälejärvi W.A., Di Russo F., McDonald J.J., Hillyard S.A. Effects of spatial congruity on audio-visual multimodal integration. Journal of Cognitive Neuroscience 2005, 17(9):1396-1409. 10.1162/0898929054985383.
47. Tiitinen H., Sivonen P., Alku P., Virtanen J., Näätänen R. Electromagnetic recordings reveal latency differences in speech and tone processing in humans. Cognitive Brain Research 1999, 8(3):355-363. 10.1016/S0926-6410(99)00028-2.
48. Tuomainen J., Andersen T.S., Tiippana K., Sams M. Audio-visual speech perception is special. Cognition 2005, 96(1):B13-22. 10.1016/j.cognition.2004.10.004.
49. van Wassenhove V., Grant K.W., Poeppel D. Visual speech speeds up the neural processing of auditory speech. Proceedings of the National Academy of Sciences of the United States of America 2005, 102(4):1181-1186. 10.1073/pnas.0408949102.
50. van Wassenhove V., Grant K.W., Poeppel D. Temporal window of integration in auditory-visual speech perception. Neuropsychologia 2007, 45(3):598-607. 10.1016/j.neuropsychologia.2006.01.001.
51. Vroomen J., Baart M. Phonetic recalibration only occurs in speech mode. Cognition 2009, 110(2):254-259. 10.1016/j.cognition.2008.10.015.
52. Vroomen J., Stekelenburg J.J. Visual anticipatory information modulates multisensory interactions of artificial audiovisual stimuli. Journal of Cognitive Neuroscience 2010, 22(7):1583-1596. 10.1162/jocn.2009.21308.
53. Vroomen J., Stekelenburg J.J. Perception of intersensory synchrony in audiovisual speech: Not that special. Cognition 2011, 118(1):75-83. 10.1016/j.cognition.2010.10.002.