Electrophysiological evidence for a multisensory speech-specific mode of perception

Neuropsychologia

Volumn 50, Issue 7, 2012, Pages 1425-1431

  Stekelenburg, Jeroen J ^a   Vroomen, Jean ^a  

^a TILBURG UNIVERSITY   (Netherlands)

Author keywords

Audiovisual speech; McGurk illusion; Mismatch negativity; Multisensory integration; Sine wave speech

Indexed keywords

ARTICLE; AUDIOVISUAL SPEECH; AUDITORY STIMULATION; BEHAVIOR; CONTROLLED STUDY; ELECTROPHYSIOLOGY; FEMALE; HUMAN; HUMAN EXPERIMENT; ILLUSION; ILLUSORY CHANGE; LIP READING; MALE; MISMATCH NEGATIVITY; NORMAL HUMAN; SENSORY SYSTEM; SOUND; SPEECH; SPEECH PERCEPTION;

ACOUSTIC STIMULATION; ATTENTION; BRAIN; BRAIN MAPPING; BRAIN WAVES; CONTINGENT NEGATIVE VARIATION; ELECTROENCEPHALOGRAPHY; FEMALE; HUMANS; ILLUSIONS; LIPREADING; MALE; PSYCHOPHYSICS; SPEECH PERCEPTION; VISUAL PERCEPTION; YOUNG ADULT;

EID: 84861221181     PISSN: 00283932     EISSN: 18733514     Source Type: Journal    
DOI: 10.1016/j.neuropsychologia.2012.02.027     Document Type: Article

References (45)

1. Alho K., Connolly J.F., Cheour M., Lehtokoski A., Huotilainen M., Virtanen J., et al. Hemispheric lateralization in preattentive processing of speech sounds. Neuroscience Letters 1998, 258(1):9-12.
2. Arnal L.H., Morillon B., Kell C.A., Giraud A.L. Dual neural routing of visual facilitation in speech processing. Journal of Neuroscience 2009, 29(43):13445-13453.
3. Bernstein L.E., Auer E.T., Takayanagi S. Auditory speech detection in noise enhanced by lipreading. Speech Communication 2004, 44:5-18.
4. Besle J., Fort A., Delpuech C., Giard M.H. Bimodal speech: Early suppressive visual effects in human auditory cortex. European Journal of Neuroscience 2004, 20(8):2225-2234.
5. Boersma, P., & Weenink, K. (1999-2005). Praat: Doing phonetics by computer. Retrieved from: http://www.fon.hum.uva.nl/praat/.
6. Callan D.E., Jones J.A., Munhall K., Callan A.M., Kroos C., Vatikiotis-Bateson E. Neural processes underlying perceptual enhancement by visual speech gestures. Neuroreport 2003, 14(17):2213-2218.
7. Callan D.E., Jones J.A., Munhall K., Kroos C., Callan A.M., Vatikiotis-Bateson E. Multisensory integration sites identified by perception of spatial wavelet filtered visual speech gesture information. Journal of Cognitive Neuroscience 2004, 16(5):805-816.
8. Calvert G.A., Brammer M.J., Bullmore E.T., Campbell R., Iversen S.D., David A.S. Response amplification in sensory-specific cortices during crossmodal binding. Neuroreport 1999, 10(12):2619-2623.
9. Calvert G.A., Campbell R., Brammer M.J. Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Current Biology 2000, 10(11):649-657.
10. Colin C., Radeau M., Soquet A., Demolin D., Colin F., Deltenre P. Mismatch negativity evoked by the McGurk-MacDonald effect: A phonetic representation within short-term memory. Clinical Neurophysiology 2002, 113(4):495-506.
11. Dehaene-Lambertz G. Electrophysiological correlates of categorical phoneme perception in adults. Neuroreport 1997, 8(4):919-924.
12. Eskelund K., Tuomainen J., Andersen T.S. Multistage audiovisual integration of speech: Dissociating identification and detection. Experimental Brain Research 2011, 208(3):447-457.
13. Fowler C.A. Listeners do hear sounds, not tongues. Journal of the Acoustical Society of America 1996, 99(3):1730-1741.
14. Gratton G., Coles M.G., Donchin E. A new method for off-line removal of ocular artifact. Electroencephalography & Clinical Neurophysiology 1983, 55(4):468-484.
15. Guthrie D., Buchwald J.S. Significance testing of difference potentials. Psychophysiology 1991, 28(2):240-244.
16. Kilian-Hütten N., Valente G., Vroomen J., Formisano E. Auditory cortex encodes the perceptual interpretation of ambiguous sound. Journal of Neuroscience 2011, 31(5):1715-1720.
17. Kislyuk D.S., Möttönen R., Sams M. Visual processing affects the neural basis of auditory discrimination. Journal of Cognitive Neuroscience 2008, 20:1-10.
18. Klucharev V., Möttönen R., Sams M. Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Brain Research, Cognitive Brain Research 2003, 18(1):65-75.
19. Kroos C., Hogan J. Visual influence on auditory perception: Is speech special?. Proceedings of the international conference on auditory-visual speech processing 2009 2009, B.J. Theobald, R. Harvey (Eds.).
20. Kuhl P.K., Williams K.A., Meltzoff A.N. Cross-modal speech-perception in adults and infants using nonspeech auditory-stimuli. Journal of Experimental Psychology-Human Perception and Performance 1991, 17(3):829-840.
21. Lee H., Noppeney U. Physical and perceptual factors shape the neural mechanisms that integrate audiovisual signals in speech comprehension. Journal of Neuroscience 2011, 31(31):11338-11350.
22. Massaro D.W. Perceiving talking faces: From speech perception to a behavioral principle 1998, MIT Press, Cambridge.
23. McGurk H., MacDonald J. Hearing lips and seeing voices. Nature 1976, 264(5588):746-748.
24. Möttönen R., Calvert G.A., Jääskeläinen I.P., Matthews P.M., Thesen T., Tuomainen J., et al. Perceiving identical sounds as speech or non-speech modulates activity in the left posterior superior temporal sulcus. Neuroimage 2006, 30(2):563-569.
25. Näätänen R., Gaillard A.W.K., Mäntysalo S. Early selective-attention effect in evoked potential reinterpreted. Acta Psychologica 1978, 42:313-329.
26. Näätänen R., Lehtokoski A., Lennes M., Cheour M., Huotilainen M., Iivonen A., et al. Language-specific phoneme representations revealed by electric and magnetic brain responses. Nature 1997, 385(6615):432-434.
27. Näätänen R., Paavilainen P., Rinne T., Alho K. The mismatch negativity (MMN) in basic research of central auditory processing: A review. Clinical Neurophysiology 2007, 118(12):2544-2590.
28. Ojanen V., Möttönen R., Pekkola J., Jääskeläinen I.P., Joensuu R., Autti T., et al. Processing of audiovisual speech in Broca's area. Neuroimage 2005, 25(2):333-338.
29. Pare M., Richler R.C., Ten Hove M., Munhall K.G. Gaze behavior in audiovisual speech perception: The influence of ocular fixations on the McGurk effect. Perception & Psychophysics 2003, 65(4):553-567.
30. Remez R.E., Rubin P.E., Pisoni D.B., Carrell T.D. Speech perception without traditional speech cues. Science 1981, 212:947-949.
31. Repp B.H. Phonetic trading relations and context effects: New experimental evidence for a speech mode of perception. Psychological Bulletin 1982, 92(1):81-110.
32. Saint-Amour D., De Sanctis P., Molholm S., Ritter W., Foxe J.J. Seeing voices: High-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion. Neuropsychologia 2007, 45(3):587-597.
33. Saldaña H.M., Rosenblum L.D. Visual influences on auditory pluck and bow judgments. Perception & Psychophysics 1993, 54(3):406-416.
34. Sams M., Aulanko R., Hämäläinen M., Hari R., Lounasmaa O.V., Lu S.T., et al. Seeing speech: Visual information from lip movements modifies activity in the human auditory cortex. Neuroscience Letters 1991, 127(1):141-145.
35. Sams M., Paavilainen P., Alho K., Näätänen R. Auditory frequency discrimination and event-related potentials. Electroencephalography & Clinical Neurophysiology 1985, 62(6):437-448.
36. Schwartz J.L., Berthommier F., Savariaux C. Seeing to hear better: Evidence for early audio-visual interactions in speech identification. Cognition 2004, 93(2):B69-B78.
37. Skipper J.I., Nusbaum H.C., Small S.L. Listening to talking faces: Motor cortical activation during speech perception. Neuroimage 2005, 25(1):76-89.
38. Stekelenburg J.J., Vroomen J. Neural correlates of multisensory integration of ecologically valid audiovisual events. Journal of Cognitive Neuroscience 2007, 19(12):1964-1973.
39. Stekelenburg J.J., Vroomen J., de Gelder B. Illusory sound shifts induced by the ventriloquist illusion evoke the mismatch negativity. Neuroscience Letters 2004, 357(3):163-166.
40. Tuomainen J., Andersen T.S., Tiippana K., Sams M. Audio-visual speech perception is special. Cognition 2005, 96(1):B13-B22.
41. van Wassenhove V., Grant K.W., Poeppel D. Visual speech speeds up the neural processing of auditory speech. Proceedings of the National Academy of Science of the USA 2005, 102:1181-1186.
42. von Kriegstein K., Giraud A.L. Implicit multisensory associations influence voice recognition. PLoS Biology 2006, 4(10):1809-1820.
43. Vroomen J., Baart M. Phonetic recalibration only occurs in speech mode. Cognition 2009, 110:254-259.
44. Vroomen J., Stekelenburg J.J. Perception of intersensory synchrony in audiovisual speech: Not that special. Cognition 2011, 118(1):75-83.
45. Winkler I. Auditory and phonetic representations for isolated vowels: Cross-language studies. Psychophysiology 1999, 36:S2.