Visemic processing in audiovisual discrimination of natural speech: A simultaneous fMRI-EEG study

Neuropsychologia

Volumn 50, Issue 7, 2012, Pages 1316-1326

  Dubois, Cyril ^a   Otzenberger, Hélène ^b   Gounot, Daniel ^b   Sock, Rudolph ^a   Metz Lutz, Marie Noëlle ^b  

^a UNIVERSITÉ DE STRASBOURG   (France)

^b UNIVERSITÉ DE STRASBOURG   (France)

Author keywords

Lipreading; Motor cortex; MT+V5; Phonetic features; Syllables; Visual distinctiveness

Indexed keywords

ADULT; ARTICLE; BRAIN FUNCTION; ELECTROENCEPHALOGRAM; EVOKED AUDITORY RESPONSE; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; HUMAN EXPERIMENT; MALE; MENTAL FUNCTION; NORMAL HUMAN; OCCIPITAL CORTEX; PHONETICS; PREMOTOR CORTEX; SPEECH ARTICULATION; SPEECH DISCRIMINATION; STIMULUS RESPONSE; TEMPORAL CORTEX; VISEMIC PROCESSING; VISUAL STIMULATION; VOICE;

ACOUSTIC STIMULATION; ADOLESCENT; ADULT; ANALYSIS OF VARIANCE; BRAIN; DISCRIMINATION (PSYCHOLOGY); ELECTROENCEPHALOGRAPHY; EVOKED POTENTIALS; FATTY ACIDS; FEMALE; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; INDOLES; MAGNETIC RESONANCE IMAGING; MALE; OXYGEN; PHONETICS; PHOTIC STIMULATION; REACTION TIME; SPEECH; TIME FACTORS; VOICE; YOUNG ADULT;

EID: 84861225001     PISSN: 00283932     EISSN: 18733514     Source Type: Journal    
DOI: 10.1016/j.neuropsychologia.2012.02.016     Document Type: Article

References (79)

1. Abry C., Boë L.-J. " Laws" for lips. Speech Communication 1986, 5(1):97-104.
2. Arnal L.H., Wyart V., Giraud A.-L. Transitions in neural oscillations reflect prediction errors generated in audiovisual speech. Nature Neuroscience 2011, 14(6):797-801.
3. Bartels A., Logothetis N.K., Moutoussis K. fMRI and its interpretations: An illustration on directional selectivity in area V5/MT. Trends in Neurosciences 2008, 31(9):444-453.
4. Beauchamp M.S., Lee K.E., Argall B.D., Martin A. Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 2004, 41(5):809-823.
5. Bernstein L.E., Auer E.T.J., Moore J.K., Ponton C.W., Don M., Singh M. Visual speech perception without primary auditory cortex activation. Neuroreport 2002, 13(3):311-315.
6. Besle J., Fort A., Delpuech C., Giard M.-H. Bimodal speech: Early suppressive visual effects in human auditory cortex. European Journal of Neuroscience 2004, 20(8):2225-2234.
7. Blumstein S.E., Cooper W.E., Zurif E.B., Caramazza A. The perception and production of voice onset time in aphasia. Neuropsychologia 1977, 15:371-383.
8. Brett, M., Anton, J.-L., Valabregue, R., & Poline, J.-B. (2002). Region of interest analysis using an SPM toolbox. Presented at the 8th international conference on functional mapping of the human brain, June 2-6, Sendai, Japan.
9. Browman C.P., Goldstein L. Gestural specification using dynamically-defined articulatory structures. Journal of Phonetics 1990, 18:299-320.
10. Buchsbaum B., Pickell B., Love T., Hatrak M., Bellugi U., Hickok G. Neural substrates for verbal working memory in deaf signers: fMRI study and lesion case report. Brain and Language 2005, 95(2):265-272.
11. Burton M.W., Small S.L., Blumstein S.E. The role of segmentation in phonological processing: An fMRI investigation. Journal of Cognitive Neuroscience 2000, 12(4):679-690.
12. Callan D.E., Jones J.A., Munhall K., Callan A.M., Kroos C., Vatikiotis-Bateson E. Neural processes underlying perceptual enhancement by visual speech gestures. Neuroreport 2003, 14(17):2213-2218.
13. Calvert G.A., Bullmore E.T., Brammer M.J., Campbell R., Williams S.C.R., McGuire P.K., et al. Activation of auditory cortex during silent lipreading. Science 1997, 276:593-596.
14. Calvert G.A., Campbell R. Reading speech from still and moving faces: The neural substrates of visible speech. Journal of Cognitive Neuroscience 2003, 15(1):57-70.
15. Campbell R. The processing of audio-visual speech: Empirical and neural bases. Philosophical Transactions of the Royal Society B: Biological Sciences 2008, 363(1493):1001-1010.
16. Campbell R., MacSweeney M., Surguladze S., Calvert G., McGuire P., Suckling J., et al. Cortical substrates for the perception of face actions: An fMRI study of the specificity of activation for seen speech and for meaningless lower-face acts (gurning). Cognitive Brain Research 2001, 12(2):233-243.
17. Dale A.M. Optimal experimental design for event-related fMRI. Human Brain Mapping 1999, 8(2-3):109-114.
18. Devlin J.T., Raley J., Tunbridge E., Lanary K., Floyer-Lea A., Narain C., et al. Functional asymmetry for auditory processing in human primary auditory cortex. Journal of Neuroscience 2003, 23(37):11516-11522.
19. Eimas P.D. The relation between identification and discrimination along speech and non-speech continua. Language and Speech 1963, 206-217.
20. Fisher C.G. Confusions among visually perceived consonants. Journal of Speech and Hearing Research 1968, 11:796-804.
21. Fowler C. An event approach of the study of speech perception from a direct-realist perspective. Journal of Phonetics 1996, 14:3-28.
22. Fridriksson J., Moss J., Davis B., Baylis G.C., Bonilha L., Rorden C. Motor speech perception modulates the cortical language areas. NeuroImage 2008, 41(2):605-613.
23. Statistical parametric mapping: The analysis of functional brain 2007, Academic Press, London. K.J. Friston, J. Ashburner, S. Kiebel, T.E. Nichols, W.D. Penny (Eds.).
24. Galantucci B., Fowler C.A., Turvey M.T. The motor theory of speech perception reviewed. Psychonomic Bulletin & Review 2006, 13(3):361-377.
25. Gallese V., Fadiga L., Fogassi L., Rizzolatti G. Action recognition in the premotor cortex. Brain 1996, 119:535-609.
26. Gerrits E., Schouten M.E.H. Categorical perception depends on the discrimination task. Perception & Psychophysics 2004, 66(3):363-376.
27. Grant K.W., Walden B.E., Seitz P.F. Auditory-visual speech recognition by hearing-impaired subjects: Consonant recognition, sentence recognition, and auditory-visual integration. Journal of the Acoustical Society of America 1998, 103(5):2677-2690.
28. Hall D.A., Fussell C., Summerfield A.Q. Reading fluent speech from talking faces: Typical brain networks and individual differences. Journal of Cognitive Neuroscience 2005, 17(6):939-953.
29. Hasnain M.K., Fox P.T., Woldorff M.G. Intersubject variability of functional areas in the human visual cortex. Human Brain Mapping 1998, 6(4):301-315.
30. Hickok G., Buchsbaum B., Humphries C., Muftuler T. Auditory-motor interaction revealed by fMRI: Speech, music, and working memory in area Spt. Journal of Cognitive Neuroscience 2003, 15(5):673-682.
31. Hickok G., Poeppel D. Dorsal and ventral streams: A framework for understanding aspects of the functional anatomy of language. Cognition 2004, 92(1-2):67-99.
32. Hickok G., Poeppel D. The cortical organization of speech processing. Nature Review Neuroscience 2007, 8(5):393-402.
33. Holle H., Obleser J., Rueschemeyer S.-A., Gunter T.C. Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions. NeuroImage 2010, 49(1):875-884.
34. Hutchison E.R., Blumstein S.E., Myers E.B. An event-related fMRI investigation of voice-onset time discrimination. NeuroImage 2008, 40(1):342-352.
35. Jones J.A., Callan D.E. Brain activity during audiovisual speech perception: An fMRI study of the McGurk effect. Neuroreport 2003, 14(8):1129-1133.
36. Kiebel S.J., Friston K.J. Statistical parametric mapping for event-related potentials: I. Generic considerations. NeuroImage 2004, 22(2):492-502.
37. Kiebel S.J., Friston K.J. Statistical parametric mapping for event-related potentials (II): A hierarchical temporal model. NeuroImage 2004, 22(2):503-520.
38. Le Goff B., Benoît C. A text-to-audiovisual-speech synthesizer for French. Proceedings of the international conference on spoken language processing, ICSLP 96 1996, 2163-2166.
39. Liberman A.M., Cooper F.S., Shankweiler D.P., Studdert-Kennedy M. Perception of the speech code. Psychological Review 1967, 74(6):431-461.
40. Liberman A.M., Mattingly I.G. The motor theory of speech perception revised. Cognition 1985, 21(1):1-36.
41. Lotto A.J., Hickok G., Holt L.L. Reflections on mirror neurons and speech perception. Trends in Cognitive Sciences 2009, 13(3):110-114.
42. MacLeod A., Summerfield Q. A procedure for measuring auditory and audiovisual speech-reception thresholds for sentences in noise: Rationale, evaluation, and recommendations for use. British Journal of Audiology 1990, 24(1):29-43.
43. Marshall J.C., Fink G.R. Spatial cognition: Where we were and where we are. NeuroImage 2001, 14(1):S2-S7.
44. McGurk H., MacDonald J. Hearing lips and seeing voices. Nature 1976, 264(5588):746-748.
45. Myatchin I., Mennes M., Wouters H., Stiers P., Lagae L. Working memory in children with epilepsy: An event-related potentials study. Epilepsy Research 2009, 86(2-3):183-190.
46. Obleser J., Eisner F. Pre-lexical abstraction of speech in the auditory cortex. Trends in Cognitive Sciences 2009, 13(1):14-19.
47. Ojanen V., Möttönen R., Pekkola J., Jääskeläinen I.P., Joensuu R., Autti T., et al. Processing of audiovisual speech in Broca's area. NeuroImage 2005, 25(2):333-338.
48. Okada K., Hickok G. Left posterior auditory-related cortices participate both in speech perception and speech production: Neural overlap revealed by fMRI. Brain and Language 2006, 98:112-117.
49. Okada K., Hickok G. Two cortical mechanisms support the integration of visual and auditory speech: A hypothesis and preliminary data. Neuroscience Letters 2009, 452(3):219-223.
50. Oldfield R.C. The assessment and analysis of handedness: The Edinburgh inventory. Neuropsychologia 1971, 9(1):97-113.
51. Otzenberger H., Gounot D., Foucher J.R. Optimisation of a post-processing method to remove the pulse artifact from EEG data recorded during fMRI: An application to P300 recordings during e-fMRI. Neuroscience Research 2007, 57(2):230-239.
52. Pa J., Hickok G. A parietal-temporal sensory-motor integration area for the human vocal tract: Evidence from an fMRI study of skilled musicians. Neuropsychologia 2008, 46(1):362-368.
53. Peeters M., Verhoeven L., de Moor J., van Balkom H. Importance of speech production for phonological awareness and word decoding: The case of children with cerebral palsy. Research in Developmental Disabilities 2009, 30(4):712-726.
54. Pekkola J., Ojanen V., Autti T., Jääskeläinen I.P., Möttönen R., Tarkiainen A., et al. Primary auditory cortex activation by visual speech: An fMRI study at 3T. Neuroreport 2005, 16(2):125-128.
55. Pilling M. Auditory event-related potentials (ERPs) in audiovisual speech perception. Journal of Speech Language and Hearing Research 2009, 52(4):1073-1081.
56. Pisoni D.B., Luce P.A. Acoustic-phonetic representations in word recognition. Cognition 1987, 25(1-2):21-52.
57. Rauschecker J.P., Scott S.K. Maps and streams in the auditory cortex: Nonhuman primates illuminate human speech processing. Nature Neuroscience 2009, 12(6):718-724.
58. Rizzolatti G., Arbib M.A. Language within our grasp. Trends in Neuroscience 1998, 21(5):188-194.
59. Rizzolatti G., Craighero L. The mirror-neuron system. Annual Review of Neuroscience 2003, 27:169-192.
60. Rizzolatti G., Fadiga L., Gallese V., Fogassi L. Premotor cortex and the recognition of motor actions. Cognitive Brain Research 1996, 3.
61. Saint-Amour D., De Sanctis P., Molholm S., Ritter W., Foxe J.J. Seeing voices: High-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion. Neuropsychologia 2007, 45(3):587-597.
62. Saito D.N., Yoshimura K., Kochiyama T., Okada T., Honda M., Sadato N. Cross-modal binding and activated attentional networks during audio-visual speech integration: A functional MRI study. Cerebral Cortex 2005, 15(11):1750-1760.
63. Schwartz J.-L., Abry C., Boë L.-J., Cathiard M.-A. Phonology in a theory of perception-for-action-control. Phonetics, phonology, and cognition 2002, Oxford University Press. J.L. Durand, B. Laks (Eds.).
64. Schwartz, J.-L., Basirat, A., Ménard, L., & Sato, M. The perception-for-action-control theory (PACT): A perceptuo-motor theory of speech perception. Journal of Neurolinguistics, in press.
65. Scott S.K., Johnsrude I.S. The neuroanatomical and functional organization of speech perception. Trends in Neuroscience 2003, 26(2):100-107.
66. Scott S.K., Wise R.J.S. The functional neuroanatomy of prelexical processing in speech perception. Cognition 2004, 92(1-2):13-45.
67. Segui J. The syllable: A basic perceptual unit in speech processing?. Attention and performance X: Control of language processes 1984, 165-182. Erlbaum, NJ. H. Bouma, D.G. Bouwhuis (Eds.).
68. Sekiyama K., Kanno I., Miura S., Sugita Y. Auditory-visual speech perception examined by fMRI and PET. Neuroscience Research 2003, 47(3):277-287.
69. Skipper J.L., Nusbaum H.C., Small S.L. Listening to talking faces: Motor activation during speech perception. NeuroImage 2005, 25:76-89.
70. Skipper J.I., van Wassenhove V., Nusbaum H.C., Small S.L. Hearing lips and seeing voices: How cortical areas supporting speech production mediate audiovisual speech perception. Cerebral Cortex 2007, 17(10):2387-2399.
71. Sumby W.H., Pollack I. Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America 1954, 26(2):212-216.
72. Turkeltaub P.E., Branch Coslett H. Localization of sublexical speech perception components. Brain and Language 2010, 114(1):1-15.
73. van Atteveldt N., Formisano E., Goebel R., Blomert L. Integration of letters and speech sounds in the human brain. Neuron 2004, 43(2):271-282.
74. van Wassenhove V., Grant K.W., Poeppel D. Visual speech speeds up the neural processing of auditory speech. PNAS 2005, 102(4):1181-1186.
75. Vidal J., Bonnet-Brilhault F., Roux S., Bruneau N. Auditory evoked potentials to tones and syllables in adults: Evidence of specific influence on N250 wave. Neuroscience Letters 2005, 378(3):145-149.
76. Watkins K.E., Strafella A.P., Paus T. Seeing and hearing speech excites the motor system involved in speech production. Neuropsychologia 2003, 41:989-994.
77. Wilms M., Eickhoff S., Specht K., Amunts K., Shah N., Malikovic A., et al. Human V5/MT+: Comparison of functional and cytoarchitectonic data. Anatomy and Embryology 2005, 210(5):485-495.
78. Wilson S.M., Molnar-Szakacs I., Iacoboni M. Beyond superior temporal cortex: Intersubject correlations in narrative speech comprehension. Cerebral Cortex 2008, 18(1):230-242.
79. Wilson S.M., Saygin A.P., Sereno M.I., Iacoboni M. Listening to speech activates motor areas involved in speech production. Nature Neuroscience 2004, 7(7):701-702.