Hearing lips and seeing voices: How cortical areas supporting speech production mediate audiovisual speech perception

Cerebral Cortex

Volumn 17, Issue 10, 2007, Pages 2387-2399

  Skipper, Jeremy I ^a   Van Wassenhove, Virginie ^b   Nusbaum, Howard C ^a   Small, Steven L ^a  

^a UNIVERSITY OF CHICAGO   (United States)

^b CALIFORNIA INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Audiovisual speech perception; Efference copy; McGurk effect; Mirror system; Motor system; Prediction

Indexed keywords

ADULT; ARTICLE; AUDIOVISUAL EQUIPMENT; AUDITORY CORTEX; BRAIN CORTEX; EVOKED CORTICAL RESPONSE; FEEDBACK SYSTEM; FRONTAL CORTEX; HEARING ACUITY; HUMAN; MOTOR PERFORMANCE; NORMAL HUMAN; PHONEME; PREDICTION; PRIORITY JOURNAL; SPEECH ARTICULATION; SPEECH DISCRIMINATION; SPEECH PERCEPTION; VISUAL CORTEX;

AUDITORY PERCEPTION; BRAIN MAPPING; CEREBRAL CORTEX; FUNCTIONAL LATERALITY; HEARING; HUMANS; ILLUSIONS; LIP; MOUTH; SOMATOSENSORY CORTEX; SPEECH; SPEECH PERCEPTION;

EID: 34249039173     PISSN: 10473211     EISSN: 14602199     Source Type: Journal    
DOI: 10.1093/cercor/bhl147     Document Type: Article

References (83)

1. Abbs JH, Sussman HM. 1971. Neurophysiological feature detectors and speech perception: a discussion of theoretical implications. J Speech Hear Res. 14:23-36.
2. Barlow HB. 1972. Single units and sensation: a neuron doctrine for perceptual psychology? Perception. 1:371-394.
3. Barlow HB, Narasimhan R, Rosenfeld A. 1972. Visual pattern analysis in machines and animals. Science. 177:567-575.
4. Belin P, Zatorre RJ, Ahad P. 2002. Human temporal-lobe response to vocal sounds. Brain Res Cogn Brain Res. 13:17-26.
5. Belin P, Zatorre RJ, Lafaille P, Ahad P, Pike B. 2000. Voice-selective areas in human auditory cortex. Nature. 403:309-312.
6. Bell CC. 1981. An efference copy which is modified by reafferent input. Science. 214:450-453.
7. Blakemore SJ, Wolpert DM, Frith CD. 1998. Central cancellation of selfproduced tickle sensation. Nat Neurosci. 1:635-640.
8. Blumstein SE, Stevens KN. 1981. Phonetic features and acoustic invariance in speech. Cognition. 10:25-32.
9. Bodznick D, Montgomery JC, Carey M. 1999. Adaptive mechanisms in the elasmobranch hindbrain. J Exp Biol. 202:1357-1364.
10. Buccino G, Lui F, Canessa N, Patteri I, Lagravinese G, Benuzzi F, Porro CA, Rizzolatti G. 2004. Neural circuits involved in the recognition of actions performed by nonconspecifics: an FMRI study. J Cogn Neurosci. 16:114-126.
11. Burton MW, Small SL, Blumstein SE. 2000. The role of segmentation in phonological processing: an fMRI investigation. J Cogn Neurosci. 12:679-690.
12. Callan DE, Jones JA, Callan AM, Akahane-Yamada R. 2004. Phonetic perceptual identification by native- and second-language speakers differentially activates brain regions involved with acoustic phonetic processing and those involved with articulatory-auditory/orosensory internal models. Neuroimage. 22:1182-1194.
13. Callan DE, Jones JA, Munhall K, Callan AM, Kroos C, Vatikiotis-Bateson E. 2003. Neural processes underlying perceptual enhancement by visual speech gestures. Neuroreport. 14:2213-2218.
14. Callan DE, Tajima K, Callan AM, Kubo R, Masaki S, Akahane-Yamada R. 2003. Learning-induced neural plasticity associated with improved identification performance after training of a difficult second-language phonetic contrast. Neuroimage. 19:113-124.
15. Calvert GA, Campbell R. 2003. Reading speech from still and moving faces: the neural substrates of visible speech. J Cogn Neurosci. 15:57-70.
16. Campbell R, MacSweeney M, Surguladze S, Calvert G, McGuire P, Suckling J, Brammer MJ, David AS. 2001. Cortical substrates for the perception of face actions: an fMRI study of the specificity of activation for seen speech and for meaningless lower-face acts (gurning). Brain Res Cogn Brain Res. 12:233-243.
17. Caviness VS, Meyer J, Makris N, Kennedy DN. 1996. MRI-based topographic parcellation of human neocortex: an anatomically specified method with estimate of reliability. J Cogn Neurosci. 8:566-587.
18. Chung MK, Robbins SM, Dalton KM, Davidson RJ, Alexander AL, Evans AC. 2005. Cortical thickness analysis in autism with heat kernel smoothing. Neuroimage. 25:1256-1265.
19. Cox RW. 1996. AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. Comput Biomed Res. 29:162-173.
20. el Manira A, Shupliakov O, Fagerstedt P, Grillner S. 1996. Monosynaptic input from cutaneous sensory afferents to fin motoneurons in lamprey. J Comp Neurol. 369:533-542.
21. Fant G. 1969. Acoustic theory of speech perception. The Hague, The Netherlands: Mouton.
22. Fischl B, Sereno MI, Dale AM. 1999. Cortical surface-based analysis. II: Inflation, flattening, and a surface-based coordinate system. Neuroimage. 9:195-207.
23. Fischl B, Sereno MI, Tootell RB, Dale AM. 1999. High-resolution intersubject averaging and a coordinate system for the cortical surface. Hum Brain Mapp. 8:272-284.
24. Fischl B, van der Kouwe A, Destrieux C, Halgren E, Segonne F, Salat DH, Busa E, Seidman LJ, Goldstein J, Kennedy D, et al. 2004. Automatically parcellating the human cerebral cortex. Cereb Cortex. 14:11-22.
25. Genovese CR, Lazar NA, Nichols T. 2002. Thresholding of statistical maps in functional neuroimaging using the false discovery rate. Neuroimage. 15:870-878.
26. Gentilucci M, Cattaneo L. 2005. Automatic audiovisual integration in speech perception. Exp Brain Res. 167:66-75.
27. Glover GH. 1999. Deconvolution of impulse response in event-related BOLD fMRI. Neuroimage. 9:416-429.
28. Gossard JP, Cabelguen JM, Rossignol S. 1991. An intracellular study of muscle primary afferents during fictive locomotion in the cat. J Neurophysiol. 65:914-926.
29. Grant KW, Greenberg S. 2001. Speech intelligibility derived from asynchronous processing of auditory-visual information. Proceedings of the Workshop on Audio-visual Speech Processing; Scheelsminde, Denmark.
30. Hatfield G. 2002. Perception as unconscious inference. In: Heyer D, Mausfeld R, editors. Perception and the physical world: psychological and philosophical issue in perception. New York: John Wiley & Sons, Ltd. p. 115-143.
31. Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P. 2001. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science. 293:2425-2430.
32. Helmholtz HL. 1867. Handbuch der physiologischen Optik. Leipzig: L. Voss.
33. Hubel DH, Wiesel TN. 1959. Receptive fields of single neurones in the cat's striate cortex. J Physiol. 148:574-591.
34. Hubel DH, Wiesel TN. 1977. Ferrier lecture. Functional architecture of macaque monkey visual cortex. Proc R Soc Lond B Biol Sci. 198:1-59.
35. Iacoboni M. 2005. Understanding others: imitation, language, empathy. In: Hurley S, Chater N, editors. Perspectives on Imitation: from Cognitive Neuroscience to Social Science, Volume 1: Mechanisms of Imitation and Imitation in Animals. Cambridge, MA: MIT Press. p. 77-99.
36. Iacoboni M, Koski LM, Brass M, Bekkering H, Woods RP, Dubeau MC, Mazziotta JC, Rizzolatti G. 2001. Reafferent copies of imitated actions in the right superior temporal cortex. Proc Natl Acad Sci USA. 98:13995-13999.
37. Jordan MI, Rumelhart DE. 1992. Forward models: supervised learning with a distal teacher. Cogn Sci. 16:307-354.
38. Kerzel D, Bekkering H. 2000. Motor activation from visible speech: evidence from stimulus response compatibility. J Exp Psychol Hum Percept Perform. 26:634-647.
39. Klatt DH. 1979. Speech perception: a model of acoustic-phonetic analysis and lexical access. J Phon. 7:279-312.
40. Knoblich G, Flach R. 2001. Predicting the effects of actions: interactions of perception and action. Psychol Sci. 12:467-472.
41. Liberman AM, Cooper FS, Shankweiler DS, Studdert-Kennedy M. 1967. Perception of the speech code. Psychol Rev. 74:431-461.
42. Liberman AM, Mattingly IG. 1985. The motor theory of speech perception revised. Cognition. 21:1-36.
43. McClelland JL, Elman JL. 1986. The TRACE model of speech perception. Cognit Psychol. 18:1-86.
44. McGurk H, MacDonald J. 1976. Hearing lips and seeing voices. Nature. 264:746-748.
45. Miall RC. 2003. Connecting mirror neurons and forward models. Neuroreport. 14:2135-2137.
46. Nishitani N, Hari R. 2002. Viewing lip forms: cortical dynamics. Neuron. 36:1211-1220.
47. Noll DC, Cohen JD, Meyer CH, Schneider W. 1995. Spiral k-space MRI imaging of cortical activation. J Magn Reson Imaging. 5:49-56.
48. Nusbaum HC, Magnuson J. 1997. Talker normalization: phonetic constancy as a cognitive process. In: Johnson K, Mullennix JW, editors. Talker variability in speech processing. San Diego, CA: Academic Press. p. 109-132.
49. Oldfield RC. 1971. The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia. 9:97-113.
50. Olson IR, Gatenby JC, Gore JC. 2002. A comparison of bound and unbound audio-visual information processing in the human cerebral cortex. Brain Res Cogn Brain Res. 14:129-138.
51. Paulesu E, Perani D, Blasi V, Silani G, Borghese NA, De Giovanni U, Sensolo S, Fazio F. 2003. A functional-anatomical model for lipreading. J Neurophysiol. 90:2005-2013.
52. Pekkola J, Laasonen M, Ojanen V, Autti T, Jaaskelainen IP, Kujala T, Sams M. 2006. Perception of matching and conflicting audiovisual speech in dyslexic and fluent readers: an fMRI study at 3 T. Neuroimage. 29:797-807.
53. Pulvermuller F, Huss M, Kherif F, Moscoso del Prado Martin F, Hauk O, Shtyrov Y. 2006. Motor cortex maps articulatory features of speech sounds. Proc Natl Acad Sci USA. 103:7865-7870.
54. Reisberg D, McLean J, Goldfield A. 1987. Easy to hear but hard to understand: a lipreading advantage with intact auditory stimuli. In: Dodd B, Campbell R, editors. Hearing by eye: the psychology of lipreading. Hillsdale, NJ: Erlbaum. p. 97-114.
55. Risberg A, Lubker J. 1978. Prosody and speechreading. Speech Transmission Laboratory Quarterly Progress Report Status Report 4. p. 1-16.
56. Rizzolatti G, Craighero L. 2004. The mirror-neuron system. Annu Rev Neurosci. 27:169-192.
57. Saad ZS, Reynolds RC, Argall B, Japee S, Cox RW. 2004. SUMA: an interface for surface-based intra- and inter-subject analysis with AFNI. IEEE International Symposium on Biomedical Imaging; Arlington, VA. p. 1510-1513.
58. Sams M, Mottonen R, Sihvonen T. 2005. Seeing and hearing others and oneself talk. Brain Res Cogn Brain Res. 23:429-435.
59. Sillar KT, Roberts A. 1988. A neuronal mechanism for sensory gating during locomotion in a vertebrate. Nature. 331:262-265.
60. Sillar KT, Skorupski P. 1986. Central input to primary afferent neurons in crayfish, Pacifastacus leniusculus, is correlated with rhythmic motor output of thoracic ganglia. J Neurophysiol. 55:678-688.
61. Skipper JI, Nusbaum HC, Small SL. 2005. Listening to talking faces: motor cortical activation during speech perception. Neuroimage. 25:76-89.
62. Skipper JI, Nusbaum HC, Small SL. 2006. Lending a helping hand to hearing: another motor theory of speech perception. In: Arbib MA, editor. Action to language via the mirror neuron system. Cambridge, MA: Cambridge University Press. p. 250-285.
63. Sperry RW. 1950. Neural basis of the spontaneous optokinetic response produced by visual inversion. J Comp Physiol Psychol. 43:482-489.
64. Stevens KN, Blumstein SE. 1978. Invariant cues for place of articulation in stop consonants. J Acoust Soc Am. 64:1358-1368.
65. Stevens KN, Halle M. 1967. Remarks on analysis by synthesis and distinctive features. In: Wathen-Dunn W, editor. Models for the perception of speech and visual form. Caimbridge, MA: MIT Press. p. 88-102.
66. Suga N, Schlegel P. 1972. Neural attenuation of responses to emitted sounds in echolocating rats. Science. 177:82-84.
67. Suga N, Shimozawa T. 1974. Site of neural attenuation of responses to self-vocalized sounds in echolocating bats. Science. 183:1211-1213.
68. Sumby WH, Pollack I. 1954. Visual contribution of speech intelligibility in noise. J Acoust Soc Am. 26:212-215.
69. Sundara M, Namasivayam AK, Chen R. 2001. Observation-execution matching system for speech: a magnetic stimulation study. Neuroreport. 12:1341-1344.
70. van Wassenhove V, Grant KW, Poeppel D. 2005. Visual speech speeds up the neural processing of auditory speech. Proc Natl Acad Sci USA. 102:1181-1186.
71. von Holst E, Mittelstaedt H. 1950. Das Reafferenzprinzip: Wechselwirkungen Zwischen Zentralnervensystem und Peripherie. Naturwissenschaften. 37:464-476.
72. Watkins K, Paus T. 2004. Modulation of motor excitability during speech perception: the role of Broca's area. J Cogn Neurosci. 16:978-987.
73. Watkins KE, Strafella AP, Paus T. 2003. Seeing and hearing speech excites the motor system involved in speech production. Neuropsychologia. 41:989-994.
74. Whitfield IC, Evans EF. 1965. Responses of auditory cortical neurons to stimuli of changing frequency. J Neurophysiol. 28:655-672.
75. Wilson SM, Iacoboni M. 2006. Neural responses to non-native phonemes varying in producibility: evidence for the sensorimotor nature of speech perception. Neuroimage. 33:316-325.
76. Wilson SM, Saygin AP, Sereno MI, Iacoboni M. 2004. Listening to speech activates motor areas involved in speech production. Nat Neurosci. 7:701-702.
77. Wolf H, Burrows M. 1995. Proprioceptive sensory neurons of a locust leg receive rhythmic presynaptic inhibition during walking. J Neurosci. 15:5623-5636.
78. Wolpert DM, Flanagan JR. 2001. Motor prediction. Curr Biol. 11:R729-R732.
79. Wolpert DM, Goodbody SJ, Husain M. 1998. Maintaining internal representations: the role of the human superior parietal lobe. Nat Neurosci. 1:529-533.
80. Yan J, Suga N. 1996. Corticofugal modulation of time-domain processing of biosonar information in bats. Science. 273:1100-1103.
81. Zaretsky M, Rowell CH. 1979. Saccadic suppression by corollary discharge in the locust. Nature. 280:583-585.
82. Zatorre RJ, Belin P. 2001. Spectral and temporal processing in human auditory cortex. Cereb Cortex. 11:946-953.
83. Zatorre RJ, Meyer E, Gjedde A, Evans AC. 1996. PET studies of phonetic processing of speech: review, replication, and reanalysis. Cereb Cortex. 6:21-30.