Phonetic Processing by the Speech Perceiving Brain

The Handbook of Speech Perception

Volumn , Issue , 2008, Pages 79-98

  Bernstein, Lynne E ^a  

^a HOUSE EAR INSTITUTE   (United States)

Author keywords

Brain; Cortical pathways; Intracortical recordings; Phonetic processing; Speech perception

Indexed keywords

EID: 84890743180     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1002/9780470757024.ch4     Document Type: Chapter

References (107)

1. Auer, E. T., Jr., Bernstein, L. E., & Singh, M. (2001). Comparing cortical activity during the perception of two forms of biological motion for language communication. In D. W. Massaro, J. Light, & K. Geraci (eds.), Proceedings of Audio Visual Speech Perception 2001 (pp. 40-4).
2. Bartels, A. & Zeki, S. (1998). The theory of multistage integration in the visual brain. Proceedings of the Royal Society of London, Series B: Biological Sciences, 265, 2327-32.
3. Belin, P. & Zattore, R. J. (2000). "What," "where" and "how" in auditory cortex. Nature Neuroscience, 3, 965-6.
4. Belin, P., Zatorre, R. J., Lafaille, P., Ahad, P., & Pike, B. (2000). Voice-selective areas in human auditory cortex, Nature, 403, 309-12.
5. Benson, R. R., Whalen, D. H., Richardson, M., Swainson, B., Clark, V. P., Lai, S., & Liberman, A. M. (2001). Parametrically dissociating speech and nonspeech perception in the brain using fMRI. Brain and Language, 78, 364-96.
6. Bernstein, L. E., Auer, E. T., Jr., Moore, J. K., Ponton, C. W., Don, M., & Singh, M. (2002). Visual speech perception without primary auditory cortex activation. NeuroReport, 13, 311-15.
7. Bernstein, L. E., Auer, E. T., Jr., Zhou, Y., & Singh, M. (2003). Cortical specialization for visual speech versus non-speech face movements in color video and point lights. Cognitive Neuroscience Society, March 30-April 1.
8. Bernstein, L. E., Auer, E. T., Jr., & Moore, J. K. (2004). Audiovisual speech binding: convergence or association? In G. A. Calvert, C. Spence, & B. E. Stein (eds.), Handbook of Multisensory Processing (pp. 203-23). Cambridge, MA: MIT Press.
9. Bernstein, L. E., Demorest, M. E., & Tucker, P. E. (2000). Speech perception without hearing. Perception & Psychophysics, 62, 233-52.
10. Binder, J. R., Frost, J. A., Hammeke, T. A., Bellgowan, P. S., Springer, J. A., Kaufman, J. N., & Possing, E. T. (2000). Human temporal lobe activation by speech and nonspeech sounds. Cerebral Cortex, 10, 512-28.
11. Bonda, E., Petrides, M., Ostry, D., & Evans, A. (1996). Specific involvement of human parietal systems and the amygdala in the perception of biological motion. Journal of Neuroscience, 16, 3737-44.
12. Braida, L. D. (1991). Crossmodal integration in the identification of consonant segments. Quarterly Journal of Experimental Psychology: Human Experimental Psychology, 43A, 647-77.
13. Brancazio, L., Miller, J. L., & Pare, M. A. (2000). Visual influences on internal structure of phonetic categories. Journal of the Acoustical Society of America, 108, 2481.
14. Brodmann, K. (1909). Vergleichende Lokalisatinslehre der Grosshirnrinde in ihren Prinzipien dargestellt auf Grund des Zellenbaues. Leipzig: J. A. Barth.
15. Büchel, C., Price, C., & Friston, K. (1998). A multimodal language region in the ventral visual pathway. Nature, 394, 274-77.
16. Calvert, G. A. (2001). Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cerebral Cortex, 11, 1110-23.
17. Calvert, G. A., Brammer, M. J., Bullmore, E. T., Campbell, R., Iversen, S. D., & David, A. S. (1999). Response amplification in sensory-specific cortices during crossmodal binding. NeuroReport, 10, 2619-23.
18. Calvert, G. A., Bullmore, E. T., Brammer, M. J., Campbell, R., Williams, S. C., McGuire, P. K., Woodruff, P. W., Iversen, S. D., & David, A. S. (1997). Activation of auditory cortex during silent lipreading. Science, 276, 593-6.
19. Calvert, G. A., Campbell, R., & Brammer, M. J. (2000). Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Current Biology, 10, 649-57.
20. Campbell, R., MacSweeney, M., Surguladze, S., Calvert, G., McGuire, P., Suckling, J., Brammer, M. J., & David, A. S. (2001). Cortical substrates for the perception of face actions: an fMRI study of the specificity of activation for seen speech and for meaningless lower-face acts (gurning). Cognitive Brain Research, 12, 233-43.
21. Celsis, P., Boulanouar, K., Doyon, B., Ranjeva, J. P., Berry, I., Nespoulous, J. L., & Chollet, F. (1999). Differential fMRI responses in the left posterior superior temporal gyrus and left supramarginal gyrus to habituation and change detection in syllables and tones. NeuroImage, 9, 135-44.
22. Creutzfeldt, O. D., Watanabe, S., & Lux, H. D. (1966). Relations between EEG phenomena and potentials of single cortical cells. I: Evoked responses after thalamic and epicortical stimulation. Electroencephalography and Clinical Neurophysiology, 20, 1-18.
23. Eggermont, J. J. (2001). Between sound and perception: reviewing the search for a neural code. Hearing Research, 157, 1-42.
24. Eggermont, J. J. & Ponton, C. W. (2002). The neurophysiology of auditory perception: from single units to evoked potentials. Audiology & Neuro-otology, 7, 71-99.
25. Falchier, A., Clavagnier, S., Barone, P., & Kennedy, H. (2002). Anatomical evidence of multimodal integration in primate striate cortex. Journal of Neuroscience, 22, 5749-59.
26. Felleman, D. J. & Van Essen, D. C. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex, 1, 1-47.
27. Fiez, J. A., Raichle, M. E., Balota, D. A., Tallal, P., & Petersen, S. E. (1996). PET activation of posterior temporal regions during auditory word presentation and verb generation. Cerebral Cortex, 6, 1-10.
28. Fowler, C. (2004). Speech as a supramodal or amodal phenomenon. In G. Calvert, C. Spence, & B. E. Stein (eds.), Handbook of Multisensory Processes (pp. 189-201). Cambridge, MA: MIT Press.
29. Foxe, J. J. & Simpson, G. V. (2002). Flow of activation from V1 to frontal cortex in humans: A framework for defining "early" visual processing. Experimental Brain Research, 142, 139-50.
30. Friston, K. J., Price, C. J., Fletcher, P., Moore, C., Frackowiak, R. S., & Dolan, R. J. (1996). The trouble with cognitive subtraction. NeuroImage, 4, 97-104.
31. Giard, M. H. & Peronnet, F. (1999). Auditory-visual integration during multimodal object recognition in humans: A behavioral and electrophysiological study. Journal of Cognitive Neuroscience, 11, 473-90.
32. Green, K. P. (1998). The use of auditory and visual information during phonetic processing: Implications for theories of speech perception. In R. Campbell, B. Dodd, & D. Burnham (eds.), Hearing by Eye II: Advances in the Psychology of Speechreading and Auditory-visual Speech (pp. 3-25). Hove, UK: Psychology Press.
33. Green, K. P. & Kuhl, P. K. (1989). The role of visual information in the processing of place and manner features in speech perception. Perception & Psychophysics, 45, 34-42.
34. Green, K. P. & Norrix, L. W. (2001). Perception of /r/and /l/in a stop cluster: Evidence of cross-modal context effects. Journal of Experimental Psychology: Human Perception and Performance, 27, 166-77.
35. Green, K. P., Kuhl, P. K., Meltzoff, A. N., & Stevens, E. B. (1991). Integrating speech information across talkers, gender, and sensory modality: Female faces and male voices in the McGurk effect. Perception & Psychophysics, 50, 524-36.
36. Grill-Spector, K., Kourtzi, Z., & Kanwisher, N. (2001). The lateral occipital complex and its role in object recognition. Vision Research, 41, 1409-22.
37. Hackett, T. A., Stepniewska, I., & Kaas, J. H. (1998). Subdivisions of auditory cortex and ipsilateral cortical connections of the parabelt auditory cortex in macaque monkeys. Journal of Comparative Neurology, 394, 475-95.
38. Halgren, E., Dale, A. M., Sereno, M. I., Tootell, R. B., Marinkovic, K., & Rosen, B. R. (1999). Location of human face-selective cortex with respect to retinotopic areas. Human Brain Mapping, 7, 29-37.
39. Hickok, G. & Poeppel, D. (2000). Towards a functional neuroanatomy of speech perception. Trends in Cognitive Sciences, 4, 131-8.
40. Howard, M. A., Volkov, I. O., Mirsky, R., Garell, P. C., Noh, M. D., Granner, M., Damasio, H., Steinschneider, M., Reale, R. A., Hind, J. E., & Brugge, J. F. (2000). Auditory cortex on the human posterior superior temporal gyrus. Journal of Comparative Neurology, 416, 79-92.
41. Huckins, S. C., Turner, C. W., Doherty, K. A., Fonte, M. M., & Szeverenyi, N. M. (1998). Functional magnetic resonance imaging measures of blood flow patterns in the human auditory cortex in response to sound. Journal of Speech, Language, and Hearing Research, 41, 538-48.
42. Jiang, J., Alwan, A., Keating, P., Auer, E. T., Jr., & Bernstein, L. E. (2002). On the relationship between face movements, tongue movements, and speech acoustics. EURASIP Journal on Applied Signal Processing: Special issue on Joint Audio-Visual Speech Processing, 2002, 1174-88.
43. Jones, E. G. & Powell, T. P. (1970). An anatomical study of converging sensory pathways within the cerebral cortex of the monkey. Brain, 93, 793-820.
44. Kaas, J. H. & Hackett, T. A. (2000). Subdivisions of auditory cortex and processing streams in primates. Proceedings of the National Academy of Sciences of the United States of America, 97, 11793-9.
45. Knight, R. T., Scabini, D., Woods, D. L., & Clayworth, C. (1988). The effects of lesions of superior temporal gyrus and inferior parietal lobe on temporal and vertex components of the human AEP. Electroencephalography and Clinical Neurophysiology, 70, 499-509.
46. Krolak-Salmon, P., Henaff, M. A., Tallon-Baudry, C., Yvert, B., Fischer, C., Vighetto, A., Betrand, O., & Mauguiere, F. (2001). How fast can the human lateral geniculate nucleus and visual striate cortex see? Society for Neuroscience Abstracts, 27, 913.
47. Liberman, A. M. & Whalen, D. H. (2000). On the relation of speech to language. Trends in Cognitive Sciences, 4, 187-96.
48. Liégeois-Chauvel, C., de Graaf, J. B., Laguitton, V., & Chauvel, P. (1999). Specialization of left auditory cortex for speech perception in man depends on temporal coding. Cerebral Cortex, 9, 484-96.
49. Liégeois-Chauvel, C., Musolino, A., Badier, J. M., Marquis, P., & Chauvel, P. (1994). Evoked potentials recorded from the auditory cortex in man: Evaluation and topography of the middle latency components. Electroencephalography and Clinical Neurophysiology, 92, 204-14.
50. Ludman, C. N., Summerfield, A. Q., Hall, D., Elliott, M., Foster, J., Hykin, J. L., Bowtell, R., & Morris, P. G. (2000). Lip-reading ability and patterns of cortical activation studied using fMRI. British Journal of Audiology, 34, 225-30.
51. Massaro, D. W. (1987). Speech Perception by Ear and Eye: A Paradigm for Psychological Inquiry. Hillsdale, NJ: Lawrence Erlbaum.
52. Massaro, D. W. (1989). Multiple book review of Speech Perception by Ear And Eye: A Paradigm for Psychological Inquiry. Behavioral and Brain Sciences, 12, 741-54.
53. Massaro, D. W. (1999). Speechreading: Illusion or window into pattern recognition. Trends in Cognitive Sciences, 3, 310-17.
54. Massaro, D. W., Cohen, M. M., & Smeele, P. M. (1996). Perception of asynchronous and conflicting visual and auditory speech. Journal of the Acoustical Society of America, 100, 1777-86.
55. McGurk, H. & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746-8.
56. Meredith, M. A. (2002). On the neuronal basis for multisensory convergence: a brief overview. Cognitive Brain Research, 14, 31-40.
57. Mesulam, M. M. (1994). Neurocognitive networks and selectively distributed processing. Revue Neurologique, 150, 564-9.
58. Mesulam, M. M. (1998). From sensation to cognition. Brain, 121, 1013-52.
59. Mitzdorf, U. (1986). The physiological causes of VEP: Current source density analysis of electrically and visually evoked potential. In R. Q. Cracco & I. Bodis-Wollner (eds.), Evoked Potentials (pp. 141-54). New York: Alan R. Liss Inc.
60. Molholm, S., Ritter, W., Murray, M. M., Javitt, D. C., Schroeder, C. E., & Foxe, J. J. (2002). Multisensory auditory-visual interactions during early sensory processing in humans: A high-density electrical mapping study. Cognitive Brain Research, 14, 115-28.
61. Mottonen, R., Krause, C. M., Tiippana, K., & Sams, M. (2002). Processing of changes in visual speech in the human auditory cortex. Cognitive Brain Research, 13, 417-25.
62. Moutoussis, K. & Zeki, S. (1997). A direct demonstration of perceptual asynchrony in vision. Proceedings of the Royal Society of London, Series B: Biological Sciences, 264, 393-9.
63. Munhall, K. G. & Tohkura, Y. (1998). Audiovisual gating and the time course of speech perception. Journal of the Acoustical Society of America, 104, 530-9.
64. Munhall, K. G., Gribble, P., Sacco, L., & Ward, M. (1996). Temporal constraints on the McGurk effect. Perception & Psychophysics, 58, 351-62.
65. Näätänen, R. & Picton, T. (1987). The N1 wave of the human electric and magnetic response to sound: A review and an analysis of the component structure. Psychophysiology, 24, 375-425.
66. Nobre, A. C., Allison, T., & McCarthy, G. (1994). Word recognition in the human inferior temporal lobe. Nature, 372, 260-3.
67. Paulesu, E., Perani, D., Blasi, V., Silani, G., Borghese, N. A., De Giovanni, U., Sensolo, S., & Fazio, F. (in press). A functional-anatomical model for lip-reading. Journal of Cognitive Neuroscience.
68. Picton, T. W., Bentin, S., Berg, P., Donchin, E., Hillyard, S. A., Johnson, R., Jr., Miller, G. A., Ritter, W., Ruchkin, D. S., Rugg, M. D., & Taylor, M. J. (2000). Guidelines for using human event-related potentials to study cognition: Recording standards and publication criteria. Psychophysiology, 37, 127-52.
69. Ponton, C. W. & Eggermont, J. J. (2001). Of kittens and kids: Altered cortical maturation following profound deafness and cochlear implant use. Audiology & Neuro-otology, 6, 363-80.
70. Ponton, C. W., Don, M., Waring, M. D., Eggermont, J. J., & Masuda, A. (1993). Spatio-temporal source modeling of evoked potentials to acoustic and cochlear implant stimulation. Electroencephalography and Clinical Neurophysiology, 88, 478-93.
71. Puce, A., Allison, T., Asgari, M., Gore, J. C., & McCarthy, G. (1996). Differential sensitivity of human visual cortex to faces, letterstrings, and textures: A functional magnetic resonance imaging study. Journal of Neuroscience, 16, 5205-15.
72. Puce, A., Allison, T., Bentin, S., Gore, J. C., & McCarthy, G. (1998). Temporal cortex activation in humans viewing eye and mouth movements. Journal of Neuroscience, 18, 2188-99.
73. Raij, T., Uutela, K., & Hari, R. (2000). Audiovisual integration of letters in the human brain. Neuron, 28, 617-25.
74. Rauschecker, J. P. (1998). Cortical processing of complex sounds. Current Opinion in Neurobiology, 8, 516-21.
75. Rauschecker, J. P. & Tian, B. (2000). Mechanisms and streams for processing of "what" and "where" in auditory cortex. Proceedings of the National Academy of Sciences of the United States of America, 97, 11800-6.
76. Rosenblum, L. D. (2002). The perceptual basis for audiovisual speech integration. Proceedings of the 7th International Conference on Spoken Language Processing, September 16-20, Denver, CO.
77. Saldaňa, H. M. & Rosenblum, L. D. (1994). Selective adaptation in speech perception using a compelling audiovisual adaptor. Journal of the Acoustical Society of America, 95, 3658-61.
78. Scherg, M. & Von Cramon, D. (1985). Two bilateral sources of the late AEP as identified by a spatio-temporal dipole model. Electroencephalography and Clinical Neurophysiology, 62, 32-44.
79. Scherg, M. & Von Cramon, D. (1986). Evoked dipole source potentials of the human auditory cortex. Electroencephalography and Clinical Neurophysiology, 65, 344-60.
80. Schwartz, J.-L., Robert-Ribes, J., & Escudier, P. (1998). Ten years after Summerfield: A taxonomy of models for audio-visual fusion in speech perception. In R. Campbell, B. Dodd, & D. Burnham (eds.), Hearing by Eye II: The Psychology of Speechreading and Auditory-visual Speech (pp. 85-108). Hove, UK: Psychology Press.
81. Scott, S. K. & Johnsrude, I. S. (2003). The neuroanatomical and functional organization of speech perception. Trends in Neurosciences, 26, 100-7.
82. Scott, S. K., Blank, C. C., Rosen, S., & Wise, R. J. (2000). Identification of a pathway for intelligible speech in the left temporal lobe. Brain, 123, 2400-6.
83. Sekiyama, K. (1997). Cultural and linguistic factors in audiovisual speech processing: The McGurk effect in Chinese subjects. Perception & Psychophysics, 59, 73-80.
84. Sekiyama, K. & Tohkura, Y. (1991). McGurk effect in non-English listeners: Few visual effects for Japanese subjects hearing Japanese syllables of high auditory intelligibility. Journal of the Acoustical Society of America, 90, 1797-1805.
85. Sekiyama, K. & Tohkura, Y. (1993). Inter-language differences in the influence of visual cues in speech perception. Journal of Phonetics, 21, 427-44.
86. Sharma, A. & Dorman, M. F. (1999). Cortical auditory evoked potential correlates of categorical perception of voice-onset time. Journal of the Acoustical Society of America, 106, 1078-83.
87. Sharma, A. & Dorman, M. F. (2000). Neurophysiologic correlates of cross-language phonetic perception. Journal of the Acoustical Society of America, 107, 2697-703.
88. Sharma, A., Marsh, C. M., & Dorman, M. F. (2000). Relationship between N1 evoked potential morphology and the perception of voicing. Journal of the Acoustical Society of America, 108, 3030-5.
89. Singer, W. (1998). Consciousness and the structure of neuronal representations. Philosophical Transactions of the Royal Society of London, Series B: Biological Sciences, 353, 1829-40.
90. Stein, B. E. (1998). Neural mechanisms for synthesizing sensory information and producing adaptive behaviors. Experimental Brain Research, 123, 124-35.
91. Stein, B. E. & Meredith, M. A. (1993). The Merging of the Senses. Cambridge, MA: MIT Press.
92. Steinschneider, M., Volkov, I. O., Noh, M. D., Garell, P. C., & Howard, M. A., 3rd (1999). Temporal encoding of the voice onset time phonetic parameter by field potentials recorded directly from human auditory cortex. Journal of Neurophysiology, 82, 2346-57.
93. Sumby, W. H. & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212-15.
94. Summerfield, Q. (1987). Some preliminaries to a comprehensive account of audio-visual speech perception. In B. Dodd & R. Campbell (eds.), Hearing by Eye: The Psychology of Lip-Reading (pp. 3-52). Hove, UK: Psychology Press.
95. Summerfield, Q. & McGrath, M. (1984). Detection and resolution of audio-visual incompatibility in the perception of vowels. Quarterly Journal of Experimental Psychology: Human Experimental Psychology, 36, 51-74.
96. Tong, F., Nakayama, K., Moscovitch, M., Weinrib, O., & Kanwisher, N. (2000). Response properties of the human fusiform face area. Cognitive Neuropsychology, 17, 257-79.
97. Trout, J. D. (2001). The biological basis of speech: What to infer from talking to the animals. Psychological Review, 108, 523-49.
98. Ungerleider, L. G. & Haxby, J. V. (1994). "What" and "where" in the human brain. Current Opinion in Neurobiology, 4, 157-65.
99. Vaughan, H. & Arezzo, J. (1988). The neural basis of event-related potentials. In T. W. Picton (ed.), Human Event-Related Potentials (pp. 45-96). Amsterdam: Elsevier Science Publishers.
100. Walker, S., Bruce, V., & O'Malley, C. (1995). Facial identity and facial speech processing: Familiar faces and voices in the McGurk effect. Perception & Psychophysics, 57, 1124-33.
101. Watson, J. D., Myers, R., Frackowiak, R. S., Hajnal, J. V., Woods, R. P., Mazziotta, J. C., Shipp, S., & Zeki, S. (1993). Area V5 of the human brain: Evidence from a combined study using positron emission tomography and magnetic resonance imaging. Cerebral Cortex, 3, 79-94.
102. Wise, R. J., Scott, S. K., Blank, S. C., Mummery, C. J., Murphy, K., & Warburton, E. A. (2001). Separate neural subsystems within "Wernicke's area." Brain, 124, 83-95.
103. Yehia, H., Kuratate, T., & Vatikiotis-Bateson, E. (1999). Using speech acoustics to drive facial motion. Proceedings of the International Congress of Phonetic Sciences (ICPhS 1999) (pp. 631-4). San Francisco, CA: IPA.
104. Yehia, H., Rubin, P., & Vatikiotis-Bateson, E. (1998). Quantitative association of vocal-tract and facial behavior. Speech Communication, 26, 23-43.
105. Yvert, B., Crouzeix, A., Bertrand, O., Seither-Preisler, A., & Pantev, C. (2001). Multiple supratemporal sources of magnetic and electric auditory evoked middle latency components in humans. Cerebral Cortex, 11, 411-23.
106. Zeki, S. (1998). Parallel processing, asynchronous perception, and a distributed system of consciousness in vision. Neuroscientist, 4, 365-72.
107. Zeki, S. (2001). Localization and globalization in conscious vision. Annual Review of Neuroscience, 24, 57-86.