Reading speech from still and moving faces: The neural substrates of visible speech

Journal of Cognitive Neuroscience

Volumn 15, Issue 1, 2003, Pages 57-70

  Calvert, Gemma A ^a   Campbell, Ruth ^b  

^a UNIVERSITY OF OXFORD   (United Kingdom)

^b NONE

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ARTICLE; AUDITORY CORTEX; CLINICAL ARTICLE; FACIES; FEMALE; HUMAN; IMAGERY; MALE; NUCLEAR MAGNETIC RESONANCE IMAGING; PHONEME; PRIORITY JOURNAL; READING; RECOGNITION; RELIABILITY; SPEECH PERCEPTION; VISUAL CORTEX;

ACOUSTIC STIMULATION; ADULT; AUDITORY CORTEX; BRAIN MAPPING; FACE; FEMALE; HUMANS; MAGNETIC RESONANCE IMAGING; MALE; PHOTIC STIMULATION; READING; SPEECH; SPEECH PERCEPTION; TIME FACTORS;

EID: 0037240789     PISSN: 0898929X     EISSN: None     Source Type: Journal    
DOI: 10.1162/089892903321107828     Document Type: Article

References (85)

1. Allison, T., Puce, A., & McCarthy, G. (2000). The neurobiology of social cognition. Trends in Cognitive Sciences, 4, 267-279.
2. Bernstein, L. E., Auer, E. T., Jr., Moore, J. K., Ponton, C. W., Don, M., & Singh, M. (2002). Visual speech perception without primary auditory cortex activation. NeuroReport, 13, 311-315.
3. Binkofski, F., Amunts, K., Stephan, K. M., Posse, S., Schormann, T., Freund, H. J., Zilles, K., & Seitz, R. J. (2000). Broca's region subserves imagery of motion: A combined cytoarchitectonic and fMRI study. Human Brain Mapping, 11, 273-285.
4. Bonda, E., Petrides, M., Ostry, D., & Evans, A. (1996). Specific involvement of human parietal systems and the amygdala in the perception of biological motion. Journal of Neuroscience, 16, 3737-3744.
5. Brunet, E., Sarfati, Y., Hardy-Baylé M.-C., & Decety, J. (2000). A PET investigation of the attributions of intention with a nonverbal task. Neuroimage, 11, 157-166.
6. Buccino, G., Binkowski, F., Fink, G. R., Fadiga, L., Fogassi, L., Gallese, V., Seitz, R. J., Zilles, K., Rizzolatti, G., & Freund, H.-J. (2001). Action observation activates premotor and parietal areas in a somatotopic manner: An fMRI study. European Journal of Neuroscience, 13, 400-404.
7. Burnham, D. K. (1993). Visual recognition of mother by young infants: Facilitation by speech. Perception, 22, 1133-1153.
8. Callan, D. E., Callan, A., Kroos, E., & Vatikiotis-Bateson, E. (2000). Multimodal contributions to speech perception revealed by independent component analysis: A single sweep EEG study. Cognitive Brain Research, 10, 349-353.
9. Calvert, G. A., Brammer, M. J., Bullmore, E. T., Campbell, R., Iversen, S. D., & David, A. S. (1999). Response amplification in sensory-specific cortices during cross-modal binding. NeuroReport, 10, 2619-2623.
10. Calvert, G. A., Bullmore, E. T., Brammer, M. J., Campbell, R., Williams, S. C., McGuire, P. K., Woodruff, P. W., Iversen, S. D., & David, A. S. (1997). Activation of auditory cortex during silent lipreading. Science, 276, 593-596.
11. Calvert, G. A., Campbell, R., & Brammer, M. J. (2000). Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Current Biology, 10, 649-657.
12. Campbell, R. (1992). The neuropsychology of lipreading. Philosophical Transactions of the Royal Society of London, Series B: Biological Sciences, 335, 39-45.
13. Campbell, R., Brooks, B., de Haan, E., & Roberts, T. (1996). Dissociating face processing skills: Decisions about lip-read speech, expression and identity. Quarterly Journal of Experimental Psychology, Section A, 49, 295-314.
14. Campbell, R., De Gelder, B., & de Haan, E. H. F. (1996). The laterality of lipreading: A second look. Neuropsychologia, 34, 1235-1240.
15. Campell, R., & Dodd, B. (1980). Hearing by eye. Quarterly Journal of Experimental Psychology, 32, 85-99.
16. Campbell, R., Landis, T., & Regard, M. (1986). Face recognition and lipreading. A neurological dissociation. Brain, 109, 509-521.
17. Campbell, R., MacSweeney, M., Surguladze, S., Calvert, G. A., McGuire, P. K., Brammer, M. J., David, A. S., & Suckling, J. (2001). Cortical substrates for the perception of face actions: An fMRI study of the specificity of activation for seen speech and for meaningless lower-face acts (gurning). Cognitive Brain Research, 12, 233-243.
18. Campbell, R., Zihl, J., Massaro, D. W., Munhall, K., & Cohen, M. M. (1997). Speechreading in the akinotopsic patient, L. M. Brain, 120, 1793-1803.
19. Cathiard, M.-A., & Tiberghien, G. (1994). Le visage de la parole: Une coherence bimodale temporalle ou configurationelle? Psychologie Francaise, 39, 357-374.
20. Cathiard, M.-A., Tiberghien, G., & Abry, C. (1992). Face and profile identification skills for liprounding in normal-hearing French subjects. Bulletin de la Communication Parlée, 2, 43-58.
21. Crozier, S., Sirigu, A., Lehericy, S., van der Moortele, P.-F., Pillon, B., Grafman, J., Agid, Y., Dubois, B., & LeBihan, D. (1999). Distinct prefrontal activations in processing sequence at the sentence and script level: An fMRI study. Neuropsychologia, 37, 1469-1476.
22. Decety, J., & Grezes, J. (1999). Neural mechanisms subserving the perception of human actions. Trends in Cognitive Sciences, 3, 172-178.
23. de Sa, V. R., & Ballard, D. H. (1997). Perceptual learning from cross-modal feedback. Psychology of Learning and Motivation, 36, 309-351.
24. Dodd, B. (1979). Lip reading in infants: Attention to speech presented in-and-out-of-synchrony. Cognitive Psychology, 11, 478-484.
25. Dodd, B., & Burnham, D. (1988). Processing speechread information. New reflections on speechreading. Volta Review, 90, 45-60.
26. Fowler, C. A., & Rosenblum, L. D. (1991). The perception of phonetic gestures. In I. G. Mattingly & M. Studdert-Kennedy (Eds.), Modularity and the motor theory of speech perception (pp. 33-59). Hillsdale, NJ: Erlbaum.
27. Friston, K., Worsley, K., Frackowiak, R., Mazziotta, J., & Evans, A. (1994). Assessing the significance of focal activations using their spatial extent. Human Brain Mapping, 1, 214-220.
28. Gallese, V., Fadiga, L., Fogassi, L., & Rizzolatti, G. (1996). Action recognition in the premotor cortex. Brain, 119, 593-609.
29. Grant, K. W., & Seitz, P. F. (2000). The use of visible speech cues for improving auditory detection of spoken sentences. Journal of the Acoustical Society of America, 108, 1197-1208.
30. Green, K. P. (1998). The use of auditory and visual information during phonetic processing: Implications for theories of speech perception. In R. Campbell, B. Dodd, & D. Burnham (Eds.), Hearing by eye II: The psychology of speechreading and audiovisual speech. Hove, UK: Psychology Press.
31. Green, K. P., Kuhl, P. K., Meltzoff, A. N., & Stevens, K. N. (1991). Integrating speech information across talkers, gender, and sensory, modality: Female faces and male voices in the McGurk effect. Perception and Psychophysics, 50, 524-536.
32. Grèzes, J., & Decety, J. (2001). Functional anatomy of execution, mental simulation, observation, and verb generation of actions: A meta-analysis. Human Brain Mapping, 12, 1-19.
33. Grossman, E., Donnelly, M., Price, R., Pickens, D., Morgan, V., Neighbor, G., & Blake, R. (2000). Brain areas involved in perception of biological motion. Journal of Cognitive Neuroscience, 12, 711-720.
34. Haist, F., So, G., Wild, K., Faber, T. L., Popp, C. A., & Morris, R. D. (2001). Linking sight and sound: fMRI evidence of primary auditory cortex activation during visual word recognition. Brain and Language, 76, 340-350.
35. Hasnain, M. K., Fox, P. T., & Woldorff, M. G. (1998). Intersubject variability of functional areas in the human visual cortex. Human Brain Mapping, 6, 301-315.
36. Haxby, J. V., Hoffman, E. A., & Gobbini, M. I. (2000). The distributed human neural system for face perception. Trends in Cognitive Sciences, 4, 223-233.
37. Hermsdörfer, J., Goldenburg, G., Wachsmuth, C., Conrad, B., Ceballos-Baumann, A. O., Partenstein, P., Schwaiger, M., & Boecker, H. (2001). Cortical correlates of gesture processing: Clues to the cerebral mechanisms underlying apraxia during the imitation of meaningless gestures. Neuroimage, 14, 149-161.
38. Howard, R., Brammer, M. J., Wright, I., Woodruff, P. W. R., Bullmore, E. T., & Zeki, S. (1996). A direct demonstration of functional specialisation within motion-related visual and auditory cortex of the human brain. Current Biology, 6, 1015-1019.
39. Iacoboni, M., Woods, R. P., Brass, M., Bekkering, H., Mazziotta, J. C., & Rizzolatti, G. (1999). Cortical mechanisms of human imitation. Science, 286, 2526-2528.
40. Jeffers, J., & Barley, M. (1971). Speechreading (lipreading). Springfield, IL: Thomas.
41. Jenkinson, M., & Smith, S. M. (2001). A global optimisation method for robust affine registration of brain images. Medical Image Analysis, 5, 143-156.
42. Jordan, T. R., & Bevan, K. M. (1997). Seeing and hearing rotated faces: Influences of facial orientation on visual and audio-visual speech recognition. Journal of Experimental Psychology: Human Perception and Performance, 23, 388-403.
43. Jordan, T. R., McCotter, M. V., & Thomas, S. M. (2000). Visual and audio-visual speech perception with color and gray scale facial images. Perception and Psychophysics, 62, 1394-1404.
44. Jordan, T. R., & Sergeant, P. C. (2000). Effects of distance on visual and audio-visual speech recognition. Language and Speech, 43, 107-124.
45. Kanwisher, N., McDermott, J., & Chun, M. (1997). The fusiform face area: A module in human extrastriate cortex specialized for the perception of faces. Journal of Neuroscience, 17, 4302-4311.
46. Kanwisher, N., & Moscovitch, M. (2000). The cognitive neuroscience of face processing: An introduction. Cognitive Neuropsychology, 1/2/3, 1-13.
47. Kourtzi, Z., & Kanwisher, N. (2000). Activation in human MT/MST by static images with implied motion. Journal of Cognitive Neuroscience, 12, 48-55.
48. Kuhl, P. K., & Meltzoff, A. N. (1982). The bimodal perception of speech in infancy. Science, 218, 1138-1141.
49. Levänen, S. (1999). Neuromagnetic studies of human auditory cortex function and reorganization. Scandinavian Audiology, 27, 2-6.
50. Lewkowicz, D. J. (1996). Infants' response to the audible and visible properties of the human face: I. Role of lexical-syntactic content, temporal synchrony, gender, and manner of speech. Developmental Psychobiology, 32, 347-366.
51. Lewkowicz, D. J. (1998). Infants' response to the audible and visible properties of the human face: II. Discrimination of differences between singing and adult-directed speech. Developmental Psychobiology, 32, 261-274.
52. Liberman, A. M., & Whalen, D. H. (2000). On the relation of speech to language. Trends in Cognitive Sciences, 4, 187-196.
53. Ludman, C. N., Summerfield, A. Q., Hall, D., Elliott, M., Foster, J., Hykin, J. L., Bowtell, R., & Morris, P. G. (2000). Lip-reading ability and patterns of cortical activation studied using fMRI. British Journal of Audiology, 34, 225-230.
54. MacSweeney, M., Amaro, E., Calvert, G. A., Campbell, R., David, A. S., McGuire, P. K., Williams, S. C. R., Woll, B., & Brammer, M. J. (2000). Activation of auditory cortex by silent speechreading in the absence of scanner noise: An event-related fMRI study. NeuroReport, 11, 1729-1734.
55. MacSweeney, M., Campbell, R., Calvert, A., McGuire, P. K., David, A. S., Suckling, J., Andrew, C., Woll, B., & Brammer, M. J. (2001). Dispersed activation in left temporal cortex for speechreading in congenitally deaf people. Proceedings of the Royal Society of London, Series B: Biological Sciences, 268, 451-457.
56. MacSweeney, M., Woll, B., Campbell, R., McGuire, P. K., David, A. S., Williams, S. C. R., Suckling, J., Calvert, G. A., & Brammer, M. J. (2002). Neural systems underlying British Sign Language and audiovisual English processing in native users. Brain, 125, 1583-1593.
57. Massaro, D. W. (1998). Perceiving talking faces: From speech perception to a behavioral principle. Cambridge: MIT Press.
58. Massaro, D. (1999). Speechreading: Illusion or window into pattern recognition? Trends in Cognitive Sciences, 3, 310-317.
59. Massaro, D. W., & Cohen, M. M. (1996). Perceiving speech from inverted faces. Perception and Psychophysics, 58, 1047-1065.
60. McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746-748.
61. Munhall, K. G., & Vatikiotis-Bateson, E. (1998). The moving face during speech communication. In R. Campbell, B. Dodd, & D. Burnham (Eds.), Hearing by eye II: Advances in the psychology of speechreading and auditory-visual speech (pp. 123-139). Hove, UK: Psychology Press.
62. Olson, I. R., Gatenby, J. G., & Gore, J. C. (2002). A comparison of bound and unbound audio-visual information processing in the human cerebral cortex. Brain Research. Cognitive Brain Research, 14, 129-138.
63. Passingham, R. (1993). The frontal lobes and voluntary action. Oxford, UK: Oxford University Press.
64. Puce, A., Allison, T., Bentin, S., Gore, J. C., & McCarthy, G. (1998). Temporal cortex activation in humans viewing eye and mouth movements. Journal of Neuroscience, 18, 2188-2199.
65. Puce, A., Castiello, U., Syngeniotis, A., & Abbott, D. (2001). The human STS integrates form and motion. Neuroimage, 13, S931.
66. Reisberg, D., McLean, J., & Goldfield, A. (1987). Easy to hear but hard to understand: A lipreading advantage with intact auditory stimuli. In B. Dodd & R. Campbell (Eds.), Hearing by eye: The psychology of lip-reading (pp. 97-113). Hillsdale, NJ: Erlbaum.
67. Rizzolatti, G., & Arbib, M. A. (1999). From grasping to speech: Imitation might provide a missing link. Trends in Neurosciences, 22, 152.
68. Rizzolatti, G., Fadiga, L., Gallese, V., & Fogassi, L. (1996). Premotor cortex and the recognition of motor actions. Brain Research. Cognitive Brain Research, 3, 131-141.
69. Rosenblum, L. D., Johnson, J. A., & Saldaña, H. M. (1996). Point-light facial displays enhance comprehension of speech in noise. Journal of Speech and Hearing Research, 39, 1159-1170.
70. Rosenblum, L. D., & Saldaña, H. M. (1996). An audio-visual test of kinematic primitives for visual speech perception. Journal of Experimental Psychology: Human Perception and Performance, 22, 318-331.
71. Rosenblum, L. D., Schmuckler, M. A., & Johnson, J. A. (1997). The McGurk effect in infants. Perception and Psychophysics, 59, 347-357.
72. Sams, M., Aulanko, R., Hämäläinen, M., Hari, R., Lounasmaa, O. V., Lu, S.-T., & Simola, J. (1991). Seeing speech: Visual information from lip movements modifies activity in the human auditory cortex. Neuroscience Letters, 127, 141-145.
73. Sams, M., & Levänen, S. (1996). When and where are the heard and seen speech integrated? In D. G. Stork & M. E. Henneke (Eds.), Speechreading in humans and machines (pp. 232-238). Berlin: Springer-Verlag.
74. Schweinberger, S. R., & Soukup, G. R. (1998). Asymmetric relationships among perceptions of facial identity, emotion, and facial speech. Journal of Experimental Psychology: Human Perception and Performance, 24, 1748-1765.
75. Sekiyama, K. (1997). Cultural and linguistic factors in audiovisual speech processing: The McGurk effect in Chinese subjects. Perception and Psychophysics, 59, 73-80.
76. Sekiyama, K., & Tohkura, Y. (1993). Inter-language differences in the influence of visual cues in speech perception. Journal of Phonetics, 21, 427-444.
77. Senior, C., Barnes, J., Giampetro, V., Simmons, A., Bullmore, E. T., Brammer, M., & David, A. S. (2000). Functional neuroanatomy of implicit motion perception or representational momentum. Current Biology, 10, 16-22.
78. Sumby, W. H., & Pollack, I. (1954). Visual contributions to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212-215.
79. Summerfield, A. Q. (1987). Some preliminaries to a comprehensive account of audiovisual speech perception. In B. Dodd & R. Campbell (Eds.), Hearing by eye: The psychology of lipreading (pp. 3-51). Hillsdale, NJ: Erlbaum.
80. Summerfield, A. Q. (1992). Lipreading and audio-visual speech perception. Philosophical Transactions of the Royal Society of London, Series B: Biological Sciences, 335, 71-78.
81. Surguladze, S., Calvert, G., Brammer, M. J., Campbell, R., Bullmore, E. T., & David, A. S. (2001). Audio-visual speech perception in schizophrenia: An fMRI study. Psychiatry Research: Neuroimaging, 106, 1-14.
82. Werker, J. F., Frost, P., & McGurk, H. (1992). Cross-language influences on bimodal speech perception. Canadian Journal of Psychology, 46, 551-568.
83. Whalen, D. H., Irwin, J. R., & Fowler, C. A. (in press). A sex difference in audio-visual integration of speech.
84. Woolrich, M., Ripley, B., Brady, J., & Smith, S. (2001). Temporal autocorrelation in univariate linear modelling of fMRI data. Neuroimage, 14, 1370-1386.
85. Yehia, H., Rubin, P., & Vatikiotis-Bateson, E. (1998). Quantitative association of vocal-tract and facial behavior. Speech Communication, 26, 23-43.