Some behavioral and neurobiological constraints on theories of audiovisual speech integration: A review and suggestions for new directions

Seeing and Perceiving

Volumn 24, Issue 6, 2011, Pages 513-539

  Altieri, Nicholas ^a   Pisoni, David B ^b   Townsend, James T ^b  

^a UNIVERSITY OF OKLAHOMA   (United States)

^b INDIANA UNIVERSITY   (United States)

Author keywords

AUDIO VISUAL SPEECH PERCEPTION; MCGURK EFFECT; MULTISENSORY ENHANCEMENT

Indexed keywords

AUDIO-VISUAL SPEECH; AUDITORY MODALITY; BRAIN CIRCUITS; BRAIN REGIONS; DYNAMIC MODELING TOOLS; FEATURE-BASED THEORIES; FILTER FUNCTION; MCGURK EFFECT; MULTISENSORY; OPTICAL PARAMETER; PROCESSING MODEL; SPEECH PERCEPTION; VISUAL SPEECH; VOCAL-TRACTS;

BRAIN; DATA PROCESSING;

INTEGRATION;

BEHAVIOR; BIOLOGICAL MODEL; HUMAN; PHONETICS; PHYSIOLOGY; REVIEW; SPEECH; SPEECH PERCEPTION; VISION; VOCAL CORD;

BEHAVIOR; HUMANS; MODELS, NEUROLOGICAL; PHONETICS; SPEECH; SPEECH PERCEPTION; VISUAL PERCEPTION; VOCAL CORDS;

EID: 84858175159     PISSN: 18784755     EISSN: 18784763     Source Type: Journal    
DOI: 10.1163/187847611X595864     Document Type: Review

References (89)

1. Allman, B. J., Keniston, L. P. and Meredith, M. A. (2009). Not just for bimodal neurons anymore: the contribution of unimodal neurons to cortical multisensory processing, Brain Topography 21, 157-167.
2. Altieri, N. (2010). Toward a Unified Theory of Audiovisual Integration in Speech Perception, Indiana University, Bloomington, IN, USA.
3. Auer, E. T. Jr. and Bernstein, L. E. (2007). Enhanced visual speech perception in individuals with early onset hearing impairment, J. Speech Hearing Lang. Res. 50, 1157-1165. (Pubitemid 350058546)
4. Bernstein, L. E. (2005). Phonetic perception by the speech perceiving brain, in: The Handbook of Speech Perception, D. B. Pisoni and R. E. Remez (Eds), pp. 79-98. Blackwell Publishing, Malden, MA, USA.
5. Bernstein, L. E., Auer, E. T., Moore, J. K., Ponton, C., Don, M. and Singh, M. (2002). Visual speech perception without primary auditory cortex activation, NeuroReport 13, 311-315. (Pubitemid 34183573)
6. Bernstein, L. E., Auer, E. T. and Moore, J. K. (2004). Audiovisual speech binding: convergence or association?, in: Handbook of Multisensory Processing, G. A. Calvert, C. Spence and B. E. Stein (Eds), pp. 203-223. MIT Press, Cambridge, MA, USA.
7. Bernstein, L. E., Lu, Z. L. and Jiang, J. (2008). Quantified acoustic-optical speech signal incongruity identifies cortical sites of audiovisual speech processing, Brain Res. 1242, 172-184.
8. Bergeson, T. R. and Pisoni, D. B. (2004). Audiovisual speech perception in deaf adults and children following cochlear implantation, in: Handbook of Multisensory Processes, G. A. Calvert, C. Spence and B. E. Stein (Eds), pp. 153-176. MIT Press, Cambridge, MA, USA.
9. Bergeson, T. R., Pisoni, D. B. and Davis, R. A. O. (2003). A longitudinal study of audiovisual speech perception by children with hearing loss who have cochlear implants, Volta Rev. 163, 347-370. (Pubitemid 33814509)
10. Besle, J., Fort, A., Delpuech, C. and Giard, M.-H. (2004). Bimodal speech: early suppressive visual effects in the human auditory cortex, Europ. J. Neurosci. 20, 2225-2234. (Pubitemid 39331143)
11. Borg, I. and Lingoes, J. (1987). Multidimensional Similarity Structure Analysis. Springer, New York, NY, USA.
12. Braida, L. D. (1991). Crossmodal integration in the identification of consonant segments, Quatr. J. Exper. Psychol. 43A, 647-677.
13. Calvert, G. A. and Campbell, R. (2003). Reading speech from still and moving faces: the neural substrates of visible speech, J. Cognit. Neurosci. 15, 57-70. (Pubitemid 36106263)
14. Calvert, G. A. and Lewis, J. W. (2004). Hemodynamic studies of audiovisual interactions, in: Handbook of Multisensory Processes, G. A. Calvert, C. Spence and B. E. Stein (Eds), pp. 483-502. MIT Press, Cambridge, MA, USA.
15. Calvert, G. A., Bullmore, E. T., Brammer, M. J., Campbell, R., Iversen, S. D., Woodruff, P., McGuire, P. Williams, S. and David, A. S. (1997). Activation of auditory cortex during silent lip-reading, Science 276, 593-596. (Pubitemid 27193683)
16. Calvert, G. A., Campbell, R. and Brammer, M. J. (2000). Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex, Curr. Biol. 10, 649-657. (Pubitemid 30354700)
17. Chandrasekaran, C. and Ghazanfar, A. A. (2009). Different neural frequency bands integrate faces and voices differently in the superior temporal sulcus, J. Neurophysiol. 101, 773-788.
18. Chandrasekaran, C., Trubanova, A., Stillittano, S., Caplier, A. and Ghazanfar, A. A. (2009). The natural statistics of audiovisual speech, PLoS Computat. Biol. 5, e1000436.
19. Colonius, H. and Diederich, A. (2010). The optimal time window of visual-auditory integration: a reaction time analysis, Frontiers Integrat. Neurosci. 4, 1-8.
20. Conrey, B. L. and Pisoni, D. B. (2006). Auditory-visual speech perception and synchrony detection for speech and non-speech signals, J. Acoust. Soc. Amer. 119, 4065-4073. (Pubitemid 43886378)
21. Dahl, C. D., Logothetis, N. K. and Kayser, C. (2009). Spatial organization of multisensory responses in temporal association cortex, J. Neurosci., 29, 11924-11932.
22. Diederich, A. and Colonius, H. (2008). Crossmodal interaction in saccadic reaction time: separating multisensory from warning effects in the time window of integration model, Exper. Brain Res. 186, 1-22.
23. Diederich, A. and Colonius, H. (2009). Crossmodal interaction in speeded responses: time window of integration model, Prog. Brain Res. 174, 119-135.
24. Dodd, B., McIntosh, B., Erdener, D. and Burnham, D. (2008). Perception of the auditory-visual illusion in speech perception by children with phonological disorders, Clin. Linguist. Phonet. 22, 69-82.
25. Drullman, R. (1995). Temporal envelope and fine structure cues for speech intelligibility, J. Acoust. Soc. Amer. 97, 585-592.
26. Drullman, R., Festen, J. M. and Plomp, R. (1994). Effect of reducing slow temporal modulations on speech reception, J. Acoust. Soc. Amer. 95, 2670-2680. (Pubitemid 24152861)
27. Eidels, A., Houpt, J., Altieri, N., Pei, L. and Townsend, J. T. (2011). Nice guys finish fast and bad guys finish last: a theory of interactive parallel processing, J. Math. Psychol. 55, 176-190.
28. Erber, N. P. (1969). Interaction of audition and vision in the recognition of oral speech stimuli, J. Speech Hearing Res. 12, 423-425.
29. Fowler, C. A. and Dekle, D. J. (1991). Listening with eye and hand: Cross-modal contributions to speech perception, J. Exper. Psychol.: Human Percept. Perform. 17, 816-828.
30. Fowler, C. A. and Rosenblum, L. D. (1991). Perception of the phonetic gesture, in: Modularity and the Motor Theory of Speech Perception, I. G. Mattingly and M. Studdert-Kennedy (Eds), pp. 33-59. Lawrence Erlbaum, Hillsdale, NJ, USA.
31. Galantucci, B., Fowler, C. A. and Turvey, M. T. (2006). The motor theory of speech perception reviewed, Psychonom. Bull. Rev. 13, 361-377. (Pubitemid 44825057)
32. Ghazanfar, A. A., Chandrasekaran, C. and Logothetis, N. K. (2008). Interactions between the superior temporal sulcus and auditory cortex mediate dynamic face/voice integration in rhesus monkeys, J. Neurosci. 28, 4457-4469.
33. Gilbert, C. D., Sigman, M. and Crist, R. E. (2001). The neural basis of perceptual learning, Neuron 31, 681-697. (Pubitemid 32907367)
34. Gori, M., Del Viva, M., Sandini, G. and Burr, D. C. (2008). Young children do not integrate visual and haptic form information, Curr. Biol. 18, 694-698.
35. Grant, K. W. (2002). Measures of auditory-visual integration for speech understanding: a theoretical perspective (L), J. Acoust. Soc. Amer. 112, 30-33. (Pubitemid 34747734)
36. Grant, K.W. and Seitz, P. F. (1998). Measures of auditory-visual integration in nonsense syllables and sentences, J. Acoust. Soc. Amer. 104, 2438-2450. (Pubitemid 28463139)
37. Grant, K.W., Tufts, J. B. and Greenberg, S. (2007). Integration efficiency for speech perception within and across sensory modalities by normal-hearing and hearing impaired individuals, J. Acoust. Soc. Amer. 121, 1164-1176. (Pubitemid 46192715)
38. Green, K. (1996). The use of auditory and visual information in phonetic perception, in: Speech Reading by Humans and Machines, D. Stork and M. Hennecke (Eds), pp. 55-78. Springer-Verlag, Berlin, Germany.
39. Green, K. P. and Miller, J. L. (1985). On the role of visual rate information in phonetic perception, Percept. Psychophys. 38, 269-276. (Pubitemid 16176689)
40. Grossberg, S., Boardman, I. and Cohen, M. (1997). Neural dynamics of variable-rate speech categorization, J. Exper. Psychol.: Human Percept. Perform. 23, 481-503. (Pubitemid 127458887)
41. Hall, D. A., Fussell, C. and Summerfield, Q. (2005). Reading fluent speech from talking faces: typical brain networks and individual differences, J. Cognit. Neurosci. 17, 939-953. (Pubitemid 40800027)
42. Hamilton, R. H., Shenton, J. T. and Coslett, H. B. (2006). An acquired deficit of audiovisual speech processing, Brain Lang. 98, 66-73. (Pubitemid 43775701)
43. Hay-McCutcheon, M. J., Pisoni, D. B. and Kirk, K. I. (2005). Audiovisual speech perception in elderly cochlear implant recipients, Laryngoscope 115, 1887-1894. (Pubitemid 41475910)
44. Hikosaka, K., Iwai, E., Saito, H. and Tanaka, K. (1988). Polysensory properties of neurons in the anterior bank of the caudal superior temporal sulcus of the macaque monkey, J. Neurophysiol. 60, 1615-1637.
45. Jones E. G. and Powell, T. P. (1970). An anatomical study of converging sensory pathways within the cerebral cortex of the monkey, Brain 93,793-820.
46. Kaiser, A. R., Kirk, K. I., Lachs, L. and Pisoni, D. B. (2003). Talker and lexical effects on audiovisual word recognition by adults with cochlear implants, J. Speech Lang. Hearing Res. 46, 390-404. (Pubitemid 36457689)
47. Kayser, C., Logothetis, N. K. and Panzeri, S. (2010). Visual enhancement of the information representation in auditory cortex, Curr. Biol. 20, 19-24.
48. Kim, J. and Davis, C. (2004). Investigating the audio-visual speech detection advantage, Speech Comm. 44, 19-30.
49. Klatt, D. H. (1979). Speech perception: a model of acoustic-phonetic analysis and lexical access, J. Phonetics 7, 279-312.
50. Lachs, L. and Pisoni, D. B. (2004a). Crossmodal source identification in speech perception, Ecolog. Psychol. 16, 159-187. (Pubitemid 39666197)
51. Lachs, L. and Pisoni, D. B. (2004b). Cross-modal source information and spoken word recognition, J. Exper. Psychol.: Human Percept. Perform. 30, 378-396. (Pubitemid 38422593)
52. Lakatos, P., Chen, C., O'Connell, M. N., Mills, A. and Schroeder, C. E. (2007). Neuronal oscillations and multisensory interaction in primary auditory cortex, Neuron 53, 279-292. (Pubitemid 46074398)
53. Laurienti, P. J., Perrault, T. J., Stanford, T. R., Wallace, M. T. and Stein, B. E. (2005). On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies, Exper. Brain Res. 166, 289-297. (Pubitemid 41502075)
54. Liberman, A. M. and Mattingly, I. G. (1985). The motor theory of speech perception, Cognition 21, 1-36.
55. Luce, R. D. (1986). Response Times. Oxford University Press, New York, NY, USA.
56. McGurk, H. and Mcdonald, J. W. (1976). Hearing lips and seeing voices, Nature 264, 746-748.
57. Marks, L. E. (1978). The Unity of the Senses. Academic Press, New York, USA.
58. Massaro, D. W. (1987). Speech perception by ear and eye, in: Hearing by Eye: The Psychology of Lip-Reading, B. Dodd and R. Campbell (Eds), pp. 53-83. Lawrence Erlbaum, Hillsdale, NJ, USA.
59. Massaro, D.W. (2004). From multisensory integration to talking heads and language learning, in: The Handbook of Multisensory Processes, G. A. Calvert, C. Spence and B. E. Stein (Eds), pp. 153-176. MIT Press, Cambridge, MA, USA.
60. Massaro, D. W. and Cohen, M. M. (2000). Tests of auditory-visual integration efficiency within the framework of the fuzzy logical model of perception, J. Acoust. Soc. Amer. 108, 784-789. (Pubitemid 30625903)
61. Meredith,M. A. (2002). On the neuronal basis for multisensory convergence: a brief overview, Cognit. Brain Res. 14, 31-40. (Pubitemid 34626795)
62. Mesulam, M. M. (1998). From sensation to cognition, Brain 121, 1013-1052.
63. Nagarajan, S. S., Cheung, S.W., Bedenbaugh, P., Beitel, R. E., Schreiner, C. E. and Merzenich, M. M. (2002). Representation of spectral and temporal envelope of twitter vocalizations in common marmoset primary auditory cortex, J. Neurophysiol. 87, 1723-1737. (Pubitemid 34670739)
64. Österbauer, R. A., Mathews, P. M., Jenkinson, M., Beckmann, C. F., Hansen, P. C. and Calvert, G. A. (2005). Color of scents: chromatic stimuli modulate odor response in the human brain, J. Neurophysiol. 93, 3434-3411.
65. Palmeri, T. J., Goldinger, S. D. and Pisoni, D. B. (1993). Episodic encoding of voice attributes and recognition memory for spoken words, J. Exper. Psychol.: Learning Memory Cognit. 19, 309-328.
66. Pekkola, J., Ojanen, V., Autti, T., Jääskeläinen, I. P., Möttönen, R., Tarkiainen, A. and Sams, M. (2005). Primary auditory cortex driven by visual speech: an fMRI study at 3T, Neuroreport 16, 125-128. (Pubitemid 40250383)
67. Pitt, M. A., Myung, J. I. and Altieri, N. (2007). Modeling the word recognition data of Vitevitch and Luce (1998): is it ARTful?, Psychonom. Bull. Rev. 14, 442-448.
68. Ponton, C. W., Bernstein, L. E. and Auer, E. T. (2009). Mismatch negativity with visual-only and audiovisual speech, Brain Topography 21, 207-215.
69. Puce, A., Allison, T., Bentin, S., Gore, J. C. and McCarthy, G. (1998). Temporal cortex activation in humans viewing eye and mouth movements, J. Neurosci. 18, 2188-2199. (Pubitemid 28141423)
70. Rosenblum, L. D. (2005). Primacy of multimodal speech perception, in: Handbook of Speech Perception, D. B. Pisoni and R. E. Remez (Eds), pp. 51-78. Blackwell Publishing, Malden, MA, USA.
71. Rosenblum, L. D., Miller, R. M. and Sanchez, K. (2007). Lip-read me now, hear me better later: cross-modal transfer of talker-familiarity effects, Psycholog. Sci. 18, 392-395.
72. Ross, L. A., Saint-Amour, D., Leavitt, V. M., Javitt, D. C. and Foxe, J. J. (2007). Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments, Cerebral Cortex 17, 1147-1153. (Pubitemid 46598855)
73. Schroeder, C. E., Lakatos, P., Kajikawa, Y., Partan, S. and Puce, A. (2008). Neuronal oscillations and visual amplification of speech, Trends Cognit. Sci. 12, 106-113.
74. Scott, S. K., McGettigan, C. and Eisner, F. (2009). A little more conversation, a little less action - candidate roles for the motor cortex in speech perception, Nature Rev. Neurosci. 10, 295-302.
75. Shannon, R. V., Zeng, F.-G., Kamath, V., Wygonski, J. and Ekelid, M. (1995). Speech recognition with primarily temporal cues, Science 270, 303-304.
76. Skipper, J. I., Nusbaum, H. C. and Small, S. L. (2005). Listening to talking faces: motor cortical activation during speech perception, NeuroImage 25, 76-89. (Pubitemid 40289139)
77. Skipper, J. I., van Wassenhove, V., Nusbaum, H. C. and Small, S. L. (2007). Hearing lips and seeing voices: how cortical areas supporting speech production mediate audiovisual speech perception, Cerebral Cortex 17, 2387-2399. (Pubitemid 47517474)
78. Sommers, M., Tye-Murray, N. and Spehar, B. (2004). Time-compressed visual speech and age: a first report, Ear and Hearing 25, 565-572. (Pubitemid 39602244)
79. Stevenson, R. A. and James, T. W. (2009). Neuronal convergence and inverse effectiveness with audiovisual integration of speech and tools in human superior temporal sulcus: evidence from BOLD fMRI, NeuroImage 44, 1210-1223.
80. Sumby, W. H. and Pollack, I. (1954). Visual contribution to speech intelligibility in noise, J. Acoust. Soc. Amer. 26, 12-15.
81. Summerfield, Q. (1987). Some preliminaries to a comprehensive account of audio-visual speech perception, in: The Psychology of Lip-Reading, B. Dodd and R. Campbell (Eds), pp. 3-50. LEA, Hillsdale, NJ, USA.
82. Teinonen, T. Aslin, R. N., Alku, P. and Csibra, G. (2008). Visual speech contributes to phonetic learning in 6-month old infants, Cognition 108, 850-855.
83. Townsend, J. T. and Nozawa, G. (1995). Spatio-temporal properties of elementary perception: an investigation of parallel, serial and coactive theories, J. Math. Psychol. 39, 321-360.
84. Townsend, J. T. and Wenger, M. J. (2004). The serial-parallel dilemma: a case study in a linkage of theory and method, Psychonom. Bull. Rev. 11, 391-418. (Pubitemid 39234107)
85. van Wassenhove, V., Grant, K. W. and Poeppel, D. (2005). Visual speech speeds up the neural processing of auditory speech, Proc. Natl. Acad. Sci. USA 102, 1181-1186. (Pubitemid 40170746)
86. van Wassenhove, V., Grant, K. W. and Poeppel, D. (2007). Temporal window of integration in auditory-visual speech perception, Neuropsychologia 45, 598-607. (Pubitemid 44894453)
87. Wallace, M. T., Carriere, B. N., Perrault, T. J., Vaughan, J.W. and Stein, B. E. (2006). The development of cortical multisensory neurons, J. Neurosci. 15, 11844-11849.
88. Wang, X., Lu, T. and Liang, L. (2003). Cortical processing of temporal modulations, Speech Comm. 4, 107-121.
89. Wang, X., Merzenich, M. M., Beitel, R. and Schreiner, C. E. (1995). Representation of a speciesspecific vocalization in the primary auditory cortex of the common marmoset: temporal and spectral characteristics, J. Neurosci. 74, 2685-2706. (Pubitemid 26012324)