Prediction and constraint in audiovisual speech perception

Cortex

Volumn 68, Issue , 2015, Pages 169-181

  Peelle, Jonathan E ^a   Sommers, Mitchell S ^a  

^a WASHINGTON UNIVERSITY   (United States)

Author keywords

Audiovisual speech; Multisensory integration; Predictive coding; Predictive timing; Speech perception

Indexed keywords

ASSOCIATION; ATTENTION; AUDITORY CORTEX; BEHAVIOR; ELECTROSTIMULATION; HEARING; HUMAN; INFORMATION; NERVE CELL; NERVE POTENTIAL; NEUROANATOMY; NEUROIMAGING; NEUROPHYSIOLOGY; NOISE; NONHUMAN; OSCILLATION; PREDICTION; PROCESSING; RECOGNITION; REVIEW; SPEECH; SPEECH PERCEPTION; SUPERIOR TEMPORAL SULCUS; VISION; VISUAL INFORMATION; ANTICIPATION; PHYSIOLOGY;

ANTICIPATION, PSYCHOLOGICAL; AUDITORY PERCEPTION; HUMANS; SPEECH PERCEPTION; VISUAL PERCEPTION;

EID: 84930865241     PISSN: 00109452     EISSN: 19738102     Source Type: Journal    
DOI: 10.1016/j.cortex.2015.03.006     Document Type: Review

References (97)

1. Arnal L.H., Giraud A.-L. Cortical oscillations and sensory predictions. Trends in Cognitive Sciences 2012, 16:390-398.
2. Arnal L.H., Wyart V., Giraud A.-L. Transitions in neural oscillations reflect prediction errors generated in audiovisual speech. Nature Neuroscience 2011, 14:797-801.
3. Arnold P., Hill F. Bisensory augmentation: a speechreading advantage when speech is clearly audible and intact. British Journal of Audiology 2001, 92:339-355.
4. Auer E.T. The influence of the lexicon on speech read word recognition: contrasting segmental and lexical distinctiveness. Psychonomic Bulletin and Review 2002, 9:341-347.
5. Beauchamp M.S., Argall B.D., Bodurka J., Duyn J.H., Martin A. Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nature Neuroscience 2004, 7:1190-1192.
6. Bernstein L.E., Auer E.T., Takayanagi S. Auditory speech detection in noise enhanced by lipreading. Speech Communication 2004, 44:5-18.
7. Biau E., Torralba M., Fuentemilla L., de Diego Balaguer R., Soto-Faraco S. Speaker's hand gestures modulate speech perception through phase resetting of ongoing neural oscillations. Cortex 2015, 68:76-85.
8. Blamey P., Cowan R., Alcantara J., Whitford L., Clark G. Speech perception using combinations of auditory, visual, and tactile information. Journal of Rehabilitation Research and Development 1989, 26:15-24.
9. Braida L.D. Crossmodal integration in the identification of consonant segments. Quarterly Journal of Experimental Psychology Section A: Human Experimental Psychology 1991, 43:647-677.
10. Calvert G.A., Bullmore E.T., Brammer M.J., Campbell R., Williams S.C.R., McGuire P.K., et al. Activation of auditory cortex during silent lipreading. Science 1997, 276:593-596.
11. Calvert G.A., Campbell R., Brammer M.J. Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Current Biology 2000, 10:649-657.
12. Canolty R.T., Edwards E., Dalal S.S., Soltani M., Nagarajan S.S., Kirsch H.E., et al. High gamma power is phase-locked to theta oscillations in human neocortex. Science 2006, 313:1626-1628.
13. Canolty R.T., Knight R.T. The functional role of cross-frequency coupling. Trends in Cognitive Sciences 2010, 14:506-515.
14. Carlyon R.P., Cusack R., Foxton J.M., Robertson I.H. Effects of attention and unilateral neglect on auditory stream segregation. Journal of Experimental Psychology: Human Perception and Performance 2001, 27:115-127.
15. Chandrasekaran C., Trubanova A., Stillittano S., Caplier A., Ghazanfar A.A. The natural statistics of audiovisual speech. PLoS Computational Biology 2009, 5:e1000436.
16. Ding N., Simon J.Z. Neural coding of continuous speech in auditory cortex during monaural and dichotic listening. Journal of Neurophysiology 2012, 107:78-89.
17. Doelling K., Arnal L.H., Ghitza O., Poeppel D. Acoustic landmarks drive delta-theta oscillations to enable speech comprehension by facilitating perceptual parsing. NeuroImage 2014, 85:761-768.
18. Egan J.P., Greenberg G.Z., Schulman A.I. Interval of time uncertainty in auditory detection. Journal of the Acoustical Society of America 1961, 33:771-778.
19. Engel A.K., Fries P., Singer W. Dynamic predictions: oscillations and synchrony in top-down processing. Nature Reviews Neuroscience 2001, 2:704-716.
20. Erber N.P. Auditory-visual perception of speech. Journal of Speech and Hearing Disorders 1975, 40:481-492.
21. Erickson L.C., Heeg E., Rauschecker J.P., Turkeltaub P.E. An ALE meta-analysis on the audiovisual integration of speech signals. Human Brain Mapping 2014, 35:5587-5605.
22. Feld J., Sommers M.S. There goes the neighborhood: lipreading and the structure of the mental lexicon. Speech Communication 2011, 53:220-228.
23. Fisher C.G. Confusions among visually perceived consonants. Journal of Speech and Hearing Research 1968, 11:796-804.
24. Fridriksson J., Moss J., Davis B., Baylis G.C., Bonilha L., Rorden C. Motor speech perception modulates the cortical language areas. NeuroImage 2008, 41:605-613.
25. Fries P. A mechanism for cognitive dynamics: neuronal communication through neuronal coherence. Trends in Cognitive Sciences 2005, 9:474-480.
26. Friston K. The free-energy principle: a unified brain theory?. Nature Reviews Neuroscience 2010, 11:127-138.
27. Friston K., Kiebel S. Predictive coding under the free-energy principle. Philosophical Transactions of The Royal Society B 2009, 364:1211-1221.
28. Gagnepain P., Henson R.N., Davis M.H. Temporal predictive codes for spoken words in auditory cortex. Current Biology 2012, 22:615-621.
29. Ghitza O. Linking speech perception and neurophysiology: speech decoding guided by cascaded oscillators locked to the input rhythm. Frontiers in Psychology 2011, 2:130.
30. Giraud A.-L., Poeppel D. Cortical oscillations and speech processing: emerging computational principles and operations. Nature Neuroscience 2012, 15:511-517.
31. Gosselin P.A., Gagné J.-P. Older adults expend more listening effort than younger adults recognizing audiovisual speech in noise. International Journal of Audiology 2011, 50:786-792.
32. Grant K.W., Seitz P.F. Measures of auditory-visual integration in nonsense syllables and sentences. Journal of the Acoustical Society of America 1998, 104:2438-2450.
33. Grant K.W., Seitz P.F. The use of visible speech cues for improving auditory detection of spoken sentences. Journal of the Acoustical Society of America 2000, 108:1197-1208.
34. Grant K.W., Walden B.E., Seitz P.F. Auditory-visual speech recognition by hearing-impaired subjects: consonant recognition, sentence recognition, and auditory-visual integration. Journal of the Acoustical Society of America 1998, 103:2677-2690.
35. Gross J., Hoogenboom N., Thut G., Schyns P., Panzeri S., Belin P., et al. Speech rhythms and multiplexed oscillatory sensory coding in the human brain. PLoS Biology 2013, 11:e1001752.
36. Hall D.A., Fussell C., Summerfield A.Q. Reading fluent speech from talking faces: typical brain networks and individual differences. Journal of Cognitive Neuroscience 2005, 17:939-953.
37. Henry M.J., Herrmann B., Obleser J. Entrained neural oscillations in multiple frequency bands comodulate behavior. Proceedings of the National Academy of Sciences 2014, 111:14935-14940.
38. Henry M.J., Obleser J. Frequency modulation entrains slow neural oscillations and optimizes human listening behavior. Proceedings of the National Academy of Science 2012, 109:20095-20100.
39. Jensen O., Colgin L.L. Cross-frequency coupling between neuronal oscillations. Trends in Cognitive Sciences 2007, 11:267-269.
40. Kayser C., Petkov C.I., Logothetis N.K. Visual modulation of neurons in auditory cortex. Cerebral Cortex 2008, 18:1560-1574.
41. Kerlin J.R., Shahin A.J., Miller L.M. Attentional gain control of ongoing cortical speech representations in a "cocktail party". Journal of Neuroscience 2010, 30:620-628.
42. Lakatos P., Chen C.-M., O'Connell M.N., Mills A., Schroeder C.E. Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 2007, 53:279-292.
43. Lakatos P., Karmos G., Mehta A.D., Ulbert I., Schroeder C.E. Entrainment of neuronal oscillations as a mechanism of attentional selection. Science 2008, 320:110-113.
44. Lakatos P., Shah A.S., Knuth K.H., Ulbert I., Karmos G., Schroeder C.E. An oscillatory hierarchy controlling neuronal excitability and stimulus processing in the auditory cortex. Journal of Neurophysiology 2005, 94:1904-1911.
45. Luce P.A., Pisoni D.B. Recognizing spoken words: the neighborhood activation model. Ear and Hearing 1998, 19:1-36.
46. Luo H., Liu Z., Poeppel D. Auditory cortex tracks both auditory and visual stimulus dynamics using low-frequency neuronal phase modulation. PLoS Biology 2010, 8:e1000445.
47. Luo H., Poeppel D. Phase patterns of neuronal responses reliably discriminate speech in human auditory cortex. Neuron 2007, 54:1001-1010.
48. Macleod A., Summerfield Q. Quantifying the contribution of vision to speech perception in noise. British Journal of Audiology 1987, 21:131-141.
49. Marslen-Wilson W.D., Tyler L. The temporal structure of spoken language processing. Cognition 1980, 8:1-71.
50. Massaro D.W. Speechreading: illusion or window into pattern recognition. Trends in Cognitive Sciences 1999, 3:310-317.
51. Mattys S.L., Bernstein L.E., Auer E.T. Stimulus-based lexical distinctiveness as a general word-recognition mechanism. Perception and Psychophysics 2002, 64:667-679.
52. McGettigan C., Faulkner A., Altarelli I., Obleser J., Baverstock H., Scott S.K. Speech comprehension aided by multiple modalities: behavioural and neural interactions. Neuropsychologia 2012, 50:762-775.
53. McGurk H., MacDonald J. Hearing lips and seeing voices. Nature 1976, 264:746-748.
54. Mesgarani N., Chang E.F. Selective cortical representation of attended speaker in multi-talker speech perception. Nature 2012, 485:233-237.
55. Miller J.L., Grosjean F., Lomanto C. Articulation rate and its variability in spontaneous speech: a reanalysis and some implications. Phonetica 1984, 41:215-225.
56. Möttönen R., Schürmann M., Sams M. Time course of multisensory interactions during audiovisual speech perception in humans: a magnetoencephalographic study. Neuroscience Letters 2004, 363:112-115.
57. Möttönen R., Watkins K.E. Motor representations of articulators contribute to categorical perception of speech sounds. Journal of Neuroscience 2009, 29:9819-9825.
58. Nath A.R., Beauchamp M.S. Dynamic changes in superior temporal sulcus connectivity during perception of noisy audiovisual speech. Journal of Neuroscience 2011, 31:1704-1714.
59. Oden G.C., Massaro D.W. Integration of featural information in speech perception. Psychological Review 1978, 85:172-191.
60. ten Oever S., Schroeder C.E., Poeppel D., van Atteveldt N., Zion-Golumbic E. Rhythmicity and cross-modal temporal cues facilitate detection. Neuropsychologia 2014, 63:43-50.
61. Okada K., Venezia J.H., Matchin W., Saberi K., Hickok G. An fMRI study of audiovisual speech perception reveals multisensory interactions in auditory cortex. PLoS One 2013, 8:e68959.
62. Peelle J.E. The hemispheric lateralization of speech processing depends on what "speech" is: a hierarchical perspective. Frontiers in Human Neuroscience 2012, 6:309.
63. Peelle J.E. Methodological challenges and solutions in auditory functional magnetic resonance imaging. Frontiers in Neuroscience 2014, 8:253.
64. Peelle J.E., Davis M.H. Neural oscillations carry speech rhythm through to comprehension. Frontiers in Psychology 2012, 3:320.
65. Peelle J.E., Gross J., Davis M.H. Phase-locked responses to speech in human auditory cortex are enhanced during comprehension. Cerebral Cortex 2013, 23:1378-1387.
66. Perrodin C., Kayser C., Logothetis N.K., Petkov C.I. Natural asynchronies in audiovisual communication signals regulate neuronal multisensory interactions in voice-sensitive cortex. Proceedings of the National Academy of Science 2015, 112:273-278.
67. Reisberg D., McLean J., Goldfield A. Easy to hear but hard to understand: a speechreading advantage with intact stimuli. Hearing by eye: The psychology of lip-reading 1987, 97-113. Erlbaum, London. R. Campbell, B. Dodd (Eds.).
68. Romei V., Gross J., Thut G. On the role of prestimulus alpha rhythms over occipito-parietal areas in visual input regulation: correlation or causation?. Journal of Neuroscience 2010, 30:8692-8697.
69. Schroeder C.E., Foxe J. Multisensory contributions to low-level, 'unisensory' processing. Current Opinion in Neurobiology 2005, 15:454-458.
70. Schroeder C.E., Lakatos P., Kajikawa Y., Partan S., Puce A. Neuronal oscillations and visual amplification of speech. Trends in Cognitive Sciences 2008, 12:106-113.
71. Schwartz J.-L., Savariaux C. No, there is no 150ms lead of visual speech on auditory speech, but a range of audiovisual asynchronies varying from small audio lead to large audio lag. PLoS Computational Biology 2014, 10:e1003743.
72. Sekiyama K., Kanno I., Miura S., Sugita Y. Auditory-visual speech perception examined by fMRI and PET. Neuroscience Research 2003, 47:277-287.
73. Seltzer B., Pandya D.N. Afferent cortical connections and architectonics of the superior temporal sulcus and surrounding cortex in the rhesus monkey. Brain Research 1978, 149:1-24.
74. Skipper J.I., Nusbaum H., Small S.L. Listening to talking faces: motor cortical activation during speech perception. NeuroImage 2005, 25:76-89.
75. Sohoglu E., Peelle J.E., Carlyon R.P., Davis M.H. Predictive top-down integration of prior knowledge during speech perception. Journal of Neuroscience 2012, 32:8443-8453.
76. Sommers M.S., Tye-Murray N., Spehar B. Auditory-visual speech perception and auditory-visual enhancement in normal-hearing younger and older adults. Ear and Hearing 2005, 26:263-275.
77. Stevenson R.A., Geoghegan M.L., James T.W. Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects. Experimental Brain Research 2007, 179:85-95.
78. Stevenson R.A., Ghose D., Fister J.K., Sarko D.K., Altieri N.A., Nidiffer A.R., et al. Identifying and quantifying multisensory integration: a tutorial review. Brain Topography 2014, 27:707-730.
79. Sumby W.H., Pollack I. Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America 1954, 26:212-215.
80. Thut G., Nietzel A., Brandt S.A., Pascual-Leone A. α-Band electroencephalographic activity over occipital cortex indexes visuospatial attention bias and predicts visual target detection. Journal of Neuroscience 2006, 26:9494-9502.
81. Tian X., Poeppel D. Mental imagery of speech: linking motor and perceptual systems through internal simulation and estimation. Frontiers in Human Neuroscience 2012, 6:314.
82. Tye-Murray N., Sommers M.S., Spehar B. Audiovisual integration and lipreading abilities of older adults with normal and impaired hearing. Ear and Hearing 2007, 28:656-668.
83. Tye-Murray N., Sommers M.S., Spehar B. Auditory and visual lexical neighborhoods in audiovisual speech perception. Trends in Amplification 2007, 11:233-241.
84. Tye-Murray N., Spehar B.P., Myerson J., Hale S., Sommers M.S. Reading your own lips: common-coding theory and visual speech perception. Psychonomic Bulletin and Review 2013, 20:115-119.
85. Tye-Murray N., Spehar B.P., Myerson J., Hale S., Sommers M.S. The self-advantage in visual speech processing enhances audiovisual speech recognition in noise. Psychonomic Bulletin and Review 2015, (in press).
86. Tye-Murray N., Spehar B., Myerson J., Sommers M.S., Hale S. Crossmodal enhancement of speech detection in young and older adults: does signal content matter?. Ear and Hearing 2011, 32:650-655.
87. Van Engen K.J., Phelps J.E.B., Smiljanic R., Chandrasekaran B. Enhancing speech intelligibility: interactions among context, modality, speech style, and masker. Journal of Speech, Language, and Hearing Research 2014, 57:1908-1918.
88. Vatakis A., Ghazanfar A.A., Spence C. Facilitation of multisensory integration by the "unity effect" reveals that speech is special. Journal of Vision 2008, 8:1-11.
89. Volgushev M., Chistiakova M., Winger W. Modification of discharge patterns of neocortical neurons by induced oscillations of the membrane potential. Neuroscience 1998, 83:15-25.
90. van Wassenhove V., Grant K.W., Poeppel D. Visual speech speeds up the neural processing of auditory speech. Proceedings of the National Academy of Science 2005, 102:1181-1186.
91. van Wassenhove V., Grant K.W., Poeppel D. Temporal window of integration in auditory-visual speech perception. Neuropsychologia 2007, 45:598-607.
92. Watson C.S., Nichols T.L. Detectability of auditory signals presented without defined observation intervals. Journal of the Acoustical Society of America 1976, 59:655-668.
93. Wayne R.V., Johnsrude I.S. The role of visual speech information in supporting perceptual learning of degraded speech. Journal of Experimental Psychology: Applied 2012, 18:419-435.
94. Wright T.M., Pelphrey K.A., Allison T., McKeown M.J., McCarthy G. Polysensory interactions along lateral temporal regions evoked by audiovisual speech. Cerebral Cortex 2003, 13:1034-1043.
95. Yi H.-G., Smiljanic R., Chandrasekaran B. The neural processing of foreign-accented speech and its relationship to listener bias. Frontiers in Human Neuroscience 2014, 8:768.
96. Zion Golumbic E., Cogan G.B., Schroeder C.E., Poeppel D. Visual input enhances selective speech envelope tracking in auditory cortex at a "cocktail party". Journal of Neuroscience 2013, 33:1417-1426.
97. Zion Golumbic E., Ding N., Bickel S., Lakatos P., Schevon C.A., McKhann G.M., et al. Mechanisms underlying selective neuronal tracking of attended speech at a "cocktail party". Neuron 2013, 77:980-991.