Congruent visual speech enhances cortical entrainment to continuous auditory speech in noise-free conditions

Journal of Neuroscience

Volumn 35, Issue 42, 2015, Pages 14195-14204

  Crosse, Michael J ^a   Butler, John S ^a,b   Lalor, Edmund C ^a  

^a TRINITY COLLEGE DUBLIN   (Ireland)

^b DUBLIN INSTITUTE OF TECHNOLOGY   (Ireland)

Author keywords

Audiovisual speech; EEG; Multisensory integration; Stimulus reconstruction; Temporal coherence; Temporal response function

Indexed keywords

ACOUSTIC STRESS; ADULT; ARTICLE; AUDITORY STIMULATION; BEHAVIORAL SCIENCE; CONGRUENT VISUAL SPEECH; CONTINUOUS AUDITORY SPEECH; ELECTROENCEPHALOGRAPHY; ELECTROPHYSIOLOGY; FEMALE; HUMAN; MAGNETOENCEPHALOGRAPHY; MALE; NORMAL HUMAN; PHONETICS; PRIORITY JOURNAL; REACTION TIME; SENSORY FEEDBACK; SPEECH; SPEECH PERCEPTION; VISUAL CORTEX; VISUAL STIMULATION; ANALYSIS OF VARIANCE; BRAIN MAPPING; ELECTROMYOGRAPHY; EVOKED AUDITORY RESPONSE; EVOKED VISUAL RESPONSE; PHOTOSTIMULATION; PHYSIOLOGY; VISION; YOUNG ADULT;

ACOUSTIC STIMULATION; ADULT; ANALYSIS OF VARIANCE; BRAIN MAPPING; ELECTROENCEPHALOGRAPHY; ELECTROMYOGRAPHY; EVOKED POTENTIALS, AUDITORY; EVOKED POTENTIALS, VISUAL; FEMALE; HUMANS; MALE; PHOTIC STIMULATION; REACTION TIME; SPEECH PERCEPTION; VISUAL PERCEPTION; YOUNG ADULT;

EID: 84944929752     PISSN: 02706474     EISSN: 15292401     Source Type: Journal    
DOI: 10.1523/JNEUROSCI.1829-15.2015     Document Type: Article

References (69)

1. Abrams DA, Nicol T, Zecker S, Kraus N (2008) Right-hemisphere auditory cortex is dominant for coding syllable patterns in speech. J Neurosci 28:3958-3965.CrossRef Medline
2. Ahissar E, Nagarajan S, Ahissar M, Protopapas A, Mahncke H, Merzenich MM (2001) Speech comprehension is correlated with temporal response patterns recorded from auditory cortex. Proc Natl Acad Sci U S A 98:13367-13372.CrossRef Medline
3. Arnal LH, Morillon B, Kell CA, Giraud AL (2009) Dual neural routing of visual facilitation in speech processing. J Neurosci 29:13445-13453.CrossRef Medline
4. Beauchamp MS, Lee KE, Argall BD, Martin A (2004) Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41:809-823.CrossRef Medline
5. Bernstein LE, Liebenthal E (2014) Neural pathways for visual speech perception. Front Neurosci 8:386.CrossRef Medline
6. Bernstein LE, Auer ET, Wagner M, Ponton CW (2008) Spatiotemporal dynamics of audiovisual speech processing. Neuroimage 39:423-435.CrossRef Medline
7. Besle J, Fort A, Giard M-H (2004a) Interest and validity of the additive model in electrophysiological studies of multisensory interactions. Cogn Process 5:189-192.
8. Besle J, Fort A, Delpuech C, Giard MH (2004b) Bimodal speech: early suppressive visual effects in human auditory cortex. Eur J Neurosci 20:2225-2234.CrossRef Medline
9. Besle J, Fischer C, Bidet-Caulet A, Lecaignard F, Bertrand O, Giard MH (2008) Visual activation and audiovisual interactions in the auditory cortex during speech perception: intracranial recordings in humans. J Neurosci 28:14301-14310.CrossRef Medline
10. Brandwein AB, Foxe JJ, Butler JS, Russo NN, Altschuler TS, Gomes H, Molholm S (2013) The development of multisensory integration in highfunctioning autism: high-density electrical mapping and psychophysical measures reveal impairments in the processing of audiovisual inputs. Cereb Cortex 23:1329-1341.CrossRef Medline
11. Calvert GA, Campbell R (2003) Reading speech from still and moving faces: the neural substrates of visible speech. J Cogn Neurosci 15:57-70.CrossRef Medline
12. Campbell R (2008) The processing of audio-visual speech: empirical and neural bases. Philos Trans R Soc Lond B Biol Sci 363:1001-1010.CrossRef Medline
13. Chandrasekaran C, Trubanova A, Stillittano S, Caplier A, Ghazanfar AA (2009) The natural statistics of audiovisual speech. PLoS Comput Biol 5:e1000436.CrossRef Medline
14. Chang EF, Rieger JW, Johnson K, Berger MS, Barbaro NM, Knight RT (2010) Categorical speech representation in human superior temporal gyrus. Nat Neurosci 13:1428-1432.CrossRef Medline
15. Crosse MJ, ElShafei HA, Foxe JJ, Lalor EC (2015) Investigating the temporal dynamics of auditory cortical activation to silent lipreading. Paper presented at 7th International IEEE/EMBS Conference on Neural Engineering, Montpellier, France, April.
16. Delorme A, Makeig S (2004) EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J Neurosci Methods 134:9-21.CrossRef Medline
17. Di Liberto GM, O’Sullivan JA, Lalor EC (2015) Low frequency cortical entrainment to speech reflects phonemic level processing. Curr Biol. Advance onlinepublication. Retrieved Sept. 24, 2015CrossRef
18. Ding N, Simon JZ (2012) Emergence of neural encoding of auditory objects while listening to competing speakers. Proc Natl Acad Sci U S A 109: 11854-11859.CrossRef Medline
19. Ding N, Simon JZ (2013) Adaptive temporal encoding leads to a background-insensitive cortical representation of speech. J Neurosci 33: 5728-5735.CrossRef Medline
20. Ding N, Simon JZ (2014) Cortical entrainment to continuous speech: functional roles and interpretations. Front Hum Neurosci 8:311.CrossRef Medline
21. Ding N, Chatterjee M, Simon JZ (2014) Robust cortical entrainment to the speech envelope relies on the spectro-temporal fine structure. Neuroimage 88:41-46.CrossRef Medline
22. Elhilali M, Ma L, Micheyl C, Oxenham AJ, Shamma SA (2009) Temporal coherence in the perceptual organization and cortical representation of auditory scenes. Neuron 61:317-329.CrossRef Medline
23. Giraud AL, Poeppel D (2012) Cortical oscillations and speech processing: emerging computational principles and operations. Nat Neurosci 15: 511-517.CrossRef Medline
24. Grant KW, Seitz PF (2000) The use of visible speech cues for improving auditory detection of spoken sentences. J Acoust Soc Am 108:1197-1208.CrossRef Medline
25. Grant KW, Walden BE, Seitz PF (1998) Auditory-visual speech recognition by hearing-impaired subjects: consonant recognition, sentence recognition, and auditory-visual integration. J Acoust Soc Am 103:2677-2690CrossRef Medline
26. Hairston WD, Burdette JH, Flowers DL, Wood FB, Wallace MT (2005) Altered temporal profile of visual-auditory multisensory interactions in dyslexia. Exp Brain Res 166:474-480.CrossRef Medline
27. Haufe S, Meinecke F, Görgen K, Dähne S, Haynes JD, Blankertz B, Bießmann F (2014) On the interpretation of weight vectors of linear models in multivariate neuroimaging. Neuroimage 87:96-110.CrossRef Medline
28. Jiang J, Bernstein LE (2011) Psychophysics of the McGurk and other audiovisual speech integration effects. J Exp Psychol Hum Percept Perform 37:1193-1209.CrossRef Medline
29. Jiang J, Alwan A, Keating PA, Auer ET, Bernstein LE (2002) On the relationship between face movements, tongue movements, and speech acoustics. EURASIP J Appl Signal Process 11:1174-1188.
30. Kayser C, Logothetis NK (2009) Directed interactions between auditory and superior temporal cortices and their role in sensory integration. Front Integr Neurosci 3:7.CrossRef Medline
31. Kayser C, Petkov CI, Logothetis NK (2008) Visual modulation of neurons in auditory cortex. Cereb Cortex 18:1560-1574.CrossRef Medline
32. Klucharev V, Möttönen R, Sams M (2003) Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Cogn Brain Res 18:65-75.CrossRef Medline
33. Kruskal JB, Wish M (1978) Multidimensional scaling. Newbury Park, CA: Sage.
34. Lalor EC, Foxe JJ (2010) Neural responses to uninterrupted natural speech can be extracted with precise temporal resolution. Eur J Neurosci 31:189-193.CrossRef Medline
35. Luo H, Liu Z, Poeppel D (2010) Auditory cortex tracks both auditory and visual stimulus dynamics using low-frequency neuronal phase modulation. PLoS Biol 8:e1000445.CrossRef Medline
36. McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264:746-748.CrossRef Medline
37. Mesgarani N, David SV, Fritz JB, Shamma SA (2009) Influence of context and behavior on stimulus reconstruction from neural activity in primary auditory cortex. J Neurophysiol 102:3329-3339.CrossRef Medline
38. Miller J (1982) Divided attention: evidence for coactivation with redundant signals. Cogn Psychol 14:247-279.CrossRef Medline
39. Möttönen R, Krause CM, Tiippana K, Sams M (2002) Processing of changes in visual speech in the human auditory cortex. Cogn Brain Res 13:417-425. CrossRef Medline
40. Nock HJ, Iyengar G, Neti C (2002) Assessing face and speech consistency for monologue detection in video. Paper presented at 10th ACM International Conference on Multimedia, Juan-les-Pins, France, December.
41. O’Sullivan JA, Crosse MJ, Power AJ, Lalor EC (2013) The effects of attention and visual input on the representation of natural speech in EEG. Paper presented at 35th Annual International Conference of the IEEE/EMBS, Osaka, Japan, July.
42. O’Sullivan JA, Power AJ, Mesgarani N, Rajaram S, Foxe JJ, Shinn-Cunningham BG, Slaney M, Shamma SA, Lalor EC (2015) Attentional selection in a cocktail party environment can be decoded from single-trial EEG. Cereb Cortex 25:1697-1706.CrossRef Medline
43. Pasley BN, David SV, Mesgarani N, Flinker A, Shamma SA, Crone NE, Knight RT, Chang EF (2012) Reconstructing speech from human auditory cortex. PLoS Biol 10:e1001251.CrossRef Medline
44. Peelle JE, Sommers MS (2015) Prediction and constraint in audiovisual speech perception. Cortex 68:169-181.CrossRef Medline
45. Power AJ, Foxe JJ, Forde EJ, Reilly RB, Lalor EC (2012) At what time is the cocktail party? A late locus of selective attention to natural speech. Eur J Neurosci 35:1497-1503.CrossRef Medline
46. Raab DH (1962) Statistical facilitation of simple reaction times. Trans N Y Acad Sci 24:574-590.CrossRef Medline
47. Rabiner LR (1989) A tutorial on hidden Markov models and selected applications in speech recognition. Proc IEEE 77:257-286.CrossRef
48. Reisberg D, McLean J, Goldfield A (1987) Easy to hear but hard to understand: a lip-reading advantage with intact auditory stimuli. In: Hearing by eye: the psychology of lip-reading (Dodd B, Campbell R, eds), pp 97-114.
49. Hillsdale NJ Erlbaum, Rieke F, Bodnar D, Bialek W (1995) Naturalistic stimuli increase the rate and efficiency of information transmission by primary auditory afferents. Philos Trans R Soc Lond B Biol Sci 262:259-265.CrossRef Medline
50. Rosen S (1992) Temporal information in speech: acoustic, auditory and linguistic aspects. Philos Trans R Soc Lond B Biol Sci 336:367-373.CrossRef Medline
51. Ross LA, Saint-Amour D, Leavitt VM, Javitt DC, Foxe JJ (2007a) Doyou see what I am saying? Exploring visual enhancement of speech comprehension in noisy environment. Cereb Cortex 17:1147-1153.CrossRef Medline
52. Ross LA, Saint-Amour D, Leavitt VM, Molholm S, Javitt DC, Foxe JJ (2007b) Impaired multisensory processing in schizophrenia: deficits in the visual enhancement of speech comprehension under noisy environmental conditions. Schizophr Res 97:173-183.CrossRef Medline
53. Sams M, Aulanko R, Hämäläinen M, Hari R, Lounasmaa OV, Lu ST, Simola J (1991) Seeing speech: visual information from lip movements modifies activity in the human auditory cortex. Neurosci Lett 127:141-145.CrossRef Medline
54. Schroeder CE, Lakatos P, Kajikawa Y, Partan S, Puce A (2008) Neuronal oscillations and visual amplification of speech. Trends Cogn Sci 12:106-113.CrossRef Medline
55. Shamma SA, Elhilali M, Micheyl C (2011) Temporal coherence and attention in auditory scene analysis. Trends Neurosci 34:114-123.CrossRef Medline
56. Sheedy CM, Power AJ, Reilly RB, Crosse MJ, Loughnane GM, Lalor EC (2014) Endogenous auditory frequency-based attention modulates electroencephalogram-based measures of obligatory sensory activity in humans. Neuroreport 25:219-225.CrossRef Medline
57. Stein BE, Meredith MA (1993) The merging of the senses. Cambridge, MA: MIT.
58. Stekelenburg JJ, Vroomen J (2007) Neural correlates of multisensory integration of ecologically valid audiovisual events. J Cogn Neurosci 19:1964-1973.CrossRef Medline
59. Stekelenburg JJ, Maes JP, Van Gool AR, Sitskoorn M, Vroomen J (2013) Deficient multisensory integration in schizophrenia: an event-related potential study. Schizophr Res 147:253-261.CrossRef Medline
60. Sumby WH, Pollack I (1954) Visual contribution to speech intelligibility in noise. J Acoust Soc Am 26:212-215.
61. CrossRef Summerfield Q (1987) Some preliminaries to a comprehensive account of audio-visual speech perception. Hillsdale, NJ: Erlbaum.
62. Summerfield Q (1992) Lipreading and audio-visual speech perception. Philos Trans R Soc Lond B Biol Sci 335:71-78.CrossRef Medline
63. Talsma D, Senkowski D, Soto-Faraco S, Woldorff MG (2010) The multifaceted interplay between attention and multisensory integration. Trends Cogn Sci 14:400-410.CrossRef Medline
64. Teki S, Chait M, Kumar S, von Kriegstein K, Griffiths TD (2011) Brain bases for auditory stimulus-driven figure-ground segregation. J Neurosci 31: 164-171.CrossRef Medline
65. Tjan BS, Chao E, Bernstein LE (2014) A visual or tactile signal makes auditory speech detection more efficient by reducing uncertainty. Eur J Neurosci 39:1323-1331.CrossRef Medline
66. Ulrich R, Miller J, Schröter H (2007) Testing the race model inequality: an algorithm and computer programs. Behav Res Methods 39:291-302.CrossRef Medline
67. van Wassenhove V, Grant KW, Poeppel D (2005) Visual speech speeds up the neural processing of auditory speech. Proc Natl Acad Sci U S A 102: 1181-1186.CrossRef Medline
68. Welch RB, Warren DH (1980) Immediate perceptual response to intersensory discrepancy. Psychol Bull 88:638-667.CrossRef Medline
69. Zion Golumbic EM, Cogan GB, Schroeder CE, Poeppel D (2013) Visual input enhances selective speech envelope tracking in auditory cortex at a “cocktail party.” J Neurosci 33:1417-1426.CrossRef Medline