Visual input enhances selective speech envelope tracking in auditory cortex at a "Cocktail Party"

Journal of Neuroscience

Volumn 33, Issue 4, 2013, Pages 1417-1426

  Zion Golumbic, Elana ^a,b   Cogan, Gregory B ^c   Schroeder, Charles E ^a,b   Poeppel, David ^c  

^a New York Presbyterian Columbia University Medical Center   (United States)

^b NATHAN S KLINE INSTITUTE FOR PSYCHIATRIC RESEARCH   (United States)

^c NEW YORK UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ARTICLE; ATTENTION; AUDITORY CORTEX; AUDITORY STIMULATION; BEHAVIOR; EVOKED AUDITORY RESPONSE; FACE; FEMALE; HUMAN; HUMAN EXPERIMENT; MAGNETOENCEPHALOGRAPHY; MALE; NORMAL HUMAN; NUCLEAR MAGNETIC RESONANCE IMAGING; PRIORITY JOURNAL; SPEECH PERCEPTION; TASK PERFORMANCE; WAVEFORM;

AUDITORY CORTEX; FEMALE; HUMANS; MAGNETOENCEPHALOGRAPHY; MALE; PHOTIC STIMULATION; SPEECH PERCEPTION; YOUNG ADULT;

EID: 84872703974     PISSN: 02706474     EISSN: 15292401     Source Type: Journal    
DOI: 10.1523/JNEUROSCI.3675-12.2013     Document Type: Article

References (75)

1. Aiken SJ., Picton TW. (2008). Human cortical responses to the speech envelope. Ear Hear 29:139-157. CrossRef Medline
2. Arnal LH., Morillon B, Kell CA, Giraud AL. (2009). Dual neural routing of visual facilitation in speech processing. J Neurosci 29:13445-13453. CrossRef Medline
3. Arnal LH., Wyart V, Giraud AL. (2011). Transitions in neural oscillations reflect prediction errors generated in audiovisual speech. Nat Neurosci 14:797-801. CrossRef Medline
4. Beaman CP., Bridges AM, Scott SK. (2007). From dichotic listening to the irrelevant sound effect: a behavioural and neuroimaging analysis of the processing of unattended speech. Cortex 43:124-134. CrossRef Medline
5. Beauchamp MS., Lee KE, Argall BD, MartinA. (2004). Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41:809-823. CrossRef Medline
6. Bennett AT. (1980). Rhythmic analysis of multiple levels of communicative behavior in face to face interaction. In: Aspects of nonverbal communication (Raffler-Engel W, ed), pp 237-251. Lisse, the Netherlands: Swets and Zeitleinger.
7. Besle J., Fort A, Delpuech C, Giard MH. (2004). Bimodal speech: early suppressive visual effects in human auditory cortex. Eur J Neurosci 20:2225-2234. CrossRef Medline
8. Birdwhistell RL. (1970). Kinesics and context: essays on body motion communication. Philadelphia: University of Pennsylvania.
9. Bishop CW., Miller LM. (2009). A multisensory cortical network for understanding speech in noise. J Cogn Neurosci 21:1790-1805. Medline
10. Buzsáki G, Chrobak JJ. (1995). Temporal structure in spatially organized neuronal ensembles: a role for interneuronal networks. Curr Opin Neurobiol 5:504-510. CrossRef Medline
11. Callan D., Jones JA, Munhall K, Callan AM, Kroos C, Vatikiotis-Bateson E (2003) Neural processes underlying perceptual enhancement by visual speech gestures. Neuroreport 14:2213-2218. CrossRef Medline
12. Chandrasekaran C., Trubanova A, Stillittano S, Caplier A, Ghazanfar AA (2009) The natural statistics of audiovisual speech. PLoS Comput Biol 5:e1000436. CrossRef Medline
13. Cherry EC. (1953). Some experiments on the recognition of speech, with one and two ears. J Acoust Soc Am 25:975-979. CrossRef
14. David SV., Mesgarani N, Shamma SA. (2007). Estimating sparse spectrotemporal receptive fields with natural stimuli. Network 18:191-212. CrossRef Medline
15. Davis C., Kislyuk D, Kim J, Sams M. (2008). The effect of viewing speech on auditory speech processing is different in the left and right hemispheres. Brain Res 1242:151-161. CrossRef Medline
16. de Cheveigné A, Simon JZ. (2007). Denoising based on time-shift pca. J Neurosci Methods 165:297-305. CrossRef Medline
17. Ding N., Simon JZ. (2012). Neural coding of continuous speech in auditory cortex during monaural and dichotic listening. J Neurophysiol 107:78-89. CrossRef Medline
18. Drullman R. (2006). The significance of temporal modulation frequencies for speech intelligibility. In: Listening to speech: an auditory perspective. (Greenberg S, Ainsworth W, eds), pp 39-48. Mahwah, NJ: Lawrence Erlbaum.
19. Elhilali M., Shamma SA. (2008). A cocktail party with a cortical twist: how cortical mechanisms contribute to sound segregation. J Acoust Soc Am 124:3751-3771. CrossRef Medline
20. Elhilali M., Ma L, Micheyl C, Oxenham AJ, Shamma SA. (2009). Temporal coherence in the perceptual organization and cortical representation of auditory scenes. Neuron 61:317-329. CrossRef Medline
21. Fritz JB., Elhilali M, David SV, Shamma SA. (2007). Auditory attention-focusing the searchlight on sound. Curr Opin Neurobiol 17:437-455. CrossRef Medline
22. Ghitza O. (2011). Linking speech perception and neurophysiology: speech decoding guided by cascaded oscillators locked to the input rhythm. Front Psychol 2:130. Medline
23. Giraud AL., Poeppel D. (2012). Cortical oscillations and speech processing: emerging computational principles and operations. Nat Neurosci 15: 511-517. CrossRef Medline
24. Grant KW. (2001). The effect of speechreading on masked detection thresholds for filtered speech. J Acoust Soc Am 109:2272-2275. CrossRef Medline
25. Grant KW., Seitz PF. (2000). The use of visible speech cues for improving auditory detection of spoken sentences. J Acoust Soc Am 108:1197-1208. CrossRef Medline
26. Hadar U., Steiner TJ, Grant EC, Rose FC. (1983). Head movement correlates of juncture and stress at sentence level. Lang Speech 26:117-129. Medline
27. Hafter E., Sarampalis A, Loui P. (2007). Auditory attention and filters. In: Auditory perception of sound sources. (Yost W, Popper AN, Fay RR, eds), pp 115-142. New York: Springer.
28. Hämäläinen MS, Ilmoniemi RJ. (1994). Interpreting magnetic fields of the brain: minimum norm estimates. Med Biol Eng Comput 32:35-42. CrossRef Medline
29. Helfer KS., Freyman RL. (2005). The role of visual speech cues in reducing energetic and informational masking. J Acoust Soc Am 117:842-849. CrossRef Medline
30. Hertrich I., Dietrich S, Trouvain J, Moos A, Ackermann H. (2012). Magnetic brain activity phase-locked to the envelope, the syllable onsets, and the fundamental frequency of a perceived speech signal. Psychophysiology 49:322-334. CrossRef Medline
31. Hillyard SA., Hink RF, Schwent VL, Picton TW. (1973). Electrical signs of selective attention in the human brain. Science 182:177-180. CrossRef Medline
32. Hubel DH., Henson CO, Rupert A, Galambos R. (1959). "Attention" units in the auditory cortex. Science 129:1279-1280. CrossRef Medline
33. Jones MR., Johnston HM, Puente J. (2006). Effects of auditory pattern structure on anticipatory and reactive attending. Cogn Psychol 53:59-96. CrossRef Medline
34. Jung TP., Makeig S, Westerfield M, Townsend J, Courchesne E, Sejnowski TJ. (2000). Removal of eye activity artifacts from visual event-related potentials in normal and clinical subjects. Clin Neurophysiol 111: 1745-1758. CrossRef Medline
35. Kayser C., Petkov CI, LogothetisNK. (2008). Visual modulation of neurons in auditory cortex. Cereb Cortex 18:1560-1574. Medline
36. Kerlin JR., Shahin AJ, Miller LM. (2010). Attentional gain control of ongoing cortical speech representations in a "cocktail party." J Neurosci 30:620-628. CrossRef Medline
37. Kim J., Davis C. (2003). Hearing foreign voices: does knowing what is said affect visual-masked-speech detection? Perception 32:111-120. CrossRef Medline
38. Lakatos P., Shah AS, Knuth KH, Ulbert I, Karmos G, Schroeder CE. (2005). An oscillatory hierarchy controlling neuronal excitability and stimulus processing in the auditory cortex. J Neurophysiol 94:1904-1911. CrossRef Medline
39. Lakatos P., Chen CM, O'Connell MN, Mills A, Schroeder CE. (2007). Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53:279-292. CrossRef Medline
40. Lakatos P., O'Connell MN, Barczak A, Mills A, Javitt DC, Schroeder CE (2009) The leading sense: supramodal control of neurophysiological context by attention. Neuron 64:419-430. CrossRef Medline
41. Lalor EC., Foxe JJ. (2010). Neural responses to uninterrupted natural speech can be extracted with precise temporal resolution. Eur J Neurosci 31:189-193. CrossRef Medline
42. Large EW., Jones MR. (1999). The dynamics of attending: how people track time-varying events. Psychological Rev 106:119-159. CrossRef
43. Luo H., Poeppel D. (2007). Phase patterns of neuronal responses reliably discriminate speech in human auditory cortex. Neuron 54:1001-1010. CrossRef Medline
44. Luo H., Liu Z, Poeppel D. (2010). Auditory cortex tracks both auditory and visual stimulus dynamics using low-frequency neuronal phase modulation. PLoS Biol 8:e1000445. CrossRef Medline
45. McDermott JH. (2009). The cocktail party problem. Curr Biol 19: R1024-R1027. CrossRef Medline
46. McGettigan C., Faulkner A, Altarelli I, Obleser J, Baverstock H, Scott SK (2012) Speech comprehension aided by multiple modalities: behavioural and neural interactions. Neuropsychologia 50:762-776. CrossRef Medline
47. Mesgarani N., Chang EF. (2012). Selective cortical representation of attended speaker in multi-talker speech perception. Nature 485:233-236. CrossRef Medline
48. Mizuseki K., Sirota A, Pastalkova E, BuzsákiG. (2009). Theta oscillations provide temporal windows for local circuit computation in the entorhinalhippocampal loop. Neuron 64:267-280. CrossRef Medline
49. Moore BCJ., Gockel H. (2002). Factors influencing sequential stream segregation. Acta Acustica 88:320-332.
50. MorayN. (1959). Attention in dichotic listening: affective cues and the influence of instructions. Q J Exp Psychol 11:56-60. CrossRef
51. Munhall KG., Jones JA, Callan DE, Kuratate T, Vatikiotis-Bateson E. (2004). Visual prosody and speech intelligibility. Psychol Sci 15:133-137. CrossRef Medline
52. Musacchia G., Schroeder CE. (2009). Neuronal mechanisms, response dynamics and perceptual functions of multisensory interactions in auditory cortex. Hear Res 258:72-79. CrossRef Medline
53. Nobre AC., Coull JT. (2010). Attention and time. Oxford, UK: Oxford UP.
54. Nobre A., Correa A, Coull J. (2007). The hazards of time. Curr Opin Neurobiol 17:465-470. CrossRef Medline
55. Oldfield RC. (1971). The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia 9:97-113. CrossRef Medline
56. O'Neill JJ. (1954). Contributions of the visual components of oral symbols to speech comprehension. J Speech Hearing Disord 19:429-439. Medline
57. Oostenveld R., Fries P, Maris E, Schoffelen J-M. (2011). Fieldtrip: open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data. Comput Intel Neurosci 2011.
58. Peelle JE., Gross J, Davis MH. (2012). Phase-locked responses to speech in human auditory cortex are enhanced during comprehension. Cereb Cortex. Advance online publication. Retrieved May 17, 2012. doi:10.1093/ cercor/bhs118. CrossRef
59. Rosen S. (1992). Temporal information in speech: acoustic, auditory and linguistic aspects. Philos Trans R Soc Lond B Biol Sci 336:367-373. CrossRef Medline
60. Ross LA., Saint-Amour D, Leavitt VM, Javitt DC, Foxe JJ. (2007). Do you see what i am saying? Exploring visual enhancement of speech comprehension in noisy environments. Cereb Cortex 17:1147-1153. Medline
61. Scarborough R., Keating P, Mattys SL, Cho T, Alwan A. (2009). Optical pho-netics and visual perception of lexical and phrasal stress in English. Lang Speech 52:135-175. CrossRef Medline
62. Schoffelen JM., Gross J. (2009). Source connectivity analysis with MEG and EEG. Hum Brain Mapp 30:1857-1865. CrossRef Medline
63. Schroeder CE., Lakatos P, Kajikawa Y, Partan S, Puce A. (2008). Neuronal oscillations and visual amplification of speech. Trends Cogn Sci 12:106-113. CrossRef Medline
64. Schwartz JL., Berthommier F, Savariaux C. (2004). Seeing to hear better: evidence for early audio-visual interactions in speech identification. Cognition 93:B69-B78. CrossRef Medline
65. Sekiyama K., Kanno I, Miura S, Sugita Y. (2003). Auditory-visual speech perception examined by fMRI and PET. Neurosci Res 47:277-287. CrossRef Medline
66. Shamma SA., Elhilali M, Micheyl C. (2011). Temporal coherence and attention in auditory scene analysis. Trends Neurosci 34:114-123. CrossRef Medline
67. Shannon RV., Zeng FG, Kamath V, Wygonski J, Ekelid M. (1995). Speech recognition with primarily temporal cues. Science 270:303-304. CrossRef Medline
68. Studebaker GA. (1985). A "rationalized" arcsine transform. J Speech Hear Res 28:455-462. Medline
69. Sumby WH., Pollack I. (1954). Visual contribution to speech intelligibility in noise. J Acoust Soc Am 26:212-215. CrossRef
70. Theunissen FE., David SV, Singh NC, Hsu A, Vinje WE, Gallant JL. (2001). Estimating spatio-temporal receptive fields of auditory and visual neurons from their responses to natural stimuli. Network 12:289-316. CrossRef Medline
71. Tiitinen H., Sinkkonen J, Reinikainen K, Alho K, Lavikainen J, Näätänen R (1993) Selective attention enhances the auditory 40-hz transient response in humans. Nature 364:59-60. CrossRef Medline
72. van Wassenhove V, Grant KW, Poeppel D. (2005). Visual speech speeds up the neural processing of auditory speech. Proc Natl Acad Sci U S A 102: 1181-1186. CrossRef Medline
73. Woldorff MG., Gallen CC, Hampson SA, Hillyard SA, Pantev C, Sobel D, Bloom FE. (1993). Modulation of early sensory processing in human auditory cortex during auditory selective attention. Proc Natl Acad Sci U S A 90:8722-8726. CrossRef Medline
74. Wood N., Cowan N. (1995). The cocktail party phenomenon revisited: how frequent are attention shifts to one's name in an irrelevant auditory channel? J Exp Psychol Learn Mem Cogn 21:255-260. CrossRef Medline
75. Zion Golumbic EM, Poeppel D, Schroeder CE. (2012). Temporal context in speech processing and attentional stream selection: a behavioral and neural perspective. Brain Lang 122:151-161. CrossRef Medline