Quantified acoustic-optical speech signal incongruity identifies cortical sites of audiovisual speech processing

Brain Research

Volumn 1242, Issue , 2008, Pages 172-184

  Bernstein, Lynne E ^a,b   Lu, Zhong Lin ^b   Jiang, Jintao ^a  

^a HOUSE EAR INSTITUTE   (United States)

^b UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

Author keywords

Audiovisual speech perception; fMRI; Multisensory processing; Speech signal processing

Indexed keywords

ADULT; ARTICLE; AUDITORY STIMULATION; BRAIN CORTEX; CINGULATE GYRUS; CONTROLLED STUDY; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; HUMAN EXPERIMENT; MALE; OUTCOME ASSESSMENT; PRIORITY JOURNAL; SPEECH ANALYSIS; SPEECH PERCEPTION; VISUAL STIMULATION;

ADULT; BRAIN; FEMALE; HUMANS; MAGNETIC RESONANCE IMAGING; MALE; SPEECH PERCEPTION; VISUAL PERCEPTION;

EID: 55649097537     PISSN: 00068993     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.brainres.2008.04.018     Document Type: Article

References (61)

1. Allott R. The motor theory of language. In: von Raffler-Engel W., Wind J., and Jonker A. (Eds). Studies in Language Origins II. (1991), Benjamins, Amsterdam
2. Barker J.P., and Berthommier F. Estimation of speech acoustics from visual speech features: a comparison of linear and non-linear models. In: Massaro D.W. (Ed). AVSP 1999 International Conference on Auditory-Visual Speech Processing (1999) 112-117 Santa Cruz, CA
3. Beauchamp M.S. Statistical criteria in FMRI studies of multisensory integration. Neuroinformatics 3 (2005) 93-113
4. Bernstein L.E. Visual speech perception. In: Vatikiotis-Bateson E., Bailly G., and Perrier P. (Eds). Audio-Visual Speech Processing (2006), MIT, Cambridge
5. Bernstein L.E., Demorest M.E., and Tucker P.E. Speech perception without hearing. Percept. Psychophys. 62 (2000) 233-252
6. Bernstein L.E., Auer Jr. E.T., and Moore J.K. Audiovisual speech binding: convergence or association?. In: Calvert G.A., Spence C., and Stein B.E. (Eds). Handbook of Multisensory Processing (2004), MIT, Cambridge, MA 203-223
7. Bernstein L.E., Auer Jr. E.T., Wagner M., and Ponton C.W. Spatio-temporal dynamics of audiovisual speech processing. NeuroImage 39 (2008) 423-435
8. Callan D.E., Jones J.A., Munhall K., Callan A.M., Kroos C., and Vatikiotis-Bateson E. Neural processes underlying perceptual enhancement by visual speech gestures. NeuroReport 14 (2003) 2213-2218
9. Calvert G.A. Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb. Cortex 11 (2001) 1110-1123
10. Calvert G.A., Brammer M.J., Bullmore E.T., Campbell R., Iversen S.D., and David A.S. Response amplification in sensory-specific cortices during crossmodal binding. NeuroReport 10 (1999) 2619-2623
11. Calvert G.A., Campbell R., and Brammer M.J. Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr. Biol. 10 (2000) 649-657
12. Campbell, R., in press. The processing of audio-visual speech: empirical and neural bases. Philos. Trans. R. Soc. Lond. B Biol. Sci. (Electronic publication ahead of print).
13. Fadiga L.G., Craighero G., Buccino, and Rizzolatti G. Speech listening specifically modulates the excitability of tongue muscles: a TMS study. Eur. J. Neurosci. 15 (2002) 399-402
14. Fischl B., Sereno M.I., Tootell R.B., and Dale A.M. High-resolution intersubject averaging and a coordinate system for the cortical surface. Hum. Brain Mapp. 8 (1999) 272-284
15. Genovese C.R., A., L.N., T., and N. Thresholding of statistical maps in functional neuroimaging using the false discovery rate. NeuroImage 15 (2002) 870-878
16. Golestani N., and Zatorre R.J. Learning new sounds of speech: reallocation of neural substrates. Neuroimage 21 (2004) 494-506
17. Green K.P., and Kuhl P.K. The role of visual information in the processing of place and manner features in speech perception. Percept. Psychophys. 45 (1989) 34-42
18. Green K.P., and Kuhl P.K. Integral processing of visual place and auditory voicing information during phonetic perception. J. Exp. Psychol.: Hum. Percept. Perform. 17 (1991) 278-288
19. Green K.P., and Norrix L.W. Acoustic cues to place of articulation and the McGurk effect: the role of release bursts, aspiration, and formant transitions. J. Speech Lang. Hear. Res. 40 (1997) 646-665
20. Hall D.A., Fussell C., and Summerfield A.Q. Reading fluent speech from talking faces: typical brain networks and individual differences. J. Cogn. Neurosci. 17 (2005) 939-953
21. Hasson U., Skipper J.I., Nusbaum H.C., and Small S.L. Abstract coding of audiovisual speech: beyond sensory representation. Neuron 56 (2007) 1116-1126
22. Hickok G., and Poeppel D. Dorsal and ventral streams: a framework for understanding aspects of the functional anatomy of language. Cognition 92 (2004) 67-99
23. Jacquemot C., Pallier C., LeBihan D., Dehaene S., and Dupoux E. Phonological grammar shapes the auditory cortex: a functional magnetic resonance imaging study. J. Neurosci. 23 (2003) 9541-9546
24. Jiang J., Alwan A., Keating P., Auer Jr. E.T., and Bernstein L.E. On the relationship between face movements, tongue movements, and speech acoustics. EURASIP J. Appl. Signal Processing: Special issue on Joint Audio-Visual Speech Processing (2002) 1174-1188
25. Jiang J., Bernstein L.E., and Auer Jr. E.T. Perception of congruent and incongruent audiovisual speech stimuli. In: Vatikiotis-Bateson E., Burnham D., and Fels S. (Eds). AVSP '05 (2005), Causal Productions, British Columbia, Canada
26. Jiang J., Alwan A., Keating P.A., Auer Jr. E.T., and Bernstein L.E. Similarity structure in visual speech perception and optical phonetics. Percept. Psychophys. 69 (2007) 1070-1083
27. Jiang, J., Bernstein, L.E., in preparation. Quantification of the physical relationships between acoustic and optical signals that drive audiovisual speech perception.
28. Jones J.A., and Callan D.E. Brain activity during audiovisual speech perception: an fMRI study of the McGurk effect. NeuroReport 14 (2003) 1129-1133
29. Kailath T., Sayed A.H., and Hassibi B. Linear Estimation (2000), Prentice Hall
30. Kaiser J., Hertrich I., Ackermann H., and Lutzenberger W. Gamma-band activity over early sensory areas predicts detection of changes in audiovisual speech stimuli. NeuroImage 30 (2006) 1376-1382
31. Kim J.-O., and Mueller C.W. Introduction to Factor Analysis: What it is and How to do it Vol. 07-013 (1985), SAGE, Beverly Hills
32. Kohler E., Keysers C., Umilta M.A., Fogassi L., Gallese V., and Rizzolatti G. Hearing sounds understanding actions: action representation in mirror neurons. Science 297 (2002) 846-848
33. Lamoré P.J., Huiskamp T.M., van Son N.J., Bosman A.J., and Smoorenburg G.F. Auditory, visual and audiovisual perception of segmental speech features by severely hearing-impaired children. Audiology 37 (1998) 396-419
34. Laurienti P.J., Perrault T.J., Stanford T.R., Wallace M.T., and Stein B.E. On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies. Exp. Brain Res. 166 (2005) 289-297
35. Liberman A.M., and Mattingly I.G. The motor theory of speech perception revised. Cognition 21 (1985) 1-36
36. Ludman C.N., Summerfield A.Q., Hall D., Elliott M., Foster J., Hykin J.L., Bowtell R., and Morris P.G. Lip-reading ability and patterns of cortical activation studied using fMRI. Br. J. Audiol. 34 (2000) 225-230
37. McGurk H., and MacDonald J. Hearing lips and seeing voices. Nature 264 (1976) 746-748
38. Mesulam M.M. From sensation to cognition. Brain 121 (1998) 1013-1052
39. Miller L.M., and d'Esposito M. Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J. Neurosci 25 (2005) 5884-5893
40. Moody-Antonio S., Takayanagi S., Masuda A., Auer J., E.T., Fisher L., and Bernstein L.E. Improved speech perception in adult prelingually deafened cochlear implant recipients. Otol. Neurotol. 26 (2005) 649-654
41. Mottonen R., Krause C.M., Tiippana K., and Sams M. Processing of changes in visual speech in the human auditory cortex. Cogn. Brain Res. 13 (2002) 417-425
42. Nichols T., Brett M., Andersson J., Wager T., and Poline J.B. Valid conjunction inference with the minimum statistic. Neuroimage 25 (2005) 653-660
43. Nock H.J., Iyengar G., and Neti C. Assessing face and speech consistency for monologue detection in video. Proceedings of the Tenth ACM International Conference on Multimedia (2002) 303-306 Juan-les-Pins, France.
44. Ojanen V., Mottonen R., Pekkola J., Jaaskelainen I.P., Joensuu R., Autti T., and Sams M. Processing of audiovisual speech in Broca's area. NeuroImage 25 (2005) 333-338
45. Paulesu E., Frith C.D., and Frackowiak R.S. The neural correlates of the verbal component of working memory. Nature 362 (1993) 342-345
46. Pekkola J., Laasonen M., Ojanen V., Autti T., Jaaskelainen I.P., Kujala T., and Sams M. Perception of matching and conflicting audiovisual speech in dyslexic and fluent readers: an fMRI study at 3 T. Neuroimage 29 (2006) 797-807
47. Raichle M.E., and Mintun M.A. Brain work and brain imaging. Annu. Rev. Neurosci. 29 (2006) 449-476
48. Raichle M.E., and Snyder A.Z. A default mode of brain function: a brief history of an evolving idea. Neuroimage 37 (2007) 1083-1090 Discussion 1097-1099
49. Rizzolatti G., Fadiga L., Gallese V., and Fogassi L. Premotor cortex and the recognition of motor actions. Cogn. Brain Res. 3 (1996) 131-141
50. Scott S.K., and Johnsrude I.S. The neuroanatomical and functional organization of speech perception. Trends Neurosci. 26 (2003) 100-107
51. Scott S.K., Blank C.C., Rosen S., and Wise R.J. Identification of a pathway for intelligible speech in the left temporal lobe. Brain 123 Pt 12 (2000) 2400-2406
52. Skipper J.I., Goldin-Meadow S., Nusbaum H.C., and Small S.L. Speech-associated gestures, Broca's area, and the human mirror system. Brain Lang. 101 (2007) 260-277
53. Skipper J.I., van Wassenhove V., Nusbaum H.C., and Small S.L. Hearing lips and seeing voices: how cortical areas supporting speech production mediate audiovisual speech perception. Cereb. Cortex 17 (2007) 2387-2399
54. Stein B.E., and Meredith M.A. The Merging of the Senses (1993), MIT, Cambridge, MA
55. Stevens K.N. Acoustic Phonetics (1998), MIT Press, Cambridge, MA
56. Sugamura N., and Itakura F. Speech analysis and synthesis methods developed at ECL in NTT - from LPC to LSP. Speech Commun. 5 (1986) 199-215
57. Sumby W.H., and Pollack I. Visual contribution to speech intelligibility in noise. J. Acoust. Soc. Am. 26 (1954) 212-215
58. Szycik, G.R., Tausche, P., Munte, T.F., in press. A novel approach to study audiovisual integration in speech perception: localizer fMRI and sparse sampling. Brain Res. (Electronic publication ahead of print).
59. van Atteveldt N.M., Formisano E., Goebel R., and Blomert L. Integration of letters and speech sounds in the human brain. Neuron 43 (2004) 271-282
60. Wright T.M., Pelphrey K.A., Allison T., McKeowin M.J., and McCarthy G. Polysensory interactions along lateral temporal regions evoked by audiovisual speech. Cereb. Cortex 13 (2003) 1034-1043
61. Yehia H., Rubin P., and Vatikiotis-Bateson E. Quantitative association of vocal-tract and facial behavior. Speech Commun. 26 (1998) 23-43