Cortical integration of audio-visual speech and non-speech stimuli

Brain and Cognition

Volumn 74, Issue 2, 2010, Pages 97-106

  Wyk, Brent C Vander ^a   Ramsay, Gordon J ^a   Hudac, Caitlin M ^a   Jones, Warren ^a   Lin, David ^a   Klin, Ami ^a   Lee, Su Mei ^a   Pelphrey, Kevin A ^a  

^a YALE UNIVERSITY SCHOOL OF MEDICINE   (United States)

Author keywords

Audio visual; FMRI; Multi modal processing; Speech

Indexed keywords

ADULT; ARTICLE; AUDITORY STIMULATION; CONTROLLED STUDY; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; FUSIFORM GYRUS; HUMAN; HUMAN EXPERIMENT; INFERIOR FONTAL GYRUS; LANGUAGE PROCESSING; MALE; MIDDLE TEMPORAL GYRUS; NERVE CELL NETWORK; NORMAL HUMAN; OCCIPITAL CORTEX; POSTERIOR SUPERIOR TEMPORAL SULCUS; PRIORITY JOURNAL; SPEECH PERCEPTION; TASK PERFORMANCE; TELENCEPHALON; VISUAL INFORMATION; VISUAL STIMULATION;

ACOUSTIC STIMULATION; ADULT; ANALYSIS OF VARIANCE; AUDITORY PERCEPTION; BRAIN MAPPING; CEREBRAL CORTEX; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; MAGNETIC RESONANCE IMAGING; SPEECH; VISUAL PERCEPTION;

EID: 77956175452     PISSN: 02782626     EISSN: 10902147     Source Type: Journal    
DOI: 10.1016/j.bandc.2010.07.002     Document Type: Article

References (74)

1. Alpert G.F., Hein G., Tsai N., Naumer M.J., Knight R.T. Temporal characteristics of audiovisual information processing. Journal of Neuroscience 2008, 28:5344-5349.
2. Alsius A., Navarra J., Campbell R., Soto-Faraco S. Audiovisual integration of speech falters under high attention demands. Current Biology 2005, 15:839-843.
3. Arnold P., Hill F. Bisensory augmentation: A speech reading advantage when speech is clearly audible and intact. British Journal of Psychology 2001, 92:339-355.
4. Barraclough N.E., Xiao D., Baker C.I., Oram M.W., Perrett D.I. Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions. Journal of Cognitive Neuroscience 2005, 17:377-391.
5. Baumann O., Greenlee M.W. Neural correlates of coherent audiovisual motion perception. Cerebral Cortex 2007, 17:1433-1443.
6. Beauchamp M.S., Lee K.E., Argall B.D., Martin A. Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 2004, 41:809-823.
7. Bernstein L.E., Lu Z.L., Jiang J.T. Quantified acoustic-optical speech signal incongruity identifies cortical sites of audiovisual speech processing. Brain Research 2008, 1242:172-184.
8. Bischoff M., Walter B., Blecker C.R., Morgen K., Vaitl D., Sammer G. Utilizing the ventriloquism-effect to investigate audio-visual binding. Neuropsychologia 2007, 45:578-586.
9. Boynton G.M., Engel S.A., Glover G.H., Heeger D.J. Linear systems analysis of functional magnetic resonance imaging in human V1. Journal of Neuroscience 1996, 16:4207-4221.
10. Calvert G.A., Brammer M.J., Bullmore E.T., Campbell R., Iversen S.D., David A.S. Response amplification in sensory-specific cortices during crossmodal binding. Neuroreport 1999, 10:2619-2623.
11. Calvert G.A., Campbell R., Brammer J. Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Current Biology 2000, 10:649-657.
12. Calvert G.A., Hansen P.C., Iversen S.D., Brammer M.J. Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage 2001, 14:427-438.
13. Capek C.M., MacSweeney M., Woll B., Waters D., McGuire P.K., David A.S., et al. Cortical circuits for silent speechreading in deaf and hearing people. Neuropsychologia 2008, 46:1233-1241.
14. Cotton J.C. Normal 'visual hearing'. Science 1935, 82:592-593.
15. Degerman A., Rinne T., Pekkola J., Autti T., Jääskeläinen I.P., Sams M., et al. Human brain activity associated with audiovisual perception and attention. Neuroimage 2007, 34:1683-1691.
16. Dhamala M., Assisi C.G., Jirsa V.K., Steinberg F.L., Kelso J.A.S. Multisensory integration for timing engages different brain networks. Neuroimage 2007, 34:764-773.
17. Dixon N.F., Spitz L.T. The detection of audiovisual desynchrony. Perception 1980, 9:719-721.
18. Eckert M.A., Kamdar N.V., Chang C.E., Beckmann C.F., Greicius M.D., Menon V. A cross-modal system linking primary auditory and visual cortices: Evidence from intrinsic fMRI connectivity analysis. Human Brain Mapping 2008, 29:848-857.
19. Fairhall S.L., Macaluso E. Spatial attention can modulate audiovisual integration at multiple cortical and subcortical sites. European Journal of Neuroscience 2009, 29:1247-1257.
20. Forman S.D., Cohen J.D., Fitzgerald M., Eddy W.F., Mintun M.A., Noll D.C. Improved assessment of significant activation in functional magnetic resonance imaging (fMRI): Use of a cluster-size threshold. Magnetic Resonance Medicine 1995, 33:636-647.
21. Frens M.A., Van Opstal A.J., Van der Willigen R.F. Spatial and temporal factors determine auditory-visual interactions in human saccadic eye movements. Perception & Psychophysics 1995, 57:802-816.
22. Goebel R., Esposito F., Formisano E. Analysis of functional image analysis contest (FIAC) data with brainvoyager QX: From single-subject to cortically aligned group general linear model analysis and self-organizing group independent component analysis. Human Brain Mapping 2006, 27:392-401.
23. Grant K.W., Ardell L.H., Kuhl P.K., Sparks D.W. The contribution of fundamental frequency, amplitude envelope, and voicing duration cues to speechreading in normal-hearing subjects. Journal of the Acoustic Society of America 1985, 77:671-676.
24. Grant K.W., Seitz P.F. The use of visible speech cues for improving auditory detection of spoken sentences. Journal of the Acoustic Society of America 2000, 108:1197-1208.
25. Grant K.W., van Wassenhove V., Poeppel D. Detection of auditory (cross-spectral) and auditory-visual (cross-modal) synchrony. Speech Communication 2004, 44:43-53.
26. Hein G., Doehrmann O., Muller N.G., Kaiser J., Muckli L., Naumer M.J. Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. Journal of Neuroscience 2007, 27:7881-7887.
27. Hertrich I., Dietrich S., Ackermann H. Cross-modal interactions during perception of audiovisual speech and non-speech signals: An fMRI study. Journal of Cognitive Neuroscience 2010, 10.1162/jocn.2010.21421.
28. Hocking J., Price C.J. The role of the posterior superior temporal sulcus in audiovisual processing. Cerebral Cortex 2008, 18:2439-2449.
29. Hocking J., Price C.J. Dissociating verbal and nonverbal audiovisual object processing. Brain Language 2009, 108(2):89-96.
30. Hughes H.C., Reuter-Lorenz P.A., Nozawa G., Fendrich R. Visual-auditory interactions in sensorimotor processing: Saccades versus manual responses. Journal of Experimental Psychology: Human Perception and Performance 1994, 20:131-153.
31. Jones J.A., Callan D.E. Brain activity during audiovisual speech perception: An fMRI study of the McGurk effect. Neuroreport 2003, 14:1129-1133.
32. Kamachi M., Hill H., Lander L., Vatikiotis-Bateson E. 'Putting the face to the voice': Matching identity across modality. Current Biology 2003, 13:1709-1714.
33. Kang E., Lee D.S., Kang H.J., Hwang C.H., Oh S.H., Kim C.S., et al. The neural correlates of cross-modal interaction in speech perception during a semantic decision task on sentences: A PET study. Neuroimage 2006, 32:423-431.
34. Kayser C., Petkov C.I., Logothetis N.K. Visual modulation of neurons in auditory cortex. Cerebral Cortex 2008, 18:1560-1574.
35. King A.J., Palmer A.R. Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Experimental Brain Research 1985, 60:492-500.
36. Lancaster J.L., Woldorff M.G., Parsons L.M., Liotti M., Freitas C.S., Rainey L., Kochunov P.V., Nickerson D., Mikiten S.A., Fox P.T. Automated Talairach Atlas labels for functional brain mapping. Human Brain Mapping 2000, 10:120-131.
37. Laurienti P.J., Wallace M.T., Maldjian J.A., Susi C.M., Stein B.E., Burdette J.H. Cross-modal sensory processing in the anterior cingulate and medial prefrontal cortices. Human Brain Mapping 2003, 19:213-223.
38. Macaluso E., George N., Dolan R., Spence C., Driver J. Spatial and temporal factors during processing of audiovisual speech: A PET study. NeuroImage 2004, 21(2):725-732.
39. Martuzzi R., Murray M.M., Michel C.M., Thiran J.P., Maeder P.P., Clarke S., et al. Multisensory interactions within human primary cortices revealed by BOLD dynamics. Cerebral Cortex 2007, 17:1672-1679.
40. Massarro D.W., Cohen M.M. Speech perception in perceivers with hearing loss: Synergy of multiple modalities. Journal of Speech and Language Hearing Research 1983, 42:21-41.
41. McGrath M., Summerfield Q. Intermodal timing relations and audio-visual speech recognition by normal-hearing adults. Journal of the Acoustic Society of America 1985, 77:678-685.
42. McGurk H., MacDonald J. Hearing lips and seeing voices. Nature 1976, 265:746-748.
43. Meredith M.A., Stein B.E. Interactions among converging sensory inputs in the superior colliculus. Science 1983, 221:389-391.
44. Meredith M.A., Stein B.E. Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. Journal of Neurophysiology 1986, 56:640-662.
45. Miller J. Divided attention: Evidence for coactivation with redundant signals. Cognitive Psychology 1982, 14:247-279.
46. Miller L.M., D'Esposito M. Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. Journal of Neuroscience 2005, 25:5884-5893.
47. Neely K.K. Effect of visual factors on the intelligibility of speech. Journal of the Acoustic Society of America 1956, 28:1275-1277.
48. Ojanen V., Mottonen R., Pekkola J., Jaaskelainen I.P., Joensuu R., Autti T., et al. Processing of audiovisual speech in Broca's area. Neuroimage 2005, 25:333-338.
49. Olson I.R., Gatenby J.C., Gore J.C. A comparison of bound and unbound audio-visual information processing in the human cerebral cortex. Cognitive Brain Research 2002, 14:129-138.
50. Reisberg D., McLean J., Goldfield A. Easy to hear but hard to understand: A speechreading advantage with intact auditory stimuli. Hearing by eye: The psychology of lip-reading 1987, 97-113. Lawrence Erlbaum Associates, London. B. Dodd, R. Campbell (Eds.).
51. Remez R.E., Rubin P.E., Pisoni D.B., Carrell T.D. Speech perception without traditional speech cues. Science 1981, 212:947-949.
52. Robert-Ribes J., Schwartz J.-L., Escudier P. A comparison of models for fusion of the auditory and visual sensors in speech perception. Artificial Intelligence Review 1995, 9:81-104.
53. Robins D.L., Hunyadi E., Schultz R.T. Superior temporal activation in response to dynamic audio-visual emotional cues. Brain and Cognition 2009, 69:269-278.
54. Rosen S.M., Fourcin A.J., Moore B.C.J. Voice pitch as an aid to lipreading. Nature 1981, 291:150-152.
55. Saito D.N., Yoshimura K., Kochiyama T., Okada T., Honda M., Sadato N. Cross-modal binding and activated attentional networks during audio-visual speech integration: A functional MRI study. Cerebral Cortex 2005, 15:1750-1760.
56. Sekiyama K., Kanno I., Miura S., Sugita Y. Auditory-visual speech perception examined by fMRl and PET. Neuroscience Research 2003, 47:277-287.
57. Sestieri C., Di Matteo R., Ferretti A., Del Gratta C., Caulo M., Tartaro A., et al. 'What' versus 'Where' in the audiovisual domain: An fMRI study. Neuroimage 2006, 33:672-680.
58. Shannon R.V., Zeng F.-G., Kamath V., Wygonski J., Ekelid M. Speech recognition with primarily temporal cues. Science 1995, 270:303-304.
59. Stanford T.R., Stein B.E. Superadditivity in multisensory integration: Putting the computation in context. Neuroreport 2007, 18:787-792.
60. Stevenson R.A., Altieri N.A., Kim S., Pisoni D.B., James T.W. Neural processing of asynchronous audiovisual speech perception. NeuroImage 2010, 49:3308-3318.
61. Stevenson R.A., Geoghegan M.L., James T.W. Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects. Experimental Brain Research 2007, 179:85-95.
62. Stevenson R.A., James T.W. Audiovisual integration in human superior temporal sulcus: Inverse effectiveness and the neural processing of speech and object recognition. Neuroimage 2009, 44:1210-1223.
63. Stevenson R.A., Kim S., James T.W. An additive-factors design to disambiguate neuronal and areal convergence: Measuring multisensory interactions between audio, visual, and haptic sensory streams using fMRI. Experimental Brain Research 2009, 198(2):183-194.
64. Sumby W.H., Pollack I. Visual contribution to speech intelligibility in noise. Journal of the Acoustic Society of America 1954, 26:212-215.
65. Summerfield Q. Use of visual information for phonetic perception. Phonetica 1979, 36:314-331.
66. Summerfield Q. Some preliminaries to a comprehensive account of audio-visual speech perception. Hearing by eye: The psychology of lip-reading 1987, 3-51. Lawrence Erlbaum Associates, London. B. Dodd, R. Campbell (Eds.).
67. Summerfield Q., McGrath M. Detection and resolution of audio-visual incompatibility in the perception of vowels. Journal of Experimental Psychology 1984, 36(A):51-74.
68. Szycik G.R., Tausche P., Munte T.F. A novel approach to study audiovisual integration in speech perception: Localizer fMRI and sparse sampling. Brain Research 2008, 1220:142-149.
69. Talairach J., Tournoux P. Co-planar stereotaxic atlas of the human brain: 3-Dimensional proportional system: An approach to cerebral imaging 1988, Stuttgart, Thieme.
70. Taylor K.I., Moss H.E., Stamatakis E.A., Tyler L.K. Binding crossmodal object features in perirhinal cortex. Proceedings of the National Academy of Sciences of the United States of America 2006, 103:8239-8244.
71. Watkins S., Shams L., Josephs O., Rees G. Activity in human V1 follows multisensory perception. Neuroimage 2007, 37:572-578.
72. Watkins S., Shams L., Tanaka S., Haynes J.D., Rees G. Sound alters activity in human V1 in association with illusory visual perception. Neuroimage 2006, 31:1247-1256.
73. Werner, S., & Noppeney, U. (2009). Superadditive responses in superior temporal sulcus predict audiovisual benefits in object categorization. Cerebral Cortex. doi: Retrieved 29.06.10 [Advance online publication]. doi:10.1093/cercor/bhp248.
74. Wright T.M., Pelphrey K.A., Allison T., McKeown M.J., McCarthy G. Polysensory interactions along lateral temporal regions evoked by audiovisual speech. Cerebral Cortex 2003, 13:1034-1043.