Audiovisual integration in human superior temporal sulcus: Inverse effectiveness and the neural processing of speech and object recognition

NeuroImage

Volumn 44, Issue 3, 2009, Pages 1210-1223

  Stevenson, Ryan A ^a   James, Thomas W ^a  

^a INDIANA UNIVERSITY   (United States)

Author keywords

Auditory; fMRI; Multisensory integration; Neuroimaging; Perception; Speech perception; Superadditivity; Vision

Indexed keywords

ADULT; ARTICLE; AUDIOVISUAL EQUIPMENT; AUDITORY STIMULATION; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; HUMAN EXPERIMENT; MALE; PRIORITY JOURNAL; SENSORY NERVE CELL; SPEECH; TEMPORAL CORTEX; VISUAL STIMULATION;

ADULT; BRAIN MAPPING; FEMALE; HUMANS; MALE; PATTERN RECOGNITION, VISUAL; SPEECH PERCEPTION; TASK PERFORMANCE AND ANALYSIS; TEMPORAL LOBE;

EID: 57649200617     PISSN: 10538119     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neuroimage.2008.09.034     Document Type: Article

References (75)

1. Allman B.L., and Meredith M.A. Multisensory processing in "unimodal" neurons: cross-modal subthreshold auditory effects in cat extrastriate visual cortex. J. Neurophysiol. 98 (2007) 545-549
2. Barraclough N.E., Xiao D., Baker C.I., Oram M.W., and Perrett D.I. Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions. J. Cogn. Neurosci. 17 (2005) 377-391
3. Beauchamp M.S. Statistical criteria in FMRI studies of multisensory integration. Neuroinformatics 3 (2005) 93-113
4. Beauchamp M.S., Argall B.D., Bodurka J., Duyn J.H., and Martin A. Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nat. Neurosci. 7 (2004) 1190-1192
5. Beauchamp M.S., Lee K.E., Argall B.D., and Martin A. Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41 (2004) 809-823
6. Benevento L.A., Fallon J., Davis B.J., and Rezak M. Auditory-visual interaction in single cells in the cortex of the superior temporal sulcus and the orbital frontal cortex of the macaque monkey. Exp. Neurol. 57 (1977) 849-872
7. Binder J.R., Frost J.A., Hammeke T.A., Rao S.M., and Cox R.W. Function of the left planum temporale in auditory and linguistic processing. Brain 119 Pt 4 (1996) 1239-1247
8. Bolognini N., Frassinetti F., Serino A., and Ladavas E. "Acoustical vision" of below threshold stimuli: interaction among spatially converging audiovisual inputs. Exp. Brain Res. 160 (2005) 273-282
9. Brainard D.H. The psychophysics toolbox. Spat. Vis. 10 (1997) 433-436
10. Bruce C., Desimone R., and Gross C.G. Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque. J. Neurophysiol. 46 (1981) 369-384
11. Bushara K.O., Grafman J., and Hallett M. Neural correlates of auditory-visual stimulus onset asynchrony detection. J. Neurosci. 21 (2001) 300-304
12. Callan D.E., Jones J.A., Munhall K., Kroos C., Callan A.M., and Vatikiotis-Bateson E. Multisensory integration sites identified by perception of spatial wavelet filtered visual speech gesture information. J. Cogn. Neurosci. 16 (2004) 805-816
13. Calvert G.A. Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb. Cortex 11 (2001) 1110-1123
14. Calvert G.A., Bullmore E.T., Brammer M.J., Campbell R., Williams S.C., McGuire P.K., Woodruff P.W., Iversen S.D., and David A.S. Activation of auditory cortex during silent lipreading. Science 276 (1997) 593-596
15. Calvert G.A., Campbell R., and Brammer M.J. Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr. Biol. 10 (2000) 649-657
16. Calvert G.A., Hansen P.C., Iversen S.D., and Brammer M.J. Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage 14 (2001) 427-438
17. Driver J., and Noesselt T. Multisensory interplay reveals crossmodal influences on 'sensory-specific' brain regions, neural responses, and judgments. Neuron 57 (2008) 11-23
18. Fowler C.A. Listeners do hear sounds, not tongues. J. Acoust. Soc. Am. 99 (1996) 1730-1741
19. Frens M.A., Van Opstal A.J., and Van der Willigen R.F. Spatial and temporal factors determine auditory-visual interactions in human saccadic eye movements. Percept. Psychophys 57 (1995) 802-816
20. Friston K.J., Rothshtein P., Geng J.J., Sterzer P., and Henson R.N. A critique of functional localizers. Neuroimage 30 (2006) 1077-1087
21. Hairston W.D., Laurienti P.J., Mishra G., Burdette J.H., and Wallace M.T. Multisensory enhancement of localization under conditions of induced myopia. Exp. Brain Res. 152 (2003) 404-408
22. Hairston W.D., Wallace M.T., Vaughan J.W., Stein B.E., Norris J.L., and Schirillo J.A. Visual localization ability influences cross-modal bias. J. Cogn. Neurosci. 15 (2003) 20-29
23. Hairston W.D., Hodges D.A., Burdette J.H., and Wallace M.T. Auditory enhancement of visual temporal order judgment. Neuroreport 17 (2006) 791-795
24. Hikosaka K., Iwai E., Saito H., and Tanaka K. Polysensory properties of neurons in the anterior bank of the caudal superior temporal sulcus of the macaque monkey. J. Neurophysiol. 60 (1988) 1615-1637
25. Hocking J., and Price C.J. The role of the posterior superior temporal sulcus in audiovisual processing. Cereb. Cortex 18 10 (2008) 1685-1694
26. Holmes N.P. The law of inverse effectiveness in neurons and behaviour: multisensory integration versus normal variability. Neuropsychologia 45 (2007) 3340-3345
27. James T.W., and Gauthier I. Repetition-induced changes in BOLD response reflect accumulation of neural activity. Hum. Brain Mapp. 27 (2006) 37-46
28. James T.W., Humphrey G.K., Gati J.S., Menon R.S., and Goodale M.A. The effects of visual object priming on brain activation before and after recognition. Curr. Biol. 10 (2000) 1017-1024
29. James T.W., Humphrey G.K., Gati J.S., Menon R.S., and Goodale M.A. Differential effects of viewpoint on object-driven activation in dorsal and ventral streams. Neuron 35 (2002) 793-801
30. James T.W., Culham J., Humphrey G.K., Milner A.D., and Goodale M.A. Ventral occipital lesions impair object recognition but not object-directed grasping: an fMRI study. Brain 126 (2003) 2463-2475
31. Johnson-Frey S.H., Newman-Norlund R., and Grafton S.T. A distributed left hemisphere network active during planning of everyday tool use skills. Cereb. Cortex 15 (2005) 681-695
32. Kayser C., Petkov C.I., Augath M., and Logothetis N.K. Integration of touch and sound in auditory cortex. Neuron 48 (2005) 373-384
33. Kuhl P.K., Williams K.A., and Meltzoff A.N. Cross-modal speech perception in adults and infants using nonspeech auditory stimuli. J. Exp. Psychol. Hum. Percept. Perform. 17 (1991) 829-840
34. Lachs L., and Hernandez L.R. Update: the Hoosier audiovisual multitalker database. In: Pisoni D.B. (Ed). Research on Spoken Language Processing (1998), Speech Research Laboratory, Indiana University, Bloomington, IN, 377-388
35. Laurienti P.J., Burdette J.H., Wallace M.T., Yen Y.F., Field A.S., and Stein B.E. Deactivation of sensory-specific cortex by cross-modal stimuli. J. Cogn. Neurosci. 14 (2002) 420-429
36. Laurienti P.J., Kraft R.A., Maldjian J.A., Burdette J.H., and Wallace M.T. Semantic congruence is a critical factor in multisensory behavioral performance. Experimental Brain Research 158 4 (2004) 405-414
37. Laurienti P.J., Perrault T.J., Stanford T.R., Wallace M.T., and Stein B.E. On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies. Exp. Brain Res. 166 (2005) 289-297
38. Liberman A.M., and Mattingly I.G. The motor theory of speech perception revised. Cognition 21 (1985) 1-36
39. Liberman A.M., and Mattingly I.G. A specialization for speech perception. Science 243 (1989) 489-494
40. Macaluso E., George N., Dolan R., Spence C., and Driver J. Spatial and temporal factors during processing of audiovisual speech: a PET study. Neuroimage 21 (2004) 725-732
41. Malach R., Reppas J.B., Benson R.R., Kwong K.K., Jiang H., Kennedy W.A., Ledden P.J., Brady T.J., Rosen B.R., and Tootell R.B. Object-related activity revealed by functional magnetic resonance imaging in human occipital cortex. Proc. Natl. Acad. Sci. U. S. A. 92 (1995) 8135-8139
42. Massaro D.W. Perceiving Talking Faces: From Speech Perception to a Behavioral Principle (1998), MIT Press, Cambridge, Mass
43. Materna S., Dicke P.W., and Their P. Dissociable roles of the superior temporal sulcus and the intraparietal sulcus in joint attention: a functional magnetic resonance imaging study. J Cogn. Neurosci. 20 1 (2007) 108-119
44. Meredith M.A., and Stein B.E. Interactions among converging sensory inputs in the superior colliculus. Science 221 (1983) 389-391
45. Meredith M.A., and Stein B.E. Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res. 365 (1986) 350-354
46. Meredith M.A., and Stein B.E. Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J. Neurophysiol. 56 (1986) 640-662
47. Meredith M.A., and Stein B.E. Spatial determinants of multisensory integration in cat superior colliculus neurons. J. Neurophysiol. 75 (1996) 1843-1857
48. Meredith M.A., Nemitz J.W., and Stein B.E. Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. J. Neurosci. 7 (1987) 3215-3229
49. Miller L.M., and D'Esposito M. Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J. Neurosci. 25 (2005) 5884-5893
50. Moore B.C.J. An Introduction to the Psychology of Hearing, 5th ed (2003), Academic Press, Amsterdam; Boston
51. Mozolic J.L., Hugenschmidt C.E., Peiffer A.M., and Laurienti P.J. Modality-specific selective attention attenuates multisensory integration. Exp. Brain Res. 184 1 (2007) 39-52
52. Pelli D.G. The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spat. Vis. 10 (1997) 437-442
53. Pelphrey K.A., Morris J.P., Michelich C.R., Allison T., and McCarthy G. Functional anatomy of biological motion perception in posterior temporal cortex: an FMRI study of eye, mouth and hand movements. Cereb. Cortex 15 (2005) 1866-1876
54. Perrault Jr. T.J., Vaughan J.W., Stein B.E., and Wallace M.T. Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. J. Neurophysiol. 93 (2005) 2575-2586
55. Puce A., Allison T., Bentin S., Gore J.C., and McCarthy G. Temporal cortex activation in humans viewing eye and mouth movements. J. Neurosci. 18 (1998) 2188-2199
56. Remez R.E., Rubin P.E., Pisoni D.B., and Carrell T.D. Speech perception without traditional speech cues. Science 212 (1981) 947-949
57. Remez R.E., Rubin P.E., Berns S.M., Pardo J.S., and Lang J.M. On the perceptual organization of speech. Psychol. Rev. 101 (1994) 129-156
58. Repp B.H. Phonetic trading relations and context effects: new experimental evidence for a speech mode of perception. Psychol. Bull. 92 (1982) 81-110
59. Saxe R., Brett M., and Kanwisher N. Divide and conquer: a defense of functional localizers. Neuroimage, 30 (2006) 1088-1099
60. Scott S.K., and Johnsrude I.S. The neuroanatomical and functional organization of speech perception. Trends Neurosci. 26 (2003) 100-107
61. Scott S.K., Blank C.C., Rosen S., and Wise R.J. Identification of a pathway for intelligible speech in the left temporal lobe. Brain 123 Pt 12 (2000) 2400-2406
62. Semple M.N., and Scott B.H. Cortical mechanisms in hearing. Curr. Opin. Neurobiol. 13 (2003) 167-173
63. Senkowski D., Talsma D., Grigutsch M., Herrmann C.S., and Woldorff M.G. Good times for multisensory integration: effects of the precision of temporal synchrony as revealed by gamma-band oscillations. Neuropsychologia 45 (2007) 561-571
64. Serino A., Bassolino M., Farne A., and Ladavas E. Extended multisensory space in blind cane users. Psychol. Sci. 18 (2007) 642-648
65. Sheffert S.M., Lachs L., and Hernandez L.R. The Hoosier audiovisual multitalker database. In: Pisoni D.B. (Ed). Research on Spoken Language Processing (1996), Speach Research Laboratory, Indiana University, Bloomington, IN 578-583
66. Skipper J.I., Nusbaum H.C., and Small S.L. Listening to talking faces: motor cortical activation during speech perception. Neuroimage 25 (2005) 76-89
67. Specht K., and Reul J. Functional segregation of the temporal lobes into highly differentiated subsystems for auditory perception: an auditory rapid event-related fMRI-task. Neuroimage 20 (2003) 1944-1954
68. Stanford T.R., Quessy S., and Stein B.E. Evaluating the operations underlying multisensory integration in the cat superior colliculus. J. Neurosci. 25 (2005) 6499-6508
69. Stein B.E., and Stanford T.R. Multisensory integration: current issues from the perspective of the single neuron. Nat. Rev. Neurosci. 9 (2008) 255-266
70. Stekelenburg J.J., and Vroomen J. Neural correlates of multisensory integration of ecologically valid audiovisual events. J. Cogn. Neurosci. 19 12 (2007) 1964-1973
71. Stevenson R.A., Geoghegan M.L., and James T.W. Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects. Exp. Brain Res. 179 (2007) 85-95
72. Talaraich J., and Tournoux P. Co-Planar Stereotaxic Atlas of the Human Brain (1988), Thieme Medical Publishers, New York, New York
73. Tuomainen J., Andersen T.S., Tiippana K., and Sams M. Audio-visual speech perception is special. Cognition 96 (2005) B13-22
74. Winston J.S., Henson R.N., Fine-Goulden M.R., and Dolan R.J. fMRI-adaptation reveals dissociable neural representations of identity and expression in face perception. J. Neurophysiol. 92 (2004) 1830-1839
75. Zekveld A.A., Heslenfeld D.J., Festen J.M., and Schoonhoven R. Top-down and bottom-up processes in speech comprehension. Neuroimage 32 (2006) 1826-1836