Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions

Journal of Cognitive Neuroscience

Volumn 17, Issue 3, 2005, Pages 377-391

  Barraclough, Nick E ^a   Xiao, Dengke ^a   Baker, Chris I ^b   Oram, Mike W ^a   Perrett, David I ^a  

^a UNIVERSITY OF ST ANDREWS   (United Kingdom)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ANIMAL CELL; ARTICLE; AUDITORY RESPONSE; AUDITORY STIMULATION; BODY MOVEMENT; BRAIN FUNCTION; BRAIN NERVE CELL; BRAIN REGION; EVOKED VISUAL RESPONSE; FACE; HAND MOVEMENT; INFORMATION PROCESSING; MONKEY; NONHUMAN; PRIORITY JOURNAL; SOUND; SUPERIOR TEMPORAL SULCUS; TEMPORAL CORTEX; VISION; VISUAL STIMULATION;

ACOUSTIC STIMULATION; ANALYSIS OF VARIANCE; ANIMALS; AUDITORY PERCEPTION; BEHAVIOR, ANIMAL; BRAIN MAPPING; CELL COUNT; EVOKED POTENTIALS, SOMATOSENSORY; MACACA MULATTA; NEURONS; PHOTIC STIMULATION; REACTION TIME; TEMPORAL LOBE; TIME FACTORS; VISUAL PERCEPTION;

EID: 14944354656     PISSN: 0898929X     EISSN: None     Source Type: Journal    
DOI: 10.1162/0898929053279586     Document Type: Article

References (73)

1. Averbeck, B. B., & Romanski, L. M. (2004). Principle and independent components of macaque vocalizations: Constructing stimuli to probe high-level sensory processing. Journal of Neurophysiology, 91, 2897-2909.
2. Baylis, G. C., Rolls, E. T., & Leonard, C. M. (1987). Functional subdivisions of the temporal neocortex. Journal of Neuroscience, 7, 330-342.
3. Beauchamp, M. S., Lee, K. E., Argall, B. D., & Martin, A. (2004). Integration of auditory and visual information about objects in superior temporal sulcus. Neuron, 41, 809-823.
4. Bekkering, H., & Wohlschlaeger, A. (2002). Action perception and imitation: A tutorial. In W. Prinx, & B. Hommel (Eds.), Attention and performance: XIX. Common mechanisms in perception and action (pp. 294-333). Oxford: Oxford University Press.
5. Bell, A. H., Cornell, B. D., Munoz, D. P., & Meredith, M. A. (2003). Engagement of visual fixation suppresses sensory responsiveness and multisensory integration in the primate superior colliculus. European Journal of Neuroscience, 18, 2867-2873.
6. Benevento, L. A., & Fallon, J. H. (1975). The ascending projections of the superior colliculus in the rhesus monkey (Macaca mulatta). Journal of Comparative Neurology, 160, 339-362.
7. Benevento, L. A., Fallon, J., Davis, B. J., & Rezak, M. (1977). Auditory-visual interaction in single cells in the cortex of the superior temporal sulcus and the orbital frontal cortex of the macaque monkey. Experimental Neurology, 57, 849-872.
8. Bruce, C. J., Desimone, R., & Gross, C. G. (1981). Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque. Journal of Neurophysiology, 46, 369-384.
9. Bruce, C. J., Desimone, R., & Gross, C. G. (1986). Both striate cortex and superior colliculus contribute to visual properties of neurons in superior temporal polysensory area of macaque monkey. Journal of Neurophysiology, 55, 1057-1075.
10. Burnett, L. R., Stein, B. E., Chaponis, D., & Wallace, M. T. (2004). Superior colliculus lesions preferentially disrupt multisensory orientation. Neuroscience, 124, 535-547.
11. Burton, H., & Jones, E. G. (1976). The posterior thalamic region and its cortical projection in New World and Old World monkeys. Journal of Comparative Neurology, 168, 249-302.
12. Bushara, K. O., Hanakawa, T., Immisch, I., Toma, K., Kansaku, K., & Hallett, M. (2003). Neural correlates of cross-modal binding. Nature Neuroscience, 6, 190-195.
13. Calvert, G. A. (2000). Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Current Biology, 10, 649-657.
14. Calvert, G. A. (2001). Crossmodal processing in the human brain: Insights from functional neuroimaging studies. Cerebral Cortex, 11, 1110-1123.
15. Colombo, M., & Gross, C. G. (1994). Responses of inferior temporal cortex and hippocampal neurons during delayed matching to sample in monkeys (Macaca fascicularis). Behavioural Neuroscience, 108, 443-455.
16. Desimone, R., & Gross, C. G. (1979). Visual areas in the temporal cortex of the macaque. Brain Research, 178, 363-380.
17. Desimone, R., & Ungerleider, L. G. (1986). Multiple visual areas in the caudal superior temporal sulcus of the macaque. Journal of Comparative Neurology, 248, 164-189.
18. Pelligrino, G., Fadiga, L., Fogassi, L., Gallese, V., & Rizzolatti, G. (1992). Understanding motor events: A neurophysiological study. Experimental Brain Research, 91, 176-180.
19. Distler, C., Boussaoud, D., Desimone, R., & Ungerleider, L. G. (1993). Cortical connections of inferior temporal area TEO in macaque monkeys. Journal of Comparative Neurology, 334, 125-150.
20. Dodd, B. (1977). The role of vision in the perception of speech. Perception, 6, 31-40.
21. Downing, P. E., Jiang, Y., Shuman, M., & Kanwisher, N. (2001). A cortical area selective for the visual processing of the human body. Science, 293, 2470-2473.
22. Driver, J. (1996). Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading. Nature, 381, 66-68.
23. Edwards, R., Xiao, D.-K., Keysers, C., Foldiak, P., & Perrett, D. I. (2003). Color sensitivity of cells responsive to complex stimuli in the temporal cortex. Journal of Neurophysiology, 90, 1245-1256.
24. Ferrari, P. F., Gallese, V., Rizzolatti, G., & Fogassi, L. (2003). Mirror neurons responding to the observation of ingestive and communicative mouth actions in the monkey ventral premotor cortex. European Journal of Neuroscience, 17, 1703-1714.
25. Foldiak, P., Xiao, D.-K., Keysers, C., Edwards, R., & Perrett, D. I. (2003). Rapid serial visual presentation for the determination of neural selectivity in area STSa. Progress in Brain Research, 144, 107-116.
26. Fort, A., Delpuech, C., Pernier, J., & Giard, M. H. (2002). Dynamics of cortico-subcortical cross-modal operations involved in audio-visual object detection in humans. Journal of Cognitive Neuroscience, 12, 1031-1039.
27. Gallese, V., Fadiga, L., Fogassi, L., & Rizzolatti, G. (1996). Action recognition in the premotor cortex. Brain, 119, 593-609.
28. Ghazanfar, A. A., & Logothetis, N. K. (2003). Facial expressions Linked to monkey calls. Nature, 423, 937-938.
29. Giard, M. H., & Peronnet, F. (1999). Auditory-visual integration during multimodal object recognition in humans: A behavioural and electrophysiological study. Journal of Cognitive Neuroscience, 11, 473-490.
30. Gibson, J. R., & Maunsell, J. H. R. (1997). Sensory modality specificity of neural activity related to memory in visual cortex. Journal of Neurophysiology, 78, 1263-1275.
31. Grossman, E., Donnelly, M., Price, R., Pickens, D., Morgan, V., Neighbor, G., & Blake, R. (2000). Brain areas involved in perception of biological motion. Journal of Cognitive Neuroscience, 12, 711-720.
32. Hasselmo, M. E., Rolls, E. T., & Baylis, G. C. (1989). The role of expression and identity in the face-selective responses of neurons in the temporal visual cortex of the monkey. Behavioural Brain Research, 32, 203-218.
33. Hikosaka, K., Iwai, E., Saito, H., & Tanaka, K. (1988). Polysensory properties of neurons in the anterior bank of the caudal superior temporal sulcus of the macaque monkey. Journal of Neurophysiology, 60, 1615-1637.
34. Iacoboni, M., Woods, R. P., Brass, M., Bekkering, H., Mazzionta, J. C., & Rizzolatti, G. (1999). Cortical mechanisms of human imitation. Science, 286, 2526-2528.
35. Jay, M. F., & Sparks, D. L. (1984). Auditory receptive fields in primate superior colliculus shift with changes in eye position. Nature, 309, 345-347.
36. Jiang, W., Jiang, H., & Stein, B. E. (2002). Two corticotectal areas facilitate multisensory orientation behaviour. Journal of Cognitive Neuroscience, 14, 1240-1255.
37. Jiang, W., & Stein, B. E. (2003). Cortex controls multisensory depression in superior colliculus. Journal of Neurophysiology, 90, 2123-2135.
38. Jiang, W., Wallace, M. T., Jiang, H., Vaughan, J. W., & Stein, B. E. (2001). Two cortical areas mediate multisensory integration in superior colliculus neurons. Journal of Neurophysiology, 85, 506-522.
39. Kadunce, D. C., Vaughan, J. W., Wallace, M. T., Benedek, G., & Stein, B. E. (1997). Mechanisms of within- and cross-modality suppression in the superior colliculus. Journal of Neurophysiology, 78, 2834-2847.
40. Kohler, E., Keysers, C., Umilta, M. A., Fogassi, L., Gallese, V., & Rizzolatti, G. (2002). Hearing sounds, understanding actions: Action representation in mirror neurons. Science, 297, 846-848.
41. Lui, F., Gregory, K. M., Blanks, R. H. L, & Giolli, R. A. (1995). Projections from visual areas of the cerebral cortex to pretectal nuclear complex, terminal accessory optic nuclei, and superior colliculus in macaque monkey. Journal of Comparative Neurology, 363, 439-460.
42. McGurk, H., & McDonald, J. (1976). Hearing Lips and seeing voices. Nature, 264, 746-748.
43. Meredith, M. A., & Stein, B. E. (1983). Interactions among converging sensory inputs in the superior colliculus. Science, 221, 389-391.
44. Meredith, M. A., & Stein, B. E. (1986a). Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Research, 365, 350-354.
45. Meredith, M. A., & Stein, B. E. (1986b). Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. Journal of Neurophysiology, 56, 640-662.
46. Meredith, M. A., & Stein, B. E. (1996). Spatial determinants of multisensory integration in cat superior colliculus. Journal of Neurophysiology, 75, 1843-1857.
47. Nishitani, N., & Hari, R. (2000). Temporal dynamics of cortical representation for action. Proceedings of the National Academy of Sciences, U.S.A., 97, 913-918.
48. Oram, M. W., & Perrett, D. I. (1992). Time course of neural responses discriminating views of the face and head. Journal of Neurophysiology, 68, 70-84.
49. Oram, M. W., & Perrett, D. I. (1994). Responses of anterior superior temporal polysensory (STPa) neurons to "biological motion" stimuli. Journal of Cognitive Neuroscience, 6, 99-116.
50. Oram, M. W., & Perrett, D. I. (1996). Integration of form and motion in the anterior superior temporal polysensory area (STPa) of the macaque monkey. Journal of Neurophysiology, 76, 109-129.
51. Oram, M. W., Perrett, D. I., & Hietanen, J. K. (1993). Directional tuning of motion-sensitive cells in the anterior superior temporal polysensory area of the macaque. Experimental Brain Research, 97, 274-294.
52. Perrett, D. I., Harries, M. H., Bevan, R., Thomas, S., Benson, P. J., Mistlin, A. J., Chitty, A. J., Hietanen, J. K., & Ortega, J. E. (1989). Frameworks of analysis for the neural representation of animate objects and actions. Journal of Experimental Biology, 146, 87-113.
53. Perrett, D. I., Mistlin, A. J., Harries, M. H., & Chitty, A. J. (1990). Understanding the visual appearance and consequences of hand actions. In M. A. Goodale (Ed.), Vision and action: The control of action (pp. 163-180). Norwood, NJ: Ablex.
54. Perrett, D. I., Smith, P. A. J., Mistlin, A. J., Chitty, A. J., Head, A. S., Potter, D. D., Broennimann, R., Milner, A. D., & Jeeves, M. A. (1985). Visual analysis of body movements by neurons in the temporal cortex of the macaque monkey: A preliminary report. Behavioural Brain Research, 16, 153-170.
55. Perrett, D. I., Smith, P. A. J., Potter, D. D., Mistlin, A. J., Head, A. S., Milner, A. D., & Jeeves, M. A. (1984). Neurons responsive to faces in the temporal cortex: Studies of functional organization, sensitivity to identity and relation to perception. Human Neurobiology, 3, 197-208.
56. Prinz, W. (2002). Experimental approaches to immitation. In A. N. Melzoff & W. Prinz (Eds.), The imitative mind. Development, evolution and brain bases (pp. 143-162). Cambridge: Cambridge University Press.
57. Puce, A., Allison, T., Bentin, S., Gore, J. C., & McCarthy, G. (1998). Temporal cortex activation in humans viewing eye and mouth movements. Journal of Neuroscience, 18, 2188-2199.
58. Puce, A., & Perrett, D. I. (2003). Electrophysiology and brain imaging of biological motion. Philosophical Transactions of the Royal Society B, 358, 435-445.
59. Rauschecker, J. P., Tian, B., & Hauser, M. (1995). Processing of complex sounds in the macaque nonprimary auditory cortex. Science, 268, 111-114.
60. Saleem, K. S., Suzuki, W., Tanaka, K., & Hashikawa, T. (2000). Connections between anterior inferotemporal cortex and superior temporal sulcus regions in the macaque monkey. Journal of Neuroscience, 20, 5083-5101.
61. Schroeder, C. E., & Foxe, J. J. (2002). The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. Cognitive Brain Research, 14, 187-198.
62. Seltzer, B., & Pandya, D. N. (1978). Afferent cortical connections and architectonics of the superior temporal sulcus and surrounding cortex. Brain Research, 149, 1-24.
63. Seltzer, B., & Pandya, D. N. (1984). Further observations on parietal-temporal connections in the rhesus monkey. Experimental Brain Research, 55, 301-312.
64. Seltzer, B., & Pandya, D. N. (1989). Intrinsic connections and architectonics of the superior temporal sulcus in the rhesus monkey. Journal of Comparative Neurology, 290, 451-471.
65. Seltzer, B., & Pandya, D. N. (1994). Parietal, temporal and occipital projections to cortex of the superior temporal sulcus in the rhesus monkey: A retrograde tracer study. Journal of Comparative Neurology, 15, 445-463.
66. Stein, B. E., & Meredith, M. A. (1993). The merging of the senses. Cambridge: MIT Press.
67. Sugase, Y., Yamane, S., Ueno, S., & Kawano, K. (1999). Global and fine information coded by single neurons in the temporal visual cortex. Nature, 400, 869-873.
68. Tipper, S. P., Phillips, N., Dancer, C., Lloyd, D., Howard, L. A., & McGlone, F. (2001). Vision influences tactile perception at body sites that cannot be viewed directly. Experimental Brain Research, 139, 160-167.
69. Ungerleider, L. G., & Desimone, R. (1986). Cotical connections of visual area MT in the macaque. Journal of Comparative Neurology, 248, 190-222.
70. Viviani, P. (2002). Motor competence in the perception of dynamic events: A tutorial. In W. Prinz & B. Hommel (Eds.), Attention and performance XIX. Common mechanisms in perception and action (pp. 406-442). Oxford: Oxford University Press.
71. Von Bonin, G., & Bailey, P. (1947). The neocortex of macaca mullata. Urbana, IL: University of Illinois Press.
72. Wallace, M. T., Meredith, M. A., & Stein, B. E. (1992). Integration of multiple sensory modalities in cat cortex. Experimental Brain Research, 91, 484-488.
73. Wallace, M. T., Wilkinson, L. K., & Stein, B. E. (1996). Representation and integration of multiple sensory inputs in primate superior colliculus. Journal of Neurophysiology, 76, 1246-1266.