Interactions between auditory and visual semantic stimulus classes: Evidence for common processing networks for speech and body actions

Journal of Cognitive Neuroscience

Volumn 23, Issue 9, 2011, Pages 2291-2308

  Meyer, Georg F ^a   Greenlee, Mark ^b   Wuerger, Sophie ^a  

^a UNIVERSITY OF LIVERPOOL   (United Kingdom)

^b UNIVERSITY OF REGENSBURG   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ARTICLE; AUDITORY ORIENTATION; BOLD SIGNAL; FEMALE; FRONTAL GYRUS; HUMAN; HUMAN EXPERIMENT; MALE; NORMAL HUMAN; PREMOTOR CORTEX; PRIORITY JOURNAL; RECOGNITION; SEMANTICS; SIGNAL TRANSDUCTION; SPEECH; STIMULUS; TASK PERFORMANCE; VISUAL ACUITY;

ACOUSTIC STIMULATION; ADULT; ANALYSIS OF VARIANCE; AUDITORY PERCEPTION; BRAIN MAPPING; CEREBRAL CORTEX; FEMALE; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; MAGNETIC RESONANCE IMAGING; MALE; OXYGEN; PHOTIC STIMULATION; REACTION TIME; SEMANTICS; SPEECH; VISUAL PERCEPTION; YOUNG ADULT;

EID: 79960137734     PISSN: 0898929X     EISSN: 15308898     Source Type: Journal    
DOI: 10.1162/jocn.2010.21593     Document Type: Article

References (98)

1. Aziz-Zadeh, L., Wilson, S. M., Rizzolatti, G., & Iacoboni, M. (2006). Congruent embodied representations of visually presented actions and linguistic phrases describing actions. Current Biology, 16, 1818-1823.
2. Baddeley, A. D. (2000). The episodic buffer: A new component of working memory. Trends in Cognitive Sciences, 4, 417-423.
3. Barraclough, N. E., Xiao, D., Baker, C. I., Oram, M. W., & Perrett, D. I. (2005). Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions. Journal of Cognitive Neuroscience, 17, 377-391.
4. Beauchamp, M. S. (2005). See me, hear me, touch me: Multisensorial integration in lateral occipital temporal cortex. Current Opinion in Neurobiology, 15, 145-153.
5. Beauchamp, M. S., Argall, B. D., Bodurka, J., Duyn, J. H., & Martin, A. (2004). Unraveling multisensory integration: Patchy organization within human STS multisensory cortex. Nature Neuroscience, 7, 1190-1192.
6. Beauchamp, M. S., Lee, K. E., Argall, B. D., & Martin, A. (2004). Integration of auditory and visual information about objects in superior temporal sulcus. Neuron, 41, 809-823.
7. Bidet-Caulet, A., Voisin, J., Bertrand, O., & Fonlup, P. (2005). Listening to a walking human activates the temporal biological motion area. Neuroimage, 28, 132-139.
8. Blesser, B. A. (1969). Perception of spectrally rotated speech. Ph.D. thesis, Department of Electrical Engineering, Massachusetts Institute of Technology.
9. Brett, M., Anton, J.-L., Valabregue, R., & Poline, J. P. (2002). Region of interest analysis using an SPM toolbox [Abstract]. Presented at the 8th International Conference on Functional Mapping of the Human Brain, June 2-6, 2002, Sendai, Japan. Available on CD-ROM in Neuroimage, 16(2).
10. Brooks, A., van der Zwan, R., Billard, A., Petreska, B., Clarke, S., & Blanke, O. (2007). Auditory motion affects visual biological motion processing. Neuropsychologia, 45, 523-530.
11. Buccino, G., Binkowski, F., & Riggio, L. (2004). The mirror neuron system and action recognition. Brain and Language, 89, 370-376.
12. Bunzeck, N., Wuestenberg, T., Lutz, K., Heinze, H., & Jäncke, J. (2005). Scanning silence: Mental imagery of complex sounds. Neuroimage, 26, 1119-1127.
13. Cabeza, R., & Nyberg, L. (2000). Imaging cognition: II. An empirical review of 275 PET and fMRI studies. Journal of Cognitive Neuroscience, 12, 1-47.
14. Calvert, G. A. (2001). Crossmodal processing in the human brain: Insights from functional neuroimaging studies. Cerebral Cortex, 11, 1110-1123.
15. Calvert, G. A., Campbell, R., & Brammer, M. J. (2000). Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Current Biology, 10, 649-657.
16. Chung, G. H., Man, Y. M., Jeong, S. H., & Jack, C. R. (2005). Functional heterogeneity of the supplementary motor area. AJNR, American Journal of Neuroradiology, 26, 1819-1823.
17. Damasio, A. R., & Geschwind, N. (1984). The neural basis of language. Annual Reviews of Neuroscience, 7, 127-147.
18. Doehrmann, O., Weigelt, S., Altmann, C. F., Kaiser, J., & Naumer, M. J. (2010). Audiovisual functional magnetic resonance imaging adaptation reveals multisensory integration effects in object-related sensory cortices. Journal of Neuroscience, 30, 3370-3379.
19. Eickhoff, S., Stephan, K. E., Mohlberg, H., Grefkes, C., Fink, G. R., Amunts, K., et al. (2005). A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data. Neuroimage, 25, 1325-1335.
20. Eickhoff, S. B., Heim, S., Zilles, K., & Amunts, K. (2006). Testing anatomically specified hypotheses in functional imaging using cytoarchitectonic maps. Neuroimage, 32, 570-582.
21. Eickhoff, S. B., Paus, T., Caspers, S., Grosbras, M. H., Evans, A., Zilles, K., et al. (2007). Assignment of functional activations to probabilistic cytoarchitectonic areas revisited. Neuroimage, 36, 511-521.
22. Fadiga, L., & Craighero, L. (2006). Hand actions and speech representation in Broca's area. Cortex, 42, 486-490.
23. Friston, K. J., Holmes, A. P., Price, C. J., Buchel, C., &Worsley, K. J. (1999). Multisubject fMRI studies and conjunction analyses. Neuroimage, 10, 385-396.
24. Friston, K. J., Penny, W. D., & Glaser, D. E. (2005). Conjunction revisited. Neuroimage, 25, 661-667.
25. Gaab, N., Gaser, C., Zaehle, T., Jancke, L., & Schlaug, G. (2003). Functional anatomy of pitch memory-An fMRI study with sparse temporal sampling. Neuroimage, 19, 1417-1426.
26. Gallese, V., Fadiga, L., Fogassi, L., & Rizzolatti, G. (1996). Action recognition in the premotor cortex. Brain, 119, 593-609.
27. Grezes, J., Armony, J. L., Rowe, J., & Passingham, R. E. (2003). Activations related to ''mirror'' and ''canonical'' neurones in the human brain: An fMRI study. Neuroimage, 18, 928-937.
28. Grossman, E. D., Batelli, L., & Pascual-Leone, A. (2005). Repetitive TMS over posterior STS disrupts perception of biological motion. Vision Research, 45, 2847-2853.
29. Grossman, E. D., & Blake, R. (2001). Brain activity evoked by inverted and imagined biological motion. Vision Research, 41, 1475-1482.
30. Grossman, E. D., Donnelly, M., Price, P., Morgan, V., Pickens, D., Neighbor, G., et al. (2000). Brain areas involved in the perception of biological motion. Journal of Cognitive Neuroscience, 12, 711-720.
31. Hall, D. A., Fussell, C., & Summerfield, A. Q. (2005). Reading fluent speech from talking faces: Typical brain networks and individual differences. Journal of Cognitive Neuroscience, 17, 939-953.
32. Hall, D. A., Haggard, M. P., Akeroyd, M. A., Palmer, A. R., Summerfield, A. Q., Elliott, M. R., et al. (1999). ''Sparse'' temporal sampling in auditory fMRI. Human Brain Mapping, 7, 213-223.
33. Hamzei, F., Rijntjes, M., Dettmers, C., Glauche, V., Weiller, C., & Büchel, C. (2003). The human action recognition system and its relationship to Broca's area: An fMRI study. Neuroimage, 19, 637-644.
34. Hein, G., & Knight, R. T. (2008). Superior temporal sulcus-It's my area: Or is it? Journal of Cognitive Neuroscience, 20, 2125-2136.
35. Henson, R. N. A., Rugg, M. D., & Friston, K. J. (2001). The choice of basis functions in event-related fMRI. Neuroimage, 13, 149.
36. Hickok, G., & Buchsbaum, B. (2003). Temporal lobe speech perception systems are part of the verbal working memory circuit: Evidence from two recent fMRI studies. Behavioral and Brain Sciences, 26, 740-741.
37. Hickok, G., & Poeppel, D. (2004). Dorsal and ventral streams: A framework for understanding aspects of the functional anatomy of language. Cognition, 92, 67-99.
38. Hickok, G., & Poeppel, D. (2007). The cortical organization of speech processing. Nature Reviews Neuroscience, 8, 393-402.
39. Hikosaka, K., Iwai, E., Saito, H., & Tanaka, K. (1988). Polysensory properties of neurons in the anterior bank of the caudal superior temporal sulcus of the macaque monkey. Journal of Neurophysiology, 60, 1615-1637.
40. Hocking, J., & Price, C. J. (2008). The role of the posterior superior temporal sulcus in audiovisual processing. Cerebral Cortex, 18, 2439-2449.
41. Iacoboni, M. (2008). The role of premotor cortex in speech perception: Evidence from fMRI and rTMS. Journal of Physiology, Paris, 102, 31-34.
42. Jacquemot, C., & Scott, S. K. (2006). What is the relationship between phonological short-term memory and speech processing? Trends in Cognitive Sciences, 10, 480-486.
43. Keysers, C., Kohler, E., Umiltà, M. A., Nanetti, L., Fogassi, L., & Gallese, V. (2003). Audiovisual mirror neurons and action recognition. Experimental Brain Research, 153, 628-636.
44. Kohler, E., Keysers, C., Umiltà, M. A., Fogassi, L., Gallese, V., & Rizzolatti, G. (2002). Hearing sounds, understanding actions: Action representation in mirror neurons. Science, 297, 846-848.
45. Krainik, A., Lehéricy, S., Duffau, H., Capelle, L., Chainay, H., Cornu, P., et al. (2003). Postoperative speech disorder after medial frontal surgery: Role of the supplementary motor area. Neurology, 60, 587-594.
46. Laurienti, P. J., Kraft, R. A., Maldjian, J. A., Burdette, J. H., & Wallace, M. T. (2004). Semantic congruence is a critical factor in multisensory behavioral performance. Experimental Brain Research, 158, 405-414.
47. Lewis, J., Wightman, F., Brefczynski, J., Phinney, R., Binder, J., & DeYoe, E. (2004). Human brain regions involved in recognizing environmental sounds. Cerebral Cortex, 14, 1008-1021.
48. Liberman, A. M. (1996). Speech: A special code. Learning, development, and conceptual change series. Cambridge, MA: MIT Press.
49. McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746-748.
50. Meister, I. G., Boroojerdi, B., Foltys, H., Sparing, R., Huber, W., & Topper, R. (2003). Motor cortex hand area and speech: Implications for the development of language. Neuropsychologia, 41, 401-406.
51. Meister, I. G., & Iacoboni, M. (2007). No language-specific activation during linguistic processing of observed actions. PLoS One, 9, e891.
52. Meister, I. G., Wilson, S. M., Deblieck, C., Wu, A. D., & Iacoboni, M. (2007). The essential role of the premotor cortex in speech perception. Current Biology, 17, 1692-1696.
53. Meredith, M. A., Nemitz, J.W., & Stein, B. E. (1987). Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. Journal of Neuroscience, 7, 3215-3229.
54. Meredith, M. A., & Stein, B. E. (1996). Spatial determinants of multisensory integration in cat superior colliculus neurons. Journal of Neurophysiology, 75, 1843-1857.
55. Meyer, G., & Morse, R. (2003). The intelligibility of consonants in noisy vowel-consonant-vowel sequences when vowels are selectively enhanced. Speech Communication, 41, 429-440.
56. Meyer, G. F., Wuerger, S. M., Röhrbein, F., & Zetzsche, C. (2005). Low-level integration of auditory and visual motion signals requires spatial co-localisation. Experimental Brain Research, 166, 538-547.
57. Obleser, J., & Eisner, F. (2009). Pre-lexical abstraction of speech in the auditory cortex. Trends in Cognitive Sciences, 13, 14-19.
58. Pelphrey, K. A., Morris, J. P., Michelich, J. R., Allison, T., & McCarthy, G. (2005). Functional anatomy of biological motion perception in posterior temporal cortex: An fMRI study of eye, mouth and hand movements. Cerebral Cortex, 15, 1866-1876.
59. Petrides, M., & Pandya, D. N. (1997). Comparative architectonic analysis of the human and the macaque frontal cortex. In F. Boller & J. Grafman (Eds.), Handbook of neuropsychology (Vol. IX, pp. 17-58). New York: Elsevier.
60. Pineda, J. A. (2008). Sensorimotor cortex as a critical component of an ''extended'' mirror neuron system: Does it solve the development, correspondence, and control problems in mirroring? Behavioral and Brain Functions, 4, 47.
61. Pizzamiglio, L., Aprile, T., Spitoni, G., Pitzalis, S., Bates, E., D'Amico, S., et al. (2005). Separate neural systems for processing action- or non-action-related sounds. Neuroimage, 24, 852-861.
62. Postle, B. R. (2006). Working memory as an emergent property of the mind and brain. Neuroscience, 139, 23-38.
63. Puce, A., Allison, T., Bentin, S., Gore, J. C., & McCarthy, G. (1998). Temporal cortex activation in humans viewing eye and mouth movements. Journal of Neuroscience, 18, 2188-2199.
64. Puce, A., & Perrett, D. (2003). Electrophysiology and brain imaging of biological motion. Philosophical Transactions of the Royal Society of London, Series B, Biological Sciences, 358, 435-445.
65. Pulvermüller, F. (2005). Brain mechanisms linking language and action. Nature Reviews Neuroscience, 6, 576-582.
66. Pulvermüller, F., Huss, M., Kherif, F., Moscoso del Prado Martin, F., Hauk, O., & Shtyrov, Y. (2006). Motor cortex maps articulatory features of speech sounds. Proceedings of the National Academy of Sciences, U.S.A., 103, 7865-7870.
67. Rizzolatti, G., & Arbib, M. A. (1998). Language within our grasp. Trends in Neurosciences, 21, 188-194.
68. Sadaghiani, S., Maier, J. X., & Noppeney, U. (2009). Natural, metaphoric and linguistic auditory direction signals have distinct influences on visual motion processing. Journal of Neuroscience, 29, 6490-6499.
69. Santi, A., Servos, P., Vatikiotis-Bateson, E., Kuratate, T., & Munhall, K. (2003). Perceiving biological motion: Dissociating visible speech from walking. Journal of Cognitive Neuroscience, 15, 800-809.
70. Saygin, A. P. (2007). Superior temporal and premotor brain areas necessary for biological motion perception. Brain, 130, 2452-2461.
71. Saygin, A. P., Driver, J., & de Sa, V. R. (2008). In the footsteps of biological motion and multisensory perception: Judgements of audiovisual temporal relations are enhanced for upright walkers. Psychological Science, 19, 469-475.
72. Saygin, A. P., Wilson, S. M., Hagler, D. J., Bates, E., & Sereno, M. I. (2004). Point-light biological motion perception activates human premotor cortex. Journal of Neuroscience, 24, 6181-6188.
73. Schubotz, R. I., & von Cramon, D. Y. (2001). Functional organization of the lateral premotor cortex: fMRI reveals different regions activated by anticipation of object properties, location and speed. Cognitive Brain Research, 11, 97-112.
74. Schubotz, R. I., von Cramon, D. Y., & Lohmann, G. (2003). Auditory what, where, and when: A sensory somatotopy in lateral premotor cortex. Neuroimage, 20, 173-185.
75. Scott, S. K., McGettigan, C., & Eisner, F. (2009). A little more conversation, a little less action-Candidate roles for the motor cortex in speech perception. Nature Reviews Neuroscience, 10, 295-302.
76. Seltzer, B., & Pandya, D. N. (1989). Frontal lobe connections of the superior temporal sulcus in the rhesus monkey. Journal of Comparative Neurology, 281, 97-113.
77. Skipper, J. I., Goldin-Meadow, S., Howard, C., Nusbaum, H. C., & Small, S. L. (2007). Speech-associated gestures, Broca's area, and the human mirror system. Brain and Language, 101, 260-277.
78. Skipper, J. I., Nusbaum, H. C., & Small, S. L. (2005). Listening to talking faces: Motor cortical activation during speech perception. Neuroimage, 25, 76-89.
79. Skipper, J. I., van Wassenhove, V., Howard, C., Nusbaum, H. C., & Small, S. L. (2007). Hearing lips and seeing voices: How cortical areas supporting speech production ediate audiovisual speech perception. Cerebral Cortex, 17, 2387-2399.
80. Soto-Faraco, S., Navarra, J., & Alsius, A. (2004). Assessing automaticity in audiovisual speech integration: Evidence from the speeded classification task. Cognition, 92, B13-B23.
81. Stevenson, R. A., Kim, S., & James, T.W. (2009). An additive-factors design to disambiguate neuronal and areal convergence: Measuring multisensory interactions between audio, visual, and haptic sensory streams using fMRI. Experimental Brain Research, 198, 183-194.
82. Szycik, G. R., Jansma, H., & Münte, T. F. (2009). Audiovisual integration during speech comprehension: An fMRI study comparing ROI-based and hole brain analyses. Human Brain Mapping, 30, 1990-1999.
83. Szycik, G. R., Tausche, P., & Münte, T. F. (2008). A novel approach to study audiovisual integration in speech perception: Localizer fMRI and sparse sampling. Brain Research, 1220, 142-149.
84. Teder-Salejärvi, W. A., Di Russo, F., McDonald, J. J., & Hillyard, S. A. (2005). Effects of spatial congruity on audio-visual multimodal integration. Journal of Cognitive Neuroscience, 17, 1396-1409.
85. Thompson, J. C., Jillian, E., Hardee, J. E., Panayiotou, A., Crewther, D., & Puce, A. (2007). Common and distinct brain activation to viewing dynamic sequences of face and hand movement. Neuroimage, 37, 966-973.
86. Troje, N. F., & Westhoff, C. (2006). The inversion effect in biological motion perception: Evidence for a ''life detector''. Current Biology, 16, 821-824.
87. Vaina, L. M., Solomon, J., Chowdhury, S., Sinha, P., & Belliveau, J. W. (2001). Functional neuroanatomy of biological motion perception in humans. Proceedings of the National Academy of Sciences, U.S.A., 98, 1165-1166.
88. van Atteveldt, N., Formisano, E., Goebel, R., & Blomert, L. (2004). Integration of letters and speech sounds in the human brain. Neuron, 43, 271-282.
89. Van Essen, D. C., Dickson, J., Harwell, J., Hanlon, D., Anderson, C. H., & Drury, H. A. (2001). An integrated software system for surface-based analyses of cerebral cortex. Journal of American Medical Informatics Association, 8, 443-459.
90. van Wassenhove, V., Grant, K. W., & Poeppel, D. (2005). Visual speech speeds up the neural processing of auditory speech. Proceedings of the National Academy of Sciences, U.S.A., 102, 1181-1186.
91. Watkins, K. E., Strafella, A. P., & Paus, T. (2003). Seeing and hearing speech excites the motor system involved in speech production. Neuropsychologia, 41, 989-994.
92. Werner, S., & Noppeney, U. (2009). Superadditive responses in superior temporal sulcus predict audiovisual benefits in object categorization. Cerebral Cortex, 20, 1829-1842.
93. Werner, S., & Noppeney, U. (2010a). Distinct functional contributions of primary sensory and association areas to audiovisual integration in object categorization. Journal of Neuroscience, 30, 2662-2675.
94. Werner, S., & Noppeney, U. (2010b). Superadditive responses in superior temporal sulcus predict audiovisual benefits in object categorization. Cerebral Cortex, 20, 1892-1842.
95. Wilke, M., & Lidzba, K. (2007). LI-tool: A new toolbox to assess lateralization in functional MR-data. Journal of Neuroscience Methods, 163, 128-136.
96. Wilson, M. W., Saygin, A. P., Sereno, M. I., & Iacoboni, M. (2004). Listening to speech activates motor areas involved in speech production. Nature Neuroscience, 7, 701-702.
97. Wise, R. J. S., Scott, S. K., Blank, S. C., Mummery, C. J., Murphy, K., & Warburton, E. A. (2001). Separate neural subsystems within ''Wernicke's area''. Brain, 124, 83-95.
98. Zaehle, T., Wüstenberg, T., Meyer, M., & Jäncke, L. (2004). Evidence for rapid auditory perception as the foundation of speech processing: A sparse temporal sampling fMRI study. European Journal of Neuroscience, 20, 1460-9568.