Giving speech a hand: Gesture modulates activity in auditory cortex during speech perception

Human Brain Mapping

Volumn 30, Issue 3, 2009, Pages 1028-1037

  Hubbard, Amy L ^a,b   Wilson, Stephen M ^a   Callan, Daniel E ^b   Dapretto, Mirella ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b ADVANCED TELECOMMUNICATIONS RESEARCH INSTITUTE INTERNATIONAL   (Japan)

Author keywords

Auditory cortex; Gestures; Magnetic resonance imaging; Nonverbal communication; Speech perception

Indexed keywords

ADULT; ARTICLE; AUDITORY CORTEX; BRAIN FUNCTION; CONTROLLED STUDY; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; GESTURE; HAND MOVEMENT; HUMAN; HUMAN EXPERIMENT; MALE; NONVERBAL COMMUNICATION; PRIORITY JOURNAL; SPEECH PERCEPTION; TASK PERFORMANCE;

ACOUSTIC STIMULATION; ADULT; AUDITORY CORTEX; AUDITORY PERCEPTION; BRAIN MAPPING; FEMALE; GESTURES; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; MAGNETIC RESONANCE IMAGING; MALE; MOTOR ACTIVITY; PHOTIC STIMULATION; SPEECH;

EID: 61449250679     PISSN: 10659471     EISSN: None     Source Type: Journal    
DOI: 10.1002/hbm.20565     Document Type: Article

References (67)

1. Barrett DJ, Hall DA (2006): Response preferences for "what" and "where" in human non-primary auditory cortex. Neuroimage 32:968-977.
2. Bidet-Caulet A, Voisin J, Bertrand O, Fonlupt P (2005): Listening to a walking human activates the temporal biological motion area. Neuroimage 28:132-139.
3. Blake R, Shiffrar M (2007): Perception of human motion. Annu Rev Psychol 58:47-73.
4. Brett M, Anton JL, Valabregue R, Poline JB (2002): Region of interest analysis using an SPM toolbox. Neuroimage 16:S497 [abstract].
5. Bruce C, Desimone R, Gross CG (1981): Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque. J Neurophysiol 46:369-384.
6. Callan DE, Callan AM, Kroos C, Vatikiotis-Bateson E (2001): Multimodal contribution to speech perception revealed by independent component analysis: A single-sweep EEG case study. Brain Res Cogn Brain Res 10:349-353.
7. Callan DE, Jones JA, Munhall K, Callan AM, Kroos C, Vatikiotis-Bateson E (2003): Neural processes underlying perceptual enhancement by visual speech gestures. Neuroreport 14:2213-2218.
8. Callan DE, Jones JA, Munhall K, Kroos C, Callan AM, Vatikiotis-Bateson E (2004): Multisensory integration sites identified by perception of spatial wavelet filtered visual speech gesture information. J Cogn Neurosci 16:805-816.
9. Callan DE, Tsytsarev V, Hanakawa T, Callan AM, Katsuhara M, Fukuyama H, Turner R (2006): Song and speech: Brain regions involved with perception and covert production. Neuroimage 31:1327-1342.
10. Calvert GA, Campbell R (2003): Reading speech from still and moving faces: The neural substrates of visible speech. J Cogn Neurosci 15:57-70.
11. Calvert GA, Bullmore ET, Brammer MJ, Campbell R, Williams SC, McGuire PK, Woodruff PW, Iversen SD, David AS (1997): Activation of auditory cortex during silent lipreading. Science 276:593-596.
12. Calvert GA, Brammer MJ, Bullmore ET, Campbell R, Iversen SD, David AS (1999): Response amplification in sensory-specific cortices during crossmodal binding. Neuroreport 10:2619-2623.
13. Calvert GA, Campbell R, Brammer MJ (2000): Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10:649-657.
14. Campbell R, MacSweeney M, Surguladze S, Calvert G, McGuire P, Suckling J, Brammer MJ, David AS (2001): Cortical substrates for the perception of face actions: An fMRI study of the specificity of activation for seen speech and for meaningless lower-face acts (gurning). Brain Res Cogn Brain Res 12:233-243.
15. Chen JL, Zatorre RJ, Penhune VB (2006): Interactions between auditory and dorsal premotor cortex during synchronization to musical rhythms. Neuroimage 32:1771-1781.
16. Collins DL, Neelin P, Peters TM, Evans AC (1994): Automatic 3D intersubject registration of MR volumetric data in standardized Talairach space. J Comput Assist Tomogr 18:192-205.
17. Cook SW, Mitchell Z, Goldin-Meadow S (2007): Gesturing makes learning last. Cognition doi: 10.1016. 6/j.cognition.2007.04.010.
18. Davis MH, Johnsrude IS (2003): Hierarchical processing in spoken language comprehension. J Neurosci 23:3423-3431.
19. Desimone R, Gross CG (1979): Visual areas in the temporal cortex of the macaque. Brain Res 178:363-380.
20. Emmorey K, Grabowski T, McCullough S, Damasio H, Ponto L, Hichwa R, Bellugi U (2004): Motor-iconicity of sign language does not alter the neural systems underlying tool and action naming. Brain Lang 89:27-37.
21. Emmorey K, Grabowski T, McCullough S, Ponto LL, Hichwa RD, Damasio H (2005): The neural correlates of spatial language in English and American sign language: A PET study with hearing bilinguals. Neuroimage 24:832-840.
22. Emmorey K, Mehta S, Grabowski TJ (2007): The neural correlates of sign versus word production. Neuroimage 36:202-208.
23. Friston KJ, Holmes AP, Price CJ, Buchel C, Worsley KJ (1999): Multisubject fMRI studies and conjunction analyses. Neuroimage 10:385-396.
24. Gentilucci M, Bernardis P, Crisi G, Dalla Volta R (2006): Repetitive transcranial magnetic stimulation of Broca's area affects verbal responses to gesture observation. J Cogn Neurosci 18:1059-1074.
25. Goldin-Meadow S, Singer MA (2003): From children's hands to adults' ears: Gesture's role in the learning process. Dev Psychol 39:509-520.
26. Grossman ED, Blake R (2002): Brain areas active during visual perception of biological motion. Neuron 35:1167-1175.
27. Grossman ED, Blake R, Kim CY (2004): Learning to see biological motion: Brain activity parallels behavior. J Cogn Neurosci 16:1669-1679.
28. Holle H, Gunter TC (2007): The role of iconic gestures in speech disambiguation: ERP evidence. J Cogn Neurosci 19:1175-1192.
29. Holle H, Gunter TC, Ruschemeyer SA, Hennenlotter A, Iacoboni M (2007): Neural correlates of the processing of co-speech gestures. Neuroimage doi:10.1016/2007.10.055.
30. Humphries C, Love T, Swinney D, Hickok G (2005) Response of anterior temporal cortex to syntactic and prosodic manipulations during sentence processing. Hum Brain Mapp 26:128-38.
31. Iacoboni M, Dapretto M (2006): The mirror neuron system and the consequences of its dysfunction. Nat Rev Neurosci 21:191-199.
32. Indefrey P, Cutler A (2004): Prelexical and lexical processing in listening. In: Gazzaniga M, editor. The Cognitive Neurosciences. Cambridge, Massachusetts: MIT Press.
33. Kelly SD, Barr D, Church RB, Lynch K (1999): Offering a hand to pragmatic understanding: The role of speech and gesture in comprehension and memory. J Mem Lang 40:577-592.
34. Kelly SD, Kravitz C, Hopkins M (2004): Neural correlates of bi- modal speech and gesture comprehension. Brain Lang 89:253-260.
35. Kelly SD, Ward S, Creigh P, Bartolotti J (2006): In intentional stance modulates the integration of gesture and speech during comprehension. Brain Lang 101:222-233.
36. Kendon A (1972): Some relationships between body motion and speech: An analysis of an example. In: Siegman AW, Pope B, editors. Studies in Dyadic Communication. Elmsford, New York: Pergamon. pp 177-216.
37. Kotz SA, Meyer M, Paulmann S (2006): Materalization of emotional prosody in the brain: An overview and synopsis on the impact of study design. Prog Brain Res 156:285-294.
38. Krahmer E, Swerts M (2007) Effects of visual beats on prosodic prominence: acoustic analyses, auditory perception and visual perception. Journal of Memory and Language 57:396-414.
39. Laurienti PJ, Perrault TJ, Stanford TR, Wallace MT, Stein BE (2005): In the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies. Exp Brain Res 166:289-297.
40. Q, Nakano Y, Nishida T (2003): Gestures realization for embodied conversational agents. Presented at The 17th Annual Conference of the Japanese Society for Artificial Intelligence, Niigata, Japan.
41. MacSweeney M, Campbell R, Woll B, Giampietro V, David AS, McGuire PK, Calvert GA, Brammer MJ (2004): Dissociating linguistic and nonlinguistic gestural communication in the brain. Neuroimage 22:1605-1618.
42. MacSweeney M, Campbell R, Woll B, Brammer MJ, Giampietro V, David AS, Calvert GA, McGuire PK (2006): Lexical and sentential processing in British sign language. Hum Brain Mapp 27:63-76.
43. Mazziotta J, Toga A, Evans A, Fox P, Lancaster J, Zilles K., Woods R., Paus T, Simpson G, Pike B, Holmes C, Collins L, Thompson P, MacDonald D, Iacoboni M, Schormann T, Amunts K, Palomero-Gallagher N, Geyer S, Parsons L, Narr K, Kabani N, Le Goualher G, Boomsma D, Cannon T, Kawashima R, Mazoyer B (2001): A probabilistic atlas and reference system for the human brain: International Consortium for Brain Mapping (ICBM). Philos Trans R Soc Lond B Biol Sci 356:1293-322.
44. McGurk H, MacDonald J (1976): Hearing lips and seeing voices. Nature 264:746-748.
45. McNeill D (1992): Hand and Mind: What Gestures Reveal About Thought. Chicago: University of Chicago Press. McNeill D (2005): Gesture and Thought. Chicago: University of Chicago Press.
46. McNeill D, Cassell J, McCoullough KE (1994): Communicative effects of speech mismatched gestures. Res Lang Soc Interact 27:223-237.
47. Meyer M, Steinhauer K, Alter K, Friederici AD, von Cramon DY (2004): Brain activity varies with modulation of dynamic pitch variance in sentence melody. Brain Lang 89:277-289.
48. Özyürek A, Willems RM, Kita S, Hagoort P (2007): In-line integration of semantic information from speech and gesture: Insights from event-related brain potentials. J Cogn Neurosci 19:605-616.
49. Padberg J, Seltzer B, Cusick CG (2003): Architectonics and cortical connections of the upper bank of the superior temporal sulcus in the rhesus monkey: An analysis in the tangential plane. J Comp Neurol 467:418-434.
50. Pekkola J, Ojanen V, Autti T, Jaaskelainen IP, Mottonen R, Sams M (2006): Attention to visual speech gestures enhances hemodynamic activity in the left planum temporale. Hum Brain Mapp 27:471-477.
51. Quintilian (1856) Institutes of oratory. Honeycutt L, Ed. (Watson JS, Trans.). Rizzolatti G, Craighero L (2004): The mirror-neuron system. Ann Rev Neurosci 27:169-192.
52. Saito Y, Ishii K, Yagi K, Tatsumi IF, Mizusawa H (2006): Cerebral networks for spontaneous and synchronized singing and speaking. Neuroreport 17:1893-1897.
53. Scott SK, Blank CC, Rosen S, Wise RJ (2000): Identification of a pathway for intelligible speech in the left temporal lobe. Brain 123:2400-2406.
54. Shattuck-Hufnagel S, Yasinnik Y, Veilleux N, Renwick M (2007): A method for studying the time alignment of gestures and prosody in American English: 'Hits' and pitch accents in academic-lecture-style speech. In: Esposito A, Bratanic M, Keller E, Marinaro M, editors. Fundamentals of Verbal and Nonverbal Communication and the Biometric Issue. Amsterdam: IOS Press.
55. So C, Coppola M, Licciardello V, Goldin-Meadow S (2005): The seeds of spatial grammar in the manual modality. Cog Sci 29:1029-1043.
56. Stein BE, Meredith MA, Wallace MT (1993): The visually responsive neuron and beyond: Multisensory integration in cat and monkey. Prog Brain Res 95:79-90.
57. Sumby WH, Pollack I (1954): Visual contribution to speech intelligibility in noise. J Acoust Soc Am 26:212-215.
58. Sweet RA, Dorph-Petersen KA, Lewis DA (2005): Mapping auditory core, lateral belt, and parabelt cortices in the human superior temporal gyrus. J Comp Neurol 491:270-289.
59. Tuite K (1993): The production of gesture. Semiotica 93:83-105.
60. Warren JD, Jennings AR, Griffiths TD (2005): Analysis of the spectral envelope of sounds by the human brain. Neuroimage 24:1052-1057.
61. Westbury CF, Zatorre RJ, Evans AC (1999): Quantifying variability in the planum temporale: a probability map. Cereb Cortex 9:392-405.
62. Willems RM, Hagoort P (2007): Neural evidence for the interplay between language, gesture, and action: A review. Brain Lang 101:278-289.
63. Willems RM, Ozyurek A, Hagoort P (2006): When language meets action: The neural integration of gesture and speech. Cereb Cortex doi:10.1093/cercor/ bhl141.
64. Wilson SM, Molnar-Szakacs I, Iacoboni M (2007): Beyond superior temporal cortex: Intersubject correlations in narrative speech comprehension. Cereb Cortex doi:10.1093/cercor/bhm049.
65. Wu YC, Coulson S (2005): Meaningful gestures: Electrophysiological indices of iconic gesture comprehension. Psychophysiology 42:654-667.
66. Yasinnik Y, Renwick M, Shattuck-Hufnagel S (2004): The timing of speech-accompanying gestures with respect to prosody. Proceedings of Sound to Sense, MIT.
67. Zatorre RJ, Belin P, Penhune VB (2002): Structure and function of auditory cortex: Music and speech. Trends Cog Neurosci 6:37-46.