Neural integration of speech and gesture in schizophrenia: Evidence for differential processing of metaphoric gestures

Human Brain Mapping

Volumn 34, Issue 7, 2013, Pages 1696-1712

  Straube, Benjamin ^a   Green, Antonia ^a   Sass, Katharina ^b,c   Kirner Veselinovic, André ^b,c   Kircher, Tilo ^a  

^a PHILIPPS UNIVERSITY MARBURG   (Germany)

^b RWTH AACHEN UNIVERSITY   (Germany)

^c JARA Translational Brain Medicine   (Germany)

Author keywords

Co verbal gestures; Iconic gestures; Integration; Metaphoric gestures; Schizophrenia

Indexed keywords

ADULT; ARTICLE; CLINICAL ARTICLE; COMPARATIVE STUDY; CONTROLLED STUDY; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; GESTURE; HIPPOCAMPUS; HUMAN; INFERIOR FRONTAL GYRUS; INFORMATION PROCESSING; MALE; NERVE CELL NETWORK; PRIORITY JOURNAL; SCHIZOPHRENIA; SPEECH; TEMPORAL GYRUS;

ADULT; BRAIN; COMMUNICATION DISORDERS; FEMALE; FUNCTIONAL LATERALITY; GESTURES; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; MAGNETIC RESONANCE IMAGING; MALE; MEMORY; METAPHOR; MIDDLE AGED; NEUROPSYCHOLOGICAL TESTS; OXYGEN; PHOTIC STIMULATION; SCHIZOPHRENIA; SPEECH; YOUNG ADULT;

EID: 84879199911     PISSN: 10659471     EISSN: 10970193     Source Type: Journal    
DOI: 10.1002/hbm.22015     Document Type: Article

References (76)

1. Allen P, Seal ML, Valli I, Fusar-Poli P, Perlini C, Day F, Wood SJ, Williams SC, McGuire PK (2011): Altered prefrontal and hippocampal function during verbal encoding and recognition in people with prodromal symptoms of psychosis. Schizophr Bull 37: 746-756.
2. Amedi A, von Kriegstein K, van Atteveldt NM, Beauchamp MS, Naumer MJ (2005): Functional imaging of human crossmodal identification and object recognition. Exp Brain Res 166: 559-571.
3. Andreasen NC (1983): Scale for the Assessment of Negative Symptoms (SANS). Iowa City, IA: University of Iowa.
4. Andreasen NC (1984). Scale for the Assessment of Positive Symptoms (SAPS). Iowa City, IA: University of Iowa.
5. Andreasen NC, Pressler M, Nopoulos P, Miller D, Ho BC (2010). Antipsychotic dose equivalents and dose-years: A standardized method for comparing exposure to different drugs. Biol Psychiatry 67: 255-262.
6. Beauchamp MS, Argall BD, Bodurka J, Duyn JH, Martin A (2004): Unraveling multisensory integration: Patchy organization within human STS multisensory cortex. Nat Neurosci 7: 1190-1192.
7. Beauchamp MS, Yasar NE, Frye RE, Ro T (2008): Touch, sound and vision in human superior temporal sulcus. Neuroimage 41: 1011-1020.
8. Bergemann N, Parzer P, Jaggy S, Auler B, Mundt C, Maier-Braunleder S (2008): Estrogen and comprehension of metaphoric speech in women suffering from schizophrenia: Results of a double-blind, placebo-controlled trial. Schizophr Bull 34: 1172-1181.
9. Bernardis P, Salillas E, Caramelli N (2008): Behavioural and neurophysiological evidence of semantic interaction between iconic gestures and words. Cogn Neuropsychol 25: 1114-1128.
10. Berndl K, von Cranach M, Grüsser OJ (1986): Impairment of perception and recognition of faces, mimic expression and gestures in schizophrenic patients. Eur Arch Psychiatry Neurol Sci 235: 282-291.
11. Bonner-Jackson A, Yodkovik N, Csernansky JG, Barch DM (2008. Episodic memory in schizophrenia: The influence of strategy use on behavior and brain activation. Psychiatry Res 164: 1-15.
12. Bucci S, Startup M, Wynn P, Baker A, Lewin TJ (2008): Referential delusions of communication and interpretations of gestures. Psychiatry Res 158: 27-34.
13. Bunge SA, Helskog EH, Wendelken C (2009): Left, but not right, rostrolateral prefrontal cortex meets a stringent test of the relational integration hypothesis. Neuroimage 46: 338-342.
14. Bunge SA, Wendelken C, Badre D, Wagner AD (2005): Analogical reasoning and prefrontal cortex: Evidence for separable retrieval and integration mechanisms. Cereb Cortex 15: 239-249.
15. Calvert GA (2001): Crossmodal processing in the human brain: Insights from functional neuroimaging studies. Cereb Cortex 11: 1110-1123.
16. Crow TJ (2000): Schizophrenia as the price that homo sapiens pays for language: A resolution of the central paradox in the origin of the species. Brain Res Brain Res Rev 31: 118-129.
17. de Bonis M, Epelbaum C, Deffez V, Féline A (1997): The comprehension of metaphors in schizophrenia. Psychopathology 30: 149-154.
18. Dick AS, Goldin-Meadow S, Hasson U, Skipper JI, Small SL (2009): Co-speech gestures influence neural activity in brain regions associated with processing semantic information. Hum Brain Mapp 30: 3509-3526.
19. Eickhoff SB, Stephan KE, Mohlberg H, Grefkes C, Fink GR, Amunts K, Zilles K (2005): A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data. Neuroimage 25: 1325-1335.
20. Eviatar Z, Just MA (2006): Brain correlates of discourse processing: An fMRI investigation of irony and conventional metaphor comprehension. Neuropsychologia 44: 2348-2359.
21. Green A, Straube B, Weis S, Jansen A, Willmes K, Konrad K, Kircher T (2009): Neural integration of iconic and unrelated coverbal gestures: A functional MRI study. Hum Brain Mapp 30: 3309-3324.
22. Green AE, Fugelsang JA, Dunbar KN (2006a): Automatic activation of categorical and abstract analogical relations in analogical reasoning. Mem Cognit 34: 1414-1421.
23. Green AE, Fugelsang JA, Kraemer DJ, Shamosh NA, Dunbar KN (2006b): Frontopolar cortex mediates abstract integration in analogy. Brain Res 1096: 125-137.
24. Grüsser OJ, Kirchhoff N, Naumann A (1990): Brain mechanisms for recognition of faces, facial expression, and gestures: Neuropsychological and electroencephalographic studies in normals, brain-lesioned patients, and schizophrenics. Res Publ Assoc Res Nerv Ment Dis 67: 165-193.
25. Hagoort P, Baggio G, Willems R (2009): Semantic unification. In: Gazzaniga M, editor. The Cognitive Neurosciences IV. Cambridge: MIT Press.
26. Hein G, Doehrmann O, Müller NG, Kaiser J, Muckli L, Naumer MJ (2007): Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. J Neurosci 27: 7881-7887.
27. Holle H, Gunter TC, Ruschemeyer SA, Hennenlotter A, Iacoboni M (2008): Neural correlates of the processing of co-speech gestures. Neuroimage 39: 2010-2024.
28. Holle H, Obleser J, Rueschemeyer SA, Gunter TC (2010): Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions. Neuroimage 49: 875-884.
29. Hubbard AL, Wilson SM, Callan DE, Dapretto M (2009): Giving speech a hand: Gesture modulates activity in auditory cortex during speech perception. Hum Brain Mapp 30: 1028-1037.
30. Iakimova G, Passerieux C, Denhière G, Laurent JP, Vistoli D, Vilain J, Hardy-Baylé MC (2010): The influence of idiomatic salience during the comprehension of ambiguous idioms by patients with schizophrenia. Psychiatry Res 177: 46-54.
31. Kable JW, Chatterjee A (2006): Specificity of action representations in the lateral occipitotemporal cortex. J Cogn Neurosci 18: 1498-1517.
32. Kable JW, Kan IP, Wilson A, Thompson-Schill SL, Chatterjee A (2005): Conceptual representations of action in the lateral temporal cortex. J Cogn Neurosci 17: 1855-1870.
33. Kay SR, Fiszbein A, Opler LA (1987): The positive and negative syndrome scale (PANSS) for schizophrenia. Schizophr Bull 13: 261-276.
34. Kelly SD, Ozyürek A, Maris E (2010): Two sides of the same coin: Speech and gesture mutually interact to enhance comprehension. Psychol Sci 21: 260-267.
35. Kircher T, Straube B, Leube D, Weis S, Sachs O, Willmes K, Konrad K, Green A (2009): Neural interaction of speech and gesture: Differential activations of metaphoric co-verbal gestures. Neuropsychologia 47: 169-179.
36. Kircher T, Whitney C, Krings T, Huber W, Weis S (2008): Hippocampal dysfunction during free word association in male patients with schizophrenia. Schizophr Res 101: 242-255.
37. Kircher TT, Leube DT, Erb M, Grodd W, Rapp AM (2007): Neural correlates of metaphor processing in schizophrenia. Neuroimage 34: 281-289.
38. Kircher TT, Liddle PF, Brammer MJ, Williams SC, Murray RM, McGuire PK (2002): Reversed lateralization of temporal activation during speech production in thought disordered patients with schizophrenia. Psychol Med 32: 439-449.
39. Leube DT, Rapp A, Buchkremer G, Bartels M, Kircher TT, Erb M, Grodd W (2003): Hippocampal dysfunction during episodic memory encoding in patients with schizophrenia-An fMRI study. Schizophr Res 64: 83-85.
40. Lewis JW (2006): Cortical networks related to human use of tools. Neuroscientist 12: 211-231.
41. Lewis JW, Brefczynski JA, Phinney RE, Janik JJ, DeYoe EA (2005): Distinct cortical pathways for processing tool versus animal sounds. J Neurosci 25: 5148-158.
42. Luck D, Danion JM, Marrer C, Pham BT, Gounot D, Foucher J (2010): Abnormal medial temporal activity for bound information during working memory maintenance in patients with schizophrenia. Hippocampus 20: 936-948.
43. Luo Q, Perry C, Peng D, Jin Z, Xu D, Ding G, Xu S (2003): The neural substrate of analogical reasoning: An fMRI study. Brain Res Cogn Brain Res 17: 527-534.
44. Martin P, Tewesmeier M, Albers M, Schmid G, Scharfetter C (1994): Investigation of gestural and pantomime performance in chronic schizophrenic inpatients. Eur Arch Psychiatry Clin Neurosci 244: 59-64.
45. Matthews N, Gold BJ, Sekuler R, Park S (2011): Gesture imitation in schizophrenia. Schizophr Bull.
46. McNeill D, editor (1992): Hand and Mind: What Gestures Reveal About Thought. Chicago, IL: University of Chicago Press.
47. Mittal VA, Tessner KD, McMillan AL, Delawalla Z, Trotman HD, Walker EF (2006): Gesture behavior in unmedicated schizotypal adolescents. J Abnorm Psychol 115: 351-358.
48. Naumer MJ, Doehrmann O, Müller NG, Muckli L, Kaiser J, Hein G (2009): Cortical plasticity of audio-visual object representations. Cereb Cortex 19: 1641-1653.
49. Nestor PG, Kubicki M, Kuroki N, Gurrera RJ, Niznikiewicz M, Shenton ME, McCarley RW (2007): Episodic memory and neuroimaging of hippocampus and fornix in chronic schizophrenia. Psychiatry Res 155: 21-28.
50. Nichols T, Brett M, Andersson J, Wager T, Poline JB (2005): Valid conjunction inference with the minimum statistic. Neuroimage 25: 653-660.
51. Ozyurek A, Willems RM, Kita S, Hagoort P (2007): On-line integration of semantic information from speech and gesture: Insights from event-related brain potentials. J Cogn Neurosci 19: 605-616.
52. Rapp AM, Leube DT, Erb M, Grodd W, Kircher TT (2004): Neural correlates of metaphor processing. Brain Res Cogn Brain Res 20: 395-402.
53. Rapp AM, Leube DT, Erb M, Grodd W, Kircher TT (2007): Laterality in metaphor processing: Lack of evidence from functional magnetic resonance imaging for the right hemisphere theory. Brain Lang 100: 142-149.
54. Silverstein SM, Berten S, Essex B, Kovács I, Susmaras T, Little DM (2009): An fMRI examination of visual integration in schizophrenia. J Integr Neurosci 8: 175-202.
55. Skipper JI, Goldin-Meadow S, Nusbaum HC, Small SL (2007): Speech-associated gestures, Broca's area, and the human mirror system. Brain Lang 101: 260-277.
56. Slotnick SD, Moo LR, Segal JB, Hart J Jr (2003): Distinct prefrontal cortex activity associated with item memory and source memory for visual shapes. Brain Res Cogn Brain Res 17: 75-82.
57. Soriano MF, Jiménez JF, Román P, Bajo MT (2008): Cognitive substrates in semantic memory of formal thought disorder in schizophrenia. J Clin Exp Neuropsychol 30: 70-82.
58. Stevenson RA, VanDerKlok RM, Pisoni DB, James TW (2011): Discrete neural substrates underlie complementary audiovisual speech integration processes. Neuroimage 55: 1339-1345.
59. Straube B, Green A, Bromberger B, Kircher T(2011a): The differentiation of iconic and metaphoric gestures: Common and unique integration processes. Hum Brain Mapp 32: 520-533.
60. Straube B, Green A, Chatterjee A, Kircher T (2011b): Encoding social interactions: The neural correlates of true and false memories. J Cogn Neurosci 23: 306-324.
61. Straube B, Green A, Jansen A, Chatterjee A, Kircher T (2010): Social cues, mentalizing and the neural processing of speech accompanied by gestures. Neuropsychologia 48: 382-393.
62. Straube B, Green A, Weis S, Chatterjee A, Kircher T (2009): Memory effects of speech and gesture binding: Cortical and hippocampal activation in relation to subsequent memory performance. J Cogn Neurosci 21: 821-836.
63. Stringaris AK, Medford NC, Giampietro V, Brammer MJ, David AS (2007): Deriving meaning: Distinct neural mechanisms for metaphoric, literal, and non-meaningful sentences. Brain Lang 100: 150-162.
64. Surguladze SA, Calvert GA, Brammer MJ, Campbell R, Bullmore ET, Giampietro V, David AS (2001): Audio-visual speech perception in schizophrenia: An fMRI study. Psychiatry Res 106: 1-14.
65. Szycik GR, Münte TF, Dillo W, Mohammadi B, Samii A, Emrich HM, Dietrich DE (2009): Audiovisual integration of speech is disturbed in schizophrenia: An fMRI study. Schizophr Res 110: 111-118.
66. Takahashi H, Kato M, Sassa T, Shibuya T, Koeda M, Yahata N, Matsuura M, Asai K, Suhara T, Okubo Y (2010): Functional deficits in the extrastriate body area during observation of sports-related actions in schizophrenia. Schizophr Bull 36: 642-647.
67. Troisi A, Spalletta G, Pasini A (1998): Non-verbal behaviour deficits in schizophrenia: An ethological study of drug-free patients. Acta Psychiatr Scand 97: 109-115.
68. van Veelen NM, Vink M, Ramsey NF, Sommer IE, van Buuren M, Hoogendam JM, Kahn RS (2011): Reduced language lateralization in first-episode medication-naive schizophrenia. Schizophr Res 127: 195-201.
69. Weiss AP, Ellis CB, Roffman JL, Stufflebeam S, Hamalainen MS, Duff M, Goff DC, Schacter DL (2009): Aberrant frontoparietal function during recognition memory in schizophrenia: A multimodal neuroimaging investigation. J Neurosci 29: 11347-11359.
70. Wendelken C, Nakhabenko D, Donohue SE, Carter CS, Bunge SA (2008): "Brain is to thought as stomach is to ??": Investigating the role of rostrolateral prefrontal cortex in relational reasoning. J Cogn Neurosci 20: 682-693.
71. Willems RM, Hagoort P (2007): Neural evidence for the interplay between language, gesture, and action: A review. Brain Lang 101: 278-289.
72. Willems RM, Ozyurek A, Hagoort P (2007): When language meets action: The neural integration of gesture and speech. Cereb Cortex 17: 2322-2333.
73. Willems RM, Ozyurek A, Hagoort P (2009): Differential roles for left inferior frontal and superior temporal cortex in multimodal integration of action and language. Neuroimage 47: 1992-2004.
74. Wu YC, Coulson S (2007): Iconic gestures prime related concepts: An ERP study. Psychon Bull Rev 14: 57-63.
75. Wu YC, Coulson S (2010): Gestures modulate speech processing early in utterances. Neuroreport 21: 522-526.
76. Xu J, Gannon PJ, Emmorey K, Smith JF, Braun AR (2009): Symbolic gestures and spoken language are processed by a common neural system. Proc Natl Acad Sci USA 106: 20664-20669.