Impaired multisensory processing in schizophrenia: Deficits in the visual enhancement of speech comprehension under noisy environmental conditions

Schizophrenia Research

Volumn 97, Issue 1-3, 2007, Pages 173-183

  Ross, Lars A ^a,b,c   Saint Amour, Dave ^b,d   Leavitt, Victoria M ^b,e   Molholm, Sophie ^a,b   Javitt, Daniel C ^a,b   Foxe, John J ^a,b,e  

^a CITY COLLEGE OF NEW YORK   (United States)

^b NATHAN S KLINE INSTITUTE FOR PSYCHIATRIC RESEARCH   (United States)

^c RAMAPO COLLEGE OF NEW JERSEY   (United States)

^d UNIVERSITÉ DE MONTRÉAL   (Canada)

^e CITY UNIVERSITY OF NEW YORK   (United States)

Author keywords

Audiovisual; Crossmodal; Multisensory; Schizophrenia; Sensory integration; Speech

Indexed keywords

ADULT; ARTICLE; CLINICAL ARTICLE; COMPREHENSION; CONTROLLED STUDY; FEMALE; HUMAN; MALE; NOISE; PARANOID SCHIZOPHRENIA; PRIORITY JOURNAL; SCHIZOAFFECTIVE PSYCHOSIS; SCHIZOPHRENIA; SIGNAL NOISE RATIO; SPEECH DISORDER; SPEECH PERCEPTION;

ADULT; APTITUDE; ATTENTION; COMPREHENSION; FEMALE; HUMANS; LIPREADING; MALE; MIDDLE AGED; PERCEPTUAL MASKING; PSYCHIATRIC STATUS RATING SCALES; SCHIZOPHRENIA; SCHIZOPHRENIC PSYCHOLOGY; SPEECH PERCEPTION; VISUAL PERCEPTION;

EID: 36048970381     PISSN: 09209964     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.schres.2007.08.008     Document Type: Article

References (94)

1. Alain C., Bernstein L.J., Cortese F., Yu H., and Zipursky R.B. Deficits in automatically detecting changes in conjunction of auditory features in patients with schizophrenia. Psychophysiology 39 (2002) 599-606
2. Allison T.P.A., and McCarthy G. Social perception from visual cues: role of the STS region. Trends Cogn. Sci. 4 (2000) 267-278
3. Baltaxe C.A., and Simmons III J.Q. Speech and language disorders in children and adolescents with schizophrenia. Schizophr. Bull. 21 (1995) 677-692
4. Barnett. Overview of speech intelligibility. Proc. IOA 21 (1999) 1-15
5. Bernstein L.J., Auer E.T., and Moore J.K. Audiovisual Speech Binding: Convergence or Association? (2004), MIT Press, Cambridge, MA
6. Buchan J.N., Paré M., and Munhall K.G. Spatial statistics of gaze fixations during dynamic face processing. Social Neurosci. 2 (2007) 1-13
7. Bull H.C., and Venables P.H. Speech perception in schizophrenia. Br. J. Psychiatry 125 (1974) 350-354
8. Butler P.D., Hoptman M.J., Nierenberg J., Foxe J.J., Javitt D.C., and Lim K.O. Visual white matter integrity in schizophrenia. Am. J. Psychiatry 163 (2006) 2011-2013
9. Butler P.D., Martinez A., Foxe J.J., Kim D., Silipo G., Mahony J., Shpaner M., Jalbrikowski M., and Javitt D.C. Subcortical visual dysfunction in schizophrenia drives secondary cortical impairments. Brain 130 (2007) 417-430
10. Callan D.E., Jones J.A., Munhall K., Callan A.M., Kroos C., and Vatikiotis-Bateson E. Neural processes underlying perceptual enhancement by visual speech gestures. NeuroReport 14 (2003) 2213-2218
11. Calvert G.A. Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb. Cortex 11 (2001) 1110-1123
12. Calvert G.A., and Campbell R. Reading speech from still and moving faces: the neural substrates of visible speech. J. Cogn. Neurosci. 15 (2003) 57-70
13. Campbell R., and MacSweeney M. Neuroimaging Studies of Cross-Modal Plasticity and Language Processing in Deaf People (2004), MIT Press, Cambridge, MA
14. Cannon M., Caspi A., and Moffitt T.E. Evidence for early-childhood, pan-developmental impairment specific to schizophreniform disorder: results from a longitudinal birth cohort. Arch. Gen. Psychiatry 59 (2002) 449-456
15. CHABA, Working Group on Communication Aids for the Hearing-Impaired. Speech perception aids forth the hearing impaired people: current status and needed research. J. Acoust. Soc. Am. 90 (1991) 637-685
16. Chen Y., Bidwell L.C., and Holzman P.S. Visual motion integration in schizophrenia patients, their first-degree relatives, and patients with bipolar disorder. Schizophr. Res. 74 (2005) 271-281
17. Cienkowski K.M., and Carney A.E. Auditory-visual speech perception and aging. Ear Hear. 23 (2002) 439-449
18. Condray R. Language disorder in schizophrenia as a developmental learning disorder. Schizophr. Res. 73 (2005) 5-20
19. de Gelder B., Vroomen J., Annen L., Masthof E., and Hodiamont P. Audio-visual integration in schizophrenia. Schizophr. Res. 59 (2003) 211-218
20. Dekle D.J., Fowler C.A., and Funnell M.G. Audiovisual integration in perception of real words. Percept. Psychophys. 51 (1992) 355-362
21. Doniger G.M., Foxe J.J., Murray M.M., Higgins B.A., and Javitt D.C. Impaired visual object recognition and dorsal/ventral stream interaction in schizophrenia. Arch. Gen. Psychiatry 59 (2002) 1011-1020
22. Dudley H. The automatic synthesis of speech. Proc. Natl. Acad. Sci. U.S.A. 25 7 (1939) 377-383
23. Erber N.P. Interaction of audition and vision in the recognition of oral speech stimuli. J. Speech Hear. Res. 12 (1969) 423-425
24. First M.B., Spitzer R.L., Benjamin L., Gibbon M., and Williams J.B.W. Structured Clinical Interview for DSM-IV (1997), American Psychiatric Publishers INC
25. Foxe J.J., and Schroeder C.E. The case for a feedforward component in multisensory integration mechanisms. NeuroReport 16 (2005) 419-423
26. Foxe J.J., Doniger G.M., and Javitt D.C. Early visual processing deficits in schizophrenia: impaired P1 generation revealed by high-density electrical mapping. NeuroReport 12 (2001) 3815-3820
27. Foxe J.J., Murray M.M., and Javitt D.C. Filling-in in schizophrenia: a high-density electrical mapping and source-analysis investigation of illusory contour processing. Cereb. Cortex 15 (2005) 1914-1927
28. French N.R., and Steinberg J.C. Factors governing the Intelligibility of Speech Sounds. J. Acoust. Soc. Am. 19 (1947) 90-119
29. Gagné J.-P., Querengesser C., Folkeard P., Munhall K., and Masterson V.M. Auditory, visual, and audiovisual speech intelligibility for sentence-length stimuli: an investigation of conversational and clear speech. Volta Rev. 97 (1995) 33-51
30. Grant K., and Seitz P.F. The use of visible speech cues for improving auditory detection of spoken sentences. J. Acoust. Soc. Am. 108 (2000) 1197-1208
31. Greenberg S., and Arai T. What are the essential cues for understanding spoken language?. IEICE Trans. Inf. Syst., E 87 (2004) 1059-1070
32. Hoffman R.E., Rapaport J., Mazure C.M., and Quinlan D.M. Selective speech perception alterations in schizophrenic patients reporting hallucinated "voices". Am. J. Psychiatry 156 (1999) 393-399
33. Hyman S.E., Arana G.W., and Rosenbaum J.F. Handbook of Psychiatric Drug Therapy (1995), Little, Brown and Company, Boston
34. Iacoboni M., Koski L.M., Brass M., Bekkering H., Woods R.P., Dubeau M.C., Mazziotta J.C., and Rizzolatti G. Reafferent copies of imitated actions in the right superior temporal cortex. Proc. Natl. Acad. Sci. U. S. A. 98 (2001)
35. Iarocci G., and McDonald J. Sensory integration and the perceptual experience of persons with autism. J. Autism Dev. Disord. 36 (2006) 77-90
36. Javitt D.C., Doneshka P., Zylberman I., Ritter W., and Vaughan Jr. H.G. Impairment of early cortical processing in schizophrenia: an event-related potential confirmation study. Biol. Psychiatry 33 (1993) 513-519
37. Javitt D.C., Doneshka P., Grochowski S., and Ritter W. Impaired mismatch negativity generation reflects widespread dysfunction of working memory in schizophrenia. Arch. Gen. Psychiatry 52 (1995) 550-558
38. Javitt D.C., Strous R.D., Grochowski S., Ritter W., and Cowan N. Impaired precision, but normal retention, of auditory sensory ("echoic") memory information in schizophrenia. J. Abnorm. Psychology 106 (1997) 315-324
39. Javitt D.C., Grochowski S., Shelley A.M., and Ritter W. Impaired mismatch negativity (MMN) generation in schizophrenia as a function of stimulus deviance, probability, and interstimulus/interdeviant interval. Electroencephalogr. Clin. Neurophysiol. 108 (1998) 143-153
40. Javitt D.C., Shelley A., and Ritter W. Associated deficits in mismatch negativity generation and tone matching in schizophrenia. Clin. Neurophysiol. 111 (2000) 1733-1737
41. Javitt D.C., Shelley A.M., Silipo G., and Lieberman J.A. Deficits in auditory and visual context-dependent processing in schizophrenia: defining the pattern. Arch. Gen. Psychiatry 57 (2000) 1131-1137
42. Jiang W., Wallace M., Jiang H., Vaughan J., and Stein B. Two cortical areas mediate multisensory integration in superior colliculus neurons. J. Neurophysiol. 85 (2001) 506-522
43. Jibson M.D., and Tendon R. New atypical antipsychotic medications. J. Psychiatr. Res. 32 (1998) 215-228
44. Kern J.K. The possible role of the cerebellum in autism/PDD: disruption of a multisensory feedback loop. Med. Hypotheses 59 (2002) 255-260
45. Kim J., Doop M.L., Blake R., and Park S. Impaired visual recognition of biological motion in schizophrenia. Schizophr. Res. 77 (2005) 299-307
46. Kim D., Wylie G., Pasternak R., Butler P.D., and Javitt D.C. Magnocellular contributions to impaired motion processing in schizophrenia. Schizophr. Res. 82 (2006) 1-8
47. Kircher T., Liddle P.F., Brammer M.J., Williams S.C., Murray R.M., and McGuire P.K. Reversed lateralization of temporal activation during speech production in thought disordered patients with schizophrenia. Psychol. Med. 32 (2002) 439-449
48. Kucera H., and Francis W.N. Computational Analysis of Present-Day American English (1967), Brown University Press, Providence, I
49. Kuperberg G.R., Sitnikova T., Goff D., and Holcomb P.J. Making sense of sentences in schizophrenia: electrophysiological evidence for abnormal interactions between semantic and syntactic processing. J. Abnorm. Psychology 115 (2006) 251-265
50. Lebib R., Papo D., de Bode S., and Baudonniere P.M. Evidence of a visual-to-auditory cross-modal sensory gating phenomenon as reflected by the human P50 event-related brain potential modulation. Neurosci. Lett. 341 (2003) 185-188
51. Lee S.H., Chung Y.C., Kim Y.K., and Suh K.Y. Abnormal speech perception in schizophrenia with auditory hallucinations. Acta Neuropsychiatr. 16 (2004) 154-159
52. Leitman D.I., Foxe J.J., Butler P.D., Saperstein A., Revheim N., and Javitt D.C. Sensory contributions to impaired prosodic processing in schizophrenia. Biol. Psychiatry 58 (2005) 56-61
53. Manoach D.S. Prefrontal cortex dysfunction during working memory performance in schizophrenia: reconciling discrepant findings. Schizophr. Res. 60 (2003) 285-298
54. Manoach D.S., Press D.Z., and Thangaraj V. Schizophrenic subjects activate dorsolateral prefrontal cortex during a working memory task, as measured by fMRI. Biol. Psychiatry 45 (1999) 1128-1137
55. Massaro D.W. Perceiving Talking Faces: Insights into Auditory Attention (1998), MIT Press, Cambridge, MA
56. McGuire P., Quested D.J., Spence S.A., Murray R.M., Frith C.D., and Liddle P.F. Pathophysiology of 'positive' thought disorder in schizophrenia. Br. J. Psychiatry 173 (1998) 231-235
57. McGurk H., and MacDonald J. Hearing lips and seeing voices. Nature 264 (1976) 746-748
58. Meredith M.A., and Stein B.E. Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J. Neurophysiol. 56 (1986) 640-662
59. Michie P.T. What has MMN revealed about the auditory system in schizophrenia?. Int. J. Psychophysiol. 42 (2001) 177-194
60. Michie P.T., Innes-Brown H., Todd J., and Jablensky A.V. Duration mismatch negativity in biological relatives of patients with schizophrenia spectrum disorders. Biol. Psychiatry 52 (2002) 749-758
61. Molholm S., and Foxe J.J. Look 'hear', primary auditory cortex is active during lip-reading. NeuroReport 16 (2005) 123-124
62. Molholm S., Ritter W., Javitt D.C., and Foxe J.J. Multisensory visual-auditory object recognition in humans: a high-density electrical mapping study. Cereb. Cortex 14 (2004) 452-465
63. Molholm S., Sehatpour P., and Mehta A.D. Audio-visual multisensory integration in superior parietal lobule revealed by human intracranial recordings. J. Neurophysiol. 96 (2006) 721-729
64. Munhall K.G., Servos P., Santi A., and Goodale M.A. Dynamic visual speech perception in a patient with visual form agnosia. NeuroReport 13 (2002) 1793-1796
65. Munhall K.G., Jones J.A., Callan D.E., Kuratate T., and Vatikiotis-Bateson E. Visual prosody and speech intelligibility: head movement improves auditory speech perception. Psychol. Sci. 15 (2004) 133-137
66. Munhall K.G., Jones J.A., Callan D.E., Kuratate T., and Vatikiotis-Bateson E. Visual prosody and speech intelligibility: head movement improves auditory speech perception. Psychol. Sci. 15 (2004) 133-137
67. Myslobodsky M.S., Goldberg T., Johnson F., Hicks L., and Weinberger D.R. Lipreading in patients with schizophrenia. J. Nerv. Ment. Dis. 180 (1992) 168-171
68. O'Neill J.J. Contributions of the visual component of oral symbols to speech comprehension. J. Speech Hear. Disord. 19 (1954) 429
69. Paré M., Richler R.C., Hove M., and Munhall K.G. Gaze behavior in audiovisual speech perception: the influence of ocular fixations on the McGurk effect. Percept. Psychophys. 65 (2003) 553-567
70. Peuskens J., and Link C.G.G. A comparison of quetiapine and chlorpromazine in the treatment of schizophrenia. Acta Psychiatr. Scand. 96 (1997) 265-273
71. Rorden C., Heutnik J., Greenfield E., and Robertson I.H. When a rubber hand 'feels' what a real hand cannot. Neuroreport. 10 1 (1999) 135-138
72. Rosburg T., Kreitschmann-Andermahr I., and Sauer H. Mismatch negativity in schizophrenia research. An indicator of early processing disorders of acoustic information. Nervenarzt 75 (2004) 633-641
73. Ross L.A., Saint-Amour D., Leavitt V.M., Javitt D.C., and Foxe J.J. Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. Cereb. Cortex 17 (2007) 1147-1153
74. Ruchsow M., Trippel N., Groen G., Spitzer M., and Kiefer M. Semantic and syntactic processes during sentence comprehension in patients with schizophrenia: evidence from event-related potentials. Schizophr. Res. 64 (2003) 147-156
75. Saint-Amour D., De Sanctis P., Molholm S., Ritter W., and Foxe J.J. Seeing voices: High-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion. Neuropsychologia 45 3 (2006) 587-597
76. Salisbury D.F., Shenton M.E., Griggs C.B., Bonner-Jackson A., and McCarley R.W. Mismatch negativity in chronic schizophrenia and first-episode schizophrenia. Arch. Gen. Psychiatry 59 (2002) 686-694
77. Schlumberger E., Narbona J., and Manrique M. Non-verbal development of children with deafness with and without cochlear implants. Dev. Med. Child Neurol. 46 (2004) 599-606
78. Schonauer K., Achtergarde D., and Reker T. Lipreading in prelingually deaf and hearing patients with schizophrenia. J. Nerv. Ment. Dis. 186 (1998) 247-249
79. Schroeder C.E., and Foxe J. Multisensory contributions to low-level, "Unisensory" processing. Curr. Opin. Neurobiol. 15 (2005) 454-458
80. Schwartz B.D., Tomlin H.R., Evans W.J., and Ross K.V. Neurophysiologic mechanisms of attention: a selective review of early information processing in schizophrenics. Front. Biosci. 6 (2001) D120-D134
81. Shenton M.E., Dickey C.C., Frumin M., and McCarley R.W. A review of MRI findings in schizophrenia. Schizophr. Res. 49 (2001) 1-52
82. Stein B.E., Jiang W., Wallace M.T., and Stanford T.R. Nonvisual influences on visual-information processing in the superior colliculus. Prog. Brain Res. 134 (2001) 143-156
83. Stein B.E., Wallace M.W., Stanford T.R., and Jiang W. Cortex governs multisensory integration in the midbrain. Neuroscientist 8 (2002) 306-314
84. Sumby W.H., and Pollack I. Visual contribution to speech intelligibility in noise. J. Acoust. Soc. Am. 26 (1954) 212-215
85. Surguladze S.A., Calvert G.A., and Brammer M.J. Audio-visual speech perception in schizophrenia: an fMRI study. Psychiatry Res. 106 (2001) 1-14
86. Titone D., and Levy D.L. Lexical competition and spoken word identification in schizophrenia. Schizophr. Res. 68 (2004) 75-85
87. Vatikiotis-Bateson E., Eigsti I.M., Yano S., and Munhall K.G. Eye movement of perceivers during audiovisual speech perception. Percept. Psychophys. 60 (1998) 926-940
88. Wallace M.T., and Stein B.E. Development of multisensory neurons and multisensory integration in cat superior colliculus. J. Neurosci. 17 (1997) 2429-2444
89. Wallace M., and Stein B.E. Early experience determines how the senses will interact. J. Neurophysiol. 97 1 (2007) 921-926
90. Wallace M., Perrault II T.J., Hairston W.D., and Stein B.E. Visual experience is necessary for the development of multisensory integration. J. Neurosci. 24 (2004) 9580-9584
91. Watson C.S., Qiu W.W., Chamberlain M.M., and Li X. Auditory and visual speech perception: confirmation of a modality-independent source of individual differences in speech recognition. J. Acoust. Soc. Am. 100 (1996) 1153-1162
92. Weinstein S., Werker J.F., Vouloumanos A., Woodward T.S., and Ngan E.T. Do you hear what I hear? Neural correlates of thought disorder during listening to speech in schizophrenia. Schizophr. Res. 86 (2006) 130-137
93. Woods S.W. Chlorpromazine equivalent doses for the newer atypical antipsychotics. J. Clin. Psychiatry 64 (2003) 663-667
94. Yeap S., Kelly S.P., Sehatpour P., Magno E., Javitt D.C., Garavan H., Thakore J.H., and Foxe J.J. Early visual sensory deficits as endophenotypes for schizophrenia: high-density electrical mapping in clinically unaffected first-degree relatives. Arch. Gen. Psychiatry 63 (2006) 1180-1188