Audiovisual spoken word training can promote or impede auditory-only perceptual learning: Results from prelingually deafened adults with late-acquired cochlear implants and normal-hearing adults

Frontiers in Psychology

Volumn 5, Issue AUG, 2014, Pages

  Bernstein, Lynne E ^a   Eberhardt, Silvio P ^a   Auer Jr , Edward T ^a  

^a GEORGE WASHINGTON UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84906354531     PISSN: None     EISSN: 16641078     Source Type: Journal    
DOI: 10.3389/fpsyg.2014.00934     Document Type: Article

References (106)

1. Ahissar, M., & Hochstein, S. (1997). Task difficulty and the specificity of perceptual learning. Nature, 387(6631), 401-406.
2. Ahissar, M., Nahum, M., Nelken, I., & Hochstein, S. (2008). Reverse hierarchies and sensory learning. Philosophical Transactions of the Royal Society B, 364, 285-299.
3. Allen, T. E. (1986). Patterns of academic achievement amoung hearing impaired students: 1974- 1983. In A. N. Schildroth & M. A. Karchmer (Eds.), Deaf Children in America (pp. 161- 206). San Diego, CA: College-Hill Press.
4. Aparicio, M., Peigneux, P., Charlier, B., Neyrat, C., & Leybaert, J. (2012). Early experience of Cued Speech enhances speechreading performance in deaf. Scandinavian Journal of Psychology, 53(1), 41-46.
5. Auer, E. T., Jr., & Bernstein, L. E. (1997). Speechreading and the structure of the lexicon: computationally modeling the effects of reduced phonetic distinctiveness on lexical uniqueness. Journal of the Acoustical Society of America, 102(6), 3704-3710.
6. Auer, E. T., Jr., & Bernstein, L. E. (2007). Enhanced visual speech perception in individuals with early-onset hearing impairment. Journal of Speech, Language, and Hearing Research, 50(5), 1157-1165.
7. Auer, E. T., Jr., & Bernstein, L. E. (in preparation). A corpus of audiovisual disyllabic speech tokens spoken by two talkers and visual perceptual identification data obtained from skilled hearing and deaf lipreaders
8. Auer, E. T., Jr., Bernstein, L. E., Sungkarat, W., & Singh, M. (2007). Vibrotactile activation of the auditory cortices in deaf versus hearing adults. Neuroreport, 18(7), 645-648.
9. Bernstein, L. E., Auer, E. T., Jr., Eberhardt, S. P., & Jiang, J. (2013). Auditory perceptual learning for speech perception can be enhanced by audiovisual training. Frontiers in Neuroscience, 7, 34.
10. Bernstein, L. E., Auer, E. T., Jr., & Takayanagi, S. (2004). Auditory speech detection in noise enhanced by lipreading. Speech Communication, 44(1-4), 5-18.
11. Bernstein, L. E., Auer, E. T., Jr., & Tucker, P. E. (2001). Enhanced speechreading in deaf adults: can short-term training/practice close the gap for hearing adults? Journal of Speech, Language, and Hearing Research, 44(1), 5-18.
12. Bernstein, L. E., Auer, E. T., Jr., Wagner, M., & Ponton, C. W. (2008). Spatio-temporal dynamics of audiovisual speech processing. Neuroimage, 39, 423-435.
13. Bernstein, L. E., Demorest, M. E., & Tucker, P. E. (2000). Speech perception without hearing. Percept Psychophys, 62(2), 233-252.
14. Bernstein, L. E., Jiang, J., Pantazis, D., Lu, Z. L., & Joshi, A. (2011). Visual phonetic processing localized using speech and nonspeech face gestures in video and point-light displays. Human Brain Mapping, 32(10), 1660-1676.
15. Bernstein, L. E., Tucker, P. E., & Auer, E. T., Jr. (1998). Potential perceptual bases for successful use of a vibrotactile speech perception aid. Scandinavian Journal of Psychology, 39(3), 181-186.
16. Binder, J. R., Frost, J. A., Hammeke, T. A., Bellgowan, P. S., Springer, J. A., Kaufman, J. N., & Possing, E. T. (2000). Human temporal lobe activation by speech and nonspeech sounds. Cerebral Cortex, 10(5), 512-528.
17. Bodmer, D., Shipp, D. B., Ostroff, J. M., Ng, A. H., Stewart, S., Chen, J. M., & Nedzelski, J. M. (2007). A comparison of postcochlear implantation speech scores in an adult population. Laryngoscope, 117(8), 1408-1411
18. Boothroyd, A., & Cawkwell, S. (1970). Vibrotactile thresholds in pure tone audiometry. Acta Oto Laryngologica, 69, 381-387.
19. Bottari, D., Heimler, B., Caclin, A., Dalmolin, A., Giard, M. H., & Pavani, F. (2014). Visual change detection recruits auditory cortices in early deafness. Neuroimage, 94.
20. Cornett, R. O. (1967). Cued Speech. American Annals of the Deaf, 112, 3-13.
21. Davis, M. H., & Johnsrude, I. S. (2007). Hearing speech sounds: top-down influences on the interface between audition and speech perception. HearingResearch, 229(1-2), 132-147.
22. Davis, M. H., Johnsrude, I. S., Hervais-Adelman, A., Taylor, K., & McGettigan, C. (2005). Lexical information drives perceptual learning of distorted speech: Evidence from the comprehension of noise-vocoded sentences. Journal of Experimental Psychology: General, 134(2), 222-241.
23. Dettman, S., Wall, E., Constantinescu, G., & Dowell, R. (2013). Communication outcomes for groups of children using cochlear implants enrolled in auditory-verbal, aural-oral, and bilingual-bicultural early intervention programs. Otology & Neurotology, 34(3), 451- 459.
24. Dorman, M. F., Loizou, P. C., & Rainey, D. (1997). Speech intelligibility as a function of the number of channels of stimulation for signal processors using sine-wave and noise-band outputs. Journal of the Acoustical Society of America, 102(4), 2403-2411.
25. Driver, J., & Noesselt, T. (2008). Multisensory interplay reveals crossmodal influences on 'sensory-specific' brain regions, neural responses, and judgments. Neuron, 57(1), 11-23.
26. Dunn, L. M., & Dunn, L. M. (1981). Peabody Picture Vocabulary Test-Revised. Pines, MN: American Guidance Service. Eberhardt, S. P., Auer, E. T., Jr., & Bernstein, L. E. (submitted). Multisensory training can promote or impede unisensory perceptual learning of speech stimuli.
27. Erber, N. P. (1975). Auditory-visual perception of speech. Journal of Speech and Hearing Disorders, 40, 481-492.
28. Falchier, A., Cappe, C., Barone, P., & Schroeder, C. E. (2012). Sensory convergence in low-level cortices. In B. E. Stein (Ed.), The New Handbook of Multisensory Processing. Cambridge, MA: MIT.
29. Faulkner, A., Rosen, S., & Smith, C. (2000). Effects of the salience of pitch and periodicity information on the intelligibility of four-channel vocoded speech: implications for cochlear implants. Journal of the Acoustical Society of America, 108(4), 1877-1887.
30. Felleman, D. J., & Van Essen, D. C. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex, 1(1), 1-47.
31. Files, B. T., Auer, E. T., Jr., & Bernstein, L. E. (2013). The visual mismatch negativity elicited with visual speech stimuli. Frontiers in Human Neuroscience, 7, 371.
32. Files, B. T., Tjan, B., & Bernstein, L. E. (in preparation). Visual speech discrimination and identification of natural and synthetic stimuli
33. Finney, E. M., Fine, I., & Dobkins, K. R. (2001). Visual stimuli activate auditory cortex in the deaf. Nature Neuroscience, 4(12), 1171-1173.
34. Foxe, J. J., & Schroeder, C. E. (2005). The case for feedforward multisensory convergence during early cortical processing. Neuroreport, 16(5), 419-423.
35. Fryauf-Bertschy, H., Tyler, R. S., Kelsay, D. M. R., Gantz, B. J., & Woodworth, G. G. (1997). Cochlear implant use by prelingually deafened children: The influences of age at implant and length of device use. Journal of Speech and Hearing Research, 40, 183-199.
36. Fu, Q. J., & Galvin, J. J., 3rd. (2007). Perceptual learning and auditory training in cochlear implant recipients. Trends in Amplification, 11(3), 193-205.
37. Fu, Q. J., & Galvin, J. J., 3rd. (2008). Maximizing cochlear implant patients' performance with advanced speech training procedures. Hearing Research, 242(1-2), 198-208.
38. Ghazanfar, A. A., Chandrasekaran, C., & Logothetis, N. K. (2008). Interactions between the superior temporal sulcus and auditory cortex mediate dynamic face/voice integratn in Rhesus mondeys. The Journal of Neuroscience, 28(17), 4457-4469.
39. Ghazanfar, A. A., & Schroeder, C. E. (2006). Is neocortex essentially multisensory? Trends in Cognitive Science, 10(6), 278-285.
40. Gilley, P. M., Sharma, A., Mitchell, T. V., & Dorman, M. F. (2010). The influence of a sensitive period for auditory-visual integration in children with cochlear implants. Restorative Neurology and Neuroscience, 28(2), 207-218.
41. Giraud, A.-L., Price, C. J., Graham, J. M., Truy, E., & Frackowiak, S. J. (2001). Cross-modal plasticity underpins language recovery after cochlear implantation. Neuron, 30, 657-663.
42. Hammill, D. D., Pearson, N. A., & Wiederholt, J. L. (1996). Comprehensive test of Nonverbal Intelligence. Austin, TX: Pro-Ed.
43. Hasson, U., Skipper, J. I., Nusbaum, H. C., & Small, S. L. (2007). Abstract coding of audiovisual speech: beyond sensory representation. Neuron, 56(6), 1116-1126.
44. Hervais-Adelman, A., Davis, M. H., Johnsrude, I. S., & Carlyon, R. P. (2008). Perceptual learning of noise vocoded words: Effects of feedback and lexicality. Journal of Experimental Psychology: Human Performance and Perception, 34(2), 460-474.
45. Hervais-Adelman, A., Davis, M. H., Johnsrude, I. S., Taylor, K. J., & Carlyon, R. P. (2011). Generalization of perceptual learning of vocoded speech. Journal of Experimental Psychology: Human Performance and Perception, 37(1), 293-295.
46. Hubel, D. H., & Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. Journal of Physiololgy, 160, 106-154.
47. Huyse, A., Berthommier, F., & Leybaert, J. (2013). Degradation of labial information modifies audiovisual speech perception in cochlear-implanted children. Ear & Hearing, 34(1), 110-121.
48. Iverson, P., Bernstein, L. E., & Auer, E. T., Jr. (1998). Modeling the interaction of phonemic intelligibility and lexical structure in audiovisual word recognition. Speech Communication, 26(1-2), 45-63.
49. Jackson, P. L., Montgomery, A. A., & Binnie, C. A. (1976). Perceptual dimensions underlying vowel lipreading performance. Journal of Speech and Hearing Research, 19(4), 796-812.
50. Jiang, J., Alwan, A., Keating, P., Auer, E. T., Jr., & Bernstein, L. E. (2002). On the relationship between face movements, tongue movements, and speech acoustics. EURASIP Journal on Applied Signal Processing: Special issue on Joint Audio-Visual Speech Processing, 2002(11), 1174-1188.
51. Jiang, J., & Bernstein, L. E. (2011). Psychophysics of the McGurk and other audiovisual speech integration effects. Journal of Experimental Psychology: Human Performance and Perception, 37(4), 1193-1209.
52. Jiang, X., Bradley, E. D., Rini, R. A., Zeffiro, T., VanMeter, J., & Riesenhuber, M. (2007). Categorization training results in shape- and category-selective human neural plasticity. Neuron, 53(6), 891-903.
53. Kaas, J. H., & Hackett, T. A. (2000). Subdivisions of auditory cortex and processing streams in primates. Proceedings of the National Academy of SciencesU S A, 97(22), 11793-11799.
54. Karns, C. M., Dow, M. W., & Neville, H. J. (2012). Altered cross-modal processing in the primary auditory cortex of congenitally deaf adults: A visual-somatosensory fMRI study with a double-flass illusion. The Journal of Neuroscience, 32(28), 9626-9638.
55. Kayser, C., Petkov, C. I., Remedios, R., & Logothetis, N. K. (2012). Multisensory Influences on Auditory Processing: Perspectives from fMRI and Electrophysiology. In M. M. Murray & W. M. T. (Eds.), The Neural Bases of Multisensory Processes. Boca Raton, FL: CRC.
56. King, C. M., & Quigley, S. P. (1985). Reading and Deafness. San Diego, CA: College-Hill Press.
57. Knudsen, E. I. (2004). Sensitive periods in the development of the brain and behavior. Journal of Cognitive Neuroscience, 16(8), 1412-1425.
58. Kral, A., & Eggermont, J. J. (2007). What's to lose and what's to learn: Development under auditory deprivation, cochlear implants and limits of cortical plasticity. Brain Research Reviews, 56(1), 259-269.
59. Kral, A., & Sharma, A. (2012). Developmental neuroplasticity after cochlear implantation. Trends in Neuroscience, 35(2), 111-122.
60. Kyle, F. E., Campbell, R., Mohammed, T., Coleman, M., & Macsweeney, M. (2013). Speechreading development in deaf and hearing children: introducing the test of child speechreading. Journal of Speech and Hearing Research, 56(2), 416-426.
61. Lamoré, P. J., Huiskamp, T. M., van Son, N. J., Bosman, A. J., & Smoorenburg, G. F. (1998). Auditory, visual and audiovisual perception of segmental speech features by severely hearing-impaired children. Audiology, 37(6), 396-419.
62. Logothetis, N. K., & Sheinberg, D. L. (1996). Visual object recognition. Annual Review of Neuroscience, 19, 577-621.
63. Manrique, M., Cervera-Paz, F. J., Huarte, A., Perez, N., Molina, M., & Garcia-Tapia, R. (1999). Cerebral auditory plasticity and cochlear implants. International Journal of Pediatric Otorhinolaryngology, 49(Suppl.1), S193-S197.
64. Massaro, D. W., Cohen, M. M., Tabain, M., & Beskow, J. (2012). Animated speech: Research progress and applications. In R. B. Clark, J., P. Perrier & E. Vatikiotis-Bateson (Eds.), Audiovisual Speech Processing (pp. 246-272). Cambridge: Cambridge University.
65. McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264(5588), 746- 748.
66. Merabet, L. B., Rizzo, J. R., Amedi, A., Somers, D. C., & Pascual-Leone, A. (2005). What blindness can tell us about seeing again: merging neuroplasticity and neuroprostheses. Nature Reviews: Neuroscience, 6, 71-77.
67. Miller, L. M., & d'Esposito, M. (2005). Perceptual fusion and stimulus coincidence in the crossmodal integration of speech. The Journal of Neuroscience, 25(25), 5884-5893.
68. Mohammed, T., Campbell, R., Macsweeney, M., Barry, F., & Coleman, M. (2006). Speechreading and its association with reading among deaf, hearing and dyslexic individuals. Clinical Linguistics and Phonetics, 20(7-8), 621-630 .
69. Moody-Antonio, S., Takayanagi, S., Masuda, A., Auer, Jr., E. T., Fisher, L., & Bernstein, L. E. (2005). Improved speech perception in adult prelingually deafened cochlear implant recipients. Otology and Neurotology, 26, 649-654
70. Nath, A. R., & Beauchamp, M. S. (2011). Dynamic changes in superior temporal sulcus connectivity during perception of noisy audiovisual speech. The Journal of Neuroscience, 31, 1704-1714.
71. Nath, A. R., & Beauchamp, M. S. (2012). A neural basis for interindividual differences in the McGurk effect, a multisensory speech illusion. Neuroimage, 59(1), 781-787.
72. Nilsson, M., Soli, S. D., & Sullivan, J. A. (1994). Development of the hearing in noise test for the measurement of speech reception thresholds in quiet and in noise. Journal of the Acoustical Society of America, 95(1085-1099).
73. Niparko, J. K., Tobey, E. A., Thal, D. J., Eisenberg, L. S., Wang, N. Y., Quittner, A. L., & Fink, N.E. (2010). Spoken language development in children following cochlear implantation. The Journal of the American Medical Association, 303(15), 1498-1506.
74. Nober, E. H. (1967). Vibrotactile sensitivity of deaf children to high intensity sound. Laryngoscope, 78, 2128-2146.
75. Obleser, J., Zimmermann, J., Van Meter, J., & Rauschecker, J. P. (2007). Multiple stages of auditory speech perception reflected in event-related FMRI. Cerebral Cortex, 17(10), 2251-2257.
76. Osberger, M. J., Fisher, L., Zimmerman-Phillips, S., Geier, L., & Barker, M. J. (1998). Speech recognition performance of older children with cochlear implants. American Journal of Otology, 19, 152-157.
77. Ponton, C. W., Don, M., Eggermont, J. J., Waring, M. D., & Masuda, A. (1996). Maturation of human cortical auditory function: differences between normal-hearing children and children with cochlear implants. Ear & Hearing, 17(5), 430-437.
78. Ponton, C. W., Moore, J. K., & Eggermont, J. J. (1999). Prolonged deafness limits auditory system developmental plasticity: evidence from an evoked potential study in children with cochlear implants. Scandanavian Journal of Audiology, 28(Suppl. 51), 13-22.
79. Ross, L. A., Saint-Amour, D., Leavitt, V. N., Javitt, D. C., & Foxe, J. J. (2007). Do you see what i am saying? Exploring visual enhancement of speech comprehension in noisy environments. Cerebral Cortex, 17, 1147-1153.
80. Rouger, J., Fraysse, B., Deguine, O., & Barone, P. (2008). McGurk effects in cochlear-implanted deaf subjects. Brain Research, 1188, 87-99.
81. Rouger, J., Lagleyre, S., Fraysse, B., Deneve, S., Deguine, O., & Barone, P. (2007). Evidence that cochlear-implanted deaf patients are better multisensory integrators. Proceedings of the National Academy of ScienciesU S A, 104(17), 7295-7300.
82. Samuel, A. G., & Kraljic, T. (2009). Perceptual learning for speech. Attention, Perception & Psychophysics, 71(6), 1207-1218.
83. Samuel, A. G., & Lieblich, J. (2014). Visual speech acts differently than lexical context in supporting speech perception. Journal of Experimental Psychology: Human Perception and Performance.
84. Schramm, D., Fitzpatrick, E., & Seguin, C. (2002). Cochlear implantation for adolescents and adults with prelinguistic deafness. Otology & Neurotology, 23, 698-703.
85. Schroeder, C. E., Lakatos, P., Kajikawa, Y., Partan, S., & Puce, A. (2008). Neuronal oscillations and visual amplification of speech. Trends in Cognitive Science, 12(3), 106-113.
86. Shannon, R. V., Zeng, F. G., Kamath, V., Wygonski, J., & Ekelid, M. (1995). Speech recognition with primarily temporal cues. Science, 270(5234), 303-304.
87. Sharma, A., Dorman, M. F., & Spahr, A. J. (2002). A sensitive period for the development of the central auditory system in children with cochlear implants: Implications for age of implantation. Ear & Hearing, 23(6), 532-539.
88. Snik, A. F. M., Machdoum, M. J. A., Vermeulen, A. M., Brokx, J. P. L., & Van Den Broek, P. (1997). The relation between age at the time of cochlear implantation and long-term speech perception abilities in congenitally deaf subjects. International Journal of Pediatric Otorhinolaryngology, 41, 121-131
89. Song, J. J., Lee, H. J., Kang, H., Lee, D. S., Chang, S. O., & Oh, S. H. (2014). Effects of congruent and incongruent visual cues on speech perception and brain activity in cochlear implant users. Brain Structure and Function.
90. Stevens, K. N. (1998). Acoustic Phonetics. Cambridge, MA: MIT Press.
91. Sumby, W. H., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26(2), 212-215.
92. Summerfield, Q. (1987). Some preliminaries to a comprehensive account of audio-visual speech perception. In B. Dodd & R. Campbell (Eds.), Hearing by Eye: The Psychology of Lip- Reading (pp. 3-52). London: Lawrence Erlbaum Associates, Inc.
93. Teoh, S. W., Pisoni, D. B., & Miyamoto, R. T. (2004). Cochlear implantation in adults with prelingual deafness. Part I. Clinical results. Laryngoscope, 114(9), 1536-1540.
94. Tjan, B. S., Chao, E., & Bernstein, L. E. (2014). A visual or tactile signal makes auditory speech detection more efficient by reducing uncertainty. European Journal of Neuroscience, 39, 1323-1331.
95. Trybus, R. J., & Karchmer, M. A. (1977). School achievement scores of hearing impaired children: National data on achievement status and growth patterns. American Annals of the Deaf, 122(2), 62-69.
96. Ungerleider, L. G., & Haxby, J. V. (1994). 'What' and 'where' in the human brain. Current Opinion in Neurobiology, 4(2), 157-165.
97. Waltzman, S., & Cohen, N. L. (1999). Implantation of patients with prelingual long-term deafness. Annals of Otology, Rhinology & Laryngology, Suppl 177, 84-87.
98. Waltzman, S., Roland, J. T., Jr., & Cohen, N. L. (2002). Delayed implantation in congenitally deaf children and adults. Otology & Neurotology, 23, 333-340.
99. Waltzman, S. B., Cohen, N. L., & Shapiro, W. H. (1992). Use of a multichannel cochlear implant in the congenitally and prelingually deaf population. Laryngoscope, 102(4), 395-399.
100. Wayne, R. V., & Johnsrude, I. S. (2012). The role of visual speech information in supporting perceptual learning of degraded speech. Journal of Experimental Psychology: Applied, 18(4), 419-435.
101. Williams, P., & Simons, D. (2000). Detecting changes in novel, complex three-dimensional objects. Visual Cognition, 7, 297-322.
102. Wilson, B. S., Dorman, M. F., Woldorff, M. G., & Tucci, D. L. (2011). Cochlear implants matching the prosthesis to the brain and facilitating desired plastic changes in brain function. Progress in Brain Research, 194, 117-129.
103. Yehia, H., Rubin, P., & Vatikiotis-Bateson, E. (1998). Quantitative association of vocal-tract and facial behavior. Speech Communication, 26(1-2), 23-43.
104. Yoon, Y. S., Li, Y., Kang, H. Y., & Fu, Q. J. (2011). The relationship between binaural benefit and difference in unilateral speech recognition performance for bilateral cochlear implant users. International Journal of Audiology, 50(8), 554-565.
105. Zeki, S. (2005). The Ferrier Lecture 1995: Behind the Seen: The functional specialization of the brain in space and time. Philosophical Transactions: Biological Sciences, 360(1458), 1145-1183.
106. Zeng, F.-G., Popper, A. N., & Fay, R. R. (2004). Cochlear Implants: Auditory Prostheses and Electrical Hearing. NY: Springer.