The role of visual speech information in supporting perceptual learning of degraded speech

Journal of Experimental Psychology: Applied

Volumn 18, Issue 4, 2012, Pages 419-435

  Wayne, Rachel V ^a   Johnsrude, Ingrid S ^a,b  

^a QUEEN S UNIVERSITY   (Canada)

^b LINKÖPING UNIVERSITY   (Sweden)

Author keywords

Audiovisual speech; Cochlear implant rehabilitation; Cross modal integration; Listening effort; Speech perception

Indexed keywords

ADAPTATION; ADOLESCENT; ADULT; ARTICLE; COCHLEA PROSTHESIS; FEMALE; HUMAN; IMPLANTATION; LEARNING; MALE; PHYSIOLOGY; SENSORY FEEDBACK; SPEECH; SPEECH PERCEPTION;

ADAPTATION, PHYSIOLOGICAL; ADOLESCENT; ADULT; COCHLEAR IMPLANTATION; COCHLEAR IMPLANTS; FEEDBACK, SENSORY; FEMALE; HUMANS; LEARNING; MALE; SPEECH; SPEECH PERCEPTION;

EID: 84874677289     PISSN: 1076898X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0031042     Document Type: Article

References (91)

1. Ahissar, M., & Hochstein, S. (1993). Attentional control of early perceptual learning. Proceedings of the National Academy of Sciences of the United States of America, 90, 5718-5722. doi:10.1073/pnas.90.12.5718
2. Altmann, G., & Young, D. (1993). Factors affecting adaptation to timecompressed speech. In Proceedings of Third European Conference on Speech Communication and Technology, EUROSPEECH 1993, Berlin, Germany, September 22-25, 1993.
3. Anderson Gosselin, P., & Gagne, J. P. (2011). Older adults expend more listening effort than young adults recognizing speech in noise. Journal of Speech, Language, and Hearing Research, 54, 944-958. doi:10.1044/ 1092-4388(2010/10-0069)
4. Arnold, P., & Hill, F. (2001). Bisensory augmentation: A speechreading advantage when speech is clearly audible and intact. British Journal of Psychology, 92, 339-355. doi:10.1348/000712601162220
5. Bernstein, L. E., Auer, E. T., Jr., & Moore, J. K. (2004). Audiovisual speech binding: Convergence or association? In G. A. Calvert, C. Spence, & B. E. Stein (Eds.), The handbook of multisensory processes (pp. 203-223). Cambridge, MA: MIT Press.
6. Bertelson, P., Vroomen, J., & De Gelder, B. (2003). Visual recalibration of auditory speech identification: A McGurk aftereffect. Psychological Science, 14, 592-597. doi:10.1046/j.0956-7976.2003.psci_1470.x
7. Boothroyd, A. (2007). Adult aural rehabilitation: What is it and does it work? Trends in Amplification, 11, 63-71. doi:10.1177/1084713807301073
8. Bradlow, A. R., & Bent, T. (2008). Perceptual adaptation to non-native speech. Cognition, 106, 707-729. doi:10.1016/j.cognition.2007.04.005 Neuroreport, 14, 2213-2218.
9. Callan, D. E., Jones, J. A., Munhall, K., Callan, A. M., Kroos, C., & Vatikiotis-Bateson, E. (2003). Neural processes underlying perceptual enhancement by visual speech gestures.
10. Clark, G. M. (2003). Rehabilitation and habilitation. In R. T. Beyer (Ed.), Cochlear implants: Fundamentals and applications (pp. 654-706). New York, NY: Springer-Verlag.
11. Davis, M. H., & Johnsrude, I. S. (2007). Hearing speech sounds: Top-down influences on the interface between audition and speech perception. Hearing Research, 229, 132-147.
12. Davis, M. H., Johnsrude, I. S., Hervais-Adelman, A., Taylor, K., & McGettigan, C. (2005). Lexical information drives perceptual learning of distorted speech: Evidence from the comprehension of noise-vocoded sentences. Journal of Experimental Psychology: General, 134, 222-241. doi:10.1037/0096-3445.134.2.222
13. Dodd, B., Plant, G., & Gregory, M. (1989). Teaching lip-reading: The efficacy of lessons on video. British Journal of Audiology, 23, 229-238. doi:10.3109/03005368909076504
14. Dupoux, E., & Green, K. (1997). Perceptual adjustment to highly compressed speech: Effects of talker and rate changes. Journal of Experimental Psychology: Human Perception and Performance, 23, 914-927. doi:10.1037/0096-1523.23.3.914
15. Edwards, B. (2007). The future of hearing aid technology. Trends in Amplification, 11, 31-45. doi:10.1177/1084713806298004
16. Erber, N. P. (1975). Auditory-visual perception of speech. The Journal of Speech and Hearing Disorders, 40, 481-492.
17. Fadiga, L., & Craighero, L. (2006). Hand actions and speech representation in Broca's area. Cortex: A Journal Devoted to the Study of the Nervous System and Behavior, 42, 486-490. doi:10.1016/S0010-9452(08)70383-6
18. Faulkner, A., Rosen, S., & Smith, C. (2000). Effects of the salience of pitch and periodicity information on the intelligibility of four-channel vocoded speech: Implications for cochlear implants. Journal of the Acoustical Society of America, 108, 1877-1887. doi:10.1121/1.1310667
19. Fernandes, T., Kolinsky, R., & Ventura, P. (2010). The impact of attention load on the use of statistical information and coarticulation as speech segmentation cues. Attention, Perception, & Psychophysics, 72, 15221532. doi:10.3758/APP.72.6.1522
20. Feuerstein, J. F. (1992). Monaural versus binaural hearing: Ease of listening, word recognition, and attentional effort. Ear and Hearing, 13, 80-86. doi:10.1097/00003446-199204000-00003
21. Fishman, K. E., Shannon, R. V., & Slattery, W. H. (1997). Speech recognition as a function of the number of electrodes used in the SPEAK cochlear implant speech processor. Journal of Speech, Language, and Hearing Research, 40, 1201-1215.
22. Francis, H. W., Chee, N., Yeagle, J., Cheng, A., & Niparko, J. K. (2002). Impact of cochlear implants on the functional health status of older adults. Laryngoscope, 112, 1482-1488. doi:10.1097/00005537-200208000-00028
23. Fu, Q. J., & Galvin, J. J., 3rd. (2003). The effects of short-term training for spectrally mismatched noise-band speech. Journal of the Acoustical Society of America, 113, 1065-1072. doi:10.1121/1.1537708
24. Fu, Q. J., & Galvin, J. J., 3rd. (2007). Perceptual learning and auditory training in cochlear implant recipients. Trends in Amplification, 11, 193-205. doi:10.1177/1084713807301379
25. Grant, K. W., & Braida, L. D. (1991). Evaluating the articulation index for auditory-visual input. Journal of the Acoustical Society of America, 89, 2952-2960. doi:10.1121/1.400733
26. Grant, K. W., & Seitz, P. F. (1998). Measures of auditory-visual integration in nonsense syllables and sentences. Journal of the Acoustical Society of America, 104, 2438-2450. doi:10.1121/1.423751
27. Grant, K. W., & Seitz, P. F. (2000). The use of visible speech cues for improving auditory detection of spoken sentences. Journal of the Acoustical Society of America, 108, 1197-1208. doi:10.1121/1.1288668
28. Greenspan, S. L., Nusbaum, H. C., & Pisoni, D. B. (1988). Perceptual learning of synthetic speech produced by rule. Journal of Experimental Psychology: Learning, Memory, and Cognition, 14, 421.
29. Greenwood, D. D. (1990). A cochlear frequency-position function for several species-29 years later. Journal of the Acoustical Society of America, 87, 2592-2605. doi:10.1121/1.399052
30. Hällgren, M., Larsby, B., Lyxell, B., & Arlinger, S. (2005). Speech understanding in quiet and noise, with and without hearing aids. International Journal of Audiology, 44, 574-583. doi:10.1080/ 14992020500190011
31. Heider, F., & Heider, G. (1940). An experimental investigation of lipreading. Psychological Monographs, 52, 124-153. doi:10.1037/h0093465
32. Hervais-Adelman, A., Davis, M. H., Johnsrude, I. S., & Carlyon, R. P. (2008). Perceptual learning of noise vocoded words: Effects of feedback and lexicality. Journal of Experimental Psychology: Human Perception and Performance, 34, 460-474. doi:10.1037/0096-1523.34.2.460
33. Hervais-Adelman, A., Davis, M. H., Johnsrude, I. S., Taylor, K. J., & Carlyon, R. P. (2011). Generalization of perceptual learning of vocoded speech. Journal of Experimental Psychology: Human Perception and Performance, 37, 283-295. doi:10.1037/a0020772
34. Hicks, C. B., & Tharpe, A. M. (2002). Listening effort and fatigue in school-age children with and without hearing loss. Journal of Speech, Language, and Hearing Research, 45, 573-584. doi:10.1044/1092-4388(2002/046)
35. Horn, K. L., McMahon, N. B., McMahon, D. C., Lewis, J. S., Barker, M., & Gherini, S. (1991). Functional use of the Nucleus 22-channel cochlear implant in the elderly. Laryngoscope, 101, 284-288. doi:10.1288/ 00005537-199103000-00011
36. Howard, C. S., Munro, K. J., & Plack, C. J. (2010). Listening effort at signal-to-noise ratios that are typical of the school classroom. International Journal of Audiology, 49, 928-932. doi:10.3109/14992027.2010 .520036
37. Humes, L. E., Christensen, L., Thomas, T., Bess, F. H., Hedley-Williams, A., & Bentler, R. (1999). A comparison of the aided performance and benefit provided by a linear and a two-channel wide dynamic range compression hearing aid. Journal of Speech, Language, and Hearing Research, 42, 65-79.
38. Huyck, J. J., Smith, R., Hawkins, S., & Johnsrude, I. (2012, February). Does perceptual learning of degraded speech generalize to a novel voice and accent? Paper presented at the 35th Midwinter Meeting of the Association for Research in Otolaryngology, San Diego, CA.
39. Kidd, G. R., Watson, C. S., & Gygi, B. (2007). Individual differences in auditory abilities. Journal of the Acoustical Society of America, 122, 418-435. doi:10.1121/1.2743154
40. Kraljic, T., & Samuel, A. G. (2006). Generalization in perceptual learning for speech. Psychonomic Bulletin & Review, 13, 262-268. doi:10.3758/ BF03193841
41. Larsby, B., Hällgren, M., Lyxell, B., & Arlinger, S. (2005). Cognitive performance and perceived effort in speech processing tasks: Effects of different noise backgrounds in normal-hearing and hearing-impaired subjects. International Journal or Audiology, 44, 131-143. doi:10.1080/ 14992020500057244
42. Liberman, A. M., & Mattingly, I. G. (1985). The motor theory of speech perception revised. Cognition, 21, 1-36.
43. Loebach, J. L., Pisoni, D. B., & Svirsky, M. A. (2010). Effects of semantic context and feedback on perceptual learning of speech processed through an acoustic simulation of a cochlear implant. Journal of Experimental Psychology: Human Perception and Performance, 36, 224-234. doi:10.1037/a0017609
44. Loizou, P. C., Dorman, M., & Tu, Z. (1999). On the number of channels needed to understand speech. Journal of the Acoustical Society of America, 106, 2097-2103. doi:10.1121/1.427954
45. Mackersie, C. L., & Cones, H. (2011). Subjective and psychophysiological indexes of listening effort in a competing-talker task. Journal of the American Academy of Audiology, 22, 113-122. doi:10.3766/jaaa.22.2.6
46. MacLeod, A., & Summerfield, Q. (1987). Quantifying the contribution of vision to speech perception in noise. British Journal of Audiology, 21, 131-141. doi:10.3109/03005368709077786
47. MacLeod, A., & Summerfield, Q. (1990). A procedure for measuring auditory and audiovisual speech-reception thresholds for sentences in noise: Rationale, evaluation, and recommendations for use. British Journal of Audiology, 24, 29-43. doi:10.3109/03005369009077840
48. McConkey-Robbins, A. (2000). Rehabilitation after cochlear implantation. In J. Niparko, K. Kirk, N. Mellon, A. McConkey, D. Tucci, & B. Wilson (Eds.), Cochlear implants: Principles and practices (pp. 323-367). Philadelphia, PA: Lippincott, Williams & Wilkins.
49. McCoy, S. L., Tun, P. A., Cox, L. C., Colangelo, M., Stewart, R. A., & Wingfield, A. (2005). Hearing loss and perceptual effort: Downstream effects on older adults' memory for speech. The Quarterly Journal of Experimental Psychology A: Human Experimental Psychology, 58, 2233. doi:10.1080/02724980443000151
50. Mehler, J., Sebastian, N., Altmann, G., Dupoux, E., Christophe, A., & Pallier, C. (1993). Understanding compressed sentences: The role of rhythm and meaning. Annals of the New York Academy of Sciences, 682, 272-282. doi:10.1111/j.1749-6632.1993.tb22975.x
51. Miller, G. A., Heise, G. A., & Lichten, W. (1951). The intelligibility of speech as a function of the context of the test materials. Journal of Experimental Psychology, 41, 329-335. doi:10.1037/h0062491
52. Munhall, K. G., Jones, J. A., Callan, D. E., Kuratate, T., & VatikiotisBateson, E. (2004). Visual prosody and speech intelligibility: Head movement improves auditory speech perception. Psychological Science, 15, 133-137. doi:10.1111/j.0963-7214.2004.01502010.x
53. Murphy, D. R., Craik, F. I., Li, K. Z., & Schneider, B. A. (2000). Comparing the effects of aging and background noise on short-term memory performance. Psychology and Aging, 15, 323-334. doi: 10.1037/0882-7974.15.2.323
54. Nath, A. R., & Beauchamp, M. S. (2011). Dynamic changes in superior temporal sulcus connectivity during perception of noisy audiovisual speech. Journal of Neuroscience, 31, 1704-1714. doi:10.1523/ JNEUROSCI.4853-10.2011
55. Nath, A. R., & Beauchamp, M. S. (2012). A neural basis for interindividual differences in the McGurk effect, a multisensory speech illusion. NeuroImage, 59, 781-787. doi:10.1016/j.neuroimage.2011.07.024
56. Pallier, C., Sebastian-Galles, N., Dupoux, E., Christophe, A., & Mehler, J. (1998). Perceptual adjustment to time-compressed speech: A crosslinguistic study. Memory & Cognition, 26, 844-851. doi:10.3758/ BF03211403
57. Pasanisi, E., Bacciu, A., Vincenti, V., Guida, M., Barbot, A., Berghenti, M. T., & Bacciu, S. (2003). Speech recognition in elderly cochlear implant recipients. Clinical Otolaryngology and Allied Sciences, 28, 154-157. doi:10.1046/j.1365-2273.2003.00681.x
58. Pichora-Fuller, M. K., Schneider, B. A., & Daneman, M. (1995). How young and old adults listen to and remember speech in noise. Journal of the Acoustical Society of America, 97, 593-608. doi:10.1121/1.412282
59. Pilling, M., & Thomas, S. (2011). Audiovisual cues and perceptual learning of spectrally distorted speech. Language and Speech, 54, 487-497. doi:10.1177/0023830911404958
60. Pollatsek, A., & Well, A. D. (1995). On the use of counterbalanced designs in cognitive research: A suggestion for a better and more powerful analysis. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21, 785-794. doi:10.1037/0278-7393.21.3.785
61. Prince, M. (2004). Does active learning work? A review of the research. Journal of Engineering Education, 93, 223-231.
62. Rabbitt, P. (1990). Mild hearing loss can cause apparent memory failures which increase with age and reduce with IQ. Acta Oto-Laryngologica. Supplementum, 476, 167-175. doi:10.3109/00016489109127274
63. Rakerd, B., Seitz, P. F., & Whearty, M. (1996). Assessing the cognitive demands of speech listening for people with hearing losses. Ear and Hearing, 17, 97-106. doi:10.1097/00003446-199604000-00002
64. Reisberg, D., McLean, J., & Goldfield, A. (Eds.). (1987). Easy to hear but hard to understand: A lip-reading advantage with intact auditory stimuli. Hillsdale, NJ: Lawrence Erlbaum.
65. Remez, R. E. (2005). Perceptual organization of speech. In D. B. Pisoni & R. E. Remez (Eds.), The handbook of speech perception (pp. 28-50). Oxford, UK: Blackwell. doi:10.1002/9780470757024.ch2
66. Remez, R. E., Rubin, P. E., Pisoni, D. B., & Carrell, T. D. (1981). Speech perception without traditional speech cues. Science, 212, 947-949. doi: 10.1126/science.7233191
67. Rodd, J. M., Davis, M. H., & Johnsrude, I. S. (2005). The neural mechanisms of speech comprehension: FMRI studies of semantic ambiguity. Cerebral Cortex, 15, 1261-1269. doi:10.1093/cercor/bhi009
68. Rosen, S., Faulkner, A., & Wilkinson, L. (1999). Adaptation by normal listeners to upward spectral shifts of speech: Implications for cochlear implants. Journal of the Acoustical Society of America, 106, 3629-3636. doi:10.1121/1.428215
69. Rosenblum, L. D., Miller, R. M., & Sanchez, K. (2007). Lip-read me now, hear me better later: Cross-modal transfer of talker-familiarity effects. Psychological Science, 18, 392-396. doi:10.1111/j.1467-9280.2007 .01911.x
70. Rouger, J., Fraysse, B., Deguine, O., & Barone, P. (2008). McGurk effects in cochlear-implanted deaf subjects. Brain Research, 1188, 87-99. doi: 10.1016/j.brainres.2007.10.049
71. Rouger, J., Lagleyre, S., Fraysse, B., Deneve, S., Deguine, O., & Barone, P. (2007). Evidence that cochlear-implanted deaf patients are better multisensory integrators. Proceedings of the National Academy of Sciences of the United States of America, 104, 7295-7300. doi:10.1073/ pnas.0609419104
72. Rudner, M., Lunner, T., Behrens, T., Thoren, E. S., & Ronnberg, J. (2012). Working memory capacity may influence perceived effort during aided speech recognition in noise. Journal of the American Academy of Audiology, 23, 577-589. doi:10.3766/jaaa.23.7.7
73. Sarampalis, A., Kalluri, S., Edwards, B., & Hafter, E. (2009). Objective measures of listening effort: Effects of background noise and noise reduction. Journal of Speech, Language, and Hearing Research, 52, 1230-1240. doi:10.1044/1092-4388(2009/08-0111)
74. Schwartz, J. L. (2010). A reanalysis of McGurk data suggests that audiovisual fusion in speech perception is subject-dependent. Journal of the Acoustical Society of America, 127, 1584-1594. doi:10.1121/1.3293001
75. Seitz, A. R., & Dinse, H. R. (2007). A common framework for perceptual learning. Current Opinion in Neurobiology, 17, 148-153. doi:10.1016/ j.conb.2007.02.004
76. Shannon, R. V., Zeng, F. G., Kamath, V., Wygonski, J., & Ekelid, M. (1995). Speech recognition with primarily temporal cues. Science, 270, 303-304. doi:10.1126/science.270.5234.303
77. Skipper, J. I., Nusbaum, H. C., & Small, S. L. (2005). Listening to talking faces: Motor cortical activation during speech perception. Neuroimage, 25, 76-89.
78. Skipper, J. I., Van Wassenhove, V., Nusbaum, H. C., & Small, S. L. (2007). Hearing lips and seeing voices: How cortical areas supporting speech production mediate audiovisual speech perception. Cerebral Cortex, 17, 2387-2399.
79. Sommers, M. S., Tye-Murray, N., & Spehar, B. (2005). Auditory-visual speech perception and auditory-visual enhancement in normal-hearing younger and older adults. Ear and Hearing, 26, 263-275. doi:10.1097/ 00003446-200506000-00003
80. Strauss, D. J., Corona-Strauss, F. I., Trenado, C., Bernarding, C., Reith, W., Latzel, M., & Froehlich, M. (2010). Electrophysiological correlates of listening effort: Neurodynamical modeling and measurement. Cognitive Neurodynamics, 4, 119-131. doi:10.1007/s11571-010-9111-39111
81. Sumby, W. H., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212-215. doi:10.1121/1.1907309
82. Summerfield, Q. (1987). Some preliminaries to a comprehensive account of audiovisual speech perception. In B. Dodd & R. Campbell (Eds.), Hearing by eye: The psychology of lipreading (pp. 3-51). Hillsdale, NJ: Lawrence Erlbaum.
83. Toro, J. M., Sinnett, S., & Soto-Faraco, S. (2005). Speech segmentation by statistical learning depends on attention. Cognition, 97, B25-B34. doi: 10.1016/j.cognition.2005.01.006df
84. Tun, P. A., McCoy, S., & Wingfield, A. (2009). Aging, hearing acuity, and the attentional costs of effortful listening. Psychology and Aging, 24, 761-766. doi:10.1037/a0014802
85. Tye-Murray, N., Sommers, M. S., & Spehar, B. (2007). Audiovisual integration and lipreading abilities of older adults with normal and impaired hearing. Ear and Hearing, 28, 656-668. doi:10.1097/AUD .0b013e31812f7185
86. Walden, B. E., Busacco, D. A., & Montgomery, A. A. (1993). Benefit from visual cues in auditory-visual speech recognition by middle-aged and elderly persons. Journal of Speech & Hearing Research, 36, 431-436.
87. Wild, C. J., Yusuf, A., Wilson, D., Peelle, J. E., Davis, M. H., & Johnsrude, I. (2012). Effortful listening: The processing of degraded speech depends critically on attention. Journal of Neuroscience, 32, 1401014021. doi:10.1523/JNEUROSCI.1528-12.2012
88. Wingfield, A., McCoy, S. L., Peelle, J. E., Tun, P. A., & Cox, L. C. (2006). Effects of adult aging and hearing loss on comprehension of rapid speech varying in syntactic complexity. Journal of the American Academy of Audiology, 17, 487-497. doi:10.3766/jaaa.17.7.4
89. Zekveld, A. A., George, E. L., Kramer, S. E., Goverts, S. T., & Houtgast, T. (2007). The development of the text reception threshold test: A visual analogue of the speech reception threshold test. Journal of Speech, Language, and Hearing Research, 50, 576-584. doi:10.1044/1092-4388(2007/040)
90. Zekveld, A. A., Kramer, S. E., & Festen, J. M. (2010). Pupil response as an indication of effortful listening: The influence of sentence intelligibility. Ear and Hearing, 31, 480-490. doi:10.1097/AUD.0b013e3181d4f251
91. Zekveld, A. A., Kramer, S. E., & Festen, J. M. (2011). Cognitive load during speech perception in noise: The influence of age, hearing loss, and cognition on the pupil response. Ear and Hearing, 32, 498-510. doi:10.1097/AUD.0b013e31820512bb