Audio-visual speech perception: A developmental ERP investigation

Developmental Science

Volumn 17, Issue 1, 2014, Pages 110-124

  Knowland, Victoria C P ^a,b   Mercure, Evelyne ^c   Karmiloff Smith, Annette ^b   Dick, Fred ^b   Thomas, Michael S C ^b  

^a CITY UNIVERSITY   (United Kingdom)

^b UNIVERSITY OF LONDON   (United Kingdom)

^c UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ANALYSIS OF VARIANCE; ASSOCIATION; AUDITORY STIMULATION; CHILD; EVOKED RESPONSE; FEMALE; HEARING; HUMAN; MALE; PHYSIOLOGY; SPEECH PERCEPTION; VISION; VOICE; YOUNG ADULT;

ACOUSTIC STIMULATION; ADULT; ANALYSIS OF VARIANCE; AUDITORY PERCEPTION; CHILD; CUES; EVOKED POTENTIALS; FEMALE; HUMANS; MALE; SPEECH PERCEPTION; VISUAL PERCEPTION; VOICE; YOUNG ADULT;

EID: 84890553912     PISSN: 1363755X     EISSN: 14677687     Source Type: Journal    
DOI: 10.1111/desc.12098     Document Type: Article

References (75)

1. Alais, D., & Burr, D. (2004). The ventriloquist effect results from near optimal bimodal integration. Current Biology, 14 (3), 257-262.
2. Bernstein, L.E., Auer, E.T., Wagner, M., & Ponton, C.W. (2007). Spatiotemporal dynamics of audio-visual speech processing. NeuroImage, 39, 423-435.
3. Besle, J., Bertrand, O., & Giard, M.H. (2009). Electrophysiological (EEG, sEEG, MEG) evidence for multiple audio-visual interactions in the human auditory cortex. Hearing Research, 258, 143-151.
4. Besle, J., Fischer, C., Bidet-Caulet, A., Lecaignard, F., Bertrand, O., & Giard, M.-H. (2008). Visual activation and audio-visual interactions in the auditory cortex during speech perception: intercranial recording in humans. Journal of Neuroscience, 28 (52), 14301-14310.
5. Besle, J., Fort, A., Delpuech, C., & Giard, M.-H. (2004). Bimodal speech: early suppressive visual effects in human auditory cortex. European Journal of Neuroscience, 20, 2225-2234.
6. Bishop, D.V.M., Hardiman, M., Uwer, R., & von Suchodeltz, W. (2007). Maturation of the long-latency auditory ERP: step function changes at start and end of adolescence. Developmental Science, 10 (5), 565-575.
7. Bristow, D., Dehaene-Lambertz, G., Mattout, J., Soares, C., Gilga, T., Baillet, S., & Mangin, F. (2008). Hearing faces: how the infant brain matches the face it sees with the speech it hears. Journal of Cognitive Neuroscience, 21 (5), 905-921.
8. Burnham, D., & Dodd, B. (2004). Auditory-visual speech integration by pre-linguistic infants: perception of an emergent consonant in the McGurk effect. Developmental Psychobiology, 44 (4), 204-220.
9. Bushara, K.O., Hanawaka, T., Immisch, I., Toma, K., Kansaku, K., & Hallett, M. (2003). Neural correlates of cross-modal binding. Nature Neuroscience, 6 (2), 190-195.
10. Callan, D.E., Jones, J.A., Munhall, K., Kroos, C., Callan, A.M., & Vatikiotis-Bateson, E. (2004). Multisensory integration sites identified by perception of spatial wavelet filtered visual speech gesture information. Journal of Cognitive Neuroscience, 16, 805-816.
11. Callaway, E., & Halliday, R. (1982). The effect of attentional effort on visual evoked potential N1 latency. Psychiatry Research, 7, 299-308.
12. Calvert, G. (2001). Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cerebral Cortex, 11 (12), 1110-1123.
13. Calvert, G.A., Bullmore, E., Brammer, M.J., Campbell, R., Woodruff, P., McGuire, P., Williams, S., Iversen, S.D., & David, A.S. (1997). Activation of auditory cortex during silent speechreading. Science, 276, 593-596.
14. Calvert, G.A., Campbell, R., & Brammer, M. J. (2000). Evidence from functional magnetic resonance imaging of crossmodal binding in human heteromodal cortex. Current Biology, 10, 649-657.
15. Campbell, R. (2008). The processing of audio-visual speech: empirical and neural bases. Philosophical Transactions of the Royal Society, B, 363, 1001-1010. doi: 10.1098/rstb.2007.2155.
16. Capek, C.M., Bavelier, D., Corina, D., Newman, A.J., Jezzard, P., & Neville, H.J. (2004). The cortical organization of audio-visual sentence comprehension: an fMRI study at 4 Tesla. Cognitive Brain Research, 20, 111-119.
17. Chandrasekaran, C., Trubanova, A., Stillittano, S., Caplier, A., & Ghazanfar, A. A. (2009). The natural statistics of audiovisual speech. PLOS Computational Biology, 5 (7), e1000436.
18. Desjardins, R., & Werker, J.F. (2004). Is the integration of heard and seen speech mandatory for infants? Developmental Psychobiology, 45 (4), 187-203.
19. Dick, A.S., Solodkin, A., & Small, S. (2010). Neural development of networks for audiovisual speech comprehension. Brain and Language, 114 (2), 101-114.
20. Fort, M., Spinelli, E., Savariaux, C., & Kandel, S. (2012). Audiovisual vowel monitoring and the word superiority effect in children. International Journal of Behavioral Development, 36 (6), 457-467.
21. Giard, M.-H., Perrin, F., Echallier, J.F., Thevenet, M., Fromenet, J.C., & Pernier, J. (1994). Dissociation of temporal and frontal components in the human auditory N1 wave: a scalp current density and dipole model analysis. Electroencephalography & Clinical Neurophysiology, 92, 238-252.
22. Gori, M., Del Viva, M., Sandini, G., & Burr, D.C. (2008). Young children do not integrate visual and haptic form information. Current Biology, 18 (9), 694-698.
23. Gotgay, N., Giedd, J., Lusk, L., Hayashi, K.M., Greenstein, D., Vaituzis, A.C., Nugent, T.F., III, Herman, D.H., Clasen, L.S., Toga, A.W., Rapopport, J.L., & Thompson, P.M. (2004). Dynamic mapping of human cortical development during childhood through early adulthood. Proceedings of the National Academy of Sciences, USA, 101 (21), 1874-1879.
24. Grant, K.W., & Greenberg, S. (2001). Speech intelligibility derived from asynchronous processing of auditory-visual information. Proceedings of the Workshop on Audio-Visual Speech Processing (AVSP-2001).
25. Grant, K.W., & Seitz, P.F. (2000). The use of visible speech cues for improving auditory detection of spoken sentences. Journal of the Acoustical Society of America, 108 (3), 1197-1208.
26. Green, K.P., Kuhl, P.K., Meltzoff, A.N., & Stevens, E.B. (1991). Integrating speech information across talkers, gender, and sensory modality: female faces and male voices in the McGurk effect. Perception & Psychophysics, 50, 524-536.
27. Hall, D.A., Fussell, C., & Summerfield, A.Q. (2005). Reading fluent speech from talking faces: typical brain networks and individual differences. Journal of Cognitive Neuroscience, 17, 939-953.
28. Hoonhorst, I., Serniclaes, W., Collet, G., Colin, C., Markessis, E., Radeau, M., & Deltenrea, P. (2009). N1b and Na subcomponents of the N100 long latency auditory evoked-potential: Neurophysiological correlates of voicing in French-speaking subjects. Clinical Neurophysiology, 120 (5), 897-903.
29. Hocking, J., & Price, C.J. (2008). The role of the posterior temporal sulcus in audiovisual processing. Cerebral Cortex, 18 (10), 2439-2449.
30. Hockley, N.S., & Polka, L. (1994). A developmental study of audiovisual speech perception using the McGurk paradigm. Journal of the Acoustical Society of America, 96 (5), 3309-3309.
31. Jasper, H.H. (1958). The ten-twenty electrode system of the International Federation. Electroencephalography and Clinical Neurophysiology, 10, 371-375.
32. Jerger, S., Damian, M.F., Spence, M.J., Tye-Murray, N., & Abdi, H. (2009). Developmental shifts in children's sensitivity to visual speech: a new multisensory picture word task. Journal of Experimental Child Psychology, 102, 40-59.
33. Kawabe, T., Shirai, N., Wada, Y., Miura, K., Kanazawa, S., & Yamaguci, M.K. (2010). The audiovisual tau effect in infancy. PLoS ONE, 5 (3): e9503. doi: 10.1371/journal.pone.0009503
34. Kim, J., & Davis, C. (2004). Investigating the audio-visual speech detection advantage. Speech Communication, 44, 19-30.
35. Klucharev, K., Mottonen, R., & Sams, M. (2003). Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audio-visual speech perception. Cognitive Brain Research, 18, 65-75.
36. Kuhl, P., & Meltzoff, A. (1982). The bimodal perception of speech in infancy. Science, 218, 1138-1141.
37. Kuperman, V., Stadthagen-Gonzalez, H., & Brysbaert, M. (2012). Age of acquisition ratings for 30, 000 English words. Behavioural Research Methods, 44 (4), 978-990.
38. Kushnerenko, E., Teinonen, T., Volien, A., & Csibra, G. (2008). Electrophysiological evidence of illusory audiovisual speech percept in human infants. Proceedings of the National Academy of Sciences, USA, 105 (32), 11442-11445.
39. Lange, K. (2009). Brain correlates of early auditory processing are attenuated by expectations for time and pitch. Brain and Cognition, 69 (1), 127-137.
40. Leech, R., Holt, L.L., Devlin, J.T., & Dick, F. (2009). Expertise with artificial non-speech sounds recruits speech-sensitive cortical regions. Journal of Neuroscience, 29 (16), 5234-5239.
41. Lenroot, R.K., & Giedd, J.N. (2006). Brain development in children and adolescents: insights from anatomical magnetic resonance imaging. Neuroscience and Biobehavioral Reviews, 30, 718-729.
42. Lewkowicz, D.J., & Hansen-Tift, A.M. (2012). Infants deploy selective attention to the mouth of a talking face when learning speech. Proceedings of the National Academy of Sciences, USA, 109 (5), 1431-1436.
43. Liebenthal, E., Desai, R., Ellinson, M.M., Ramachandran, B., Desai, A., & Binder, J.R. (2010). Specialisation along the left superior temporal sulcus for auditory categorisation. Cerebral Cortex, 20 (12), 2958-2970.
44. Lippe, S., Kovacevic, N., & McIntosh, A.R. (2009). Differential maturation of brain signal complexity in the human auditory and visual system. Frontiers in Human Neuroscience, 3, 48.
45. McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746-748.
46. Martin, L., Barajas, J.J., Fernandez, R., & Torres, E. (1988). Auditory event-related potentials in well-characterized groups of children. Electroencephalography and Clinical Neurophysiology: Evoked Potentials, 71, 375-381.
47. Massaro, D., Thompson, L., Barron, B., & Laren, E. (1986). Developmental changes in visual and auditory contributions to speech perception. Journal of Experimental Child Psychology, 41, 93-113.
48. Moore, J.K. (2002). Maturation of human auditory cortex: implications for speech perception. Annals of Otology, Rhinology and Laryngology, 111, 7-10.
49. Musacchia, G., Sams, M., Nicol, T., & Kraus, N. (2006). Seeing speech affects acoustic information processing in the human brainstem. Experimental Brain Research, 168 (1-2), 1-10.
50. Nath, A.R., & Beauchamp, M.S. (2011). Dynamic changes in superior temporal sulcus connectivity during perception of noisy audiovisual speech. Journal of Neuroscience, 31 (5), 1704-1714.
51. Nath, A.R., Fava, E.E., & Beauchamp, M.S. (2011). Neural correlates of interindividual differences in children's audiovisual speech perception. Journal of Neuroscience, 31 (39), 13963-13971.
52. Pang, E.W., & Taylor, M.J. (2000). Tracking the development of the N1 from age 3 to adulthood: an examination of speech and non-speech stimuli. Clinical Neurophysiology, 111 (3), 388-397.
53. Patterson, M., & Werker, J. (1999). Matching phonetic information in lips and voice is robust in 4.5-month-old infants. Infant Behavior and Development, 22, 237-247.
54. Picton, T.W., Hillyard, S.A., Krausz, H.I., & Galambos, R. (1974). Human auditory evoked potentials. Electroencephalography and Clinical Neurophysiology, 36, 179-190.
55. Pilling, M. (2009). Auditory event-related potentials (ERPs) in audio-visual speech perception. Journal of Speech, Language and Hearing Research, 52, 1073-1081.
56. Ponton, C., Eggermont, J.J., Khosla, D., Kwong, B., & Don, M. (2002). Maturation of human central auditory system activity: separating auditory evoked potentials by dipole source modeling. Clinical Neurophysiology, 113, 407-420.
57. Reale, R.A., Calvert, G.A., Thesen, T., Jenison, R.L., Kawasaki, H., Oys, H., Howard, M.A., & Brugg, J.F. (2007). Auditory-visual processing represented in the human superior temporal gyrus. Neuroscience, 145 (1), 162-184.
58. Ritter, W., Simson, R., & Vaughn, H. (1988). Effects of the amount of stimulus information processed on negative event-related potentials. Electroencephalography and Clinical Neuropsychiology, 28, 244-258.
59. Rosenblum, L.D., Schmuckler, M.A., & Johnson, J.A. (1997). The McGurk effect in infants. Perception & Psychophysics, 59 (3), 347-357.
60. Ross, L.A., Molholm, S., Blanco, D., Gomez-Ramirez, M., Saint-Amour, D., & Foxe, J. (2011). The development of multisensory speech perception continues into the late childhood years. European Journal of Neuroscience, 33 (12), 2329-2337.
61. Skipper, J.I., Nusbaum, H.C., & Small, S.L. (2005). Listening to talking faces: motor cortical activation during speech perception. NeuroImage, 25, 76-89.
62. Spreng, M. (1980). Influence of impulsive and fluctuating noise upon physiological excitations and short-time readaptation. Scandanavian Audiology, 12 (Suppl.), 299-306.
63. Stekelenburg, J.J., & Vroomen, J. (2007). Neural correlates of multisensory integration of ecologically valid audio-visual events. Journal of Cognitive Neuroscience, 19 (12), 1964-1973.
64. Sumby, W., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise, Journal of the Acoustical Society of America, 26, 212-215.
65. Tanabe, H.C., Honda, M., & Sadato, N. (2005). Functionally segregated neural substrates for arbitrary audio-visual paired-association learning. Journal of Neuroscience, 25 (27), 6409-6418.
66. Teder-Salejarvi, W.A., McDonald, J.J., DiRusso, F., & Hillyard, S.A. (2002). An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. Cognitive Brain Research, 14, 106-114.
67. Teinonen, T., Aslin, R., Alku, P., & Csibra, G. (2008). Visual speech contributes to phonetic learning in 6-month-old infants. Cognition, 108, 850-855.
68. Thomas, M.S.C., Annaz, D., Ansari, D., Serif, G., Jarrold, C., & Karmiloff-Smith, A. (2009). Using developmental trajectories to understand developmental disorders. Journal of Speech, Language, and Hearing Research, 52, 336-358.
69. Thornton, A.R.D. (2008). Evaluation of a technique to measure latency jitter in event-related potentials. Journal of Neuroscience Methods, 8 (1), 248-255.
70. Tremblay, C., Champoux, F., Voss, P., Bacon, B.A., Lapore, F., & Theoret, H. (2007). Speech and non-speech audio-visual illusions: a developmental study. PLoS, 8, 742.
71. Van Wassenhove, V., Grant, K.W., & Poeppel, D. (2005). Visual speech speeds up the neural processing of auditory speech. Proceedings of the National Academy of Sciences, USA, 102 (4), 1181-1186.
72. Viswanathan, D., & Jansen, B.H. (2010). The effect of stimulus expectancy on dishabituation of auditory evoked potentials. International Journal of Psychophysiology, 78, 251-256.
73. Wada, Y., Shirai, N., Midorikawa, A., Kanazawa, S., Dan, I., & Yamaguchi, M.K. (2009). Sound enhances detection of visual target during infancy: a study using illusory contours. Journal of Experimental Child Psychology, 102 (3), 315-322.
74. Wightman, F., Kistler, D., & Brungart, D. (2006). Informational masking of speech in children: auditory-visual integration. Journal of the Acoustical Society of America, 119 (6), 3940-3949.
75. Wright, T.M., Pelphrey, K.A., Allison, T., McKeown, M.J., & McCarthy, G. (2003). Polysensory interactions along lateral temporal regions evoked by audio-visual speech. Cerebral Cortex, 13, 1034-1043.