Multi-sensory learning and learning to read

International Journal of Psychophysiology

Volumn 77, Issue 3, 2010, Pages 195-204

  Blomert, Leo ^a   Froyen, Dries ^a  

^a MAASTRICHT UNIVERSITY   (Netherlands)

Author keywords

Audiovisual integration; Grapheme phoneme correspondences; Letter speech sound associations; Multi sensory processing; Reading development

Indexed keywords

ARTICLE; AUDIOVISUAL AID; AUDITORY CORTEX; AUDITORY SYSTEM FUNCTION; DYSLEXIA; HUMAN; INFORMATION PROCESSING; LEARNING; READING; SENSORY SYSTEM; SOUND; SPEECH; VISION; VISUAL CORTEX; AUDITORY STIMULATION; BRAIN; BRAIN MAPPING; ELECTROENCEPHALOGRAPHY; LANGUAGE; METHODOLOGY; NUCLEAR MAGNETIC RESONANCE IMAGING; PHONETICS; PHOTOSTIMULATION; PHYSIOLOGY; REVIEW; VASCULARIZATION;

ACOUSTIC STIMULATION; BRAIN; BRAIN MAPPING; ELECTROENCEPHALOGRAPHY; HUMANS; LANGUAGE; LEARNING; MAGNETIC RESONANCE IMAGING; PHONETICS; PHOTIC STIMULATION; READING; VISUAL PERCEPTION;

EID: 79952016982     PISSN: 01678760     EISSN: 18727697     Source Type: Journal    
DOI: 10.1016/j.ijpsycho.2010.06.025     Document Type: Article

References (93)

1. Amedi A., von Kriegstein K., Van Atteveldt N.M., Beauchamp M.S., Naumer M.J. Functional imaging of human crossmodal identification and object recognition. Experimental Brain Research 2005, 166:559-571.
2. Baart M., Vroomen J. Do you see what you are hearing? Cross-modal effects of speech sounds on lipreading. Neuroscience letters 2010, 471:100-103.
3. Beauchamp M.S., Argall B.D., Bodurka J., Duyn J., Martin A. Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nature Neuroscience 2004, 7:1190-1192.
4. Besle J., Fort A., Giard M. Is the auditory memory sensitive to visual information. Experimental Brain Research 2005, 166:337-344.
5. Bishop D.V.M. Using mismatch negativity to study central auditory processing in developmental language and literacy impairments: where are we, and where should we be going?. Psychological bulletin 2007, 133:651-672.
6. Blau V., Atteveldt N., Ekkebus M., Goebel R., Blomert L. Reduced neural integration of letters and speech sounds links phonological and reading deficits in adult dyslexia. Current Biology 2009, 19:503-508.
7. Blau V., van Atteveldt N., Formisano E., Goebel R., Blomert L. Task-irrelevant visual letters interact with the processing of speech sounds in heteromodal and unimodal cortex. European Journal of Neuroscience 2008, 28(3):500-509.
8. Blau V., Reithler J., van Atteveldt N., Seitz J., Gerretsen P., Goebel R., Blomert L. Deviant processing of letters and speech sounds as proximate cause of reading failure: a functional magnetic resonance imaging study of dyslexic children. Brain 2010, 133:868-879.
9. Blomert L., Vaessen A. 3DM Differentiaal diagnose voor dyslexie: Cognitieve analyse van lezen en spelling [3DM Differential Diagnostics for Dyslexia: Cognitive Analysis of Reading and Spelling] 2009, Boom Test Publishers, Amsterdam.
10. Blomert, L., & Willems, G. (in press). Is there a causal link from a phonological awareness deficit to reading failure in children at familial risk for dyslexia? Dyslexia. http://doi:10.1002/dys.405.
11. Blomert L. (submitted). Reading failure in dyslexia: the case for a causal cross-modal.
12. Blomert, L., de Doelder, N., Froyen D. (submitted). One or two reading routes: a multisensory perspective on reading.
13. Bonte M., Mitterer H., Zellagui N., Poelmans H., Blomert L. Auditory cortical tuning to statistical regularities in phonology. Clinical Neurophysiology 2005, 116(12):2765-2774.
14. Bonte M., Poelmans H., Blomert L. Deviant neurophysiological responses to phonological regularities in speech in dyslexic children. Neuropsychologia 2007, 45:1427-1437.
15. Burnham D., Dodd B. Auditory-visual speech integration by prelinguistic infants: perception of an emergent consonant in the McGurk effect. Developmental Psychobiology 2004, 45:204-220.
16. Burnham, D., & Sekiyama, K. (in press). Investigating auditory-visual speech perception development using the ontogenetic and differential language methods. In E. Vatikiotis-Bateson, P. Perrier & G. Bailly (Eds.), Advances in auditory-visual speech processing. Cambridge: Mitt Press.
17. By Byrne, Brian; Fielding-Barnsley, Ruth Phonemic awareness and letter knowledge in the child's acquisitions of the alphabetic principle. Journal of Educational Psychology 1989, 80:313-321.
18. Calvert G.A. Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cerebral Cortex 2001, 11(12):1110-1123.
19. Calvert G.A., Bullmore E.T., Brammer M.J., Campbell R., Williams S.C., McGuire P.K., et al. Activation of auditory cortex during silent lipreading. Science 1997, 276(5312):593-596.
20. Calvert G.A., Brammer M.J., Iversen S.D. Crossmodal identification. Trends in Cognitive Sciences 1998, 2:247-253.
21. Calvert G.A., Brammer M.J., Bullmore E.T., Campbell R., Iversen S.D., David A.S. Response amplification in sensory-specific cortices during crossmodal binding. Neuroreport 1999, 10(12):2619-2623.
22. Calvert G.A., Campbell R., Brammer M.J. Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Current Biology 2000, 10(11):649-657.
23. Castro-Caldas A., Petersson K.M., Reis A., Stone-Elander S., Ingvar M. The illiterate brain. Learning to read and write during childhood influences the functional organization of the adult brain. Brain 1998, 121(Pt 6):1053-1063.
24. Chein J.M., Schneider W. Neuroimaging studies of practice-related change: fMRI and meta-analytic evidence of a domain-general control network for learning. Cognitive Brain Research 2005, 25:607-623.
25. Cohen L., Dehaene S., Naccache L., Lehericy S., Dehaene-Lambertz G., Henaff M.A., et al. The visual word form area: spatial and temporal characterization of an initial stage of reading in normal subjects and posterior split-brain patients. Brain 2000, 123(Pt 2):291-307.
26. Colin C., Radeau M., Soquet A., Demolin D., Colin F., Deltenre P. Mismatch negativity evoked by the McGurk-MacDonald effect: a phonetic representation within short-term memory. Clinical Neurophysiology 2002, 113(4):495-506.
27. Coltheart M., Rastle K., Perry C., Langdon R., Ziegler J. DRC: a dual route cascaded model of visual word recognition and reading aloud. Psychological review 2001, 108:204-256.
28. Csépe V. Dyslexia: Different Brain, Different Behavior 2003, Kluwer Academic/ Plenum Publishers, New York.
29. Czigler I. Visual mismatch negativity; violation of nonattended environmental regularities. Journal of Psychophysiology 2007, 21:224-230.
30. Dijkstra T., Schreuder R., Frauenfelder U.H. Grapheme context effects on phonemic processing. Language and Speech 1989, 32:89-108.
31. Ehri L.C. Phases of development in learning to read words by sight. Journal of Research in Reading 1995, 18:116-125.
32. Fort A., Giard M. Multiple electrophysiological mechanisms of audiovisual integration in human perception. The Handbook of Multisensory Processes 2004, 503-513. The Mitt Press, London. G.A. Calvert, C. Spence, B.E. Stein (Eds.).
33. Fort A., Delpuech C., Pernier J., Giard M. Dynamics of cortico-subcortical cross-modal operations involved in audio-visual object detection in humans. Cerebral Cortex 2002, 12(10):1031-1039.
34. Fort A., Delpuech C., Pernier J., Giard M. Early auditory-visual interactions in human cortex during nonredundant target identification. Cognitive Brain Research 2002, 14(1):20-30.
35. Frith U. Beneath the surface of developmental dyslexia. Surface Dyslexia 1985, Routledge & Kegan-Paul, London. K.E. Patterson, J.C. Marshall, M. Coltheart (Eds.).
36. Froyen D., van Atteveldt N., Bonte M., Blomert L. Cross-modal enhancement of the MMN to speech sounds indicates early and automatic integration of letters and speech sounds. Neuroscience Letters 2008, 430:23-28.
37. Froyen D., Bonte M., van Atteveldt N., Blomert L. The long road to automation: neurocognitive development of letter-speech sound processing. Journal of Cognitive Neuroscience 2009, 21:567-580.
38. Froyen D., van Atteveldt N., Blomert L. Exploring the role of low level visual processing in letter-speech sound integration: a visual MMN study. Frontiers in Integrative Neuroscience 2010, 4(9).
39. Froyen, D., Willems, G., & Blomert, L. (accepted). Evidence for a specific cross-modal binding deficit in dyslexia: An MMN-study of letter-speech sound processing. Developmental Science.
40. Gabrieli J.D.E. Dyslexia: a new synergy between education and cognitive neuroscience. Science 2009, 325:280-283.
41. Ghazanfar A.A., Schroeder C.E. Is the neocortex essentially multisensory?. Trends in Cognitive Science 2006, 10:278-285.
42. Giard M.H., Peronnet F. Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. Journal of Cognitive Neuroscience 1999, 11(5):473-490.
43. Hashimoto R., Sakai K.L. Learning letters in adulthood: direct visualization of cortical plasticity for forming a new link between orthography and phonology. Neuron 2004, 42(2):311-322.
44. Hein G., Doehrmann O., Müller N.G., Kaiser J., Muckli L., Naumer M.J. Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. The Journal of Neuroscience 2007, 27:7881-7887.
45. Herdman A.T., Fujioka T., Chau W., Ross B., Pantev C., Picton T.W. Cortical oscillations related to processing congruent and incongruent grapheme-phoneme pairs. Neuroscience Letters 2006, 399:61-66.
46. Hocking J., Price C.J. The role of the posterior superior temporal sulcus in audiovisual processing. Cerebral Cortex 2008, 18:2439-2449.
47. Jusczyk P.W. The Discovery of Spoken Language 1997, Mitt Press, Cambridge, MA.
48. Kujala T., Naatanen R. The mismatch negativity in evaluating central auditory dysfunction in dyslexia. Neuroscience Biobehavioral Review 2001, 25(6):535-543.
49. Lewald J., Guski R. Cross-modal perceptual integration of spatially and temporally disparate auditory and visual stimuli. Cognitive Brain Research 2003, 16:468-478.
50. Lewald J., Ehrensteinb W.H., Guski G. Spatio-temporal constraints for auditory-visual integration. Behavioural Brain Research 2001, 121(1-2):69-79.
51. Lewkowicz D.J. Perception of auditory-visual temporal synchrony in human infants. Journal of Experimental Psychology. Human Perception and Performance 1996, 22:1094-1106.
52. Liberman I.Y. Segmentation of the spoken word and reading acquisition. Bulletin of the Orton Society 1973, 23:65-77.
53. Lieberman P. Toward an Evolutionary Biology of Language 2006, MIT Press, Cambridge, MA.
54. Macaluso E., George N., Dolan R., Spence C., Driver J. Spatial and temporal factors during processing of audiovisual speech: a PET study. NeuroImage 2004, 21:725-732.
55. Marsh G., Friedman M., Welch V., Desberg P. A cognitive-developmental theory of reading acquisition. Reading Research: Advances in Theory and Practice 1981, Academic Press, New York. G.E. MacKinnon, T.G. Waller (Eds.).
56. Massaro D.W., Cohen M.M., Smeele P.M. Perception of asynchronous and conflicting visual and auditory speech. Journal of the Acoustical Society of America 1996, 100:1777-1786.
57. Mattingly I.G. Reading, the linguistic process, and linguistic awareness. Language by Ear and by Eye: The Relationship between Speech and Reading 1972, MIT Press, Cambridge, MA. J.F. Kavanagh, I.G. Mattingly (Eds.).
58. McCandliss B.D., Cohen L., Dehaene S. The Visual Word Form Area: expertise for reading in the fusiform gyrus. Trends in Cognitive Sciences 2003, 7(7):293-299.
59. McGurk H., MacDonald J. Hearing lips and seeing voices. Nature 1976, 263:747.
60. Mitterer H., Blomert L. Coping with phonological assimilation in speech perception: evidence for early compensation. Perception & Psychophysics 2003, 65:956-969.
61. Molholm S., Ritter W., Murray M.M., Javitt D.C., Schroeder C.E., Foxe J.J. Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Brain Research; Cognitive Brain Research 2002, 14(1):115-128.
62. Morais J., Cary L., Alegria J., Bertelson P. Does awareness of speech as a sequence of phones arise spontaneously?. Cognition 1979, 49:957-958.
63. Möttönen R., Krause C.M., Tiippana K., Sams M. Processing of changes in visual speech in the human auditory cortex. Brain Research; Cognitive Brain Research 2002, 13(3):417-425.
64. Munhall K., Gribble P., Sacco L., Ward M. Temporal constraints on the McGurk effect. Perception & Psychophysics 1996, 58(3):351-362.
65. Munhall K., Vatikiotis-Bateson E. The moving face during speech communication. Hearing by Eye, Part 2: The Psychology of Speechreading and Audiovisual Speech 1998, 123-139. Taylor & Francis, Psychology Press, London, UK. R. Campbell, B. Dodd, D. Burnham (Eds.).
66. Näätänen R. The Mismatch Negativity: a powerful tool for cognitive neuroscience. Ear and Hearing 1995, 16:6-18.
67. Näätänen R. Mismatch negativity (MMN): perspectives for application. International Journal of Psychophysiology 2000, 37:3-10.
68. Näätänen R. The perception of speech sounds by the human brain as reflected by the mismatch negativity (MMN) and its magnetic equivalent (MMNm). Psychophysiology 2001, 38:1-21.
69. Naumer M.J., Doehrmann O., Müller N.G., Muckli L., Kaiser J., Hein G. Cortical plasticity of audio-visual object representations. Cerebral Cortex 2009, 19:1641-1653.
70. Pazo-Alvarez P., Cadaveira F., Amenedo E. MMN in the visual modality: a review. Biological Psychology 2003, 63(3):199-236.
71. Perfetti C.A., Beck I., Bell L., Hughes C. Phonemic knowledge and learning to read are reciprocal: a longitudinal study of first grade children. Merrill-Palmer Quarterly 1987, 33:283-319.
72. Picton T.W., Alain C., Otten L., Ritter W., Achim A. Mismatch negativity: different water in the same river. Audiological Neurootology 2000, 5:111-139.
73. Raij T. Patterns of brain activity during visual imagery of letters. Journal of Cognitive Neuroscience 1999, 11:282-299.
74. Raij T., Uutela K., Hari R. Audiovisual integration of letters in the human brain. Neuron 2000, 28:617-625.
75. Rayner K., Pollatsek A. The Psychology of Reading 1989, Prentice-Hall, New Jersey.
76. Sams M., Aulanko R., Hamalainen M., Hari R., Lounasmaa O.V., Lu S.T., et al. Seeing speech: visual information from lip movements modifies activity in the human auditory cortex. Neuroscience letters 1991, 127(1):141-145.
77. Schröger E. Measurement and interpretation of the mismatch negativity. Behavior Research Methods Instruments & Computers 1998, 30:131-145.
78. Sekiyama K., Burnham D. Impact of language on development of auditory-visual speech perception. Developmental Science 2008, 11:303-317.
79. Share D.L. Phonological recoding and self-teaching: sine qua non of reading acquisition. Cognition 1995, 55:151-218.
80. Sumby W.H., Pollack I. Visual contribution to speech intelligibility in noise. The Journal of the Acoustical Society of America 1954, 26(2):212-215.
81. Talsma D., Woldorff M.G. Selective attention and multisensory integration: multiple phases of effects on the evoked brain activity. Journal of Cognitive Neuroscience 2005, 17:1098-1114.
82. Talsma D., Doty T.J., Woldorff M.G. Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration. Cerebral Cortex 2007, 17:679-690.
83. Teder-Salejarvi W.A., McDonald J.J., Di Russo F., Hillyard S.A. An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. Brain Research; Cognitive Brain Research 2002, 14(1):106-114.
84. UNESCO Education for All: Literacy for Life 2005, UNESCO Publishing, Paris.
85. Vaessen A., Blomert L. Long-term cognitive dynamics of fluent reading development. Journal of Experimental Child Psychology 2010, 105:213-231.
86. Van Atteveldt N., Formisano E., Goebel R., Blomert L. Integration of letters and speech sounds in the human brain. Neuron 2004, 43:271-282.
87. Van Atteveldt N., Formisano E., Blomert L., Goebel R. The effect of temporal asynchrony on the multisensory integration of letters and speech sounds. Cerebral Cortex 2007, 13:962-974.
88. Van Atteveldt N., Formisano E., Goebel R., Blomert L. Top-down task effects overrule automatic multisensory responses to letter-sound pairs in auditory association cortex. NeuroImage 2007, 36:1345-1360.
89. Van Wassenhove V., Grant K.W., Poeppel D. Temporal window of integration in auditory-visual speech perception. Neuropsychologia 2007, 45:598-607.
90. Wimmer H., Landerl K., Linortner R., Hummer P. The relationship of phonemic awareness to reading acquisition: more consequence than precondition but still important. Cognition 1991, 40:219-249.
91. Winkler I., Kujala A., Tiitinen H., Sivonen P., Alku P., Lehtokoski A., et al. Brain responses reveal the learning of foreign language phonemes. Psychophysiology 1999, 36:638-642.
92. Zampini M., Guest S., Shore D., Spence C. Audio-visual simultaneity judgments. Perception and Psychophysics 2005, 67:531-544.
93. Ziegler J.C., Goswami U. Reading acquisition, developmental dyslexia, and skilled reading across languages: a psycholinguistic grain size theory. Psychological Bulletin 2005, 131(1):3-29.