The effects of visual material and temporal synchrony on the processing of letters and speech sounds

Experimental Brain Research

Volumn 211, Issue 2, 2011, Pages 287-298

  Mittag, Maria ^a   Takegata, Rika ^a   Kujala, Teija ^a  

^a UNIVERSITY OF HELSINKI   (Finland)

Author keywords

Audiovisual processing; Event related potentials (ERPs); Letter speech sound processing; Mismatch negativity (MMN); Multisensory integration

Indexed keywords

ADULT; ARTICLE; AUDITORY STIMULATION; CONSONANT; CONTROLLED STUDY; ELECTROENCEPHALOGRAPHY; ELECTROOCULOGRAM; EVENT RELATED POTENTIAL; FEMALE; HUMAN; HUMAN EXPERIMENT; LANGUAGE PROCESSING; LATENT PERIOD; MALE; MENTAL FUNCTION; MISMATCH NEGATIVITY; NORMAL HUMAN; PRIORITY JOURNAL; RESPONSE TIME; SPEECH; TEMPORAL SYNCHRONY; TIME; VISUAL STIMULATION; VOWEL; WRITTEN LANGUAGE;

ACOUSTIC STIMULATION; ADULT; EVOKED POTENTIALS, AUDITORY; EVOKED POTENTIALS, VISUAL; FEMALE; HUMANS; MALE; PHONETICS; PHOTIC STIMULATION; READING; SPEECH PERCEPTION; YOUNG ADULT;

EID: 79958272003     PISSN: 00144819     EISSN: 14321106     Source Type: Journal    
DOI: 10.1007/s00221-011-2686-z     Document Type: Article

References (50)

1. Alho K (1995) Cerebral generators of mismatch negativity (MMN) and its magnetic counterpart (MMNm) elicited by sound changes. Ear Hear 16(1):38-51
2. Alho K, Connolly JF, Cheour M, Lehtokoski A, Huotilainen M, Virtanen J, Aulanko R, Ilmoniemi RJ (1998) Hemispheric lateralization in preattentive processing of speech sounds. Neurosci Lett 258(1):9-12 (Pubitemid 29003149)
3. Alku P, Tiitinen H, Näätänen R (1999) A method for generating natural-sounding speech stimuli for cognitive brain research. Clin Neurophysiol 110(8):1329-1333 (Pubitemid 29333868)
4. Besle J, Fort A, Giard MH (2005) Is the auditory sensory memory sensitive to visual information? Exp Brain Res 166(3-4):337-344
5. Cheour M, Ceponiene R, Lehtokoski A, Luuk A, Allik J, Alho K, Näätänen R (1998) Development of language-specific phoneme representations in the infant brain. Nat Neurosci 1(5):351-353
6. Colin C, Radeau M, Soquet A, Deltenre P (2004) Generalization of the generation of a MMN by illusory McGurk percepts: Voiceless consonants. Clin Neurophysiol 115(9):1989-2000 (Pubitemid 39037287)
7. Delorme A, Makeig S (2004) Eeglab: an open source toolbox for analysis of single-trial eeg dynamics including independent component analysis. J Neurosci Methods 134(1):9-21 (Pubitemid 38224183)
8. Frost R, Repp BH, Katz L (1988) Can speech perception be influenced by simultaneous presentation of print? J Memory Lang 27:741-755
9. Froyen D, van Atteveldt N, Bonte M, Blomert L (2008) Cross-modal enhancement of the MMN to speech-sounds indicates early and automatic integration of letters and speech-sounds. Neurosci Lett 430(1):23-28
10. Froyen DJ, Bonte ML, van Atteveldt N, Blomert L (2009) The long road to automation: neurocognitive development of letter-speech sound processing. J Cogn Neurosci 21(3):567-580
11. Giard MH, Lavikainen J, Reinikainen K, Perrin F, Bertrand O, Pernier J, Näätänen R (1995) Seperate representation of stimulus frequency, intensity, and duration in auditory sensory memory: an event-related potential and dipole-model analysis. J Cognit Neurosci 7:133-143
12. Goswami U (2002) Cross-linguistic aspects of literacy development and dyslexia. Ann Dyslexia 52:141-163
13. Jaramillo M, Ilvonen T, Kujala T, Alku P, Tervaniemi M, Alho K (2001) Are different kinds of acoustic features processed differently for speech and non-speech sounds? Brain Res Cogn Brain Res 12(3):459-466
14. Kirmse U, Ylinen S, Tervaniemi M, Vainio M, Schröger E, Jacobsen T (2008) Modulation of the mismatch negativity (MMN) to vowel duration changes in native speakers of Finnish and German as a result of language experience. Int J Psychophysiol 67(2):131-143
15. Kujala T, Tervaniemi M, Schröger E (2007) The mismatch negativity in cognitive and clinical neuroscience: theoretical and methodological considerations. Biol Psychol 74(1):1-19 (Pubitemid 44794431)
16. Lovio R, Pakarinen S, Huotilainen M, Alku P, Silvennoinen S, Näätänen R, Kujala T (2009) Auditory discrimination profiles of speech sound changes in 6-year-old children as determined with the multi-feature MMN paradigm. Clin Neurophysiol 120(5):916-921
17. Luck SJ (2005) An introduction to the event-related potential technique. The MIT Press, Cambridge
18. Massaro DW, Cohen MC, Thompson LA (1988) Visible language in speech perception: lipreading and reading. Visible Lang 22:8-31
19. McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264(5588):746-748
20. Muller-Gass A, Stelmack RM, Campbell (2006) The effect of visual task difficulty and attentional direction on the detection of acoustic change as indexed by the mismatch negativity. Brain Res 1078:112-130
21. Näätänen R, Michie PT (1979) Early selective-attention effects on the evoked potential: a critical review and reinterpretation. Biol Psychol 8(2):81-136 (Pubitemid 9206181)
22. Näätänen R, Gaillard AW, Mäntysalo S (1978) Early selectiveattention effect on evoked potential reinterpreted. Acta Psychol (Amst) 42(4):313-329
23. Näätänen R, Lehtokoski A, Lennes M, Cheour M, Huotilainen M, Iivonen A, Vainio M, Alku P, Ilmoniemi RJ, Luuk A, Allik J, Sinkkonen J, Alho K (1997) Language-specific phoneme representations revealed by electric and magnetic brain responses. Nature 385(6615):432-434 (Pubitemid 27065927)
24. Näätänen R, Pakarinen S, Rinne T, Takegata R (2004) The mismatch negativity (MMN): towards the optimal paradigm. Clin Neurophysiol 115(1):140-144 (Pubitemid 38045607)
25. Näätänen R, Paavilainen P, Rinne T, Alho K (2007) The mismatch negativity (MMN) in basic research of central auditory processing: a review. Clin Neurophysiol 118(12):2544-2590 (Pubitemid 350182437)
26. Novak GP, Ritter W, Vaughan HG Jr, Wiznitzer ML (1990) Differentiation of negative event-related potentials in an auditory discrimination task. Electroencephalogr Clin Neurophysiol 75(4):255-275 (Pubitemid 20111464)
27. Paavilainen P, Alho K, Reinikainen K, Sams M, Näätänen R (1991) Right hemisphere dominance of different mismatch negativities. Electroencephalogr Clin Neurophysiol 78(6):466-479
28. Pakarinen S, Lovio R, Huotilainen M, Alku P, Näätänen R, Kujala T (2009) Fast multi-feature paradigm for recording several mismatch negativities (MMNs) to phonetic and acoustic changes in speech sounds. Biol Psychol 82(3):219-226
29. Pulvermüller F, Kujala T, Shtyrov Y, Simola J, Tiitinen H, Alku P, Alho K, Martinkauppi S, Ilmoniemi RJ, Näätänen R (2001) Memory traces for words as revealed by the mismatch negativity. Neuroimage 14(3):607-616
30. Rinne T, Alho K, Alku P, Holi M, Sinkkonen J, Virtanen J, Bertrand O, Näätänen R (1999) Analysis of speech sounds is left-hemisphere predominant at 100-150 ms after sound onset. NeuroReport 10:1113-1117 (Pubitemid 29317998)
31. Sams M, Aulanko R, Hämäläinen M, Hari R, Lounasmaa OV, Lu ST, Simola J (1991) Seeing speech: visual information from lip movements modifies activity in the human auditory cortex. Neurosci Lett 127(1):141-145
32. Shtyrov Y, Kujala T, Ahveninen J, Tervaniemi M, Alku P, Ilmoniemi RJ, Näätänen R (1998) Background acoustic noise and the hemispheric lateralization of speech processing in the human brain: magnetic mismatch negativity study. Neurosci Lett 251(2):141-144 (Pubitemid 28358274)
33. Shtyrov Y, Kujala T, Palva S, Ilmoniemi RJ, Näätänen R (2000) Discrimination of speech and of complex nonspeech sounds of different temporal structure in the left and right cerebral hemispheres. Neuroimage 12(6):657-663
34. Siegel LS, Faux D (1989) Acquisition of certain grapheme-phoneme correspondence in normally achieving and disabled readers. Read Writ : An Interdiscip J 1:37-52
35. Snowling MJ (1981) Phonemic deficits in developmental dyslexia. Psychol Res 43(2):219-234
36. Sorokin A, Alku P, Kujala T (2010) Change and novelty detection in speech and non-speech sound streams. Brain Res 1327:77-90
37. Sussman E, Ritter W, Vaughan HG Jr (1998) Attention affects the organization of auditory input associated with the mismatch negativity system. Brain Res 789(1):130-138 (Pubitemid 28206853)
38. Swets JA (1964) Signal detection and recognition by human observers. Wiley, New York
39. Takegata R, Tervaniemi M, Alku P, Ylinen S, Näätänen R (2008) Parameter-specific modulation of the mismatch negativity to duration decrement and increment: evidence for asymmetric processes. Clin Neurophysiol 119(7):1515-1523
40. Tervaniemi M, Jacobsen T, Rottger S, Kujala T, Widmann A, Vainio M, Näätänen R, Schröger E (2006) Selective tuning of cortical sound-feature processing by language experience. Eur J Neurosci 23(9):2538-2541 (Pubitemid 43742885)
41. Vainio M, Järvikivi J (2007) Focus in production: tonal shape, intensity and word order. J Acoust Soc Am 121 (2) :EL55-61
42. van Atteveldt NM, Formisano E, Blomert L, Goebel R (2007) The effect of temporal asynchrony on the multisensory integration of letters and speech sounds. Cereb Cortex 17(4):962-974 (Pubitemid 46434637)
43. Widmann A, Kujala T, Tervaniemi M, Kujala A, Schröger E (2004) From symbols to sounds: visual symbolic information activates sound representations. Psychophysiology 41(5):709-715 (Pubitemid 39295497)
44. Widmann A, Gruber T, Kujala T, Tervaniemi M, Schröger E (2007) Binding symbols and sounds: evidence from event-related oscillatory gamma-band activity. Cereb Cortex 17(11):2696-2702 (Pubitemid 47537829)
45. Winkler I, Kujala T, Tiitinen H, Sivonen P, Alku P, Lehtokoski A, Czigler I, Csepe V, Ilmoniemi RJ, Näätänen R (1999) Brain responses reveal the learning of foreign language phonemes. Psychophysiology 36(5):638-642 (Pubitemid 29357559)
46. Woldorff MG, Hackley SA, Hillyard SA (1991) The effects of channel-selective attention on the mismatch negativity wave elicited by deviant tones. Psychophysiology 28(1):30-42
47. Woldorff MG, Hillyard SA, Gallen CC, Hampson SR, Bloom FE (1998) Magnetoencephalographic recordings demonstrate attentional modulation of mismatch-related neural activity in human auditory cortex. Psychophysiology 35(3):283-292 (Pubitemid 28166678)
48. Yanz JL (1984) The application of the theory of signal detection to the assessment of speech perception. Ear Hear 5:64-71
49. Ylinen S, Shestakova A, Alku P, Huotilainen M (2005) The perception of phonological quantity based on durational cues by native speakers, second-language users and nonspeakers of Finnish. Lang Speech 48(Pt 3):313-338 (Pubitemid 43014126)
50. Yucel G, Petty C, McCarthy G, Belger A (2005) Graded visual attention modulates brain responses evoked by task-irrelevant auditory pitch changes. J Cogn Neurosci 17(12):1819-1828