Sequential audiovisual interactions during speech perception: A whole-head MEG study

Neuropsychologia

Volumn 45, Issue 6, 2007, Pages 1342-1354

  Hertrich, Ingo ^a   Mathiak, Klaus ^b   Lutzenberger, Werner ^c   Menning, Hans ^d   Ackermann, Hermann ^a  

^a UNIVERSITY OF TÜBINGEN   (Germany)

^b RWTH AACHEN UNIVERSITY   (Germany)

^c UNIVERSITY OF TÜBINGEN   (Germany)

^d UNIVERSITY OF ZURICH   (Switzerland)

Author keywords

Audiovisual interactions; Auditory processing; Evoked magnetic fields; Magnetoencephalography; Mismatch negativity; Speech perception

Indexed keywords

ADULT; ARTICLE; FEMALE; HEARING; HUMAN; HUMAN EXPERIMENT; LINGUISTICS; MAGNETOENCEPHALOGRAPHY; MALE; NORMAL HUMAN; PHONETICS; SENSORY SYSTEM; SPEECH; SPEECH PERCEPTION; STANDARDIZATION; VIDEORECORDING; VISION;

EID: 33846061812     PISSN: 00283932     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neuropsychologia.2006.09.019     Document Type: Article

References (69)

1. Ackermann H., Lutzenberger W., and Hertrich I. Hemispheric lateralization of the neural encoding of temporal speech features: A whole-head magnetencephalography study. Cognitive Brain Research 7 (1999) 511-518
2. Ackermann H., Hertrich I., Mathiak K., and Lutzenberger W. Contralaterality of cortical auditory processing at the level of the M50/M100 complex and the mismatch field: A whole-head magnetoencephalography study. NeuroReport 12 (2001) 1683-1687
3. Alho K. Cerebral generators of mismatch negativity (MMN) and its magnetic counterpart (MMNm) elicited by sound changes. Ear and Hearing 16 (1995) 38-51
4. Berti S., and Schröger E. Working memory controls involuntary attention switching: Evidence from an auditory distraction paradigm. European Journal of Neuroscience 17 (2003) 1119-1122
5. Besle J., Fort A., Delpuech C., and Giard M.H. Bimodal speech: Early suppressive visual effects in human auditory cortex. European Journal of Neuroscience 20 (2004) 2225-2234
6. Brosch M., Selezneva E., and Scheich H. Nonauditory events of a behavioral procedure activate auditory cortex of highly trained monkeys. Journal of Neuroscience 25 (2005) 6797-6806
7. Callan D.E., Jones J.A., Munhall K., Kroos C., Callan A.M., and Vatikiotis-Bateson E. Multisensory integration sites identified by perception of spatial wavelet filtered visual speech gesture information. Journal of Cognitive Neuroscience 16 (2004) 805-816
8. Calvert G.A., Bullmore E.T., Brammer M.J., Campbell R., Williams S.C., McGuire P.K., et al. Activation of auditory cortex during silent lipreading. Science 276 (1997) 593-596
9. Calvert G.A., and Campbell R. Reading speech from still and moving faces: The neural substrates of visible speech. Journal of Cognitive Neuroscience 15 (2003) 57-70
10. Calvert G.A., Campbell R., and Brammer M.J. Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Current Biology 10 (2000) 649-657
11. Clark V.P., Hillyard S.A., and Fan S. Identification of early visual evoked potential generators by retinotopic and topographic analyses. Human Brain Mapping 2 (1995) 170-187
12. Colin C., Radeau M., Soquet A., Demolin D., Colin F., and Deltenre P. Mismatch negativity evoked by the McGurk-MacDonald effect: A phonetic representation within short-term memory. Clinical Neurophysiology 113 (2002) 495-506
13. Colin C., Radeau M., Soquet A., and Deltenre P. Generalization of the generation of an MMN by illusory McGurk percepts: Voiceless consonants. Clinical Neurophysiology 115 (2004) 1989-2000
14. Driver J., and Frith C. Shifting baseline in attention research. Nature Reviews Neuroscience 1 (2000) 147-148
15. Foxe J.J., and Schroeder C.E. The case for feedforward multisensory convergence during early cortical processing. Neuroreport 16 (2005) 419-423
16. Fritz J., Shamma S., Elhilali M., and Klein D. Rapid task-related plasticity of spectrotemporal receptive fields in primary auditory cortex. Nature Neuroscience 6 (2003) 1216-1223
17. Fu K.M.G., Johnston T.A., Shah A.S., Schroeder C.E., Smiley J., Arnold L., et al. Auditory cortical neurons respond to somatosensory stimulation. Journal of Neuroscience 23 (2003) 7510-7515
18. Ghazanfar A.A., Maier J.X., Hoffman K.L., and Logothetis N.K. Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. Journal of Neuroscience 25 (2005) 5004-5012
19. Hertrich I., and Ackermann H. A vowel synthesizer based on formant sinusoids modulated by fundamental frequency. Journal of the Acoustical Society of America 106 (1999) 2988-2990
20. Hertrich I., Mathiak K., Lutzenberger W., and Ackermann H. Differential impact of periodic and aperiodic speech-like acoustic signals on magnetic M50/M100 fields. Neuroreport 11 (2000) 4017-4020
21. Hertrich I., Mathiak K., Lutzenberger W., and Ackermann H. Hemispheric lateralization of the processing of consonant-vowel syllables (formant transitions): Effects of stimulus characteristics and attentional demands on evoked magnetic fields. Neuropsychologia 40 (2002) 1902-1917
22. Hertrich I., Mathiak K., Lutzenberger W., and Ackermann H. Processing of dynamic aspects of speech and non-speech stimuli: A whole-head magnetoencephalography study. Cognitive Brain Research 17 (2003) 130-139
23. Hertrich I., Mathiak K., Lutzenberger W., and Ackermann H. Transient and phase-locked evoked magnetic fields in response to periodic acoustic signals. Neuroreport 15 (2004) 1687-1690
24. Huotilainen M., Winkler I., Alho K., Escera C., Virtanen J., Ilmoniemi R.J., et al. Combined mapping of human auditory EEG and MEG responses. Electroencephalogr. Clinical Neurophysiology 108 (1998) 370-379
25. Ikeda H., Nishijo H., Miyamoto K., Tamura R., Endo S., and Ono T. Generators of visual evoked potentials investigated by dipole tracing in the human occipital cortex. Neuroscience 84 (1998) 723-739
26. Jones J.A., and Callan D.E. Brain activity during audiovisual speech perception: an fMRI study of the McGurk effect. Neuroreport 14 (2003) 1129-1133
27. Kähkönen S., and Ahveninen J. Combination of magneto- and electroencephalography in studies of monoamine modulation on attention. Methods and Findings in Experimental and Clinical Pharmacology 24 (2002) 27-34
28. Kanwisher N., McDermott J., and Chun M.M. The fusiform face area: A module in human extrastriate cortex specialized for face perception. Journal of Neuroscience 17 (1997) 4302-4311
29. Kastner S., Pinsk M.A., De Weerd P., Ungerleider L.G., and Desimone R. Increased activity in human visual cortex during directed attention in the absence of visual stimulation. Neuron 22 (1999) 751-761
30. Kayser C., Petkov C.I., Augath M., and Logothetis N.K. Integration of touch and sound in auditory cortex. Neuron 48 (2005) 373-384
31. Klucharev V., Möttönen R., and Sams M. Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Cognitive Brain Research 18 (2003) 65-75
32. Korpilahti P., Krause C.M., Holopainen I., and Lang A.H. Early and late mismatch negativity elicited by words and speech-like stimuli in children. Brain and Language 76 (2001) 332-339
33. Liu J., Harris A., and Kanwisher N. Stages of processing in face perception: An MEG study. Nature Neuroscience 5 (2002) 910-916
34. Lueschow A., Sander T., Boehm S.G., Nolte G., Trahms L., and Curio G. Looking for faces: Attention modulates early occipitotemporal object processing. Psychophysiology 41 (2004) 350-360
35. MacDonald J., and McGurk H. Visual influences on speech perception processes. Perception and Psychophysics 24 (1978) 253-257
36. MacSweeney M., Amaro E., Calvert G.A., Campbell R., David A.S., McGuire P., et al. Silent speechreading in the absence of scanner noise: An event-related fMRI study. Neuroreport 11 (2000) 1729-1733
37. Massaro D.W. From multisensory integration to talking heads and language learning. In: Calvert G.A., Spence C., and Stein B.E. (Eds). The handbook of multisensory processes (2004), MIT Press, Cambridge, MA 153-176
38. Mathiak K., Hertrich I., Lutzenberger W., and Ackermann H. Encoding of temporal speech features (formant transients) during binaural and dichotic stimulus application: A whole-head magnetencephalography study. Cognitive Brain Research 10 (2000) 125-131
39. Mathiak K., Hertrich I., Lutzenberger W., and Ackermann H. Neural correlates of duplex perception: A whole-head magnetencephalography study. Neuroreport 12 (2001) 501-506
40. Mathiak K., Hertrich I., Lutzenberger W., and Ackermann H. Functional cerebral asymmetries of pitch processing during dichotic stimulus application: A whole-head magnetoencephalography study. Neuropsychologia 40 (2002) 585-593
41. Mathiak K., Hertrich I., Zvyagintsev M., Lutzenberger W., and Ackermann H. Selective influences of cross-modal spatial cues on preattentive auditory processing: A whole-head magnetoencephalography study. Neuroimage 28 (2005) 627-634
42. Menning H., Ackermann H., Hertrich I., and Mathiak K. Spatial auditory attention is modulated by tactile priming. Experimental Brain Research 164 (2005) 41-47
43. Miki K., Watanabe S., and Kakigi R. Interaction between auditory and visual stimulus relating to the vowel sounds in the auditory cortex in humans: A magnetoencephalographic study. Neuroscience Letters 357 (2004) 199-202
44. Miki K., Watanabe S., Kakigi R., and Puce A. Magnetoencephalographic study of occipitotemporal activity elicited by viewing mouth movements. Clinical Neurophysiology 115 (2004) 1559-1574
45. Miller L.M., and D'Esposito M. Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. Journal of Neuroscience 25 (2005) 5884-5893
46. Möttönen R., Krause C.M., Tiippana K., and Sams M. Processing of changes in visual speech in the human auditory cortex. Cognitive Brain Research 13 (2002) 417-425
47. Möttönen R., Schurmann M., and Sams M. Time course of multisensory interactions during audiovisual speech perception in humans: A magnetoencephalographic study. Neuroscience Letters 363 (2004) 112-115
48. Oldfield R.C. The assessment and analysis of handedness: The Edinburgh inventory. Neuropsychologia 9 (1971) 97-113
49. Olson I.R., Gatenby J.C., and Gore J.C. A comparison of bound and unbound audio-visual information processing in the human cerebral cortex. Cognitive Brain Research 14 (2002) 129-138
50. Pazo-Alvarez P., Cadaveira F., and Amenedo E. MMN in the visual modality: A review. Biological Psychology 63 (2003) 199-236
51. Perrin F., Pernier J., Bertrand O., and Echallier J.F. Spherical splines for scalp potential and current density mapping. Electroencephalography and Clinical Neurophysiology 72 (1989) 184-187
52. Phillips C., Pellathy T., Marantz A., Yellin E., Wexler K., Poeppel D., et al. Auditory cortex accesses phonological categories: An MEG mismatch study. Journal of Cognitive Neuroscience 12 (2000) 1038-1055
53. Puce A., Allison T., Bentin S., Gore J.C., and McCarthy G. Temporal cortex activation in humans viewing eye and mouth movements. Journal of Neuroscience 18 (1998) 2188-2199
54. Rensink R.A., and Enns J.T. Preemption effects in visual search: Evidence for low-level grouping. Psychological Revue 102 (1995) 101-130
55. Repp B.H. Perceptual restoration of a "missing" speech sound: Auditory induction or illusion?. Perception and Psychophysics 51 (1992) 14-32
56. Sams M., Aulanko R., Hamalainen M., Hari R., Lounasmaa O.V., Lu S.T., et al. Seeing speech: Visual information from lip movements modifies activity in the human auditory cortex. Neuroscience Letters 127 (1991) 141-145
57. Schroeder C.E., and Foxe J.J. The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. Cognitive Brain Research 14 (2002) 187-198
58. Schroeder C.E., Smiley J., Fu K.G., McGinnis T., O'Connell M.N., and Hackett T.A. Anatomical mechanisms and functional implications of multisensory convergence in early cortical processing. International Journal of Psychophysiology 50 (2003) 5-17
59. Schwartz J.L., Robert-Ribes J., and Escudier P. Ten years after Summerfield: A taxonomy of models for audio-visual fusion in speech perception. In: Campbell R., Dodd B., and Burnam D. (Eds). Hearing by Eye II (1998), Psychology Press, Hove, UK 85-108
60. Stagg C., Hindley P., Tales A., and Butler S. Visual mismatch negativity: The detection of stimulus change. Neuroreport 15 (2004) 659-663
61. Stufflebeam S.M., Poeppel D., Rowley H.A., and Roberts T.P. Peri-threshold encoding of stimulus frequency and intensity in the M100 latency. Neuroreport 9 (1998) 91-94
62. Sumby W.H., and Pollack I. Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America 26 (1954) 212-215
63. Teder W., Alho K., Reinikainen K., and Näätänen R. Interstimulus interval and the selective-attention effect on auditory ERPs: "N1 enhancement" versus processing negativity. Psychophysiology 30 (1993) 71-81
64. Tsuda R., and Ueno S. Source localization of visually evoked magnetic fields to stimuli in apparent motion. IEEE Transactions on Magnetics 36 (2004) 3727-3729
65. Van Wassenhove V., Grant K.W., and Poeppel D. Visual speech speeds up the neural processing of auditory speech. Proceedings of the National Academy of Science USA 102 (2005) 1181-1186
66. Wagner M., Fuchs M., and Kastner J. Evaluation of sLORETA in the presence of noise and multiple sources. Brain Topography 16 (2004) 277-280
67. Winkler I., Lehtokoski A., Alku P., Vainio M., Czigler I., Csépe V., et al. Pre-attentive detection of vowel contrasts utilizes both phonetic and auditory memory representations. Cognitive Brain Research 7 (1999) 357-369
68. Woldorff M.G., Hillyard S.A., Gallen C.C., Hampson S.R., and Bloom F.E. Magnetoencephalographic recordings demonstrate attentional modulation of mismatch-related neural activity in human auditory cortex. Psychophysiology 35 (1998) 283-292
69. Zachau S., Rinker T., rner B., Kohls G., Maas V., Hennighausen K., et al. Extracting rules: Early and late mismatch negativity to tone patterns. Neuroreport 16 (2005) 2015-2019