The role of visual spatial attention in audiovisual speech perception

Speech Communication

Volumn 51, Issue 2, 2009, Pages 184-193

  Andersen, Tobias S ^a,b   Tiippana, Kaisa ^c   Laarni, Jari ^d   Kojo, Ilpo ^e   Sams, Mikko ^c  

^a UNIVERSITY OF COPENHAGEN   (Denmark)

^b TECHNICAL UNIVERSITY OF DENMARK   (Denmark)

^c AALTO UNIVERSITY   (Finland)

^d VTT TECHNICAL RESEARCH CENTRE OF FINLAND   (Finland)

^e HELSINKI SCHOOL OF ECONOMICS   (Finland)

Author keywords

Attention; McGurk effect; Multisensory; Speech perception

Indexed keywords

VISUAL COMMUNICATION;

ATTENTION; AUDIO VISUALS; AUDIOVISUAL SPEECHES; AUDITORY PERCEPTIONS; CENTRAL FIXATIONS; LIP READINGS; MCGURK EFFECT; MCGURK EFFECTS; MULTISENSORY; SPEECH PERCEPTION; SPEECH PERCEPTIONS; VISUAL INFORMATIONS; VISUAL SPATIAL ATTENTIONS; VISUAL SPEECHES;

SPEECH;

EID: 56949094552     PISSN: 01676393     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.specom.2008.07.004     Document Type: Article

References (51)

1. Alsius A., Navarra J., Campbell R., and Soto-Faraco S. Audiovisual integration of speech falters under high attention demands. Curr. Biol. 15 9 (2005) 839-843
2. Andersen, T.S., Tiippana, K., Lampinen, J., Sams, M., 2001. Modelling of audiovisual speech perception in noise. In: Proc. AVSP 2001, pp. 172-176.
3. Andersen, T.S., Tiippana, K., Sams, M., 2002. Using the fuzzy logical model of perception in measuring integration of audiovisual speech in humans. In: Proc. NF2002.
4. Andersen T.S., Tiippana K., and Sams M. Factors influencing audiovisual fission and fusion illusions. Cogn. Brain Res. 21 3 (2004) 301-308
5. Bertelson P., Vroomen J., de Gelder B., and Driver J. The ventriloquist effect does not depend on the direction of deliberate visual attention. Percept. Psychophys. 62 2 (2000) 321-332
6. Bolognini N., Frassinetti F., Serino A., and Ladavas E. "Acoustical vision" of below threshold stimuli: interaction among spatially converging audiovisual inputs. Exp. Brain Res. 160 3 (2005) 273-282
7. Calvert G.A. Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb. Cortex 11 12 (2001) 1110-1123
8. Calvert G.A., Campbell R., and Brammer M.J. Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr. Biol. 10 11 (2000) 649-657
9. Campbell R. The neuropsychology of lipreading. Philos. Trans. Roy. Soc. London B Biological Sci. 335 1273 (1992) 39-44 discussion 44-35
10. Campbell R. Seeing brains reading speech: a review and speculations. In: Stork D.G., and Hennecke M.E. (Eds). Speechreading by Humans and Machines: Models. Systems and Applications (1996), Springer, Berlin 115-134
11. Cherry E.C. Some experiments on the recognition of speech, with one and with two ears. J. Acoust. Soc. Amer. 25 (1953) 975-979
12. Colin C., Radeau M., Soquet A., Demolin D., Colin F., and Deltenre P. Mismatch negativity evoked by the McGurk-MacDonald effect: a phonetic representation within short-term memory. Clin. Neurophysiol. 113 4 (2002) 495-506
13. Crowther C.S., Batchelder W.H., and Hu X. A measurement-theoretic analysis of the fuzzy logic model of perception. Psychol. Rev. 102 2 (1995) 396-408
14. Cutting J.E., Bruno N., Brady N.P., and Moore C. Selectivity, scope, and simplicity of models: a lesson from fitting judgments of perceived depth. J. Exp. Psychol. Gen. 121 3 (1992) 364-381
15. Frassinetti F., Bolognini N., and Ladavas E. Enhancement of visual perception by crossmodal visuo-auditory interaction. Exp. Brain Res. 147 3 (2002) 332-343
16. Ghazanfar A.A., Maier J.X., Hoffman K.L., and Logothetis N.K. Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. J. Neurosci. 25 20 (2005) 5004-5012
17. Grant K.W., and Seitz P.F. Measures of auditory-visual integration in nonsense syllables and sentences. J. Acoust. Soc. Amer. 104 4 (1998) 2438-2450
18. Lavie N. Distracted and confused: selective attention under load. Trends Cogn. Sci. 9 2 (2005) 75-82
19. Liberman A.M., and Mattingly I.G. The motor theory of speech perception revised. Cognition 21 1 (1985) 1-36
20. Liberman A.M., and Mattingly I.G. A specialization for speech perception. Science 243 4890 (1989) 489-494
21. Luck S.J., Chelazzi L., Hillyard S.A., and Desimone R. Neural mechanisms of spatial selective attention in areas V1, V2, and V4 of macaque visual cortex. J. Neurophysiol. 77 1 (1997) 24-42
22. Massaro D.W. Children's perception of visual and auditory speech. Child Develop. 55 (1984) 1777-1788
23. Massaro D.W. Speech Perception by Ear and Eye: A Paradigm for Psychological Inquiry (1987), Erlbaum, Hillsdale, NJ
24. Massaro D.W. Perceiving Talking Faces (1998), MIT Press, Cambridge, Massachusetts
25. Massaro D.W., Cohen M.M., and Smeele P.M. Perception of asynchronous and conflicting visual and auditory speech. J. Acoust. Soc. Amer. 100 3 (1996) 1777-1786
26. McGurk H., and MacDonald J. Hearing lips and seeing voices. Nature 264 (1976) 746-748
27. Möttönen R., Krause C.M., Tiippana K., and Sams M. Processing of changes in visual speech in the human auditory cortex. Brain Res. Cogn. Brain Res. 13 3 (2002) 417-425
28. Munhall K.G., Gribble P., Sacco L., and Ward M. Temporal constraints on the McGurk effect. Percept. Psychophys. 58 3 (1996) 351-362
29. Musacchia G., Sams M., Nicol T., and Kraus N. Seeing speech affects acoustic information processing in the human brainstem. Exp. Brain Res. 168 1-2 (2006) 1-10
30. Näätänen R. Attention and Brain Function (1992), Lawrence Erlbaum Associates, Hillsdale, NJ
31. Nishitani N., and Hari R. Viewing lip forms: cortical dynamics. Neuron 36 6 (2002) 1211-1220
32. Pekkola J., Ojanen V., Autti T., Jääskeläinen I., Möttönen R., Tarkiainen A., et al. Primary auditory cortex activation by visual speech: an fMRI study at 3 T. Neuroreport 16 2 (2005) 125-128
33. Pitt M.A. Data fitting and detection theory: reply to Massaro and Oden (1995). J. Exp. Psychol. Learn. Mem. Cogn. 21 4 (1995) 1065-1067
34. Posner M.I. Orienting of attention. Quart. J. Exp. Psychol. 32 1 (1980) 3-25
35. Posner M.I., Snyder C.R., and Davidson B.J. Attention and the detection of signals. J. Exp. Psychol. 109 2 (1980) 160-174
36. Remez R.E., Rubin P.E., Pisoni D.B., and Carrell T.D. Speech perception without traditional speech cues. Science 212 4497 (1981) 947-949
37. Sams M., Aulanko R., Hamalainen M., Hari R., Lounasmaa O.V., Lu S.T., et al. Seeing speech: visual information from lip movements modifies activity in the human auditory cortex. Neurosci. Lett. 127 1 (1991) 141-145
38. Schroeder C.E., and Foxe J.J. The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. Brain Res. Cogn. Brain Res. 14 1 (2002) 187-198
39. Schroeder C.E., and Foxe J. Multisensory contributions to low-level, 'unisensory' processing. Curr. Opin. Neurobiol. 15 4 (2005) 454-458
40. Schwartz J.L. The 0/0 problem in the fuzzy-logical model of perception. J. Acoust. Soc. Amer. 120 4 (2006) 1795-1798
41. Seltzer B., and Pandya D.N. Parietal, temporal, and occipital projections to cortex of the superior temporal sulcus in the rhesus monkey: a retrograde tracer study. J. Comp. Neurol. 343 3 (1994) 445-463
42. Soto-Faraco S., Navarra J., and Alsius A. Assessing automaticity in audiovisual speech integration: evidence from the speeded classification task. Cognition 92 3 (2004) B13-B23
43. Sumby W.H., and Pollack I. Visual contribution to speech intelligibility in noise. J. Acoust. Soc. Amer. 26 2 (1954) 212-215
44. Talsma D., and Woldorff M.G. Selective attention and multisensory integration: multiple phases of effects on the evoked brain activity. J. Cogn. Neurosci. 17 7 (2005) 1098-1114
45. Tiippana K., Andersen T.S., and Sams M. Visual attention modulates audiovisual speech perception. Eur. J. Cogn. Psychol. 16 3 (2004) 457-472
46. Treisman A., and Gelade G. A feature integration theory of attention. Cogn. Psychol. 12 1 (1980) 97-136
47. Tuomainen J., Andersen T.S., Tiippana K., and Sams M. Audio-visual speech perception is special. Cognition 96 1 (2005) B13-B22
48. Vroomen J., and de Gelder B. Crossmodal integration: a good fit is no criterion. Trends Cogn. Sci. 4 2 (2000) 37-38
49. Vroomen J., Bertelson P., and de Gelder B. The ventriloquist effect does not depend on the direction of automatic visual attention. Percept. Psychophys. 63 4 (2001) 651-659
50. Vroomen J., Driver J., and de Gelder B. Is cross-modal integration of emotional expressions independent of attentional resources?. Cogn. Affect Behav. Neurosci. 1 4 (2001) 382-387
51. Warren D.H. Spatial localization under conflict conditions: is there a single explanation?. Perception 8 3 (1979) 323-337