Assessing the role of attention in the audiovisual integration of speech

Information Fusion

Volumn 11, Issue 1, 2010, Pages 4-11

  Navarra, Jordi ^a,b   Alsius, Agnès ^c   Soto Faraco, Salvador ^d,e   Spence, Charles ^b  

^a Esplugues de Llobregat   (Spain)

^b UNIVERSITY OF OXFORD   (United Kingdom)

^c UNIVERSITY OF BARCELONA   (Spain)

^d INSTITUCIÓ CATALANA DE RECERCA I ESTUDIS AVANÇATS ICREA   (Spain)

^e UNIVERSITAT POMPEU FABRA   (Spain)

Author keywords

Attention; Automaticity; Multisensory integration; Speech perception

Indexed keywords

ATTENTION; AUDIO-VISUAL INTEGRATION; AUDIO-VISUAL SPEECH; AUTOMATICITY; CONTROVERSIAL TOPICS; HUMAN COMMUNICATIONS; MULTISENSORY INTEGRATION; NEW TECHNOLOGIES; SENSORY MODALITY; SENSORY SIGNALS; SPEECH PERCEPTION; VISUAL INFORMATION;

COMMUNICATION SYSTEMS; VISUAL COMMUNICATION;

INTEGRATION;

EID: 68349154634     PISSN: 15662535     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.inffus.2009.04.001     Document Type: Article

References (87)

1. Alais D., Morrone C., and Burr D. Separate attentional resources for vision and audition. Proceedings of the Royal Society B: Biological Sciences 273 (2006) 339-345
2. Alsius A., Navarra J., Campbell R., and Soto-Faraco S. Audiovisual integration of speech falters under high attention demands. Current Biology 15 (2005) 839-843
3. Alsius A., Navarra J., and Soto-Faraco S. Attention to touch reduces audiovisual speech integration. Experimental Brain Research 183 (2007) 399-404
4. Basil M.D. Multiple resource theory I: application to television viewing. Communication Research 21 (1994) 177-207
5. Bertelson P. Ventriloquism: A case of cross-modal perceptual grouping. In: Aschersleben G., Bachmann T., and Muesseler J. (Eds). Cognitive Contributions to the Perception of Spatial and Temporal Events (1999), Elsevier, Amsterdam 347-362
6. Bertelson P., Vroomen J., de Gelder B., and Driver J. The ventriloquist effect does not depend on the direction of deliberate visual attention. Perception and Psychophysics 62 (2000) 321-332
7. Calvert G.A., Bullmore E.T., Brammer M.J., Campbell R., Williams S.C., McGuire P.K., Woodruff P.W., Iversen S.D., and David A.S. Activation of auditory cortex during silent lipreading. Science 276 (1997) 593-596
8. Calvert G.A., Campbell R., and Brammer M.J. Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Current Biology 10 (2000) 649-657
9. Cinel C., Humphreys G.W., and Poli R. Cross-modal illusory conjunctions between vision and touch. Journal of Experimental Psychology: Human Perception and Performance 28 (2002) 1243-1266
10. Colin C., Radeau M., Soquet A., and Deltenre P. Generalization of the generation of an MMN by illusory McGurk percepts: voiceless consonants. Clinical Neurophysiology 115 (2004) 495-506
11. Colin C., Radeau M., Soquet A., Demolin D., Colin F., and Deltenre P. Mismatch negativity evoked by the McGurk-MacDonald effect: a phonetic representation within short-term memory. Clinical Neurophysiology 113 (2002) 495-506
12. Colin C., Radeau M., and Deltenre P. Top-down and bottom-up modulation of audiovisual integration in speech. European Journal of Cognitive Psychology 17 4 (2005) 541-560
13. de Gelder B., and Bertelson P. Multisensory integration, perception and ecological validity. Trends in Cognitive Sciences 7 (2003) 460-467
14. Fujisaki W., Koene A., Arnold D., Johnston A., and Nishida S. Visual search for a target changing in synchrony with an auditory signal. Proceedings of the Royal Society B: Biological Sciences 273 (2006) 865-874
15. Gentilucci M., and Cattaneo L. Automatic audiovisual integration in speech perception. Experimental Brain Research 167 (2005) 66-75
16. Giard M.-H., Fort A., Mouchetant-Rostaing Y., and Pernier J. Neurophysiological mechanisms of auditory selective attention in humans. Frontiers in Bioscience 5 (2000) 84-94
17. Green K.P., Kuhl P.K., Meltzoff A.N., and Stevens E.B. Integrating speech information across talkers, gender, and sensory modality: female faces and male voices in the McGurk effect. Perception and Psychophysics 50 (1991) 524-536
18. Grimes T. Mild auditory-visual dissonance in television news may exceed viewer attentional capacity. Human Communication Research 18 (1991) 268-298
19. Hardison D.M. Bimodal speech perception by native and nonnative speakers of English: factors influencing the McGurk effect. Language Learning 46 (1996) 3-73
20. Jones J.A., and Munhall K.G. The effects of separating auditory and visual sources on audiovisual integration of speech. Canadian Acoustics 25 (1997) 13-19
21. Kaiser J., Hertrich I., Ackermann H., and Lutzenberger W. Gamma-band activity over early sensory areas predicts detection of changes in audiovisual speech stimuli. Neuroimage 30 (2006) 1376-1382
22. Kastner S. Attentional response modulation in the human visual system. In: Posner M.I. (Ed). Cognitive Neuroscience of Attention (2004), Guilford Press, New York, NY 144-156
23. D. Kislyuk, Visual speech affects discrimination of syllables in the auditory cortex: an MMN study, in: Paper Presented at the 7th International Multisensory Research Forum (IMRF), Dublin, Ireland, June 18th-21st, 2006.
24. Klucharev V., Möttönen R., and Sams M. Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Cognitive Brain Research 18 (2003) 65-75
25. Koppen C., and Spence C. Audiovisual asynchrony modulates the Colavita visual dominance effect. Brain Research 1186 (2007) 224-232
26. Lavie N. Perceptual load as a necessary condition for selective attention. Journal of Experimental Psychology: Human Perception and Performance 21 (1995) 451-468
27. Lavie N. Distracted and confused? Selective attention under load. Trends in Cognitive Sciences 9 (2005) 75-82
28. Liberman A.M., and Mattingly I.G. The motor theory of speech perception revised. Cognition 21 (1985) 1-36
29. Macaluso E., George N., Dolan R., Spence C., and Driver J. Spatial and temporal factors during processing of audiovisual speech perception: a PET study. Neuroimage 21 (2004) 725-732
30. Manuel S.Y., Repp B., Studdert-Kennedy M., and Liberman A. Exploring the "McGurk effect". Journal of the Acoustical Society of America 74 (1983) S66
31. Marks L.E. Cross-modal interactions in speeded classification. In: Calvert G.A., Spence C., and Stein B.E. (Eds). The Handbook of Multisensory Processes (2004), MIT Press, Cambridge, MA 85-105
32. Massaro D.W. Children's perception of visual and auditory speech. Child Development 55 (1984) 1777-1788
33. Massaro D.W. Speech Perception by Ear and Eye: A Paradigm for Psychological Inquiry (1987), Lawrence Erlbaum Associates, Hillsdale, NJ (pp. 66-74)
34. Massaro D.W. Perceiving talking faces: from speech perception to a behavioral principle (1998), MIT Press, Cambridge, MA
35. McGurk H., and MacDonald J. Hearing lips and seeing voices. Nature 265 (1976) 746-748
36. Milton J., and Arnold J. Introduction to Probability and Statistics (1995), McGraw-Hill, Maidenhead, UK
37. Möttönen R., Krause C.M., Tiippana K., and Sams M. Processing of changes in visual speech in the human auditory cortex. Cognitive Brain Research 13 (2002) 417-425
38. Munhall K.G., Gribble P., Sacco L., and Ward M. Temporal constraints on the McGurk effect. Perception and Psychophysics 58 (1996) 351-362
39. Näätänen R., and Alho K. Mismatch negativity - a unique measure of sensory processing in audition. International Journal of Neuroscience 80 (1995) 317-337
40. J. Navarra, Visual speech interference in an auditory shadowing task: the dubbed movie effect, in: 15th International Congress of Phonetic Sciences Proceedings, Barcelona, Spain, 2003, pp. 893-896.
41. J. Navarra, A. Alsius, S. Soto-Faraco, C. Spence, Prior linguistic experience modulates the perception of audiovisual synchrony for speech signals, submitted for publication.
42. Navarra J., and Soto-Faraco S. Hearing lips in a second language: visual articulatory information enables the perception of L2 sounds. Psychological Research 71 (2005) 4-12
43. Ogmen H., Breitmeyer B.G., Todd S., and Mardon L. Target recovery in metacontrast: the effect of contrast. Vision Research 46 (2006) 4726-4734
44. C. Pallier, Rôle de la syllable dans la perception de la parole: Études attentionnelles. [The role of the syllable in the perception of words: Attentional studies.] Unpublished Ph.D. dissertation presented at the École des Hautes Études en Sciences Sociales, Paris, 1994.
45. Rees G., Frith C., and Lavie N. Processing of irrelevant visual motion during performance of an auditory attention task. Neuropsychologia 39 (2001) 937-949
46. Reisberg D., McLean J., and Goldfield A. Easy to hear but hard to understand: a lip-reading advantage with intact auditory stimuli. In: Dodd B., and Campbell R. (Eds). Hearing by Eye: The Psychology of Lip-reading (1987), LEA, London 97-113
47. Restuccia D., Marca G., Marra C., Rubino M., and Valeriani M. Attentional load of the primary task influences the frontal but not the temporal generators of mismatch negativity. Cognitive Brain Research 25 (2005) 891-899
48. Ritter W., Deacon D., Gomes H., Javitt D.C., and Vaughan Jr. H.G. The mismatch negativity of event-related potentials as a probe of transient auditory memory: a review. Ear and Hearing 16 (1995) 52-67
49. Rosenblum L.D., and Saldaña H.M. An audiovisual test of kinematic primitives for visual speech perception. Journal of Experimental Psychology: Human Perception and Performance 22 (1996) 318-331
50. Ross L.A., Saint-Amour D., Leavitt V.M., Javitt D.C., and Foxe J.J. Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. Cerebral Cortex 17 (2007) 1147-1153
51. Saint-Amour D., De Sanctis P., Molholm S., Ritter W., and Foxe J.J. Seeing voices: high-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion. Neuropsychologia 45 (2007) 587-597
52. Sams M., Aulanko R., Hämäläinen M., Hari R., Lounasmaa O.V., Lu S.T., and Simola J. Seeing speech: visual information from lip movements modifies activity in the human auditory cortex. Neuroscience Letters 127 (1991) 141-145
53. Santangelo V., and Spence C. Multisensory cues capture spatial attention regardless of perceptual load. Journal of Experimental Psychology: Human Perception and Performance 33 (2007) 1311-1321
54. Santangelo V., and Spence C. Is the exogenous orienting of spatial attention truly automatic? Evidence from unimodal and multisensory studies. Consciousness and Cognition 17 (2008) 989-1015
55. Shore D.I., Barnes M.E., and Spence C. Temporal aspects of the visuotactile congruency effect. Neuroscience Letters 392 (2006) 96-100
56. Soto-Faraco S., and Alsius A. Conscious access to the unisensory components of a crossmodal illusion. Neuroreport 18 (2007) 347-350
57. Soto-Faraco S., and Alsius A. Deconstructing the McGurk-MacDonald illusion. Journal of Experimental Psychology: Human Perception and Performance 35 (2009) 580-587
58. Soto-Faraco S., Navarra J., and Alsius A. Assessing automaticity in audiovisual speech integration: evidence from the speeded classification task. Cognition 92 (2004) B13-B23
59. Soto-Faraco S., and Spence C. Modality specific auditory and visual temporal processing deficits. Quarterly Journal of Experimental Psychology 55A (2002) 23-40
60. Soto-Faraco S., Spence C., Fairbank C., Kingstone A., Hillstrom A., and Shapiro K. A crossmodal attentional blink between vision and touch. Psychonomic Bulletin and Review 9 (2002) 731-738
61. Spence C. Audiovisual multisensory integration. Acoustical Science and Technology 28 (2007) 61-70
62. Spence C., Nicholls M.E.R., and Driver J. The cost of expecting events in the wrong sensory modality. Perception and Psychophysics 63 (2001) 330-336
63. Sumby W., and Pollack I. Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America 26 (1954) 212-215
64. Summerfield Q. Some preliminaries to a comprehensive account of audio-visual speech perception. In: Dodd B., and Campbell R. (Eds). Hearing by Eye: The Psychology of Lip-reading (1987), LEA, London, UK 3-51
65. Summerfield Q., and McGrath M. Detection and resolution of audiovisual incompatibility perception in vowels. Quarterly Journal of Experimental Psychology 36A (1984) 51-74
66. Talsma D., Doty T.J., and Woldorff M.G. Selective attention and audiovisual integration: Is attending to both modalities a prerequisite for early integration?. Cerebral Cortex 17 (2007) 679-690
67. Talsma D., and Woldorff M.G. Selective attention and multisensory integration: multiple phases of effects on the evoked brain activity. Journal of Cognitive Neuroscience 17 (2005) 1098-1114
68. Tellinghuisen D.J., and Nowak E.J. The inability to ignore auditory distractors as a function of visual task perceptual load. Perception and Psychophysics 65 (2003) 817-828
69. Tiippana K., Andersen T.S., and Sams M. Visual attention modulates audiovisual speech perception. European Journal of Cognitive Psychology 16 (2004) 457-472
70. Treisman A.M. The binding problem. Current Opinion in Neurobiology 6 (1996) 171-178
71. Treisman A.M., and Davies A. Divided attention to ear and eye. In: Kornblum S. (Ed). Attention and Performance IV (1973), Academic Press, New York 101-117
72. Treisman A.M., and Gelade G. A feature-integration theory of attention. Cognitive Psychology 12 (1980) 97-136
73. Tuomainen J., Andersen T.S., Tiippana K., and Sams M. Audio-visual speech is special. Cognition 96 (2005) B13-B22
74. van Wassenhove V., Grant K.W., and Poeppel D. Visual speech speeds up the neural processing of auditory speech. Proceedings of the National Academy of Sciences USA 102 (2005) 1181-1186
75. A. Vatakis, A. Ghazanfar, C. Spence, Facilitation of multisensory integration by the 'unity assumption': Is speech special? Journal of Vision 8 (Article 14) (2008) 1-11.
76. Vatakis A., and Spence C. Evaluating the influence of frame rate on the temporal aspects of audiovisual speech perception. Neuroscience Letters 405 (2006) 132-136
77. Vatakis A., and Spence C. Audiovisual synchrony perception for speech and music assessed using a temporal order judgment task. Neuroscience Letters 393 (2006) 40-44
78. Vatakis A., and Spence C. Crossmodal binding: evaluating the 'unity assumption' using audiovisual speech and non-speech stimuli. Perception and Psychophysics 69 (2007) 744-756
79. Vatakis A., and Spence C. Evaluating the influence of the 'unity assumption' on the temporal perception of realistic audiovisual stimuli. Acta Psychologica 127 (2008) 12-23
80. A. Vatakis, C. Spence, Assessing the effect of physical differences in the articulation of consonants and vowels on audiovisual temporal perception, submitted for publication.
81. Vroomen J., Bertelson P., and de Gelder B. The ventriloquist effect does not depend on the direction of automatic visual attention. Perception and Psychophysics 63 (2001) 651-659
82. Vroomen J., Driver J., and de Gelder B. Is cross-modal integration of emotional expressions independent of attentional resources?. Cognitive, Affective and Behavioral Neuroscience 1 (2001) 382-387
83. Walker S., Bruce V., and O'Malley C. Facial identity and facial speech processing: familiar faces and voices in the McGurk effect. Perception and Psychophysics 57 (1995) 1124-1133
84. Welch R.B. Meaning, attention, and the "unity assumption" in the intersensory bias of spatial and temporal perceptions. In: Ashersleben G., Bachmann T., and Müsseler J. (Eds). Cognitive contributions to the perception of spatial and temporal events (1999), Elsevier Science, Amsterdam, The Netherlands 371-387
85. Wickens C.D. Processing resources in attention. In: Parasuraman R., and Davies D.R. (Eds). Varieties of Attention (1984), Academic Press, San Diego, CA 63-102
86. Wickens C.D. Engineering psychology and human performance. second ed. (1992), Harper & Collins, NY
87. Wojciulik E., and Kanwisher N. Implicit but not explicit feature binding in a Balint's patient. In: Schneider W.X., and Maasen S. (Eds). Mechanisms of Visual Attention: A Cognitive Neuroscience Perspective (1998), Psychology Press, Hove, East Sussex 157-181