The influence of selective attention to auditory and visual speech on the integration of audiovisual speech information

Perception

Volumn 40, Issue 10, 2011, Pages 1164-1182

  Buchan, Julie N ^a   Munhall Kevin G, K G ^a  

^a QUEEN S UNIVERSITY   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 82455167101     PISSN: 03010066     EISSN: None     Source Type: Journal    
DOI: 10.1068/p6939     Document Type: Article

References (69)

1. Alais D, Burr D, 2004 "The ventriloquist effect results from near-optimal bimodal integration" Current Biology 14 257-262
2. Alsius A, Navarra J, Campbell R, Soto-Faraco S, 2005 "Audiovisual integration of speech falters under high attentional demands" Current Biology 15 839-843
3. Alsius A, Navarra J, Soto-Faraco S, 2007 "Attention to touch weakens audiovisual speech inte-gration" Experimental Brain Research 183 399-404
4. Andersen T S, Tiippana K, Laarni J, Kojo I, Sams M, 2009 "The role of visual spatial attention in audiovisual speech perception" Speech Communication 51 184-193
5. Benoît C, Guiard-Marigny T, Le Groff B, Adjoudani A, 1996 "Which components of the face do humans and machines best speechread?", in Speechreading by Humans and Machines: Models Systems & Applications Eds D G Stork, M Hennecke (Hague: Kluwer) pp 315-328
6. Besle J, Fort A, Delpuech C, Giard M, 2007 "Bimodal speech: early suppressive visual effects in human auditory cortex" European Journal of Neuroscience 20 2225-2234
7. Binnie C A, Montgomery A A, Jackson P L, 1974 "Auditory and visual contributions to the perception of consonants" Journal of Speech and Hearing Research 17 619-630
8. Buchan J N, Parë M, Munhall K G, 2007 "Spatial statistics of gaze fixations during dynamic face processing" Social Neuroscience 2 1-13
9. Buchan J N, Wilson A, Parë M, Munhall K G, 2005 "The effect of spatial frequency information in central and peripheral vision on natural gaze patterns and audiovisual speech perception" Journal of the Acoustical Society of America 117 2620
10. Colin C, Radeau M, Soquet A, Deltenre P, 2004 "Generalization of the generation of an MMN by illusory McGurk percepts voiceless consonants" Clinical Neurophysiology 115 1989-2000
11. Colin C, Radeau M, Soquet A, Demolin D, Colin F, Deltenre P, 2002 "Mismatch negativity evoked by the McGurk-MacDonald effect: A phonetic representation within short-term memory" Clinical Neurophysiology 113 495-506
12. Conrey B, Pisoni D B, 2006 "Auditory-visual speech perception and synchrony detection for speech and non-speech signals" Journal of the Acoustical Society of America 119 4065-4073
13. Dixon N, Spitz L, 1980 "The detection of audiovisual desynchrony" Perception 9 719-721
14. Driver J, 1996 "Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading" Nature 381 66-68
15. Erber N P, 1969 "Interaction of audition and vision in the recognition of oral speech stimuli" Journal of Speech and Hearing Research 12 423-425
16. Ernst M O, Banks M S, 2002 "Humans integrate visual and haptic information in a statistically optimal fashion" Nature 415 429-433
17. Findlay J M, Gilchrist I D, 2003 Active Vision: The Psychology of Looking and Seeing (London: Oxford University Press)
18. Grant K W, Wassenhove V van, Poeppel D, 2004 "Detection of auditory (cross-spectral) and auditory-visual (cross-modal) synchrony" Speech Communication 44 43-53
19. Green K P, Kuhl P K, Meltzoff A N, Stevens E B, 1991 "Integrating speech information across talkers gender and sensory modality: female faces and male voices in the McGurk effect" Perception & Psychophysics 50 524-536
20. Henderson J M, 2003 "Human gaze control during real-world scene perception" Trends in Cognitive Science 7 498-504
21. Henderson J M, Brockmole J R, Castelhano M S, Mack M, 2007 "Visual saliency does not account for eye movements during visual search in real-world scenes", in Eye Movement Research: Insights into Mind and Brain Eds R van Rompel, M Fischer, W Murray, R Hill (Amsterdam: Elsevier) pp 537-562
22. Henderson J M, Weeks P A Jr, Hollingworth A, 1999 "The effects of semantic consistency on eye movements during scene viewing" Journal of Experimental Psychology: Human Perception and Performance 25 210-228
23. Hillis J M, Ernst M O, Banks M S, Landy M S, 2002 "Combining sensory information: mandatory fusion within, but not between, senses" Science 298 1627-1630
24. Jones J A, Jarick M, 2006 "Multisensory integration of speech signals: the relationship between space and time" Experimental Brain Research 174 588-594
25. Jordan T R, Sergeant P C, 1998 "Effects of facial image size on visual and audio-visual speech recognition", in Hearing by Eye II: Advances in the Psychology of Speechreading and Auditory-Visual Speech Eds R Campbell, B Dodd, D K Burnham (Hove, Sussex: Psychology Press) pp 155-176
26. Kamachi M, Hill H, Lander K, Vatikiotis-Bateson E, 2003 " 'Putting the face to the voice': matching identity across modality" Current Biology 13 1709-1714
27. Kislyuk D S, Mottonen R, Sams M, 2008 "Visual processing affects the neural basis of auditory discrimination" Journal of Cognitive Neuroscience 20 2175-2184
28. Lansing C R, McConkie G W, 1999 "Attention to facial regions in segmental and prosodic visual speech perception tasks" Journal of Speech, Language, and Hearing R42 526-539
29. Liberman A M, 1982 "On the finding that speech is special" American Psychologist 37 148-167
30. MacDonald J, Andersen S, Bachmann T, 2000 "Hearing by eye: how much spatial degradation can be tolerated?" Perception 29 1155-1168
31. McGurk H, MacDonald J, 1976 "Hearing lips and seeing voices" Nature 264 746-748
32. Massaro D W, 1987 Speech Perception by Ear and Eye: A Paradigm for Psychological Inquiry (Hillsdale, NJ: Lawrence Erlbaum Associates)
33. Miller L M, D'Esposito M, 2005 "Perceptual fusion and stimulus coincidence in the cross-modal integration of speech" Journal of Neuroscience 25 5884-5893
34. Mozolic J L, Hugenschmidt C E, Peiffer A M, Laurienti P J, 2008 "Modality-specific selective attention attenuates multisensory integration" Experimental Brain Research 184 39-51
35. Munhall K G, Gribble P, Sacco L, Ward M, 1996 "Temporal constraints on the McGurk effect" Perception & Psychophysics 58 351-362
36. Munhall K G, Kroos C, Jozan G, Vatikiotis-Bateson E, 2004 "Spatial frequency requirements of audiovisual speech perception" Perception & Psychophysics 66 574-538
37. Munhall K G, Hove M W ten, Brammer M, Parë M, 2009 "Audiovisual integration of speech in a bistable illusion" Current Biology 19 735-739
38. Naatanen R, 1991 "Mismatch negativity (MMN) outside strong attentional focus: a commentary on Woldorff et al" Psychophysiology 28 478-484
39. Naatanen R, 1999 "The perception of speech sounds by the human brain as reflected by the mismatch negativity MMN and its magnetic equivalent MMNm" Psychophysiology 38 1-21
40. Naatanen R, 2000 "Mismatch negativity (MMN): perspectives for application" International Journal of Psychophysiology 37 3-10
41. Navarra J, Alsius A, Soto-Faraco S, Spence C, 2010 "Assessing the role of attention in the audio-visual integration of speech" Information Fusion 11 4-11
42. O'Neill J J, 1954 "Contributions of the visual components of oral symbols to speech comprehen-sion" Journal of Speech and Hearing Disorders 19 429-439
43. Parë M, Richler R C, Hove M ten, Munhall K G, 2003 "Gaze behavior in audiovisual speech perception: The influence of ocular fixations on the McGurk effect" Perception & Psychophysics 65 533-567
44. Parkhurst D, Law K, Neibur E, 2002 "Modeling the role of salience in the allocation of overt visual attention" Vision Research 42 107-123
45. Parkhurst D J, Neibur E, 2003 "Scene content selected by active vision" Spatial Vision 16 125-154
46. Pilling M, 2009 "Auditory event-related potentials (ERPs) in audiovisual speech perception" Journal of Speech, Language, and Hearing Research 52 1073-1081
47. Posner M I, 1980 "Orienting of attention" Quarterly Journal of Experimental Psychology 32 3-25
48. Radeau M, Bertelson P, 1974 "The after-effects of ventriloquism" Quarterly Journal of Experimental Psychology 26 63-71
49. Reisberg D, McLean J, Goldfield A, 1987 "Easy to hear but hard to understand: a lip-reading advantage with intact auditory stimuli", in Hearing by Eye: The Psychology of Lip-reading Eds B Dodd, R Campbell (Hillsdale, NJ: Lawrence Erlbaum Associates) pp 97-137
50. Remez R E, Rubin P E, Pisoni D B, Carrell T D, 1981 "Speech perception without traditional speech cues" Science 212 947-949
51. Rosenblum L D, Saldana H M, 1996 "An audiovisual test of kinematic primitives for visual speech perception" Journal of Experimental Psychology: Human Perception and Performance 22 318-331
52. Saint-Amour D, Sanctis P D, Molholm S, Ritter W, Foxe J J, 2007 "Seeing voices: high-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion" Neuropsychologia 45 587-597
53. Sams M, Aulanko R, Hamalainen M, Hari R, Lounasmaa O V, Lu S T, Simola J, 1991 "Seeing speech: Visual information from lip movements modifies activity in the human auditory cortex" Neuroscience Letters 127 141-145
54. Soto-Faraco S, Alsius A, 2007 "Conscious access to the unisensory components of a cross-modal illusion" NeuroReport 18 347-350
55. Soto-Faraco S, Navarra J, Alsius A, 2004 "Assessing automaticity in audiovisual speech integra-tion: evidence from the speeded classification task" Cognition 92 B13-B23
56. Sumby W H, Pollack I, 1954 "Visual contributions to speech intelligibility in noise" Journal of the Acoustical Society of America 26 212-215
57. Summerfield Q, McGrath M, 1984 "Detection and resolution of audio-visual incompatibility in the perception of vowels" Quarterly Journal of Experimental Psychology A 36 51-74
58. Thomas S M, Jordan T R, 2002 "Determining the influence of Gaussian blurring on inversion effects with talking faces" Perception & Psychophysics 64 932-944
59. Thomas S M, Jordan T R, 2004 "Contributions of oral and extraoral facial movement to visual and audiovisual speech perception" Journal of Experimental Psychology: Human Perception and Performance 30 873-888
60. Tiipana K, Andersen T S, Sams M, 2004 "Visual attention modulates audiovisual speech percep-tion" European Journal of Cognitive Psychology 16 457-472
61. Tuomainen J, Andersen T S, Tiippana K, Sams M, 2005 "Audio-visual speech perception is special" Cognition 96 B13-B22
62. Vatakis A, Ghazanfar A A, Spence C, 2008 "Facilitation of multisensory integration by the 'unity effect' reveals that speech is special" Journal of Vision 8(9):14, 1-11
63. Vatakis A, Spence C, 2007 "Crossmodal binding: evaluating the 'unity assumption' using audio-visual speech stimuli" Perception & Psychophysics 69 744-759
64. Vatikiotis-Bateson E, Eigsti I-M, Yano S, Munhall K G, 1998 "Eye movements of perceivers during audiovisual speech perception" Perception & Psychophysics 60 926-940
65. Walker S, Bruce V, O'Malley C, 1995 "Facial identity and facial speech processing: familiar faces and voices in the McGurk effect" Perception & Psychophysics 57 1124-1133
66. Wassenhove V van, Grant K, Poeppel D, 2005 "Visual speech speeds up the neural processing of auditory speech" Proceedings of the National Academy of Sciences of the USA 102 1181-1186
67. Wassenhove V van, Grant K, Poeppel D, 2007 "Temporal window of integration in auditory-visual speech perception" Neuropsychologia 45 598-607
68. Wozny D R, Beierholm U R, Shams L, 2008 "Human trimodal perception follows optimal statistical inference" Journal of Vision 8(3):24, 1-11
69. Yarbus A, 1967 Eye Movements and Vision (New York: Plenum Press)