No, There Is No 150 ms Lead of Visual Speech on Auditory Speech, but a Range of Audiovisual Asynchronies Varying from Small Audio Lead to Large Audio Lag

PLoS Computational Biology

Volumn 10, Issue 7, 2014, Pages

  Schwartz, Jean Luc ^a   Savariaux, Christophe ^a  

^a GIPSA LAB   (France)

Author keywords

[No Author keywords available]

Indexed keywords

COORDINATION REACTIONS; SPEECH;

ASYNCHRONY; AUDIO-VISUAL DATA; AUDIO-VISUAL SPEECH; INTEGRATION WINDOW; NATURAL SPEECH; SPEECH MATERIALS; SPEECH PERCEPTION; SPEECH UTTERANCE; TEMPORAL INTEGRATION; VISUAL SPEECH;

LINGUISTICS;

ARTICLE; AUDIOVISUAL ASYNCHRONY; AUDITORY SYSTEM PARAMETERS; CONTROLLED STUDY; GESTURE; MATHEMATICAL MODEL; SENSORIMOTOR INTEGRATION; SOUND DETECTION; SPEECH ANALYSIS; SPEECH PERCEPTION; VISUAL STIMULATION; VISUAL SYSTEM PARAMETERS; BIOLOGICAL MODEL; BIOLOGY; HEARING; HUMAN; MALE; PHYSIOLOGY; SPEECH; VISION;

AUDITORY PERCEPTION; COMPUTATIONAL BIOLOGY; HUMANS; MALE; MODELS, BIOLOGICAL; SPEECH; VISUAL PERCEPTION;

EID: 84905455453     PISSN: 1553734X     EISSN: 15537358     Source Type: Journal    
DOI: 10.1371/journal.pcbi.1003743     Document Type: Article

References (56)

1. Ghazanfar AA, Schroeder CE, (2006) Is neocortex essentially multisensory? Trends Cogn Sci 10: 278-285.
2. Driver J, Noesselt T, (2008) Multisensory interplay reveals crossmodal influences on 'sensory specific' brain regions, neural responses, and judgments. Neuron 57: 11-23.
3. Falchier A, Clavagnier S, Barone P, Kennedy H, (2002) Anatomical evidence of multimodal integration in primate striate cortex. J Neurosci 22: 5749-5759.
4. Besle J, Fort A, Delpuech C, Giard M-H, (2004) Bimodal Speech: Early Visual Effect in the Human Auditory Cortex. Eur J Neurosci 20: 2225-2234.
5. Van Wassenhove V, Grant KW, Poeppel D, (2005) Visual speech speeds up the neural processing of auditory speech. PNAS 102: 1181-1186.
6. Ghazanfar AA, Chandrasekaran C, Logothetis NK, (2008) Interactions between the Superior Temporal Sulcus and Auditory Cortex Mediate Dynamic Face/ Voice Integration in Rhesus Monkeys. J Neurosci 28: 4457-4469.
7. Arnal LH, Morillon B, Kell CA, Giraud A-L, (2009) Dual neural routing of visual facilitation in speech processing. J Neurosci 29: 13445-53.
8. Arnal LH, Wyart V, Giraud A-L, (2011) Transitions in neural oscillations reflect prediction errors generated in audiovisual speech. Nat Neurosci doi:10.1038/nn.2810.
9. Lakatos P, Chen C-M, O'Connell MN, Mills A, Schroeder CE, (2007) Neuronal Oscillations and Multisensory Interaction in Primary Auditory Cortex. Neuron 53: 279-292.
10. Schroeder CE, Lakatos P, Kajikawa Y, Partan S, Puce A, (2008) Neuronal oscillations and visual amplification of speech. Trends Cogn Sci 12: 106-113.
11. Luo H, Liu Z, Poeppel D, (2010) Auditory cortex tracks both auditory and visual stimulus dynamics using low-frequency neuronal phase modulation. PLoS Biol 8: e1000445.
12. Zion Golumbic E, Cogan GB, Schroeder CE, Poeppel D, (2013) Visual input enhances selective speech envelope tracking in auditory cortex at a "cocktail party". J Neurosci 33: 1417-26.
13. Friston KJ, (2005) A theory of cortical responses. Philos Trans R Soc Lond B Biol Sci 360: 815-836.
14. Friston KJ, Kiebel SJ, (2009) Cortical circuits for perceptual inference. Neural Networks 22: 1093-1104.
15. Arnal LH, Giraud A-L, (2012) Cortical oscillations and sensory predictions. Trends Cogn Sci 16: 390-398.
16. Yehia H, Rubin P, Vatikiotis-Bateson E, (1998) Quantitative association of vocal tract and facial behavior. Speech Comm 26: 23-43.
17. Barker J P, Berthommier F (1999) Evidence of correlation between acoustic and visual features of speech. Proc ICPhS '99(pp. 199-202). San Francisco: USA.
18. Grant KW, Seitz P, (2000) The use of visible speech cues for improving auditory detection of spoken sentences. J. Acoust. Soc. Am. 108: 1197-1208.
19. Jiang J, Alwan A, Keating P, Auer ET Jr, Bernstein LE, (2002) On the Relationship between Face Movements, Tongue Movements, and Speech Acoustics. Eurasip J Adv Sig Proc 11: 1174-1188.
20. Berthommier F, (2004) A phonetically neutral model of the low-level audiovisual interaction. Speech Comm 44: 31-41.
21. Chandrasekaran C, Trubanova A, Stillittano S, Caplier A, Ghazanfar AA, (2009) The natural statistics of audiovisual speech. PLoS Comput Biol 5: e1000436.
22. Musacchia C, Schroeder CE, (2009) Neuronal mechanisms, response dynamics and perceptual functions of multisensory interactions in auditory cortex. Hear Res 258: 72-79.
23. Blumstein SE, Stevens KN, (1980) Perceptual invariance and onset spectra for stop consonants in different vowel environments. J. Acoust. Soc. Am. 67(2): 648-662.
24. Löfqvist A (1995) Laryngeal mechanisms and interarticulator timing in voiceless consonant production. In Bell-Berti F, Raphael L, Eds. Producing Speech: Contemporary Issues (pp. 99-116). NY: AIP Press Woodbury.
25. Van Wassenhove V, Grant KW, Poeppel D, (2007) Temporal window of integration in auditory-visual speech perception. Neuropsychologia 45: 598-607.
26. Lallouache TM (1990) Un poste 'visage-parole'. Acquisition et traitement de contours labiaux (A "face-speech" workstation. Acquisition and processing of labial contours). In Proceedings XVIIIèmes Journées d'Études sur la Parole: 282-286.
27. Noiray A, Cathiard M-A, Abry C, Ménard L, Savariaux C (2008) Emergence of a vowel gesture control: Attunement of the anticipatory rounding temporal pattern in French children. In Kern S, Gayraud F & Marsico E (Eds.) Emergence of Language Abilities (pp. 100-117). Newcastle: Cambridge Scholars Pub.
28. Boersma P, Weenink D (2012) Praat: doing phonetics by computer (Version 5.3.04) [Computer program]. Retrieved May 2012, from http://www.praat.org.
29. Stekelenburg JJ, Vroomen J, (2007) Neural correlates of multisensory integration of ecologically valid audiovisual events. J Cog Neurosci 19: 1964-1973.
30. Vroomen J, Stekelenburg JJ, (2010) Visual anticipatory information modulates multisensory interactions of artificial audiovisual stimuli. J Cog Neurosci 22: 1583-1596.
31. Abry C, Lallouache TM, (1995) Modeling lip constriction anticipatory behaviour for rounding in French with the MEM. Proc. ICPhS' 95: 152-155.
32. Benguerel AP, Cowan HA, (1974) Coarticulation of upper lip protrusion in French. Phonetica 30: 41-55.
33. Bell-Berti F, Harris KS, (1982) Temporal patterns of coarticulation: Lip rounding. J. Acoust. Soc. Am 71: 449-459.
34. Perkell JS, Matthies LM, (1992) Temporal measures of anticipatory labial coarticulation for the vowel/u/: Within- and cross-subject variability. J. Acoust. Soc. Am 91: 2911-2925.
35. Abry C, Lallouache TM, Cathiard M-A (1996) How can coarticulation models account for speech sensitivity to audio-visual desynchronization? In Stork D and Hennecke M (Eds.) Speechreading by Humans and Machines, NATO ASI Series F (vol. 150, pp. 247-255). Berlin: Springer-Verlag.
36. Roy J-R, Sock R, Vaxelaire B, Hirsch F (2003) Auditory effects of anticipatory and carryover coarticulation. In Proc. 6th Int. Sem. Speech Production, Macquarie Series for Cognitive Sciences: 243-248.
37. Troille E, Cathiard M-A, Abry C, (2010) Speech face perception is locked to anticipation in speech production. Speech Comm 52: 513-524.
38. Feldhoffer G, Bárdi T, Takács G, Tihanyi A (2007) Temporal asymmetry in relations of acoustic and visual features of speech. Proc. 15th European Signal Processing Conf., Poznan.
39. Czap L, (2011) On the audiovisual asynchrony of speech. Proc. AVSP' 2011: 137-140.
40. Bernstein LE, Takayanagi S, Auer ET Jr, (2004) Auditory speech detection in noise enhanced by lipreading. Speech Comm 44: 5-18.
41. Kim J, Davis C, (2004) Investigating the audio-visual speech detection advantage. Speech Comm 44: 19-30.
42. Vroomen J, Keetels M, (2010) Perception of intersensory synchrony: a tutorial review. Attention, Perception, & Psychophysics 72: 871-884.
43. Stevenson RA, Zemtsov RK, Wallace MT, (2012) Individual differences in the multisensory temporal binding window predict susceptibility to audiovisual illusions. J Exp Psychol Human 38: 1517-1529.
44. Massaro DW, Cohen MM, Smeele PM, (1996) Perception of asynchronous and conflicting visual and auditory speech. J. Acoust. Soc. Am 100: 1777-1786.
45. Munhall KG, Gribble P, Sacco L, Ward M, (1996) Temporal constraints on the McGurk effect. Perception and Psychophysics 58: 351-362.
46. Welch RB, Warren DH, (1980) Immediate perceptual response to intersensory discrepancy. Psychol Bull 88: 638-667.
47. Vatakis A, Spence C, (2007) Crossmodal binding: Evaluating the 'unity assumption' using audiovisual speech stimuli. Perception & Psychophysics 69: 744-756.
48. Lewkowicz DJ, Flom R, (2014) The audio-visual temporal binding window narrows in early childhood. Child Development 85: 685-694.
49. Hillock AR, Powers AR, Wallace MT, (2011) Binding of sights and sounds: age-related changes in multisensory temporal processing. Neuropsychologia 49: 461-467.
50. Sekiyama K, Burnham D, (2008) Impact of language on development of auditory-visual speech perception. Dev Sci 11: 303-317.
51. Giard MH, Peronnet F, (1999) Auditory-visual integration during multimodal object recognition in humans: A behavioral and electrophysiological study. J Cog Neurosci 11: 473-490.
52. Schwartz JL, (2010) A reanalysis of McGurk data suggests that audiovisual fusion in speech perception is subject-dependent. J. Acoust. Soc. Am 127: 1584-1594.
53. Powers AR, (2009) Perceptual training narrows the temporal window of multisensory binding. J Neurosci 29: 12265-12274.
54. Petrini K, Dahl S, Rocchesso D, Waadeland C, Avanzini F, et al. (2009) Multisensory integration of drumming actions: musical expertise affects perceived audiovisual asynchrony. Experimental Brain Research 198: 339-352.
55. Schwartz JL, Savariaux C (2013) Data and simulations about audiovisual asynchrony and predictability in speech perception. Proc. AVSP'2013, 147-152.
56. Cummins F, (2012) Oscillators and syllables: a cautionary note. Front Psychol 3: 364.