Audio-visual speech scene analysis: Characterization of the dynamics of unbinding and rebinding the McGurk effect

Journal of the Acoustical Society of America

Volumn 137, Issue 1, 2015, Pages 362-377

  Nahorna, Olha ^a   Berthommier, Frédéric ^a   Schwartz, Jean Luc ^a  

^a UNIV GRENOBLE ALPES   (France)

Author keywords

[No Author keywords available]

Indexed keywords

ACOUSTICS;

AUDIO-VISUAL INTERACTIONS; AUDIO-VISUAL SPEECH; BINDING SYSTEMS; EXPERIMENT TESTS; PERCEPTUAL ORGANIZATION; PROCESSING STAGE; SCENE ANALYSIS; SPEECH PERCEPTION;

DYNAMICS;

EID: 84923770300     PISSN: 00014966     EISSN: None     Source Type: Journal    
DOI: 10.1121/1.4904536     Document Type: Article

References (64)

1. Alsius, A., and Munhall, K. (2013). " Detection of audiovisual speech correspondences without visual awareness," Psychol. Sci. 24, 423-431. 10.1177/0956797612457378
2. Alsius, A., Navarra, J., Campbell, R., and Soto-Faraco, S. S. (2005). " Audiovisual integration of speech falters under high attention demands," Curr. Biol. 15, 839-843. 10.1016/j.cub.2005.03.046
3. Alsius, A., Navarra, J., and Soto-Faraco, S. (2007). " Attention to touch weakens audiovisual speech integration," Exp. Brain Res. 183, 399-404. 10.1007/s00221-007-1110-1
4. Alsius, A., and Soto-Faraco, S. (2011). " Searching for audiovisual correspondence in multiple speaker scenarios," Exp. Brain Res. 213, 175-183. 10.1007/s00221-011-2624-0
5. Andersen, T. S., Tiippana, K., Laarni, J., Kojo, I., and Sams, M. (2009). " The role of visual spatial attention in audiovisual speech perception," Speech Commun. 51, 184-193. 10.1016/j.specom.2008.07.004
6. Andersen, T. S., Tiippana, K., Lampinen, J., and Sams, M. (2001). " Modelling of audiovisual speech perception in noise," in Proceedings of the Fourth International ESCA ETRW Conference on Auditory-Visual Speech Processing, Ålborg, Denmark, pp. 172-176.
7. Arnal, L. H., Morillon, B., Kell, C. A., and Giraud, A.-L. (2009). " Dual neural routing of visual facilitation in speech processing," J. Neurosci. 29, 13445-13 453. 10.1523/JNEUROSCI.3194-09.2009
8. Benoit, C., Mohamadi, T., and Kandel, S. (1994). " Effects of phonetic context on audio-visual intelligibility of French," J. Speech Hear. Res. 37, 1195-1203. 10.1044/jshr.3705.1195
9. Bernstein, L. E., Auer, E. T., and Moore, J. K. (2004). " Audiovisual speech binding: Convergence or association?," in The Handbook of Multisensory Processes, edited by G. A. Calvert, C. Spence, and B. E. Stein (The MIT Press, Cambridge, MA), pp. 203-224.
10. Bernstein, L. E., Lu, Z. L., and Jiang, J. (2008). " Quantified acoustic-optical speech signal incongruity identifies cortical sites of audiovisual speech processing," Brain Res. 1242, 172-184. 10.1016/j.brainres.2008.04.018
11. Bertelson, P., Vroomen, J., and De Gelder, B. (2003). " Visual recalibration of auditory speech identification: A McGurk aftereffect," Psychol. Sci. 14, 592-597. 10.1046/j.0956-7976.2003.psci-1470.x
12. Bertelson, P., Vroomen, J., Wiegeraad, G., and de Gelder, B. (1994). " Exploring the relation between McGurk interference and ventriloquism," in Proceedings of ICSLP 94 (Acoustical Society of Japan, Yokohama, Japan), Vol. 2, pp. 559-562.
13. Berthommier, F. (2004). " A phonetically neutral model of the low-level audiovisual interaction," Speech Commun. 44, 31-41. 10.1016/j.specom.2004.10.003
14. Besle, J., Fort, A., Delpuech, C., and Giard, M.-H. (2004). " Bimodal speech: Early suppressive visual effects in human auditory cortex," Eur. J. Neurosci. 20, 2225-2234. 10.1111/j.1460-9568.2004.03670.x
15. Bregman, A. S. (1990). Auditory Scene Analysis (MIT Press, Cambridge, MA), 773 pp.
16. Bregman, A. S., and Pinker, S. (1978). " Auditory streaming and the building of timbre," Can. J. Psychol. 32, 19-31. 10.1037/h0081664
17. Cathiard, M. A., Schwartz, J. L., and Abry, C. (2001). " Asking a naive question about the McGurk Effect: Why does audio [b] give more [d] percepts with visual [g] than with visual [d]?," in Proceedings AVSP-2001, edited by D. W. Massaro, J. Light, and K. Geraci, Aalborg, Denmark, pp. 138-142.
18. Colin, C., Radeau, M., Soquet, A., Demolin, D., Colin, F., and Deltenre, P. (2002). " Mismatch negativity evoked by the McGurk-MacDonald effect: A phonetic representation within short-term memory," Clin. Neurophysiol. 113, 495-506. 10.1016/S1388-2457(02)00024-X
19. Erber, N. P. (1969). " Interaction of audition and vision in the recognition of oral speech stimuli," J. Speech Hear. Res. 12, 423-425. 10.1044/jshr.1202.423
20. Eskelund, K., Tuomainen, J., and Andersen, T. S. (2011). " Multistage audiovisual integration of speech: Dissociating identification and detection," Exp. Brain Res. 208, 447-457. 10.1007/s00221-010-2495-9
21. Fuster-Duran, A. (1995). " McGurk effect in Spanish and German listeners. Influences of visual cues in the perception of Spanish and German conflicting audio-visual stimuli," in Proceedings of the Eurospeech 95, edited by J. Pardo, Madrid Spain, pp. 295-298.
22. Grant, K. W., and Seitz, P. (2000). " The use of visible speech cues for improving auditory detection of spoken sentences," J. Acoust. Soc. Am. 108, 1197-1208. 10.1121/1.1288668
23. Green, K., Kuhl, P., Meltzoff, A., and Stevens, E. (1991). " Integrating speech information across talkers, gender, and sensory modality: Female faces and male voices in the McGurk effect," Percept. Psychophys. 50, 524-536. 10.3758/BF03207536
24. Heckmann, M., Kroschel, K., Savariaux, C., and Berthommier, F. (2002). " DCT-based video features for audio-visual speech recognition," in Proceedings of ICSLP02, Denver, CO, pp. 1925-1928.
25. Hupé, J. M., and Pressnitzer, D. (2012). " The initial phase of auditory and visual scene analysis," Philos. Trans. R. Soc. B 367, 942-953. 10.1037/0096-1523.4.3.380
26. Huyse, A., Berthommier, F., and Leybaert, J. (2013). " Degradation of labial information modifies audiovisual speech perception in cochlear-implanted children," Ear Hear. 34, 110-121. 10.1097/AUD.0b013e3182670993
27. Keane, B. P., Rosenthal, O., Chun, N. H., and Shams, L. (2010). " Audiovisual integration in high functioning adults with autism," Res. Autism Spectrum Disord. 4, 276-289. 10.1016/j.rasd.2009.09.015
28. Keetels, M., Stekelenburg, J., and Vroomen, J. (2007). " Auditory grouping occurs prior to intersensory pairing: Evidence from temporal ventriloquism," Exp. Brain Res. 180, 449-456. 10.1007/s00221-007-0881-8
29. Kim, J., and Davis, C. (2003). " Hearing foreign voices: Does knowing what is said affect masked visual speech detection," Perception 32, 111-120. 10.1068/p3466
30. Kim, J., and Davis, C. (2004). " Investigating the audio-visual detection advantage," Speech Commun. 44, 19-30. 10.1016/j.specom.2004.09.008
31. Lallouache, M. T. (1990). " Un poste 'visage-parole.' Acquisition et traitement de contours labiaux" ("A 'face-speech' workstation. Acquisition and processing of labial contours"), in Proceedings XVIII Journées d'Etudes sur la Parole, Montréal, Quebec, Canada, pp. 282-286.
32. Manuel, S., Repp, B. H., Liberman, A. M., and Studdert-Kennedy, M. (1989). " Exploring the 'McGurk effecxt,' " Paper presented at the 24th meeting of the Psychonomic Society, San Diego, CA.
33. Massaro, D. W. (1987). Speech Perception by Ear and Eye (Lawrence Erlbaum Associates, Hillsdale, NJ), 320 p.
34. Massaro, D. W. (1989). " Multiple Book Review of Speech Perception by Ear and Eye: A Paradigm for Psychological Inquiry," Behav. Brain Sci. 12, 741-794. 10.1017/S0140525X00025619
35. Massaro, D. W., and Cohen, M. M. (1983). " Evaluation and integration of visual and auditorial information in speech perception," J. Exp. Psychol.: Human Percept. Perf. 9, 753-771. 10.1037//0096-1523.9.5.753
36. Massaro, D. W., Tsuzaki, M., Cohen, M. M., Gesi, A., and Heredia, R. (1993). " Bimodal speech perception: An examination across languages," J. Phonetics 21, 445-478.
37. McGurk, H., and MacDonald, J. (1976). " Hearing lips and seeing voices," Nature 264, 746-748. 10.1038/264746a0
38. Nahorna, O., Berthommier, F., and Schwartz, J. L. (2012). " Binding and unbinding the auditory and visual streams in the McGurk effect," J. Acoust. Soc. Am. 132, 1061-1077. 10.1121/1.4728187
39. Noppeney, U., Ostwald, D., and Werner, S. (2010). " Perceptual decisions formed by accumulation of audiovisual evidence in prefrontal cortex," J. Neurosci. 30, 7434-7446. 10.1523/JNEUROSCI.0455-10.2010
40. Ratcliff, R., and Rouder, J. N. (1998). " Modeling response times for two-choice decisions," Psychol. Sci. 9, 347-356. 10.1111/1467-9280.00067
41. Sanabria, D., Soto-Faraco, S., Chan, J. S., and Spence, C. (2005). " Intramodal perceptual grouping modulates multisensory integration: Evidence from the crossmodal congruency task," Neurosci. Lett. 377, 59-64. 10.1016/j.neulet.2004.11.069
42. Schwartz, J. L. (2006). " Bayesian model selection: The 0/0 problem in the fuzzy-logical model of perception," J. Acoust. Soc. Am. 120, 1795-1798. 10.1121/1.2258814
43. Schwartz, J. L. (2010). " A reanalysis of McGurk data suggests that audiovisual fusion in speech perception is subject-dependent," J. Acoust. Soc. Am. 127, 1584-1594. 10.1121/1.3293001
44. Schwartz, J. L., Berthommier, F., and Savariaux, C. (2004). " Seeing to hear better: Evidence for early audio-visual interactions in speech identification," Cognition 93, B69-B78. 10.1016/j.cognition.2004.01.006
45. Schwartz, J. L., Robert-Ribes, J., and Escudier, P. (1998). " Ten years after Summerfield. A taxonomy of models for audiovisual fusion in speech perception," in Hearing by Eye II. Perspectives and Directions in Research on Audiovisual Aspects of Language Processing, edited by R. Campbell, B. Dodd, and D. Burnham (Psychology Press, Hove, UK), pp. 85-108.
46. Schwartz, J. L., Tiippana, K., and Andersen, T. (2010). " Disentangling unisensory from fusion effects in the attentional modulation of McGurk effects: A Bayesian modeling study suggests that fusion is attention-dependent," in Proceedings AVSP2010, Tokyo, Japan, pp. 23-27.
47. Sekiyama, K., and Burnham, D. (2008). " Impact of language on development of auditory-visual speech perception," Dev. Sci. 11, 306-320. 10.1111/j.1467-7687.2008.00677.x
48. Sekiyama, K., and Tohkura, Y. (1991). " McGurk effect in non-English listeners: Few visual effects for Japanese subjects hearing Japanese syllables of high auditory intelligibility," J. Acoust. Soc. Am. 90, 1797-1805. 10.1121/1.401660
49. Sekiyama, K., and Tohkura, Y. (1993). " Inter-language differences in the influence of visual cues in speech perception," J. Phonetics 21, 427-444.
50. Smith, P. L., and Ratcliff, R. (2004). " Psychology and neurobiology of simple decisions," Trends Neurosci. 27, 161-168. 10.1016/j.tins.2004.01.006
51. Sodoyer, D., Girin, L., Jutten, C., and Schwartz, J. L. (2004). " Further experiments on audio-visual speech source separation," Speech Commun. 44, 113-125. 10.1016/j.specom.2004.10.002
52. Soto-Faraco, S., and Alsius, A. (2007). " Conscious access to the unisensory components of a cross-modal illusion," Neuroreport 18, 347-350. 10.1097/WNR.0b013e32801776f9
53. Soto-Faraco, S., and Alsius, A. (2009). " Deconstructing the McGurk-MacDonald illusion," J. Exp. Psychol.: Human Percept. Perf. 35, 580-587. 10.1037/a0013483
54. Soto-Faraco, S., Navarra, J., and Alsius, A. (2004). " Assessing automaticity in audiovisual speech integration: Evidence from the speeded classification task," Cognition 92, B13-B23. 10.1016/j.cognition.2003.10.005
55. Sumby, W., and Pollack, I. (1954). " Visual contribution to speech intelligibility in noise," J. Acoust. Soc. Am. 26, 212-215. 10.1121/1.1907309
56. Summerfield, Q. (1987). " Some preliminaries to a comprehensive account of audio-visual speech perception," in Hearing by Eye: The Psychology of Lipreading, edited by B. Dodd and R. Campbell (Lawrence Erlbaum Associates, New York), pp. 3-51.
57. Summerfield, Q., and McGrath, M. (1984). " Detection and resolution of audio-visual incompatibility in the perception of vowel," Q. J. Exp. Psychol. A 36, 51-74. 10.1080/14640748408401503
58. Tiippana, K., Andersen, T. S., and Sams, M. (2004). " Visual attention modulates audiovisual speech perception," Eur. J. Cognit. Psychol. 16, 457-472. 10.1080/09541440340000268
59. Tiippana, K., Puharinen, H., Möttönen, R., and Sams, M. (2011). " Sound location can influence audiovisual speech perception when spatial attention is manipulated," Seeing Perceiving 24, 67-90. 10.1163/187847511X557308
60. van Maanen, L., Grasman, R. P. P. P., Forstmann, B. U., and Wagenmakers, E.-J. (2012), " Piéron's Law and optimal behavior in perceptual decision-making," Front. Decision Neurosci. 5, 143.
61. van Wassenhove, V., Grant, K. W., and Poeppel, D. (2005). " Visual speech speeds up the neural processing of auditory speech," Proc. Natl. Acad. Sci. 102, 1181-1186. 10.1073/pnas.0408949102
62. van Wassenhove, V., Grant, K. W., and Poeppel, D. (2007). " Temporal window of integration in bimodal speech," Neuropsychologia 45, 598-607. 10.1016/j.neuropsychologia.2006.01.001
63. Vroomen, J., and Baart, M. (2011). " Phonetic recalibration in audiovisual speech," in Frontiers in the Neural Basis of Multisensory Processes, edited by M. M. Murray and M. T. Wallace (Taylor and Francis, Oxford, UK), pp. 363-379.
64. Yu, A. J., Dayan, P., and Cohen, J. D. (2009). " Dynamics of attentional selection under conflict: Toward a rational Bayesian account," J. Exp. Psychol.: Human Percept. Perf. 35, 700-717. 10.1037/a0013553