A reanalysis of McGurk data suggests that audiovisual fusion in speech perception is subject-dependent

Journal of the Acoustical Society of America

Volumn 127, Issue 3, 2010, Pages 1584-1594

  Schwartz, Jean Luc ^a  

^a UNIV GRENOBLE ALPES   (France)

Author keywords

[No Author keywords available]

Indexed keywords

AUDIO-VISUAL FUSION; AUDIOVISUAL PERCEPTION; BAYESIAN; EXPERIMENTAL DATA; FUSION PROCESS; HEARING IMPAIRED; LOGICAL MODELS; METHODOLOGICAL PROBLEMS; MODEL ASSESSMENT; OVERFITTING; REANALYSIS; ROOT MEAN SQUARE ERRORS; SELECTION MODEL; SPEECH PERCEPTION; SYSTEMATIC EXPLORATION; VISUAL DATA;

AUDITION;

ARTICLE; BAYES THEOREM; BIOLOGICAL MODEL; FUZZY LOGIC; HUMAN; PATIENT; PHYSIOLOGY; SPEECH PERCEPTION; VISION;

BAYES THEOREM; FUZZY LOGIC; HEARING IMPAIRED PERSONS; HUMANS; MODELS, BIOLOGICAL; SPEECH PERCEPTION; VISUAL PERCEPTION;

EID: 77950442700     PISSN: 00014966     EISSN: None     Source Type: Journal    
DOI: 10.1121/1.3293001     Document Type: Article

References (53)

1. Adjoudani, A., and Benòt, C. (1996). " On the integration of auditory and visual parameters in an HMM-based ASR.," in Speechreading by Humans and Machines, edited by, D. G. Stork, and, M. E. Hennecke, (Springer-Verlag, Berlin), pp. 461-472.
2. Alsius, A., Navarra, J., Campbell, R., and Soto-Faraco, S. (2005). " Audiovisual integration of speech falters under high attention demands.," Curr. Biol. CUBLE2 0960-9822 15, 839-43. 10.1016/j.cub.2005.03.046 (Pubitemid 40689403)
3. Alsius, A., Navarra, J., and Soto-Faraco, S. (2007). " Attention to touch weakens audiovisual speech integration.," Exp. Brain Res. EXBRAP 0014-4819 183, 399-404. 10.1007/s00221-007-1110-1 (Pubitemid 47568666)
4. Braida, L. D. (1991). " Crossmodal integration in the identification of consonant segments.," Q. J. Exp. Psychol. ZZZZZZ 0033-555X 43, 647-677.
5. Burnham, D., and Dodd, B. (1996). " Auditory-visual speech perception as a direct process: The McGurk effect in human infants and across languages.," in Speechreading by Humans and Machines, edited by, D. G. Stork, and, M. E. Hennecke, (Springer-Verlag, Berlin), pp. 103-114.
6. Burnham, D., and Dodd, B. (2004). " Auditory-visual speech integration by pre-linguistic infants: Perception of an emergent consonant in the McGurk effect.," Dev. Psychobiol. DEPBA5 0012-1630 44, 209-220. 10.1002/dev.20007 (Pubitemid 38560883)
7. Deǹve, S., and Pouget, A. (2004). " Bayesian multisensory integration and cross-modal spatial links.," J. Neurophysiol. JONEA4 0022-3077 98, 249-258.
8. Ernst, M. O., and Banks, M. S. (2002). " Humans integrate visual and haptic information in a statistically optimal fashion.," Nature (London) NATUAS 0028-0836 415, 429-433. 10.1038/415429a (Pubitemid 34100953)
9. Ernst, M. O., and Bulthoff, H. H. (2004). " Merging the senses into a robust percept.," Trends Cogn. Sci. TCSCFK 1364-6613 8, 162-169. 10.1016/j.tics.2004.02.002 (Pubitemid 38429926)
10. Fuster-Duran, A. (1995). " McGurk effect in Spanish and German listeners. Influences of visual cues in the perception of Spanish and German conflicting audio-visual stimuli.," in Proceedings of the Eurospeech 95, pp. 295-298.
11. Giard, M. H., and Peronnet, F. (1999). " Auditory-visual integration during multi-modal object recognition in humans: a behavioral and electrophysiological study.," J. Cogn. Neurosci. 11, 473-490. 10.1162/089892999563544 0898-929X (Pubitemid 29520540)
12. Gilks, W. R., Richardson, S., and Spiegelhalter, D. J. (1996). Markov Chain Monte Carlo in Practice (Chapman and Hall, New York).
13. Grant, K. W. (2002). " Measures of auditory-visual integration for speech understanding: A theoretical perspective (L).," J. Acoust. Soc. Am. JASMAN 0001-4966 112, 30-33. 10.1121/1.1482076 (Pubitemid 34747734)
14. Grant, K. W., and Seitz, P. F. (1998). " Measures of auditory-visual integration in nonsense syllables and sentences.," J. Acoust. Soc. Am. JASMAN 0001-4966 104, 2438-2449. 10.1121/1.423751 (Pubitemid 28463139)
15. Grassegger, H. (1995). " McGurk effect in German and Hungarian listeners.," in Proceedings of the 13th International Congress on Phonetic Sciences, pp. 210-113.
16. Irwin, J. R., Whalen, D. H., and Fowler, C. A. (2006). " A sex difference in visual influence on heard speech.," Percept. Psychophys. PEPSBJ 0031-5117 68, 582-592.
17. Jaeger, J., Lockwood, A., Van Valin, R. D., Jr., Kemmerer, D. L., Murphy, B. W., and Wack, D. S. (1998). " Sex differences in brain regions activated by grammatical and reading tasks.," NeuroReport NERPEZ 0959-4965 9, 2803-2807. 10.1097/00001756-199808240-00022
18. Jaynes, E. T. (1995). Probability Theory: The logic of Science (Cambridge University Press, Cambridge).
19. Kass, R. E., and Raftery, A. E. (1995). " Bayes factor.," J. Am. Stat. Assoc. JSTNAL 0162-1459 90, 773-795. 10.2307/2291091
20. Körding, K. P., Beierholm, U., Ma, W., Quartz, S., Tenenbaum, J., and Shams, L. (2007). " Causal inference in cue combination.," PLoS ONE ZZZZZZ 1932-6203 2, e943. 10.1371/journal.pone.0000943
21. Kuhl, P. K., Tsuzaki, M., Tohkura, Y., and Meltzoff, A. N. (1994). " Human processing of auditory-visual information in speech perception: Potential for multimodal human-machine interfaces.," in Proceedings of International Conference on Spoken Language Processing, Yokohama, Japan, pp. 539-542.
22. Lisker, L., and Rossi, M. (1992). " Auditory and visual cueing of the [± rounded] feature of vowels.," Lang Speech ZZZZZZ 0023-8309 35, 391-417.
23. MacKay, D. J. C. (1992). " Bayesian interpolation.," Neural Comput. NEUCEB 0899-7667 4, 415-447. 10.1162/neco.1992.4.3.415
24. Massaro, D. W. (1987). Speech Perception by Ear and Eye: A Paradigm for Psychological Inquiry (Laurence Erlbaum Associates, London).
25. Massaro, D. W. (1998). Perceiving Talking Faces (MIT, Cambridge, Ma).
26. Massaro, D. W., and Cohen, M. M. (2000). " Tests of auditory-visual integration efficiency within the framework of the fuzzy-logical model of perception.," J. Acoust. Soc. Am. JASMAN 0001-4966 108, 784-789. 10.1121/1.429611 (Pubitemid 30625903)
27. Massaro, D. W., Cohen, M. M., Campbell, C. S., and Rodriguez, T. (2001). " Bayes factor of model selection validates FLMP.," Psychon. Bull. Rev. PBUREN 1069-9384 8, 1-17.
28. Massaro, D. W., Cohen, M. M., Gesi, A., Heredia, R., and Tsuzaki, M. (1993). " Bimodal speech perception: An examination across languages.," J. Phonetics JPHNB9 0095-4470 21, 445-478.
29. McGurk, H., and MacDonald, J. (1976). " Hearing lips and seeing voices.," Nature (London) NATUAS 0028-0836 264, 746-748. 10.1038/264746a0
30. Myung, I. J., and Pitt, M. A. (1997). " Applying Occam's razor in modeling cognition: A Bayesian approach.," Psychon. Bull. Rev. PBUREN 1069-9384 4, 79-95.
31. Pitt, M. A., Kim, W., and Myung, I. J. (2003). " Flexibility versus generalizablity in model selection.," Psychon. Bull. Rev. PBUREN 1069-9384 10, 29-44.
32. Pitt, M. A., and Myung, I. J. (2002). " When a good fit can be bad.," Trends Cogn. Sci. TCSCFK 1364-6613 6, 421-425. 10.1016/S1364- 6613(02)01964-2 (Pubitemid 36160456)
33. Pugh, K. R., Shaywitz, B. A., Shaiwitz, S. E., Fulbright, R. K., Byrd, D., Skudlarski, P., Shankweiler, D. P., Katz, L., Constable, R. T., Fletcher, J., Lacadie, C., Marchione, K., and Gore, J. C. (1996). " Auditory selective attention: An fMRI investigation.," Neuroimage ZZZZZZ 1053-8119 4, 159-173. 10.1006/nimg.1996.0067 (Pubitemid 27024285)
34. Robert-Ribes, J., Schwartz, J. L., and Escudier, P. (1995). " A comparison of models for fusion of the auditory and visual sensors in speech perception.," Artif. Intell. Rev. AIRVE6 0269-2821 9, 323-346. 10.1007/BF00849043
35. Robert-Ribes, J., Schwartz, J. -L., Lallouache, T., and Escudier, P. (1998). " Complementarity and synergy in bimodal speech: Auditory, visual and audio-visual identification of French oral vowels in noise.," J. Acoust. Soc. Am. JASMAN 0001-4966 103, 3677-3689. 10.1121/1.423069 (Pubitemid 28274989)
36. Rosenblum, L. D., Schmuckler, M. A., and Johnson, J. A. (1997). " The McGurk effect in infants.," Percept. Psychophys. PEPSBJ 0031-5117 59, 347-357.
37. Rouger, J., (2007). " Perception audiovisuelle de la parole chez le sourd postlingual implant́ cochĺaire et le sujet normo-entendant: ́tude longitudinale psychophysique et neurofonctionnelle (Audiovisual speech perception in postlingually deaf cochlear-implant patients and normally-hearing subjects: Longitudinal psychophysical and neurofunctional study).," Ph.D. thesis, Universit́ Paul Sabatier, Toulouse, France.
38. Rouger, J., Lagleyre, S., Fraysse, B., Deneve, S., Deguine, O., and Barone, P. (2007). " Evidence that cochlear-implanted deaf patients are better multisensory integrators.," Proc. Natl. Acad. Sci. U.S.A. PNASA6 0027-8424 104, 7295-7300. 10.1073/pnas.0609419104 (Pubitemid 47186048)
39. Schorr, E. A., Fox, N. A., van Wassenhove, V., and Knudsen, E. I. (2005). " Auditory-visual fusion in speech perception in children with cochlear implants.," Proc. Natl. Acad. Sci. U.S.A. PNASA6 0027-8424 102, 18748-18750. 10.1073/pnas.0508862102 (Pubitemid 43011297)
40. Schwartz, J. L. (2006). " Bayesian model selection: The 0/0 problem in the fuzzy-logical model of perception.," J. Acoust. Soc. Am. JASMAN 0001-4966 120, 1795-1798. 10.1121/1.2258814 (Pubitemid 44536105)
41. Schwartz, J. L., and Cathiard, M. A. (2004). " Modeling audio-visual speech perception. Back on fusion architectures and fusion control.," in Proceedings of the ICSLP'2004, Jeju, Korea, pp. 2017-2020.
42. Schwarzer, G., and Massaro, D. W. (2001). " Modeling face identification processing in children and adults.," J. Exp. Child Psychol. ZZZZZZ 0022-0965 79, 139-161. 10.1006/jecp.2000.2574
43. Seewald, R., Ross, M., Giolas, T., and Yonovitz, A. (1985). " Primary modality for speech perception in children with normal and impaired hearing.," J. Speech Hear. Res. JSPHAH 0022-4685 28, 36-46.
44. Sekiyama, K., and Burnham, D. (2004). " Issues in the development of auditory-visual speech perception: Adults, infants and children.," in Proceedings of the Eighth International Conference on Spoken Language Processing, edited by, S. H. Kim, and, D. H. Yuon, (Sunjin Printing, Seoul, Korea), pp. 1137-1140.
45. Sekiyama, K., Burnham, D., Tam, H., and Erdener, D. (2003). " Auditory-Visual Speech Perception Development in Japanese and English Speakers.," in Proceedings of the International Conference Audio-Visual Speech Processing 2003, edited by, J. -L. Schwartz, F. Berthommier, M. -A. Cathiard, and, D. Sodoyer, St. Jorioz, France, pp. 43-47.
46. Sekiyama, K., and Tokhura, Y. (1991). " McGurk effect in non-English listeners: Few visual effects for Japanese subjects hearing Japanese syllables of high auditory intelligibility.," J. Acoust. Soc. Am. JASMAN 0001-4966 90, 1797-1825. 10.1121/1.401660
47. Sekiyama, K., and Tokhura, Y. (1993). " Inter-language differences in the influence of visual cues in speech perception.," J. Phonetics JPHNB9 0095-4470 21, 427-444.
48. Slate, E. H. (1999). " Assessing multivariate nonnormality using univariate distributions.," Biometrika BIOKAX 0006-3444 86, 191-202. 10.1093/biomet/86.1.191
49. Strelnikov, K., Rouger. J., Lagleyre, S., Fraysse, B., Deguine, O., and Barone, P. (2009). " Improvement in speech-reading ability by auditory training: Evidence from gender differences in normally-hearing, deaf, cochlear implanted subjects.," Neuropsychologia ZZZZZZ 0028-3932 47, 972-979. 10.1016/j.neuropsychologia.2008.10.017
50. Teissier, P., Robert-Ribes, J., Schwartz, J. L., and Gúrin- Dugú, A. (1999). " Comparing models for audiovisual fusion in a noisy-vowel recognition task.," IEEE Trans. Speech Audio Process. IESPEJ 1063-6676 7, 629-642. 10.1109/89.799688
51. Tiippana, K., Andersen, T. S., and Sams, M. (2004). " Visual attention modulates audiovisual speech perception.," Eur. J. Cogn. Psychol. 16, 457-472. 10.1080/09541440340000268 0954-1446 (Pubitemid 38581330)
52. Walker, A. M. (1967). " On the asymptotic behaviour of posterior distributions.," J. R. Stat. Soc. Ser. B (Methodol.) JSTBAJ 0035-9246 31, 80-88.
53. Welch, R. B., and Warren, D. H. (1986). " Intersensory interactions.," in Handbook of Perception and Human Performance, Volume I: Sensory Processes and Perception, edited by, K. R. Boff, L. Kaufman, and, J. P. Thomas, (Wiley, New York), pp. 25-1-25-36.