Ten years after Summerfield: A taxonomy of models for audio-visual fusion in speech perception

Hearing Eye II: Advances in the Psychology of Speechreading and Auditory-Visual Speech

Volumn , Issue , 2013, Pages 85-108

  Schwartz, Jean Luc ^a   Robert Ribes, Jordi ^b   Escudier, Pierre ^a  

^a Institut de la Communication Parlee   (France)

^b MACQUARIE UNIVERSITY   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85069232505     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.4324/9780203098752-12     Document Type: Chapter

References (98)

1. Abry, C. & Badin, P. (1996). Speech mapping as a framework for an integrated approach to the sensori-motor foundations of language. 4th Production Seminar, 1st ETRW on Speech Production Modeling: From Control Stategies to Acoustics (pp. 175-84). Autrans, France.
2. Adjoudani, A. & Benoît, C. (1996). On the integration of auditory and visual parameters in an HMM-based ASR. In D.G. Stork & M.E. Hennecke (Eds), Speechreading by Humans and Machines: Models, Systems and Applications (pp. 461-72). NATO ASI Series, Berlin: Springer.
3. Bailly, G. (1995). Recovering place of articulation for occlusives in VCV’s. Proceedings of the XIHth International Congress of Phonetic Sciences, 2, 230-33.
4. Basir, O.A. & Shen, H.C. (1992). Sensory data integration: A team consensus approach. Proceedings the 1992 IEEE International Conference on Robotics and Automation (pp. 1683-8). Nice, France.
5. Beckerman, M. (1992). A Bayes-maximum entropy method for multi-sensor data fusion. Proceedings the 1992 IEEE International Conference on Robotics and Automation (pp. 1668-74). Nice, France.
6. Benoît, C, Guiard-Marigny, T., Le Goff, B. & Adjoudani, A. (1996). Which components of the face do humans and machines best speechread? In D.G. Stork & M.E. Hennecke (Eds), Speechreading by Humans and Machines: Models, Systems and Applications (pp. 315-28). NATO ASI Series, Berlin: Springer.
7. Bertelson, P., Vroomen, J., Wiegeraad, G. & de Gelder, B. (1994). Exploring the relation between McGurk interference and ventriloquism. Proceedings International Conference on Spoken Language Processing (pp. 559-62). Yokohama, Japan.
8. Bloch, I. (1994). Information Combination Operators for Data Fusion: A Comparative Review with Classification. Internal report, Telecom Paris, 94 D 013.
9. Braida, L.D. (1991). Crossmodal integration in the identification of consonant segments. Quarterly Journal Experimental Psychology, 43, 647-77.
10. Braida, L.D., Picheny, M.A., Cohen, J.R., Rabinowitz, W.M. & Perkell, J.S. (1986). Use of articulatory signals in automatic speech recognition. Journal of the Acoustical Society of America, 80, S18.
11. Breeuwer, M. & Plomp, R. (1986). Speechreading supplemented with auditorily presented speech parameters. Journal of the Acoustical Society of America, 79, 481-99
12. Bregler, C, Hild, H., Manke, S. & Waibel, A. (1993). Improving connected letter recognition by lipreading. International Joint Conference of Speech and Signal Processing (pp. 557-60). Minneapolis.
13. Bregman, A.S. (1990). Auditory Scene Analysis: The Perceptual Organization of Sound. Cambridge, Mass: Bradford Books, MIT Press.
14. Brooke, M. & Petajan, E.D. (1986). Seeing speech: investigations into the synthesis and recognition of visible speech movements using automatic image processing and computer graphics. International Conference on Speech Input/Output, Techniques and Applications (pp. 104-109). London.
15. Campbell, H.W. (1974). Phoneme recognition by ear and by eye: a distinctive feature analysis. Doctoral dissertation, Katholieke Universiteit te Nijmegen.
16. Campbell, R. (1988). Tracing lip movements: making speech visible. Visible Language, 22, 33-57.
17. Campbell, R. (1996). Seing brains reading speech: a review and speculations. In D.G. Stork & M.E. Hennecke (Eds), Speechreading by Humans and Machines: Models, Systems and Applications (pp. 115-34). NATO ASI Series, Berlin: Springer.
18. Campbell, R. & Dodd, B. (1980). Hearing by eye. Quarterly Journal of Experimental Psychology, 32, 85-9.
19. Cathiard, M.A. (1994). La perception visuelle de l’anticipation des gestes vocaliques: cohérence des événements audibles et visibles dans le flux de la parole. Thèse de Doctorat en Psychologie Cognitive. Université Pierre Mendès, Grenoble, France.
20. Cipra, B. (1992). You can’t hear the shape of a drum. Science, 225, 1642-3.
21. Cosi, P. & Magno-Caldognetto, E. (1996). Lips and jaw movements for vowels and consonants: spatio-temporal characteristics and bimodal recognition applications. In D.G. Stork & M.E. Hennecke (Eds.), Speechreading by Humans and Machines: Models, Systems and Applications (pp. 291-314). NATO ASI Series, Berlin: Springer.
22. Crowley, J.L. & Demazeau, Y. (1993). Principles and techniques for sensor data fusion. Signal Processing, 32, 5-27.
23. Dalton, B., Kaucic, R. & Blake, A. (1996). Automatic speechreading using dynamic contours. In D.G. Stork & M.E. Hennecke (Eds), Speechreading by Humans and Machines: Models, Systems and Applications (pp. 373-82). NATO ASI Series, Berlin: Springer.
24. Dasarathy, B.V. (1994). Decision Fusion. Los Alamitos, California: IEEE Computer Society Press.
25. Delgutte, B. & Kiang, N.Y.S. (1984). Speech coding in the auditory nerve: IV. Sounds with consonant-like dynamic characteristics. Journal of the Acoustical Society of America, 75, 897-907.
26. Driver, J. (1996). Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading. Nature, 381, 66-8.
27. Driver, J. & Grossenbacher, P.G. (1996). Multimodal spatial constraints on tactile selective attention. In T. Inui & J.L. McClelland (Eds.), Attention and Performance XVI: Information Integration in Perception and Communication (pp. 209-36). Cambridge, Mass: Bradford Books, MIT Press.
28. Duchnowski, P., Meier, U. & Waibel, A. (1994). See me, hear me: integrating automatic speech recognition and lip-reading. International Conference on Spoken Language Processing (pp. 547-50). Yokohama, Japan.
29. Erber, N.P. (1975). Auditory-visual perception of speech. Journal of Speech and Hearing Disorders, 40, 481-92.
30. Faugeras, O., Ayache, N. & Faverjon, B. (1986). Building visual maps by combining noisy stereo measurements. IEEE International Conference on Robotics and Automation. San Francisco, CA.
31. Fisher, B.D. (1991). Integration of visual and auditory information in perception of speech events. Unpublished Doctoral Dissertation, University of California, Santa Cruz, CA.
32. Fowler, C.A. (1986). An event approach to the study of speech perception from a direct-realist perspective. Journal of Phonetics, 14, 3-28.
33. Fuster-Duran, A. (1995). McGurk effect in Spanish and German listeners. Influences of visual cues in the perception of Spanish and German conflicting audio-visual stimuli. Proceedings of Eurospeech, 95, 295-8.
34. Gibson, J.J. (1966). The Senses Considered as Perceptual Systems. Boston: Houghton Mifflin Company.
35. Girin, L., Feng, G. & Schwartz, J.L. (1996). Débruitage de parole par un filtrage utilisant l’image du locuteur: une étude de faisabilité. Traitement du Signal, 13, 319-34.
36. Göldschen, A.J., Garcia, O.N. & Petajan, E.D. (1996). Rationale for phoneme-viseme mapping and feature selection in visual speech recognition. In D.G. Stork & M.E. Hennecke (Eds), Speechreading by Humans and Machines: Models, Systems and Applications (pp. 505-18). NATO ASI Series, Berlin: Springer.
37. Grant, K.W., Ardeil, L.H., Kühl, P.K. & Sparks, D.W. (1985). The contribution of fundamental frequency, amplitude envelope, and voicing duration cues to speechreading in normal-hearing subjects. Journal of the Acoustical Society of America, 77, 671-7.
38. Grassegger, H. (1995). McGurk effect in German and Hungarian listeners. Proceedings of the XHIth International Congress of Phonetic Sciences, 3, 210-13.
39. Green, K.P. & Kuhl, P.K. (1989). The role of visual information in the processing of place and manner features in speech perception. Perception and Psychophysics, 45, 34-42.
40. Green, K.P. & Miller, J.L. (1985). On the role of visual rate information in phonetic perception. Perception and Psychophysics, 38, 269-76.
41. Green, K.P., Kuhl, P.K., Meltzoff, A.N. & Stevens, K.N. (1991). Integrating speech information across talkers, gender, and sensory modality: Female faces and male voices in the McGurk effect. Perception and Psychophysics, 50, 524-36.
42. Hatwell, Y. (1986). Toucher l’espace. La main et la perception tactile de l’espace. Lille: Presses Universitaires de Lille.
43. Hatwell, Y. (1993). Transferts intermodaux et intégration intermodale. In M. Richelle, J. Reguin & M. Robert (Eds), Traité de Psychologie Expérimentale (pp. 543-84). Paris: Presses Universitaires de France.
44. Inui, T. (1996). Mechanisms of information integration in the brain. In T. Inui & J.L. McClelland (Eds), Attention and Performance XVI: Information Integration in Perception and Communication (pp. 3-12). Cambridge, Mass: Bradford Books, MIT Press.
45. Johnson, C.J., Paivio, A.V. & Clark, J.M. (1989). Spatial and verbal abilities in children’s crossmodal recognition: a dual coding approach. Canadian Journal of Psychology, 43, 397-412.
46. Kac, M. (1966). Can one hear the shape of a drum? Ann. Math. Monthly, 73, 1-23.
47. Klatt, D.H. (1979). Speech perception: A model of acoustic-phonetic analysis and lexical access. Journal of Phonetics, 7, 279-312.
48. Knudsen, E.I. & Konishi, M. (1978a). Space and frequency are represented separately in auditory midbrain of the owl. Journal of Neurophysiology, 41, 870-84.
49. Knudsen, E.I. & Konishi, M. (1978b). A neural map of auditory space in the owl. Science, 200, 795-7.
50. Kuhl, P.K. & Meltzoff, A.N. (1982). The bimodal perception of speech in infancy. Science, 218, 1138-41.
51. Kuhl, P.K., Tsuzaki, M., Tohkura, Y. & Meltzoff, A.N. (1994). Human processing of auditory-visual information in speech perception: Potential for multimodal human-machine interfaces. International Conference on Spoken Language Processing (pp. 539-42). Yokohama, Japan.
52. Laboissière, R. & Galvan, A. (1995). Inferring the commands of an articulatory model from acoustical specifications of stop-vowel sequences. Proceedings of the XHIth International Congress of Phonetic Sciences, I, 358-61.
53. Liberman, A., & Mattingly, I. (1985). The motor theory of speech perception revised. Cognition, 21, 1-36.
54. Lisker, L. & Rossi, M. (1992). Auditory and visual cueing of the [± rounded] feature of vowels. Language and Speech, 35, 391-417.
55. McClelland, J.L. & Elman, J.L. (1986). The TRACE model of speech perception. Cognitive Psychology, 18, 1-86.
56. McGrath, M. & Summerfield, A.Q. (1985). Intermodal timing relations and audio-visual speech recognition by normal-hearing adults. Journal of the Acoustical Society of America, 77, 678-85.
57. McGurk, H. & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746-8.
58. Manuel, S.Y., Repp, B.H., Studdert-Kennedy, M. & Liberman, A.M. (1983). Exploring the “McGurk effect�?. Journal of the Acoustical Society of America, 74, SI, S66.
59. Massaro, D.W (1987). Speech Perception by Ear and Eye: A Paradigm for Psychological Inquiry. Hillsdale, NJ: Lawrence Erlbaum Associates Inc.
60. Massaro, D.W. (1989). Multiple book review of speech perception by ear and eye: A paradigm for psychological inquiry. Behavioral and Brain Sciences, 12, 741-94.
61. Massaro, D.W. & Cohen, M.M. (1993). Perceiving asynchronous bimodal speech in consonant-vowel and vowel syllables. Speech Communication, 13, 127-34.
62. Massaro, D.W. & Cohen, M.M. (1995). Modeling the perception of bimodal speech. Proceedings of the XIHth International Congress of Phonetic Sciences, 3, 106-13.
63. Massaro, D.W. & Warner, D.S. (1977). Dividing attention between auditory and visual perception. Perception and Psychophysics, 21, 569-74.
64. Massaro, D.W., Cohen, M.M., Gesi, A., Heredia, R. & Tsuzaki, M. (1993). Bimodal speech perception: an examination across languages. Journal of Phonetics, 21, 445-78.
65. Movellan, J.R. & Chadderdon, G. (1996). Channel separability in the audio-visual integration of speech: a Bayesian approach. In D.G. Stork & M.E. Hennecke (Eds), Speechreading by Humans and Machines: Models, Systems and Applications (pp. 473-88). NATO ASI Series, Berlin: Springer.
66. Pack, A.A. & Herman, L.M. (1995). Sensory integration in the bottlenosed dolphin: immediate recognition of complex shapes across the senses of echolocation and vision. Journal of the Acoustical Society of America, 98, 722-33.
67. Petajan, E.D. (1984). Automatic lipreading to enhance speech recognition. Doctoral Thesis, University of Illinois.
68. Radeau, M. (1994). Auditory-visual spatial interaction and modularity. Cahiers de Psychologie Cognitive, 13, 3-51.
69. Robert-Ribes, J. (1995). Modèles d’intégration audiovisuelle de signaux linguistiques: de la perception humaine à la reconnaissance automatique des voyelles. Thèse de Docteur de l’INPG, Signal-lmage-Parole.
70. Robert-Ribes, J., Piquemal, M., Schwartz, J.L. & Escudier, P. (1996). Exploiting sensor fusion architectures and stimuli complementarity in AV speech recognition. In D.G. Stork & M.E. Hennecke (Eds), Speechreading by Humans and Machines: Models, Systems and Applications (pp. 193-210). NATO ASI Series, Berlin: Springer.
71. Robert-Ribes, J., Schwartz, J.L. & Escudier, P. (1995). A comparison of models for fusion of the auditory and visual sensors in speech perception. Artificial Intelligence Review Journal, 9, 323-16.
72. Robert-Ribes, J., Schwartz, J.L., Lallouache, T. & Escudier, P. (submitted). Complementarity and synergy in bimodal speech: auditory, visual and audio-visual identification of French oral vowels in noise.
73. Roberts, M. & Summerfïeld, A.Q. (1981). Audio-visual presentation demonstrates that selective adaptation in speech perception is purely auditory. Perception and Psychophysics, 30, 309-14.
74. Rönnberg, J., Arlinger, S., Lyxell, B. & Kinnefords, C. (1989). Visual evoked potentials: relation to adult speechreading and cognitive function. Journal of Speech and Hearing Research, 32, 725-35.
75. Rosen, S., Fourcin, A.J. & Moore, B. (1981). Voice pitch as an aid to lipreading. Nature, 291, 150-52.
76. Savage-Rumbaugh, S., Sevcik, R.A. & Hopkins, W.D. (1988). Symbiolic cross-modal transfer in two species of chimpanzees. Child Development, 59, 617-25.
77. Sekiyama, K. (1994). Differences in auditory-visual speech perception between Japanese and Americans: McGurk effect as a function of incompatibility. Journal of the Acoustical Society of Japan (E), 15, 143-58.
78. Sekiyama, K. & Tohkura, Y. (1991). McGurk effect in non-English listeners: Few visual effects for Japanese subjects hearing Japanese syllables of high auditory intelligibility. Journal of the Acoustical Society of America, 90, 1797-825.
79. Sekiyama, K. & Tohkura, Y. (1993). Inter-language differences in the influence of visual cues in speech perception. Journal of Phonetics, 21, 427-44.
80. Shepherd, D.C., DeLavergne, R.W., Frueh, F.X. & Clobridge, C. (1977). Visual-neural correlate of speechreading ability in normal-hearing adults. Journal of Speech and Hearing Research, 20, 752-65.
81. Silsbee, P.L. & Su, Q. (1996). Audio-visual sensory integration using Hidden Markov Models. In D.G. Stork & M.E. Hennecke (Eds), Speechreading by Humans and Machines: Models, Systems and Applications (pp. 489-96). NATO ASI Series, Berlin: Springer.
82. Smeele, P.M.T. & Sittig, A.C. (1991). The contribution of vision to speech perception. Proceedings of the 2nd European Conference on Speech Communication and Technology (pp. 1495-7). Genova, Italy.
83. Stein, E.B. & Meredith, M.A. (1993). The Merging of Senses. Cambridge, London: The MIT Press.
84. Stork, D.G., Wolff, G. & Levine, E. (1992). Neural network lipreading system for improved speech recognition. IJCNN-92, Baltimore MD, 2, 285-95.
85. Sumby, W.H. & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212-15.
86. Summerfield, A.Q. (1987). Some preliminaries to a comprehensive account of audio-visual speech perception. In B. Dodd and R. Campbell (Eds), Hearing by Eye: The Psychology of Lipreading (pp. 3-51). Hove, UK: Lawrence Erlbaum Associates Ltd.
87. Summerfield, A.Q. & McGrath, M. (1984). Detection and resolution of audio-visual incompatibility in the perception of vowels. Quarterly Journal of Experimental Psychology: Human Experimental Psychology, 36, 51-74.
88. Teissier, P. (1995). Fusion audiovisuelle avec prise en compte de I’environnement. DEA Signal-Image-Parole, INPG.
89. Teissier, P., Robert-Ribes, J., Schwartz, J.L. & Guerin-Dugue, A. (submitted). Comparing models for audio-visual fusion in a noisy-vowel recognition task. Submitted to IEEE Transactions on Speech and Audio Processing.
90. Viviani, P. & Stucchi, N. (1992). Biological movements look uniform: evidence of motor-perceptual interactions. Journal of Experimental Psychology: Human Perception and Performance, 18, 603-23.
91. Vroomen, J.H.M. (1992). Hearing voices and seeing lips: Investigations in the psychology of lipreading. Doctoral Dissertation, Katolieke Univ. Brabant, 18 September.
92. Watanabe, T. and Kohda, M. (1990). Lip-reading of Japanese vowels using neural networks. Proceedings of the International Conference Spoken Language Processing, 90, Kobe, Japan, 1373-6.
93. Welch, R.B. (1989). A comparison of speech perception and spatial localization. Behavioral and Brain Sciences, 12, 116-1.
94. Welch, R.B. (1994). The dissection of intersensory bias: Weighting for Radeau. Current Psychology of Cognition, 13, 117-23.
95. Welch, R.B. & Warren, D.H. (1986). Intersensory interactions. In K.R. Boff, L. Kaufman and J.P. Thomas (Eds), Handbook of Perception and Human Performance, Volume I: Sensory Processes and Perception (pp. 25-1-25-36). New-York: Wiley.
96. Wertheimer, M. (1961). Psychomotor coordination of auditory-visual space at birth. Science, 134, 1692.
97. Yuhas, B.P., Goldstein, M.H. & Sejnowski, T.J. (1989). Integration of acoustic and visual speech signals using neural networks. IEEE Communications Magazine, November, 65-71.
98. Yuhas, B.P., Goldstein, M.H., Sejnowski, T.J. & Jenkins, R.E. (1990). Neural network models of sensory integration for improved vowel recognition. Proceedings IEEE, 78, 1658-68.