Audiovisual information fusion in human-computer interfaces and intelligent environments: A survey

Proceedings of the IEEE

Volumn 98, Issue 10, 2010, Pages 1692-1715

  Shivappa, Shankar T ^a   Trivedi, Mohan Manubhai ^a   Rao, Bhaskar D ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Audiovisual fusion; dynamic Bayesian networks (DBNs); hidden Markov models; human activity analysis; human activity modeling; information fusion; machine learning; multimodal systems

Indexed keywords

BAYESIAN NETWORKS; HIDDEN MARKOV MODELS; HIERARCHICAL SYSTEMS; INFORMATION FUSION; INTELLIGENT SYSTEMS; KNOWLEDGE BASED SYSTEMS; SEMANTICS; SPEECH RECOGNITION; SURVEYS;

AUDIO-VISUAL FUSION; DYNAMIC BAYESIAN NETWORKS; HUMAN ACTIVITIES; HUMAN ACTIVITY ANALYSIS; MULTIMODAL SYSTEM;

LEARNING SYSTEMS;

EID: 77956978396     PISSN: 00189219     EISSN: None     Source Type: Journal    
DOI: 10.1109/JPROC.2010.2057231     Document Type: Review

References (152)

1. A. M. Turing, "Computing machinery and intelligence" Mind, vol.LIX, no.236, pp. 433-460, 1950.
2. T. Chen, "Audiovisual speech processing" IEEE Signal Process. Mag., vol.18, no.1, pp. 9-21, Jan. 2001.
3. D. Roy, "Grounding words in perception and action: Computational insights" Trends Cogn. Sci., vol.9, no.8, pp. 389-396, Aug. 2005.
4. G. O. Deaÿk, M. S. Bartlett, and T. Jebara, "New trends in cognitive science: Integrative approaches to learning and development" Neurocomputing, vol.70, no.13-15, pp. 2139-2147, Jan. 2007.
5. C. Yu and D. H. Ballard, "A multimodal learning interface for grounding spoken language in sensory perceptions" ACM Trans. Appl. Perception, vol.1, no.1, pp. 57-80, 2004.
6. I. Bloch, "Information combination operators for data fusion: A comparative review with classification" IEEE Trans. Syst. Man Cybern. A, Syst. Humans, vol.26, no.1, pp. 52-67, Jan. 1996.
7. A. Jaimes and N. Sebe, "Multimodal human-computer interaction: A survey" Comput. Vis. Image Understand., vol.108, no.1-2, pp. 116-134, 2007.
8. J. Bijhold, A. Ruifrok, M. Jessen, Z. Geradts, and S. Ehrhardt, "Forensic audio and visual evidence 2004-2007: A review" in Proc. 15th INTERPOL Forens. Sci. Symp., Lyon, France, Oct. 2007, pp. 372-413.
9. R. Stiefelhagen, K. Bernardin, R. Bowers, J. S. Garofolo, D. Mostefa, and P. Soundararajan, "The CLEAR 2006 evaluation," Multimodal Technologies for Perception of Humans, vol. 4122. Berlin, Germany: Springer-Verlag, 2007, ser. Lecture Notes in Computer Science, pp. 1-44.
10. R. Stiefelhagen, R. Bowers, and J. Fiscus, Eds. Multimodal Technologies for Perception of Humans: International Evaluation Workshops CLEAR 2007 and RT 2007. Berlin, Germany: Springer-Verlag, 2008, ser. Lecture Notes in Computer Science, vol.4625.
11. B. Kane, S. Luz, and J. Su, "Capturing multimodal interaction at medical meetings in a hospital setting: Opportunities and challenges" in Proc. LREC Workshop Multimodal Corpora: Adv. Capturing, Coding Analyzing Multimodality, 2010, pp. 140-145.
12. A. Fleury, M. Vacher, F. Portet, P. Chahuara, and N. Noury, "A multimodal corpus recorded in a health smart home" in Proc. LREC Workshop Multimodal Corpora: Adv. Capturing, Coding Analyzing Multimodality, 2010, pp. 99-105.
13. C. Tran and M. M. Trivedi, "Towards a vision-based system exploring 3D driver posture dynamics for driver assistance: Issues and possibilities" in Proc. IEEE Intell. Veh. Symp., 2010, pp. 179-184.
14. A. Tawari and M. M. Trivedi, "Contextual framework for speech based emotion recognition in driver assistance system" in Proc. IEEE Intell. Veh. Symp., 2010, pp. 174-178.
15. K. Takeda, H. Erdogan, J. H. L. Hansen, and H. Abut, Eds., In-Vehicle Corpus and Signal Processing for Driver Behavior. New York: Springer-Verlag, 2009.
16. G. Potamianos, C. Neti, G. Gravier, A. Garg, and A. W. Senior, "Recent advances in the automatic recognition of audiovisual speech" Proc. IEEE, vol.91, no.9, pp. 1306-1326, Sep. 2003.
17. S. T. Shivappa, M. M. Trivedi, and B. D. Rao, "Hierarchical audio-visual cue integration framework for activity analysis in intelligent meeting rooms" in Proc. IEEE Comput. Vis. Pattern Recognit. Workshop, 2009, pp. 107-114.
18. S. T. Shivappa, ". D. Rao, and M. M. Trivedi, "Role of head pose estimation in speech acquisition from distant microphones" in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., 2009, pp. 3557-3560.
19. H. K. Maganti, D. Gatica-Perez, and I. McCowan, "Speech enhancement and recognition in meetings with an audio-visual sensor array" IEEE Trans. Audio Speech Lang. Process., vol.15, no.8, pp. 2257-2269, Nov. 2007.
20. P. Teissier, J. Robert-Ribes, J. L. Schwartz, and A. Guerin-Dugue, "Comparing models for audiovisual fusion in a noisy-vowel recognition task" IEEE Trans. Speech Audio Process., vol.7, no.6, pp. 629-642, Nov. 1999.
21. J. L. Schwartz, J. Robert-Ribes, and P. Escudier, "Ten years after summerfield: A taxonomy of models for audio-visual fusion in speech perception," Hearing by Eye II. London, U.K.: Psychology Press, 1998, pp. 85-108.
22. A. M. Martinez and A. C. Kak, "PCA versus LDA" IEEE Trans. Pattern Anal. Mach. Intell., vol.23, no.2, pp. 228-233, Feb. 2001.
23. C. Neti, G. Potamianos, J. Luettin, I. Matthews, H. Glotin, and D. Vergyri, "Large-vocabulary audio-visual speech recognition: A summary of the Johns Hopkins summer 2000 workshop" in Proc. IEEE Workshop Multimedia Signal Process., 2001, pp. 619-624.
24. I. McCowan, D. Gatica-Perez, S. Bengio, G. Lathoud, M. Barnard, and D. Zhang, "Automatic analysis of multimodal group actions in meetings" IEEE Trans. Pattern Anal. Mach. Intell., vol.27, no.3, pp. 305-317, Mar. 2005.
25. A. ODonovan and R. Duraiswami, "Microphone arrays as generalized cameras for integrated audio visual processing" in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., 2007, DOI: 10.1109/CVPR.2007. 383345.
26. T. Gehrig, K. Nickel, H. K. Ekenel, U. Klee, and J. McDonough, "Kalman filters for audio-video source localization" in Proc. IEEE Workshop Appl. Signal Process. Audio Acoust., Oct. 2005, pp. 118-121.
27. A. Garg, G. Potamianos, C. Neti, and T. S. Huang, "Frame-dependent multi-stream reliability indicators for audio-visual speech recognition" in Proc. Int. Conf. Multimedia Expo, 2003, vol.III, pp. 605-608.
28. K. W. Grant and S. Greenberg, "Speech intelligibility derived from asynchronous processing of auditory-visual information," Proc. Workshop Audio-Visual Speech Process., 2001, pp. 132-137.
29. S. Dupont and J. Luettin, "Audio-visual speech modeling for continuous speech recognition" IEEE Trans. Multimedia, vol.2, no.3, pp. 141-151, Sep. 2000.
30. A. Morris, A. Hagen, H. Glotin, and H. Bourlard, "Multi-stream adaptive evidence combination for noise robust ASR" Speech Commun., vol.34, no.1-2, pp. 25-40, Apr. 2001.
31. A. P. Varga and R. K. Moore, "Hidden Markov model decomposition of speech and noise" in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., Apr. 1990, vol.2, pp. 845-848.
32. A. V. Nefian, L. Liang, X. Pi, L. Xiaoxiang, C. Mao, and K. Murphy, "A coupled HMM for audio-visual speech recognition" in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., 2002, vol.2, pp. 2013-2016.
33. S. T. Shivappa, ". D. Rao, and M. M. Trivedi, "Multimodal information fusion using the iterative decoding algorithm and its application to audio-visual speech recognition" in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., 2008, pp. 2241-2244.
34. E. Erzin, Y. Yemez, A. M. Tekalp, A. Ercil, H. Erdogan, and H. Abut, "Multimodal person recognition for human-vehicle interaction" IEEE Multimedia Mag., vol.13, no.2, pp. 18-31, Apr./Jun. 2006.
35. C. Canton-Ferrer, C. Segura, J. R. Casas, M. Pardaÿs, and J. Hernando, "Audiovisual head orientation estimation with particle filtering in multisensor scenarios" EURASIP J. Adv. Signal Process., vol.2008, Jan. 2008, article no. 32.
36. R. Stiefelhagen, H. K. Ekenel, C. Fugen, P. Gieselmann, H. Holzapfel, F. Kraft, K. Nickel, M. Voit, and A. Waibel, "Enabling multimodal humanrobot interaction for the Karlsruhe humanoid robot" IEEE Trans. Robot., vol.23, no.5, pp. 840-851, Oct. 2007.
37. S. T. Shivappa, M. M. Trivedi, and B. D. Rao, "Person tracking with audio-visual cues using the iterative decoding framework" in Proc. 5th IEEE Int. Conf. Adv. Video Signal Based Surveillance, 2008, pp. 260-267.
38. K. Nickel, T. Gehrig, R. Stiefelhagen, and J. McDonough, "A joint particle filter for audio-visual speaker tracking" in Proc. 7th Int. Conf. Multimodal Interfaces, 2005, pp. 61-68.
39. K. Bernardin, T. Gehrig, and R. Stiefelhagen, "Multi-level particle filter fusion of features and cues for audio-visual person tracking Multimodal Technologies for Perception of Humans, vol. 4625. Berlin, Germany: Springer-Verlag, 2007, ser. Lecture Notes in Computer Science.
40. W. Zajdel, J. Krijnders, T. Andringa, and D. Gavrila, "CASSANDRA: Audio-video sensor fusion for aggression detection" in Proc. 4th IEEE Int. Conf. Adv. Video Signal Based Surveillance, 2007, pp. 200-205.
41. S. T. Shivappa, ". D. Rao, and M. M. Trivedi, "An iterative decoding algorithm for fusion of multi-modal information" Eurasip J. Adv. Signal Process., vol. 2008, Special Issue on Human-Activity Analysis in Multimedia Data, 2008, DOI: 10.1155/2008/478396, article id 478396.
42. M. Gurban, J.-P. Thiran, T. Drugman, and T. Dutoit, "Dynamic modality weighting for multi-stream HMMs in audio-visual speech recognition" in Proc. 10th Int. Conf. Multimodal Interfaces, 2008, pp. 237-240.
43. C. Berrou, A. Glavieux, and P. Thitimajshima, "Near Shannon limit error-correcting coding and decoding: Turbo-codes" in Proc. IEEE Int. Conf. Commun., May 1993, vol.2, pp. 1064-1070.
44. M. Cooke, J. Barker, S. Cunningham, and X. Shao, "An audio-visual corpus for speech perception and automatic speech recognition" J. Acoust. Soc. Amer., vol.120, no.5, pp. 2421-2424, Nov. 2006.
45. M. Liu, H. Tang, H. Ning, and T. S. Huang, "Person identification based on multichannel and multimodality fusion," Multimodal Technologies for Perception of Humans, vol. 4122. Berlin, Germany: Springer-Verlag, 2006, ser. Lecture Notes in Computer Science, pp. 241-248.
46. A. Stergiou, A. Pnevmatikakis, and L. Polymenakos, "A decision fusion system across time and classifiers for audio-visual person identification," Multimodal Technologies for Perception of Humans, vol. 4122. Berlin, Germany: Springer-Verlag, 2006, ser. Lecture Notes in Computer Science, pp. 223-232.
47. Z. Zeng, Z. Zhang, ". Pianfetti, J. Tu, and T. S. Huang, "Audio-visual affect recognition in activation-evaluation space" in Proc. IEEE Int. Conf. Multimedia Expo, Jul. 2005, DOI: 10.1109/ICME.2005. 1521551.
48. M. E. Sargin, Y. Yemez, E. Erzin, and A. M. Tekalp, "Audiovisual synchronization and fusion using canonical correlation analysis" IEEE Trans. Multimedia, vol.9, no.7, pp. 1396-1403, Nov. 2007.
49. Y. Keller, S. Lafon, R. Coifman, and S. Zucker, "Audio-visual group recognition using diffusion maps" IEEE Trans. Signal Process., vol.58, no.1, pp. 403-413, Jan. 2009.
50. T. Fong, I. Nourbakhsh, and K. Dautenhahn, "A survey of socially interactive robots" Robot. Autonom. Syst., vol.42, no.3-4, pp. 143-166, Mar. 2002.
51. A. Nijholt and G. Hondorp, "Towards communicating agents and avatars in virtual worlds" in Proc. Eurographics, 2000, pp. 91-95.
52. A. Nijholt, "Towards multi-modal interactions in virtual environments: A case study" in Actas-I, VI Simposio Internacional de Communicacion Social, 1999, pp. 25-28.
53. T. S. Huang, M. A. Hasegawa-Johnson, S. M. Chu, and Z. Zeng, "Sensitive talking heads" IEEE Signal Process. Mag., vol.26, no.4, pp. 67-72, 2009.
54. C. Breazeal, Designing Sociable Robots. Cambridge, MA: MIT Press, May 2002.
55. C. Breazeal, A. Edsinger, P. Fitzpatrick, and B. Scassellati, "Active vision for sociable robots" IEEE Trans. Syst. Man Cybern. A, Syst. Human, vol.31, no.5, pp. 443-453, Sep. 2001.
56. F. R. M. Baldauf and S. Dustdar, "A survey on context-aware systems" Int. J. Ad Hoc Ubiquitous Comput., vol.2, no.4, pp. 263-277, Jun. 2007.
57. M. M. Trivedi, K. S. Huang, and I. Mikic, "Dynamic context capture and distributed video arrays for intelligent spaces" IEEE Trans. Syst. Man Cybern. A, Syst, Humans, vol.35, no.1, pp. 145-163, Jan. 2005.
58. O. Brdiczka, J. Maisonnasse, P. Reignier, and J. Crowley, "Detecting small group activities from multimodal observations" Appl. Intell., vol.30, no.1, pp. 47-57, 2007.
59. O. Brdiczka, J. Maisonnasse, and P. Reignier, "Automatic detection of interaction groups" in Proc. 7th Int. Conf. Multimodal Interfaces, 2005, pp. 32-36.
60. D. Perzanowski, A. C. Schultz, and W. Adams, "Integrating natural language and gesture in a robotics domain" in Proc. IEEE Int. Symp. Comput. Intell. Robot. Autom., Sep. 1998, pp. 247-252.
61. S. S. Ghidary, Y. Nakata, H. Saito, M. Hattori, and T. Takamori, "Multi-modal human robot interaction for map generation" in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., 2001, vol.4, pp. 2246-2251.
62. D. Gatica-Perez, G. Lathoud, J. Odobez, and I. McCowan, "Audiovisual probabilistic tracking of multiple speakers in meetings" IEEE Trans. Audio Speech Lang. Process., vol.15, no.2, pp. 601-616, Feb. 2007.
63. N. Checka, K. W. Wilson, M. R. Siracusa, and T. Darrell, "Multiple person and speaker activity tracking with a particle filter" in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., 2004, vol.5, pp. 881-884.
64. Y. Chen and Y. Rui, "Real-time speaker tracking using particle filter sensor fusion" Proc. IEEE, vol.92, no.3, pp. 485-494, Mar. 2004.
65. R. Cutler and L. S. Davis, "Look whos talking: Speaker detection using video and audio correlation" in Proc. IEEE Int. Conf. Multimedia Expo (III), 2000, vol.3, pp. 1589-1592.
66. M. Beal, N. Jojic, and H. Attias, "A graphical model for audiovisual object tracking" IEEE Trans. Pattern Anal. Mach. Intell., vol.25, no.7, pp. 828-836, Jul. 2003.
67. H. McGurk and J. MacDonald, "Hearing lips and seeing voices" Lett. Nature, vol.264, pp. 746-748, Dec. 1976, DOI: 10.1038/ 264746a0.
68. D. G. Stork and M. E. Hennecke, Eds., Speechreading by Machines and Humans. Berlin, Germany: Springer-Verlag, 1996.
69. E. D. Petajan, "Automatic lipreading to enhance speech recognition (speech reading)" Ph.D. dissertation, Dept. Electr. Comput. Eng., Univ. Illinois at Urbana-Champaign, Urbana, IL, 1984.
70. E. Petajan, Ö. Bischoff, D. Bodoff, and N. M. Brooke, "An improved automatic lipreading system to enhance speech recognition" in Proc. SIGCHI Conf. Human Factors Comput. Syst., 1988, pp. 19-25.
71. B. Yuhas, M. Goldstein, and T. Sejnowski, "Integration of acoustic and visual speech signals using neural networks" IEEE Commun. Mag., vol.27, no.11, pp. 65-71, Nov. 1989.
72. C. Bregler, S. Manke, and A. Waibel, "Bimodal sensor integration on the example of speech-reading" in Proc. IEEE Int. Conf. Neural Netw., 1993, vol.2, pp. 667-671.
73. A. Adjoudani and C. Benoit, "On the integration of auditory and visual parameters in an HMM-based ASR," Speechreading by Humans and Machines: Systems and Applications, D. G. Stork and M. E. Hennecke, Eds. Berlin, Germany: Springer-Verlag, 1996, pp. 461-472.
74. J. Luettin, N. A. Thacker, and S. W. Beet, "Speechreading using shape and intensity information" in Proc. 4th Int. Conf. Spoken Lang., Oct. 1996, vol.1, pp. 58-61.
75. G. Potamianos, E. Cosatto, H. Graf, and D. Roe, "Speaker independent audio-visual database for bimodal ASR" in Proc. Eur. Tut. Work. Audio-Visual Speech Process., 1997, pp. 65-68.
76. L. Rabiner and B. Juang, Fundamentals of Speech Recognition. Englewood Cliffs, NJ: Prentice-Hall, 1993.
77. H. Bourlard and S. Dupont, "A new ASR approach based on independent processing and recombination of partial frequency bands" in Proc. Int. Conf. Spoken Lang., 1996, vol.1, pp. 426-429.
78. P. Beyerlein, "Discriminative model combination" in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., May 1998, vol.1, pp. 481-484.
79. G. Potamianos and H. Graf, "Discriminative training of HMM stream exponents for audio-visual speech recognition" in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., 1998, vol.6, pp. 3733-3736.
80. S. Nakamura, H. Ito, and K. Shikano, "Stream weight optimization of speech and lip image sequence for audio-visual speech recognition" in Proc. Int. Conf. Spoken Lang., 2000, vol.3, pp. 20-24.
81. G. Gravier, S. Axelrod, G. Potamianos, and C. Neti, "Maximum entropy and MCE based HMM stream weight estimation for audio-visual ASR" in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., 2002, vol.1, pp. 853-1-853-6.
82. A. V. Nefian, L. Liang, X. Pi, L. Xiaoxiang, and K. Murphy, "Dynamic Bayesian networks for audio-visual speech recognition" EURASIP J. Appl. Signal Process., vol.2002, no.1, pp. 1274-1288, 2002.
83. G. G. Zweig, "Speech recognition with dynamic Bayesian networks" Ph.D. dissertation, Dept. Comput. Sci., Univ. California, "erkeley, CA, 1998.
84. T. Stephenson, H. Bourlard, S. Bengio, and A. Morris, "Automatic speech recognition using dynamic Bayesian networks with both acoustic and articulatory variables" in Proc. Int. Conf. Spoken Lang., 2000, vol.II, pp. 951-954.
85. J. Gowdy, A. Subramanya, C. Bartels, and J. Bilmes, "DBN based multi-stream models for audio-visual speech recognition" in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., May 2004, pp. 993-996.
86. P. Yin, I. Essa, and J. M. Rehg, "Boosted audio-visual HMM for speech reading" in Proc. IEEE Int. Workshop Anal. Model. Faces Gestures, Oct. 2003, pp. 68-73.
87. M. Gurban and J. Thiran, "Audio-visual speech recognition with a hybrid SVM-HMM system" in Proc. 13th Eur. Signal Process. Conf., Antalya, Turkey, Sep. 2005.
88. K. Wilson, V. Rangarajan, N. Checka, and T. Darrell, "Audiovisual arrays for untethered spoken interfaces" in Proc. 4th IEEE Int. Conf. Multimodal Interfaces, 2002, pp. 389-394.
89. F. Asano, K. Yamamoto, I. Hara, J. Ogata, T. Yoshimura, Y. Motomura, N. Ichimura, and H. Asoh, "Detection and separation of speech event using audio and video information fusion and its application to robust speech interface" EURASIP J. Appl. Signal Process., vol.2004, pp. 1727-1738, 2004.
90. T. Gustafsson, ". D. Rao, and M. M. Trivedi, "Source localization in reverberant environments: Modeling and statistical analysis" IEEE Trans. Speech Audio Process., vol. 11, no. 6, pp. 791-803, Nov. 2003.
91. D. G. Stork, G. Wolff, and E. Levine, "Neural network lipreading system for improved speech recognition" in Proc. Int. Joint Conf. Neural Netw., Jun. 1992, vol.2, pp. 289-295.
92. Q. Su and P. L. Silsbee, "Robust audiovisual integration using semicontinuous hidden Markov models" in Proc. Int. Conf. Spoken Lang., 1996, vol.1, pp. 42-45.
93. A. Yilmaz, O. Javed, and M. Shah, "Object tracking: A survey" ACM Comput. Surv., vol.38, no.4, 2006, article no. 13.
94. J. H. DiBiase, H. F. Silverman, and M. S. Branstein, "Robust localization in reverberant rooms," Microphone Arrays: Signal Processing Techniques and Applications. New York: Springer-Verlag, 2001.
95. G. Pingali, G. Tunali, and I. Carlbom, "Audio-visual tracking for natural interactivity" in Proc. 7th ACM Int. Conf. Multimedia (Part 1), 1999, pp. 373-382.
96. S. G. Z. P. Aarabi, "Robust sound localization using multi-source audiovisual information fusion" Inf. Fusion, vol.2, no.3, pp. 209-223, Sep. 2001.
97. K. Nakadai, K. Hidai, H. Mizoguchi, H. G. Okuno, and H. Kitano, "Real-time auditory and visual multiple-object tracking for humanoids" in Proc. Int. Joint. Conf. Artif. Intell., 2001, pp. 1425-1432.
98. B. Kapralos, M. R. M. Jenkin, and E. Milios, "Audiovisual localization of multiple speakers in a video teleconferencing setting," Int. J. Imag. Syst, Technol., vol. 13, pp. 95-105, 2002.
99. S. T. Shivappa, S. D. Rao, and M. M. Trivedi, "Audio visual fusion and tracking with multilevel iterative decoding: Framework and experimental evaluation" IEEE J. Sel. Top. Signal Process., 2010, DOI: 10.1109/ JSTSP.2010.2057890.
100. E. Ettinger and Y. Freund, "Coordinate-free calibration of an acoustically driven camera pointing system" in Proc. Int. Conf. Distrib. Smart Cameras, 2008, DOI: 10.1109/ICDSC. 2008.4635685.
101. J. W. Fisher, T. Darrell, W. T. Freeman, and P. A. Viola, "Learning joint statistical models for audio-visual fusion and segregation" in Proc. Neural Inf. Process. Syst., 2000, pp. 772-778.
102. J. Hershey and J. Movellan, "Audio vision: Using audiovisual synchrony to locate sounds" in Proc. Neural Inf. Process. Syst., 2000, pp. 813-819.
103. J. Vermaak, M. Gangnet, A. Blake, and P. Perez, "Sequential Monte Carlo fusion of sound and vision for speaker tracking" in Proc. 8th IEEE Int. Conf. Comput. Vis., vol.1, pp. 741-746.
104. D. N. Zotkin, R. Duraiswami, and L. S. Davis, "Joint audio-visual tracking using particle filters" EURASIP J. Appl. Signal Process., vol.2002, pp. 1154-1164, Jan. 2002.
105. D. G. Perez, G. Lathoud, I. McCowan, J. M. Odobez, and D. Moore, "Audio-visual speaker tracking with importance particle filters," Proc. Int. Conf. Image Process., 2003, vol. 2, pp. 25-28.
106. L. S. Chen, T. S. Huang, T. Miyasato, and R. Nakatsu, "Multimodal human emotion/ expression recognition" in Proc. 3rd IEEE Int. Conf. Autom. Face Gesture Recognit., Apr. 1998, pp. 366-371.
107. L. C. De Silva and P. C. Ng, "Bimodal emotion recognition" in Proc. 4th IEEE Int. Conf. Autom. Face Gesture Recognit., 2000, pp. 332-335.
108. Y. Yoshitomi, K. Sung-Ill, T. Kawano, and T. Kilazoe, "Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face" in Proc. 9th IEEE Int. Workshop Robot Human Interactive Commun., 2000, pp. 178-183.
109. L. S. Chen and T. S. Huang, "Emotional expressions in audiovisual human computer interaction" in Proc. IEEE Int. Conf. Multimedia Expo, 2000, vol.1, pp. 423-426.
110. C. Busso, Z. Deng, S. Yildirim, M. Bulut, C. M. Lee, A. Kazemzadeh, S. Lee, U. Neumann, and S. Narayanan, "Analysis of emotion recognition using facial expressions, speech and multimodal information" in Proc. 6th Int. Conf. Multimodal Interfaces, 2004, pp. 205-211.
111. S. Hoch, F. Althoff, G. McGlaun, and G. Rigoll, "Bimodal fusion of emotional data in an automotive environment" in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., Mar. 2005, vol.2, pp. 1085-1088.
112. N. Sebe, I. Cohen, T. Gevers, and T. S. Huang, "Emotion recognition based on joint visual and audio cues" in Proc. 18th Int. Conf. Pattern Recognit., 2006, pp. 1136-1139.
113. L. C. D. Silva, T. Miyasato, and R. Nakatsu, "Facial emotion recognition using multi-modal information" in Proc. Int. Conf. Inf. Commun. Signal Process., Sep. 1997, vol.1, pp. 397-401.
114. H. Go, K. Kwak, D. Lee, and M. Chun, "Emotion recognition from the facial image and speech signal" in Proc. SICE Annu. Conf., Aug. 2003, vol.3, pp. 2890-2895.
115. Z. Zeng, J. Tu, ". M. Pianfetti, and T. S. Huang, "Audio-visual affective expression recognition through multistream fused hmm" IEEE Trans. Multimedia, vol. 10, no. 4, pp. 570-577, Jun. 2008.
116. A. Metallinou, S. Lee, and S. Narayanan, "Decision level combination of multiple modalities for recognition and analysis of emotional expression" in IEEE Int. Conf. Acoust. Speech Signal Process., 2010, pp. 2462-2465.
117. A. K. Jain, A. Ross, and S. Prabhakar, "An introduction to biometric recognition" IEEE Trans. Circuits Syst. Video Technol., vol.14, no.1, pp. 4-20, Jan. 2004.
118. P. S. Aleksic and A. K. Katsaggelos, "Audio-visual biometrics" Proc. IEEE, vol.94, no.11, pp. 2025-2044, Nov. 2006.
119. G. Chetty andM. Wagner, "Biometric person authentication with liveness detection based on audio visual fusion" Int. J. Biometrics, vol.1, no.4, pp. 463-478, 2009.
120. T. Wark and S. Sridharan, "Adaptive fusion of speech and lip information for robust speaker identification" Digital Signal Process., vol.11, no.3, pp. 169-186, Jul. 2001.
121. C. Sanderson and K. Paliwal, "Identity verification using speech and face information" Digital Signal Process., vol.14, no.5, pp. 449-480, Sep. 2004.
122. R. Snelick, U. Uludag, A. Mink, M. Indovina, and A. Jain, "Large-scale evaluation of multimodal biometric authentication using state-of-the-art systems" IEEE Trans. Pattern Anal. Mach. Intell., vol.27, no.3, pp. 450-455, Mar. 2005.
123. C. Chibelushi, F. Deravi, and J. Mason, "Voice and facial image integration for person recognition" in Proc. IEEE Int. Symp. Multimedia Technol. Future Appl., Southampton, U.K., Apr. 21-23, 1993.
124. R. Brunelli and D. Falavigna, "Person identification using multiple cues" IEEE Trans. Pattern Anal. Mach. Intell., vol.17, no.10, pp. 955-966, Oct. 1995.
125. P. Jourlin, J. Luettin, D. Genoud, and H. Wassner, "Integrating acoustic and labial information for speaker identification and verification" in Proc. Eur. Conf. Speech Commun. Technol., 1997, pp. 1603-1606.
126. T. Wark, S. Sridharan, and V. Chandran, "Robust speaker verification via fusion of speech and lip modalities" in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., Mar. 1999, vol.6, pp. 3061-3064.
127. S. Ben-Yacoub, J. Luettin, K. Jonsson, J. Matas, and J. Kittler, "Audio-visual person verification" in Proc. Conf. Comput. Vis. Pattern Recognit., 1999, DOI: 10.1109/ CVPR.1999.786997.
128. E. Erzin, Y. Yemez, and A. M. Tekalp, "Multimodal speaker identification using an adaptive classifier cascade based on modality reliability" IEEE Trans. Multimedia, vol.7, no.5, pp. 840-852, Oct. 2005.
129. M. M. Trivedi, K. S. Huang, and I. Mikic, "Activity monitoring and summarization for an intelligent meeting room" in Proc. IEEE Int. Workshop Human Motion, 2000, pp. 107-112.
130. M. Bett, R. Gross, H. Yu, X. Zhu, Y. Pan, J. Yang, and A. Waibel, "Multimodal meeting tracker" in Proc. Int. Conf. Adapt. Personalization Fusion Heterogeneous Inf., Paris, France, Apr. 12-14, 2000.
131. R. Gross, M. Bett, H. Yue, X. J. Zhu, Y. Pan, J. Yang, and A. Waibel, "Towards a multimodal meeting record," Proc. Int. Conf. Multimedia Expo, 2000, vol. 3, pp. 1593-1596.
132. F. Quek, D. McNeill, R. Bryll, C. Kirbas, H. Arslan, K. E. McCullough, N. Furuyama, and R. Ansari, "Gesture, speech, and gaze cues for discourse segmentation" in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2000, vol.2, pp. 247-254.
133. D. Zhang, D. Gatica-Perez, S. Bengio, I. McCowan, and G. Lathoud, "Modeling individual and group actions in meetings: A two-layer HMM framework" in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., Jun. 2004, DOI: 10.1109/CVPR.2004.125.
134. A. Dielmann and S. Renals, "Automatic meeting segmentation using dynamic bayesian networks" IEEE Trans. Multimedia, vol.9, no.1, pp. 25-36, Jan. 2007.
135. M. Al-Hames, C. Lenz, S. Reiter, J. Schenk, F. Wallhoff, and G. Rigoll, "Robust multi-modal group action recognition in meetings from disturbed videos with the asynchronous hidden Markov model" in Proc. IEEE Int. Conf. Image Process., 2007, vol.2, pp. 213-216.
136. M. Kipp, J. C. Martin, P. Paggio, and D. Heylen, Multimodal Corpora: From Models of Natural Interaction to Systems and Applications. Berlin, Germany: Springer-Verlag, 2009, ser. Lecture Notes on Artificial Intelligence.
137. S. Burger, V. MacLaren, and H. Yu, "The ISL meeting corpus: The impact of meeting type on speech style" in Proc. Int. Conf. Spoken Lang., Denver, CO, Sep. 16-20, 2002.
138. N. Morgan, D. Baron, S. Bhagat, H. Carvey, R. Dhillon, J. Edwards, D. Gelbart, A. Janin, A. Krupski, ". Peskin, T. Pfau, E. Shriberg, A. Stolcke, and C. Wooters, "Meetings about meetings: Research at ICSI on speech in multiparty conversations" in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., Apr. 2003, vol. 4, pp. 740-743.
139. J. Garofolo, C. Laprum, M. Michel, V. Stanford, and E. Tabassi, "The NIST meeting room pilot corpus" in Proc. Lang. Resource Evaluat. Conf., Lisbon, Portugal, May 26-28, 2004.
140. D. Mostefa, N. Moreau, K. Choukri, G. Potamianos, S. M. Chu, J. R. Casas, J. Turmo, L. Cristoferetti, F. Tobia, A. Pnevmatikakis, V. Mylonakis, F. Talantzis, S. Burger, R. Stiefelhagen, K. Bernardin, and C. Rochet, "The CHIL audiovisual corpus for lecture and meeting analysis inside smart rooms" J. Lang. Resources Evaluat., vol.41, no.3-4, pp. 389-407, Dec. 2007.
141. L. Chen, R. T. Rose, Y. Qiao, I. Kimbara, F. Parrill, T. X. Han, J. Tu, Z. Huang, M. Harper, Y. Xiong, D. Mcneill, R. Tuttle, and T. Huang, "Vace multimodal meeting corpus" in Proc. Workshop Mach. Learn. Multimodal Interaction, Edinburgh, U.K., Jul. 11-13, 2005.
142. S. Renals, T. Hain, and H. Bourlard, "Interpretation of multiparty meetings: The AMI and AMIDA projects" in Proc. Hands-Free Speech Commun. Microphone Arrays, Trento, Italy, May 6-8, 2008, pp. 115-118.
143. S. Cheamanunkul, E. Ettinger, M. Jacobsen, P. Lai, and Y. Freund, "Detecting, tracking and interacting with people in a public space" in Proc. Int. Conf. Multimodal Interfaces, 2009, pp. 79-86.
144. A. Pnevmatikakis, J. Soldatos, F. Talantzis, and L. Polymenakos, "Robust multimodal audio-visual processing for advanced context awareness in smart spaces" Personal Ubiquitous Comput., vol.13, no.1, pp. 3-14, Jan. 2009.
145. K. Ishizuka, S. Araki, K. Otsuka, T. Nakatani, and M. Fujimoto, "A speaker diarization method based on the probabilistic fusion of audio-visual location information" in Proc. Int. Conf. Multimodal Interfaces, 2009, pp. 55-62.
146. N. Oliver, E. Horvitz, and A. Garg, "Layered representations for human activity recognition" in Proc. Int. Conf. Multimodal Interfaces, Oct. 2002, p. 3.
147. N. M. Oliver, ". Rosario, and A. Pentland, "A Bayesian computer vision system for modeling human interactions" IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 8, pp. 831-843, Aug. 2000.
148. P. Dai and G. Xu, "Context-aware computing for assistive meeting system" in Proc. 1st Int. Conf. Pervasive Technol. Related Assistive Environ., 2008, article no. 4.
149. K. Bernardin, R. Stiefelhagen, and A. Waibel, "Probabilistic integration of sparse audio-visual cues for identity tracking" in Proc. 16th ACM Int. Conf. Multimedia, 2008, pp. 151-158.
150. P. Paggio and C. Navarretta, "Feedback in head gestures and speech" in Proc. LREC Workshop Multimodal Corpora: Adv. Capturing, Coding Analyzing Multimodality, Malta, May 18, 2010.
151. C. Yu and D. H. Ballard, "A unified model of early word learning: Integrating statistical and social cues" Neurocomputing, vol.70, no.13-15, pp. 2149-2165, 2007.
152. V. Best, E. J. Ozmeral, and B. G. Shinn-Cunningham, "Visually-guided attention enhances target identification in a complex auditory scene" J. Assoc. Res. Otolaryngol., vol.8, no.2, pp. 294-304, 2007.