Audio-visual fusion and tracking with multilevel iterative decoding: Framework and experimental evaluation

IEEE Journal on Selected Topics in Signal Processing

Volumn 4, Issue 5, 2010, Pages 882-894

  Shivappa, Shankar T ^a   Rao, Bhaskar D ^a   Trivedi, Mohan Manubhai ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Hierarchical frameworks; human activity analysis; humancomputer interaction; iterative decoding; person tracking

Indexed keywords

AUDIO AND VISUAL CUES; AUDIO AND VISUAL INFORMATION; AUDIO APPLICATIONS; AUDIO-VISUAL; AUDIO-VISUAL FUSION; DATA SETS; EXPERIMENTAL EVALUATION; HIERARCHICAL FRAMEWORKS; HUMAN ACTIVITY ANALYSIS; HUMAN COMMUNICATIONS; HUMAN COMPUTER INTERFACES; INTELLIGENT SPACES; MICROPHONE ARRAYS; MICROPHONE CALIBRATION; NATURAL INTERFACES; NEW APPROACHES; PARTICLE FILTER; PARTICLE FILTERING; PERSON TRACKING; SENSOR CALIBRATION; SENSOR CONFIGURATIONS; VIDEO INFORMATION;

CALIBRATION; CAMERAS; COMPUTER VISION; MICROPHONES; NONLINEAR FILTERING; SENSORS; SPEECH COMMUNICATION; SPEECH RECOGNITION; VISUAL COMMUNICATION;

ITERATIVE DECODING;

EID: 77956766546     PISSN: 19324553     EISSN: None     Source Type: Journal    
DOI: 10.1109/JSTSP.2010.2057890     Document Type: Article

References (44)

1. M. M. Trivedi, K. S. Huang, and I. Mikic, "Dynamic context capture and distributed video arrays for intelligent spaces," IEEE Trans. Syst., Man, Cybern., A, vol.35, no.1, pp. 145-163, Jan. 2005.
2. M. M. Trivedi, I. Mikic, and S. Bhonsle, "Active camera networks and semantic event databases for intelligent environments," in Proc. IEEE CVPR Workshop Human Modeling, Anal., Synth., 2000.
3. S. T. Shivappa, M. M. Trivedi, and B. D. Rao, "Person tracking with audio-visual cues using the iterative decoding framework," in Proc. 5th IEEE Int. Conf. Adv. Video Signal Based Surveill., 2008, pp. 260-267.
4. N. Oliver, A. Garg, and E. Horvitz, "Layered representations for learning and inferring office activity from multiple sensory channels," Comput. Vis. Image Understand., vol.96, no.2, pp. 163-180, 2004.
5. D. Zhang, D. Gatica-Perez, S. Bengio, I. McCowan, and G. Lathoud, "Modeling individual and group actions in meetings: A two-layer HMM framework," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognition, Jun. 2004, p. 117.
6. N. M. Oliver, B. Rosario, and A. Pentland, "A Bayesian computer vision system for modeling human interactions," IEEE Trans. Pattern Anal. Mach. Intell., vol.22, no.8, pp. 831-843, Aug. 2000.
7. P. Dai and G. Xu, "Context-aware computing for assistive meeting system," in Proc. 1st Int. Conf. Pervasive Technol. Rel. Assist. Environ., 2008.
8. K. Bernardin, R. Stiefelhagen, and A. Waibel, "Probabilistic integration of sparse audio-visual cues for identity tracking," in Proc. 16th ACM Int. Conf. Multimedia, 2008.
9. S. T. Shivappa, B. D. Rao, and M. M. Trivedi, "Role of head pose estimation in speech acquisition from distant microphones," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2009, pp. 3557-3560.
10. H. K. Maganti, D. Gatica-Perez, and I. McCowan, "Speech enhancement and recognition in meetings with an audio-visual sensor array," IEEE Trans. Audio, Speech, Lang. Process., vol.15, no.8, pp. 2257-2269, Nov. 2007.
11. S. T. Shivappa, M. Trivedi, and B. D. Rao, "Hierarchical audio-visual cue integration framework for activity analysis in intelligent meeting rooms," in Proc. IEEE CVPR Workshop: ViSU'09, 2009, pp. 107-114.
12. A. Yilmaz, O. Javed, and M. Shah, "Object tracking: A survey," ACM Comput. Surv., 2006.
13. M. Brandstein and D. Ward, Microphone Arrays. New York: Springer, 2001.
14. K. S. Huang and M. M. Trivedi, "Video arrays for real-time tracking of person, head, and face in an intelligent room," Mach. Vis. Applicat., 2003.
15. T. Gustafsson, B. D. Rao, and M. M. Trivedi, "Source localization in reverberant environments: Modeling and statistical analysis," IEEE Trans. Speech Audio Process., vol.11, no.6, pp. 791-803, Nov. 2003.
16. J. H. DiBiase, H. F. Silverman, and M. S. Brandstein, "Robust localization in reverberant rooms," in Microphone Arrays: Signal Processing Techniques and Applications. New York: Springer, 2001.
17. G. Pingali, G. Tunali, and I. Carlbom, "Audio-visual tracking for natural interactivity," in Proc. 7th ACM Int. Conf. Multimedia (Part 1), 1999.
18. S. G. Z. P. Aarabi, "Robust sound localization using multi-source audiovisual information fusion," Information Fusion, 2001.
19. J. Vermaak, M. Gangnet, A. Blake, and P. Perez, "Sequential Monte Carlo fusion of sound and vision for speaker tracking," in Proc. 8th IEEE Int. Conf. Comput. Vis., pp. 741-746.
20. K. Nakadai, K. Hidai, H. Mizoguchi, H. G. Okuno, and H. Kitano, "Real-time auditory and visual multiple-object tracking for humanoids," in Proc. IJCAI, 2001.
21. M. Beal, N. Jojic, and H. Attias, "A graphical model for audiovisual object tracking," IEEE Trans. Pattern Anal. Mach. Intell., vol.25, no.7, pp. 828-836, Jul. 2003.
22. R. Cutler and L. S. Davis, "Look who's talking: Speaker detection using video and audio correlation," in Proc. IEEE Int. Conf. Multimedia Expo (III), 2000.
23. J. W. Fisher, T. Darrell, W. T. Freeman, and P. A.Viola, "Learning joint statistical models for audio-visual fusion and segregation," in Proc. NIPS, 2000.
24. J. Hershey and J. Movellan, "Audio vision: Using audiovisual synchrony to locate sounds," in Proc. NIPS, 2000.
25. D. N. Zotkin, R. Duraiswami, and L. S. Davis, "Joint audio-visual tracking using particle filters," EURASIP J. Appl. Signal Process., vol.2002, no.11, pp. 1154-1164, 2002.
26. D. G. Perez, G. Lathoud, I. McCowan, J. M. Odobez, and D. Moore, "Audio-Visual Speaker Tracking With Importance Particle Filters," in Proc. ICIP, 2003.
27. Y. Chen and Y. Rui, "Real-time speaker tracking using particle filter sensor fusion," Proc. IEEE, vol.92, no.3, pp. 485-494, Mar. 2004.
28. N. Checka, K. W. Wilson, M. R. Siracusa, and T. Darrell, "Multiple person and speaker activity tracking with a particle filter," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2004, vol.V, pp. 881-884.
29. K. Nickel, T. Gehrig, R. Stiefelhagen, and J. McDonough, "A joint particle filter for audio-visual speaker tracking," in Proc. 7th Int. Conf. Multimodal Interfaces, 2005.
30. T. Gehrig, K. Nickel, H. K. Ekenel, U. Klee, and J. McDonough, "Kalman filters for audio-video source localization," in Proc. IEEE Workshop Applicat. Signal Process. Audio Acoust., Oct. 2005, pp. 118-121.
31. D. Gatica-Perez, G. Lathoud, J. Odobez, and I. McCowan, "Audiovisual probabilistic tracking of multiple speakers in meetings," IEEE Trans. Audio, Speech, Lang. Process., vol.15, no.2, pp. 601-616, Feb. 2007.
32. K. Bernardin and R. Stiefelhagen, "Audio-visual multi-person tracking and identification for smart environments," in Proc. ACM Int. Conf. Multimedia, 2007.
33. R. Stiefelhagen, H. K. Ekenel, C. Fugen, P. Gieselmann, H. Holzapfel, F. Kraft, K. Nickel, M. Voit, and A. Waibel, "Enabling multimodal human-robot interaction for the Karlsruhe humanoid robot," IEEE Trans. Robotics, vol.23, no.5, pp. 840-851, Oct. 2007.
34. R. Stiefelhagen, R. Bowers, and J. Fiscus, in Proc. Multimodal Technologies for Perception of Humans: International Evaluation Workshops CLEAR 2007 and RT 2007 (Lecture Notes in Computer Science).
35. A. Abad, C. Canton-Ferrer, C. Segura, J. L. Landabaso, D. Macho, J. R. Casas, J. Hernando, M. Pardas, and C. Nadeu, "Upc audio, video and multimodal person tracking systems in the clear evaluation campaign," in Proc. 1st Int. CLEAR Evaluation Workshop-Multimodal Technologies for Perception of Humans, 2007.
36. N. Katsarakis, F. Talantzis, A. Pnevmatikakis, and L. Polymenakos, "The ait 3D audio/visual person tracker for clear 2007," in Proc. 1st Int. CLEAR Evaluation Workshop-Multimodal Technologies for Perception of Humans, 2007.
37. S. T. Shivappa, B. D. Rao, and M. M. Trivedi, "An iterative decoding algorithm for fusion of multi-modal information," EURASIP J. Adv. Signal Process.-Special Iss. Human-Activity Analysis in Multimedia Data, 2008.
38. Y. Bar-Shalom and T. E. Fortmann, Tracking and Data Association. New York: Academic, 1988.
39. C. H. Knapp and G. C. Carter, "The generalized correlation method for estimation of time delay," IEEE Trans. Acoust., Speech, Signal Process., vol.ASSP-24, no.4, pp. 320-327, Aug. 1976.
40. M. Brandstein and H. Silverman, "A robust method for speech signal time-delay estimation in reverberant rooms," in Proc. ICASSP, 1997, pp. 375-378.
41. A. O'Donovan and R. Duraiswami, "Microphone arrays as generalized cameras for integrated audio visual processing," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognition, 2007, pp. 1-8.
42. L. Bahl, J. Cocke, F. Jelinek, and J. Raviv, "Optimal decoding of linear codes for minimizing symbol error rate," IEEE Trans. Inf. Theory, vol.IT-20, no.2, pp. 284-287, Mar. 1974.
43. S. Dasgupta and Y. Freund, "Random projection trees for vector quantization," IEEE Trans. Inf. Theory, vol.55, no.7, pp. 3229-3242, Jul. 2009.
44. D. Mostefa, N. Moreau, K. Choukri, G. Potamianos, S. M. Chu, J. R. Casas, J. Turmo, L. Cristoferetti, F. Tobia, A. Pnevmatikakis, V. Mylonakis, F. Talantzis, S. Burger, R. Stiefelhagen, K. Bernardin, and C. Rochet, "The CHIL audiovisual corpus for lecture and meeting analysis inside smart rooms," J. Lang. Res. Eval., vol.41, no. (3-4), pp. 389-407, Dec. 2007.