Multimodal fusion for multimedia analysis: A survey

Multimedia Systems

Volumn 16, Issue 6, 2010, Pages 345-379

  Atrey, Pradeep K ^a   Hossain, M Anwar ^b   El Saddik, Abdulmotaleb ^b   Kankanhalli, Mohan S ^c  

^a UNIVERSITY OF WINNIPEG   (Canada)

^b UNIVERSITY OF OTTAWA   (Canada)

^c NATIONAL UNIVERSITY OF SINGAPORE   (Singapore)

Author keywords

Multimedia analysis; Multimodal information fusion

Indexed keywords

BASIC CONCEPTS; CONFIDENCE LEVELS; CONTEXTUAL INFORMATION; FUSION METHODOLOGY; FUSION METHODS; FUSION STRATEGIES; MULTI-MEDIA ANALYSIS; MULTI-MODAL FUSION; MULTIMODAL INFORMATION FUSION; MULTIPLE MODALITIES;

INFORMATION FUSION; SURVEYS;

RESEARCH;

EID: 78049469733     PISSN: 09424962     EISSN: None     Source Type: Journal    
DOI: 10.1007/s00530-010-0182-0     Document Type: Article

References (158)

1. PETS: Performance evaluation of tracking and surveillance (Last access date 31 August 2009). http://www.cvg.rdg.ac.uk/slides/pets.html
2. TRECVID data availability (Last access date 02 September 2009). http://www-nlpir.nist.gov/projects/trecvid/trecvid.data.html
3. W. Adams G. Iyengar C. Lin M. Naphade C. Neti H. Nock J. Smith 2003 Semantic indexing of multimedia content using visual, audio, and text cues EURASIP J. Appl. Signal Process. 2003 2 170 185 10.1155/S1110865703211173
4. Aguilar, J.F., Garcia, J.O., Romero, D.G., Rodriguez, J.G.: A comparative evaluation of fusion strategies for multimodal biometric verification. In: International Conference on Video-Based Biometrie Person Authentication, pp. 830-837. Guildford (2003)
5. P.S. Aleksic A.K. Katsaggelos 2006 Audio-visual biometrics Proc. IEEE 94 11 2025 2044 10.1109/JPROC.2006.886017 (Pubitemid 46445568)
6. Andrieu, C., Doucet, A., Singh, S., Tadic, V.: Particle methods for change detection, system identification, and control. Proc. IEEE 92(3), 423-438 (2004)
7. Argillander, J., Iyengar, G., Nock, H.: Semantic annotation of multimedia using maximum entropy models. In: International Conference on Accoustic, Speech and Signal Processing, pp. II-153-156. Philadelphia (2005)
8. P.K. Atrey M.S. Kankanhalli R. Jain 2006 Information assimilation framework for event detection in multimedia surveillance systems Springer/ACM Multimedia Syst. J. 12 3 239 253 10.1007/s00530-006-0063-8 (Pubitemid 44876288)
9. Atrey, P.K., Kankanhalli, M.S., Oommen, J.B.: Goal-oriented optimal subset selection of correlated multimedia streams. ACM Trans. Multimedia Comput. Commun. Appl. 3(1), 2 (2007)
10. Atrey, P.K., Kankanhalli, M.S., El Saddik, A.: Confidence building among correlated streams in multimedia surveillance systems. In: International Conference on Multimedia Modeling, pp. 155-164. Singapore (2007)
11. Ayache, S., Quénot, G., Gensel, J.: Classifier fusion for svm-based multimedia semantic indexing. In: The 29th European Conference on Information Retrieval Research, pp. 494-504. Rome (2007)
12. N. Babaguchi Y. Kawai T. Kitahashi 2002 Event based indexing of broadcasted sports video by intermodal collaboration IEEE Trans. Multimed. 4 68 75 10.1109/6046.985555 (Pubitemid 34291529)
13. N. Babaguchi Y. Kawai T. Ogura T. Kitahashi 2004 Personalized abstraction of broadcasted american football video by highlight selection IEEE Trans. Multimed. 6 4 575 586 10.1109/TMM.2004.830811
14. Bailly-Bailliére, E., Bengio, S., Bimbot, F., Hamouz, M., Kittler, J., Mariéthoz, J., Matas, J., Messer, K., Popovici, V., Porée, F., Ruíz, B., Thiran, J.P.: The BANCA database and evaluation protocol. In: International Conference on Audio-and Video-Based Biometrie Person Authentication, pp. 625-638. Guildford (2003)
15. M.J. Beal N. Jojic H. Attias 2003 A graphical model for audio-visual object tracking IEEE Trans. Pattern Anal. Mach. Intell. 25 828 836 10.1109/TPAMI.2003.1206512
16. A. Bendjebbour Y. Delignon L. Fouque V. Samson W. Pieczynski 2001 Multisensor image segmentation using Dempster-Shafer fusion in markov fields context IEEE Trans. Geosci. Remote Sens. 39 8 1789 1798 10.1109/36.942557 (Pubitemid 32935693)
17. Bengio, S.: Multimodal authentication using asynchronous hmms. In: The 4th International Conference Audio and Video Based Biometric Person Authentication, pp. 770-777. Guildford (2003)
18. S. Bengio C. Marcel S. Marcel J. Mariethoz 2002 Confidence measures for multimodal identity verification Inf. Fusion 3 4 267 276 10.1016/S1566-2535(02) 00089-1
19. Bredin, H., Chollet, G.: Audio-visual speech synchrony measure for talking-face identity verification. In: IEEE International Conference on Acoustics, Speech and Signal Processing, vol. 2, pp. 233-236. Paris (2007)
20. Bredin, H., Chollet, G.: Audiovisual speech synchrony measure: application to biometrics. EURASIP J. Adv. Signal Process. 11 p. (2007). Article ID 70186
21. Brémond, F., Thonnat, M.: A context representation of surveillance systems. In: European Conference on Computer Vision. Orlando (1996)
22. Brooks, R.R., Iyengar, S.S.: Multi-sensor Fusion: Fundamentals and Applications with Software. Prentice Hall PTR, Upper Saddle River, NJ (1998)
23. C.J.C. Burges 1998 A tutorial on support vector machines for pattern recognition Data Min. Knowl. Discov. 2 2 121 167 10.1023/A:1009715923555 (Pubitemid 128126769)
24. Caruana, R., Munson, A., Niculescu-Mizil, A.: Getting the most out of ensemble selection. In: ACM International Conference on on Data Mining, pp. 828-833. Maryland (2006)
25. L. Chaisorn T.S. Chua C.H. Lee Y. Zhao H. Xu H. Feng Q. Tian 2003 A multi-modal approach to story segmentation for news video World Wide Web 6 187 208 10.1023/A:1023622605600 (Pubitemid 39020666)
26. Chang, S.F., Manmatha, R., Chua, T.S.: Combining text and audio-visual features in video indexing. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 5, pp. 1005-1008. IEEE Computer Society, Philadelphia (2005)
27. Chen, Q., Aickelin, U.: Anomaly detection using the dempster-shafer method. In: International Conference on Data Mining, pp. 232-240. Las Vegas (2006)
28. Chetty, G., Wagner, M.: Audio-visual multimodal fusion for biometric person authentication and liveness verification. In: NICTA-HCSNet Multimodal User Interaction Workshop, pp. 17-24. Sydney (2006)
29. Chieu, H.L., Lee, Y.K.: Query based event extraction along a timeline. In: International ACM Conference on Research and Development in Information Retrieval, pp. 425-432. Sheffield (2004)
30. Choudhury, T., Rehg, J.M., Pavlovic, V., Pentland, A.: Boosting and structure learning in dynamic bayesian networks for audio-visual speaker detection. In: The 16th International Conference on Pattern Recognition, vol. 3, pp. 789-794. Quebec (2002)
31. Chua, T.S., Chang, S.F., Chaisorn, L., Hsu, W.: Story boundary detection in large broadcast news video archives: techniques, experience and trends. In: ACM International Conference on Multimedia, pp. 656-659. New York, USA (2004)
32. Corradini, A., Mehta, M., Bernsen, N., Martin, J., Abrilian, S.: Multimodal input fusion in human-computer interaction. In: NATO-ASI Conference on Data Fusion for Situation Monitoring, Incident Detection, Alert and Response Management. Karlsruhe University, Germany (2003)
33. D. Crisan A. Doucet 2002 A survey of convergence results on particle filtering methods for practitioners IEEE Trans. Signal Process. 50 3 736 746 10.1109/78.984773 1895071 (Pubitemid 34295113)
34. Cutler, R., Davis, L.: Look who's talking: Speaker detection using video and audio correlation. In: IEEE International Conference on Multimedia and Expo, pp. 1589-1592. New York City (2000)
35. Darrell, T., Fisher III, J.W., Viola, P., Freeman, W.: Audio-visual segmentation and "the cocktail party effect". In: International Conference on Multimodal Interfaces. Bejing (2000)
36. Datcu, D., Rothkrantz, L.J.M.: Facial expression recognition with relevance vector machines. In: IEEE International Conference on Multimedia and Expo, pp. 193-196. Amsterdam, The Netherlands (2005)
37. R. Debouk S. Lafortune D. Teneketzis 2002 On an optimal problem in sensor selection J. Discret. Event Dyn. Syst. Theory Appl. 12 417 445 1087.93059 10.1023/A:1019770124060 1926147
38. Ding, Y., Fan, G.: Segmental hidden markov models for view-based sport video analysis. In: International Workshop on Semantic Learning Applications in Multimedia. Minneapolis (2007)
39. Fisher-III, J., Darrell, T., Freeman, W., Viola, P.: Learning joint statistical models for audio-visual fusion and segregation. In: Advances in Neural Information Processing Systems, pp. 772-778. Denver (2000)
40. Foresti, G.L., Snidaro, L.: A distributed sensor network for video surveillance of outdoor environments. In: IEEE International Conference on Image Processing. Rochester (2002)
41. Gandetto, M., Marchesotti, L., Sciutto, S., Negroni, D., Regazzoni, C.S.: From multi-sensor surveillance towards smart interactive spaces. In: IEEE International Conference on Multimedia and Expo, pp. I:641-644. Baltimore (2003)
42. Garcia Salicetti, S., Beumier, C., Chollet, G., Dorizzi, B., les Jardins, J., Lunter, J., Ni, Y., Petrovska Delacretaz, D.: BIOMET: A multimodal person authentication database including face, voice, fingerprint, hand and signature modalities. In: International Conference on Audio-and Video-Based Biometrie Person Authentication, pp. 845-853. Guildford, UK (2003)
43. Gehrig, T., Nickel, K., Ekenel, H., Klee, U., McDonough, J.: Kalman filters for audio-video source localization. In: IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, pp. 118- 121. Karlsruhe University, Germany (2005)
44. Guironnet, M., Pellerin, D., Rombaut, M.: Video classification based on low-level feature fusion model. In: The 13th European Signal Processing Conference. Antalya, Turkey (2005)
45. Hall, D.L., Llinas, J.: An introduction to multisensor fusion. In: Proceedings of the IEEE: Special Issues on Data Fusion, vol. 85, no. 1, pp. 6-23 (1997)
46. Hershey, J., Attias, H., Jojic, N., Krisjianson, T.: Audio visual graphical models for speech processing. In: IEEE International Conference on Speech, Acoustics, and Signal Processing, pp. 649-652. Montreal (2004)
47. Hershey, J., Movellan, J.: Audio-vision: using audio-visual synchrony to locate sounds. In: Advances in Neural Information Processing Systems, pp. 813-819. MIT Press, USA (2000)
48. G.E. Hinton S. Osindero Y. Teh 2006 A fast learning algorithm for deep belief nets Neural Comput. 18 7 1527 1554 1106.68094 10.1162/neco.2006.18.7.1527 2224485 (Pubitemid 44024729)
49. Holzapfel, H., Nickel, K., Stiefelhagen, R.: Implementation and evaluation of a constraint-based multimodal fusion system for speech and 3d pointing gestures. In: ACM International Conference on Multimodal Interfaces, pp. 175-182. State College, PA (2004)
50. Hossain, M.A., Atrey, P.K., El Saddik, A.: Smart mirror for ambient home environment. In: The 3rd IET International Conference on Intelligent Environments, pp. 589-596. Ulm (2007)
51. Hossain, M.A., Atrey, P.K., El Saddik, A.: Modeling and assessing quality of information in multi-sensor multimedia monitoring systems. ACM Trans. Multimed. Comput. Commun. Appl. 7(1) (2011)
52. Hsu, W., Kennedy, L., Huang, C.W., Chang, S.F., Lin, C.Y.: News video story segmentation using fusion of multi-level multi-modal features in TRECVID 2003. In: International Conference on Acoustics Speech and Signal Processing. Montreal, QC (2004)
53. Hsu, W.H.M., Chang, S.F.: Generative, discriminative, and ensemble learning on multi-modal perceputal fusion toward news stroy segmentation. In: IEEE International Conference on Multimedia and Expos, pp. 1091-1094. Taipei (2004)
54. Hu, H., Gan, J.Q.: Sensors and data fusion algorithms in mobile robotics. Technical report, CSM-422, Department of Computer Science, University of Essex, UK (2005)
55. Hua, X.S., Zhang, H.J.: An attention-based decision fusion scheme for multimedia information retrieval. In: The 5th Pacific-Rim Conference on Multimedia. Tokyo, Japan (2004)
56. Isler, V., Bajcsy, R.: The sensor selection problem for bounded uncertainty sensing models. In: International Symposium on Information Processing in Sensor Networks, pp. 151-158. Los Angeles (2005)
57. Iyengar, G., Nock, H.J., Neti, C.: Audio-visual synchrony for detection of monologue in video archives. In: IEEE International Conference on Acoustics, Speech, and Signal Processing. Hong Kong (2003)
58. Iyengar, G., Nock, H.J., Neti, C.: Discriminative model fusion for semantic concept detection and annotation in video. In: ACM International Conference on Multimedia, pp. 255-258. Berkeley (2003)
59. Jaffre, G., Pinquier, J.: Audio/video fusion: a preprocessing step for multimodal person identification. In: International Workshop on MultiModal User Authentification. Toulouse, France (2006)
60. Jaimes, A., Sebe, N.: Multimodal human computer interaction: a survey. In: IEEE International Workshop on Human Computer Interaction. Beijing (2005)
61. A. Jain K. Nandakumar A. Ross 2005 Score normalization in multimodal biometric systems Pattern Recognit. 38 12 2270 2285 10.1016/j.patcog.2005.01.012 (Pubitemid 41336698)
62. Jasinschi, R.S., Dimitrova, N., McGee, T., Agnihotri, L., Zimmerman, J., Li, D., Louie, J.: A probabilistic layered framework for integrating multimedia content and context information. In: International Conference on Acoustics, Speech and Signal Processing, vol. II, pp. 2057-2060. Orlando (2002)
63. Jeon, J., Manmatha, R.: Using maximum entropy for automatic image annotation. In: International Conference on Image and Video Retrieval, vol. 3115, pp. 24-32. Dublin (2004)
64. S. Jiang R. Kumar H.E. Garcia 2003 Optimal sensor selection for discrete event systems with partial observation IEEE Trans. Automat. Contr. 48 369 381 10.1109/TAC.2003.809144 1962246
65. Julier, S.J., Uhlmann, J.K.: New extension of the Kalman filter to nonlinear systems. In: Signal Processing, Sensor Fusion, and Target Recognition VI, vol. 3068 SPIE, pp. 182-193. San Diego (1997)
66. R.E. Kalman 1960 A new approach to linear filtering and prediction problems Trans. ASME J. Basic Eng. 82 Series D 35 45
67. M.S. Kankanhalli J. Wang R. Jain 2006 Experiential sampling in multimedia systems IEEE Trans. Multimed. 8 5 937 946 10.1109/TMM.2006.879876 (Pubitemid 44523108)
68. M.S. Kankanhalli J. Wang R. Jain 2006 Experiential sampling on multiple data streams IEEE Trans. Multimed. 8 5 947 955 10.1109/TMM.2006.879875 (Pubitemid 44523109)
69. J. Kittler M. Hatef R.P. Duin J. Matas 1998 On combining classifiers IEEE Trans. Pattern Anal. Mach. Intell. 20 3 226 239 10.1109/34.667881
70. Lam, K.Y., Cheng, R., Liang, B.Y., Chau, J.: Sensor node selection for execution of continuous probabilistic queries in wireless sensor networks. In: ACM International Workshop on Video Surveillance and Sensor Networks, pp. 63-71. NY, USA (2004)
71. T. León P. Zuccarello G. Ayala E. de Ves J. Domingo 2007 Applying logistic regression to relevance feedback in image retrieval systems Pattern Recognit. 40 10 2621 2632 1132.68642 10.1016/j.patcog.2007.02.002 (Pubitemid 46782861)
72. Li, D., Dimitrova, N., Li, M., Sethi, I.K.: Multimedia content processing through cross-modal association. In: ACM International Conference on Multimedia (2003)
73. Li, F.F., Perona, P.: A bayesian hierarchical model for learning natural scene categories. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 524-531. Washington (2005)
74. Li, M., Li, D., Dimitrove, N., Sethi, I.K.: Audio-visual talking face detection. In: International Conference on Multimedia and Expo, pp. 473-476. Baltimore, MD (2003)
75. X. Liu L. Zhang M. Li H. Zhang D. Wang 2005 Boosting image classification with lda-based feature combination for digital photograph management Pattern Recognit. 38 6 887 901 10.1016/j.patcog.2004.11.008 (Pubitemid 40308637)
76. Liu, Y., Zhang, D., Lu, G., Tan, A.H.: Integrating semantic templates with decision tree for image semantic learning. In: The 13th International Multimedia Modeling Conference, pp. 185-195. Singapore (2007)
77. Loh, A., Guan, F., Ge, S.S.: Motion estimation using audio and video fusion. In: International Conference on Control, Automation, Robotics and Vision, vol. 3, pp. 1569-1574 (2004)
78. Lucey, S., Sridharan, S., Chandran, V.: Improved speech recognition using adaptive audio-visual fusion via a stochastic secondary classifier. In: International Symposium on Intelligent Multimedia, Video and Speech Processing, pp. 551-554. Hong Kong (2001)
79. R.C. Luo C.C. Yih K.L. Su 2002 Multisensor fusion and integration: Approaches, applications, and future research directions IEEE Sens. J. 2 2 107 119 10.1109/JSEN.2002.1000251 (Pubitemid 44357916)
80. Magalhães, J., Rüger, S.: Information-theoretic semantic multimedia indexing. In: International Conference on Image and Video Retrieval, pp. 619-626. Amsterdam, The Netherlands (2007)
81. Makkook, M.A.: A multimodal sensor fusion architecture for audio-visual speech recognition. MS Thesis, University of Waterloo, Canada (2007)
82. Matas, J., Hamouz, M., Jonsson, K., Kittler, J., Li, Y., Kotropoulos, C., Tefas, A., Pitas, I., Tan, T., Yan, H., Smeraldi, F., Capdevielle, N., Gerstner, W., Abdeljaoued, Y., Bigun, J., Ben-Yacoub, S., Mayoraz, E.: Comparison of face verification results on the XM2VTS database. p. 4858. Los Alamitos, CA, USA (2000)
83. McDonald, K., Smeaton, A.F.: A comparison of score, rank and probability-based fusion methods for video shot retrieval. In: International Conference on Image and Video Retrieval, pp. 61-70. Singapore (2005)
84. Mena, J.B., Malpica, J.: Color image segmentation using the dempster-shafer theory of evidence for the fusion of texture. In: International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. XXXIV, Part 3/W8, pp. 139-144. Munich, Germany (2003)
85. G.F. Meyer J.B. Mulligan S.M. Wuerger 2004 Continuous audio-visual digit recognition using N-best decision fusion J. Inf. Fusion 5 91 101 10.1016/j.inffus.2003.07.001 (Pubitemid 38488057)
86. A.V. Nefian L. Liang X. Pi X. Liu K. Murphye 2002 Dynamic bayesian networks for audio-visual speech recognition EURASIP J. Appl. Signal Process. 11 1 15
87. Neti, C., Maison, B., Senior, A., Iyengar, G., Cuetos, P., Basu, S., Verma, A.: Joint processing of audio and visual information for multimedia indexing and human-computer interaction. In: International Conference RIAO. Paris, France (2000)
88. Ni, J.,, Ma, X., Xu, L., Wang, J.: An image recognition method based on multiple bp neural networks fusion. In: IEEE International Conference on Information Acquisition (2004)
89. Nickel, K., Gehrig, T., Stiefelhagen, R., McDonough, J.: A joint particle filter for audio-visual speaker tracking. In: The 7th International Conference on Multimodal Interfaces, pp. 61-68. Torento, Italy (2005)
90. Nock, H.J., Iyengar, G., Neti, C.: Assessing face and speech consistency for monologue detection in video. In: ACM International Conference on Multimedia. French Riviera, France (2002)
91. Nock, H.J., Iyengar, G., Neti, C.: Speaker localisation using audio-visual synchrony: an empirical study. In: International Conference on Image and Video Retrieval. Urbana, USA (2003)
92. Noulas, A.K., Krose, B.J.A.: Em detection of common origin of multi-modal cues. In: International Conference on Multimodal Interfaces, pp. 201-208. Banff (2006)
93. J. Ortega-Garcia J. Fierrez-Aguilar D. Simon J. Gonzalez M. Faundez-Zanuy V. Espinosa A. Satue I. Hernaez J.J. Igarza C. Vivaracho D. Escudero Q.I. Moro 2003 Biometric on the internet MCYT baseline corpus: a bimodal biometric database IEE Proc. Vis. Image Signal Process. 150 6 395 401 10.1049/ip-vis: 20031078
94. Y. Oshman 1994 Optimal sensor selection strategy for discrete-time state estimators IEEE Trans. Aerosp. Electron. Syst. 30 307 314 10.1109/7.272256
95. S. Oviatt 1999 Ten myths of multimodal interaction Commun. ACM 42 11 74 81 10.1145/319382.319398
96. S. Oviatt 2000 Taming speech recognition errors within a multimodal interface Commun. ACM 43 9 45 51 10.1145/348941.348979
97. Oviatt, S.L.: Multimodal interfaces. In: Jacko, J., Sears, A. (eds.) The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications. Lawrence Erlbaum Assoc., NJ (2003)
98. Pahalawatta, P., Pappas, T.N., Katsaggelos, A.K.: Optimal sensor selection for video-based target tracking in a wireless sensor network. In: IEEE International Conference on Image Processing, pp. V:3073-3076. Singapore (2004)
99. Perez, D.G., Lathoud, G., McCowan, I., Odobez, J.M., Moore, D.: Audio-visual speaker tracking with importance particle filter. In: IEEE International Conference on Image Processing (2003)
100. Pfleger, N.: Context based multimodal fusion. In: ACM International Conference on Multimodal Interfaces, pp. 265-272. State College (2004)
101. Pfleger, N.: Fade - an integrated approach to multimodal fusion and discourse processing. In: Dotoral Spotlight at ICMI 2005. Trento, Italy (2005)
102. Pitsikalis, V., Katsamanis, A., Papandreou, G., Maragos, P.: Adaptive multimodal fusion by uncertainty compensation. In: Ninth International Conference on Spoken Language Processing. Pittsburgh (2006)
103. Poh, N., Bengio, S.: How do correlation and variance of base-experts affect fusion in biometric authentication tasks? IEEE Trans. Signal Process. 53, 4384-4396 (2005)
104. N. Poh S. Bengio 2006 Database, protocols and tools for evaluating score-level fusion algorithms in biometric authentication Pattern Recognit. 39 2 223 233 10.1016/j.patcog.2005.06.011 (Part Special Issue: Complexity Reduction) (Pubitemid 41586091)
105. Potamianos, G., Luettin, J., Neti, C.: Hierarchical discriminant features for audio-visual LVSCR. In: IEEE International Conference on Acoustic Speech and Signal Processing, pp. 165-168. Salt Lake City (2001)
106. G. Potamianos C. Neti G. Gravier A. Garg A. Senior 2003 Recent advances in the automatic recognition of audiovisual speech Proc. IEEE 91 9 1306 1326 10.1109/JPROC.2003.817150 (Pubitemid 40890816)
107. I. Potamitis H. Chen G. Tremoulis 2004 Tracking of multiple moving speakers with multiple microphone arrays IEEE Trans. Speech Audio Process. 12 5 520 529 10.1109/TSA.2004.833004
108. Radova, V., Psutka, J.: An approach to speaker identification using multiple classifiers. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, 2, 1135-1138. Munich, Germany (1997)
109. Rashidi, A., Ghassemian, H.: Extended dempster-shafer theory for multi-system/sensor decision fusion. In: Commission IV Joint Workshop on Challenges in Geospatial Analysis, Integration and Visualization II, pp. 31-37. Germany (2003)
110. Reddy, B.S.: Evidential reasoning for multimodal fusion in human computer interaction (2007). MS Thesis, University of Waterloo, Canada
111. Ribeiro, M.I.: Kalman and extended Kalman filters: concept, derivation and properties. Technical report., Institute for Systems and Robotics, Lisboa (2004)
112. S. Roweis Z. Ghahramani 1999 A unifying review of linear gaussian models Neural Comput. 11 2 305 345 10.1162/089976699300016674
113. C. Sanderson K.K. Paliwal 2004 Identity verification using speech and face information Digit. Signal Process. 14 5 449 480 10.1016/j.dsp.2004.05.001
114. S. Satoh Y. Nakamura T. Kanade 1999 Name-It: naming and detecting faces in news video IEEE Multimed. 6 1 22 35 10.1109/93.752960
115. Siegel, M., Wu, H.: Confidence fusion. In: IEEE International Workshop on Robot Sensing, pp. 96-99 (2004)
116. R. Singh M. Vatsa A. Noore S.K. Singh 2006 Dempster-shafer theory based finger print classifier fusion with update rule to minimize training time IEICE Electron. Express 3 20 429 435 10.1587/elex.3.429
117. Slaney, M., Covell, M.: Facesync: A linear operator for measuring synchronization of video facial images and audio tracks. In: Neural Information Processing Society, vol. 13 (2000)
118. Smeaton, A.F., Over, P., Kraaij, W.: High-level feature detection from video in TRECVid: a 5-year retrospective of achievements. In: Divakaran, A. (ed.) Multimedia Content Analysis, Theory and Applications, pp. 151-174. Springer, Berlin (2009)
119. Snoek, C.G.M., Worring, M.: A review on multimodal video indexing. In: IEEE International Conference on Multimedia and Expo, pp. 21-24. Lusanne, Switzerland (2002)
120. C.G.M. Snoek M. Worring 2005 Multimodal video indexing: a review of the state-of-the-art Multimed. Tools Appl. 25 1 5 35 10.1023/B:MTAP.0000046380. 27575.a5
121. Snoek, C.G.M., Worring, M., Smeulders, A.W.M.: Early versus late fusion in semantic video analysis. In: ACM International Conference on Multimedia, pp. 399-402. Singapore (2005)
122. Sridharan, H., Sundaram, H., Rikakis, T.: Computational models for experiences in the arts and multimedia. In: The ACM Workshop on Experiential Telepresence. Berkeley, CA (2003)
123. Stauffer, C.: Automated audio-visual activity analysis. Tech. rep., MIT-CSAIL-TR-2005-057, Massachusetts Institute of Technology, Cambridge, MA (2005)
124. N. Strobel S. Spors R. Rabenstein 2001 Joint audio-video object localization and tracking IEEE Signal Process. Mag. 18 1 22 31 10.1109/79.911196 (Pubitemid 32287668)
125. Talantzis, F., Pnevmatikakis, A., Polymenakos, L.C.: Real time audio-visual person tracking. In: IEEE 8th Workshop on Multimedia Signal Processing, pp. 243-247. IEEE Computer Society, Victoria, BC (2006)
126. Tatbul, N., Buller, M., Hoyt, R., Mullen, S., Zdonik, S.: Confidence-based data management for personal area sensor networks. In: The Workshop on Data Management for Sensor Networks (2004)
127. Tavakoli, A., Zhang, J., Son, S.H.: Group-based event detection in undersea sensor networks. In: Second International Workshop on Networked Sensing Systems. San Diego, CA (2005)
128. P. Teissier A. Guerin-Dugue J.L. Schwartz 1998 Models for audiovisual fusion in a noisy-vowel recognition task J. VLSI Signal Process. 20 25 44 10.1023/A:1008014206206
129. Teriyan, V.Y., Puuronen, S.: Multilevel context representation using semantic metanetwork. In: International and Interdisciplinary Conference on Modeling and Using Context, pp. 21-32. Rio de Janeiro, Brazil (1997)
130. Tesic, J., Natsev, A., Lexing, X., Smith, J.R.: Data modeling strategies for imbalanced learning in visual search. In: IEEE International Conference on Multimedia and Expo, pp. 1990-1993. Beijing (2007)
131. C. Town 2007 Multi-sensory and multi-modal fusion for sentient computing Int. J. Comput. Vis. 71 235 253 10.1007/s11263-006-7834-8 (Pubitemid 44359123)
132. Vermaak, J., Gangnet, M., Blake, A., Perez, P.: Sequential monte carlo fusion of sound and vision for speaker tracking. In: The 8th IEEE International Conference on Computer Vision, vol. 1, pp. 741-746. Paris, France (2001)
133. Voorhees, E.M., Gupta, N.K., Johnson-Laird, B.: Learning collection fusion strategies. In: ACM International Conference on Research and Development in Information Retrieval, pp. 172-179. Seattle, WA (1995)
134. Wall, M.E., Rechtsteiner, A., Rocha, L.M.: Singular Value Decomposition and Principal Component Analysis, Chap. 5, pp. 91-109. Kluwel, Norwell, MA (2003)
135. Wang, J., Kankanhalli, M.S.: Experience-based sampling technique for multimedia analysis. In: ACM International Conference on Multimedia, pp. 319-322. Berkeley, CA (2003)
136. Wang, J., Kankanhalli, M.S., Yan, W.Q., Jain, R.: Experiential sampling for video surveillance. In: ACM Workshop on Video Surveillance. Berkeley (2003)
137. S. Wang M. Dash L.T. Chia M. Xu 2007 Efficient sampling of training set in large and noisy multimedia data ACM Trans. Multimed. Comput. Commun. Appl. 3 3 14 10.1145/1236471.1236473 (Pubitemid 47425936)
138. Wang, Y., Liu, Z., Huang, J.C.: Multimedia content analysis: using both audio and visual clues. In: IEEE Signal Processing Magazine, pp. 12-36 (2000)
139. Westerveld, T.: Image retrieval: content versus context. In: RIAO Content-Based Multimedia Information Access. Paris, France (2000)
140. Wu, H.: Sensor data fusion for context-aware computing using dempster-shafer theory. Ph.D. thesis, The Robotics Institute, Carnegie Mellon University, Pittsburgh, PA (2003)
141. Wu, K., Lin, C.K., Chang, E., Smith, J.R.: Multimodal information fusion for video concept detection. In: IEEE International Conference on Image Processing, pp. 2391-2394. Singapore (2004)
142. Wu, Y., Chang, E., Tsengh, B.L.: Multimodal metadata fusion using causal strength. In: ACM International Conference on Multimedia, pp. 872-881. Singapore (2005)
143. Wu, Y., Chang, E.Y., Chang, K.C.C., Smith, J.R.: Optimal multimodal fusion for multimedia data analysis. In: ACM International Conference on Multimedia, pp. 572-579. New York City, NY (2004) (Pubitemid 40211831)
144. Wu, Z., Cai, L., Meng, H.: Multi-level fusion of audio and visual features for speaker identification. In: International Conference on Advances in Biometrics, pp. 493-499 (2006) (Pubitemid 43856410)
145. Xie, L., Kennedy, L., Chang, S.F., Divakaran, A., Sun, H., Lin, C.Y.: Layered dynamic mixture model for pattern discovery in asynchronous multi-modal streams. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 2, pp. 1053-1056. Philadelphia, USA (2005)
146. N. Xiong P. Svensson 2002 Multi-sensor management for information fusion: issues and approaches Inf. Fusion 3 163 186(24) 10.1016/S1566-2535(02)00055-6
147. C. Xu J. Wang H. Lu Y. Zhang 2008 A novel framework for semantic annotation and personalized retrieval of sports video IEEE Trans. Multimed. 10 3 421 436 10.1109/TMM.2008.917346 (Pubitemid 351459505)
148. C. Xu Y.F. Zhang G. Zhu Y. Rui H. Lu Q. Huang 2008 Using webcast text for semantic event detection in broadcast sports video IEEE Trans. Multimed. 10 7 1342 1355 10.1109/TMM.2008.2004912
149. H. Xu T.S. Chua 2006 Fusion of AV features and external information sources for event detection in team sports video ACM Trans. Multimed. Comput. Commun. Appl. 2 1 44 67 10.1145/1126004.1126007
150. Yan, R.: Probabilistic models for combining diverse knowledge sources in multimedia retrieval. Ph.D. thesis. Carnegie Mellon University (2006)
151. Yan, R., Yang, J., Hauptmann, A.: Learning query-class dependent weights in automatic video retrieval. In: ACM International Conference on Multimedia, pp. 548-555. New York, USA (2004)
152. M.T. Yang S.C. Wang Y.Y. Lin 2005 A multimodal fusion system for people detection and tracking International Journal of Imaging Systems and Technology 15 131 142 10.1002/ima.20046 (Pubitemid 41633027)
153. W. Zhao R. Chellappa P.J. Phillips A. Rosenfeld 2003 Face recognition: a literature survey ACM Comput. Surv. 35 4 399 458 10.1145/954339.954342
154. Q. Zhou J. Aggarwal 2006 Object tracking in an outdoor environment using fusion of features and cameras Image Vis. Comput. 24 11 1244 1255 10.1016/j.imavis.2005.06.008 (Pubitemid 44485264)
155. Zhou, Z.H.: Learning with unlabeled data and its application to image retrieval. In: The 9th Pacific Rim International Conference on Artificial Intelligence, pp. 5-10. Guilin (2006)
156. Zhu, Q., Yeh, M.C., Cheng, K.T.: Multimodal fusion using learned text concepts for image categorization. In: ACM International Conference on Multimedia, pp. 211-220. Santa Barbara (2006)
157. Zotkin, D.N., Duraiswami, R., Davis, L.S.: Joint audio-visual tracking using particle filters. EURASIP J. Appl. Signal Process. (11), 1154-1164 (2002)
158. Zou, X., Bhanu, B.: Tracking humans using multimodal fusion. In: IEEE Conference on Computer Vision and Pattern Recognition, p. 4. Washington (2005)