Video retrieval based on semantic concepts

Proceedings of the IEEE

Volumn 96, Issue 4, 2008, Pages 602-622

  Hauptmann, Alexander G ^a   Christel, Michael G ^a   Yan, Rong ^b  

^a Carnegie Mellon University   (United States)

^b IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

Concept based video retrieval; High level semantic concepts; Information retrieval; Multimedia computing

Indexed keywords

INFORMATION RETRIEVAL; SEMANTICS; TEXTURES;

CONCEPT SELECTION; HIGH LEVEL SEMANTICS; INTERACTIVE RETRIEVAL; MULTIMEDIA COMPUTING; PSEUDO RELEVANCE FEEDBACK; REASONABLE ACCURACY; RELEVANCE FEEDBACK; VIDEO RETRIEVAL;

SEARCH ENGINES;

EID: 64549150985     PISSN: 00189219     EISSN: None     Source Type: Journal    
DOI: 10.1109/JPROC.2008.916355     Document Type: Article

References (73)

1. A. W. M. Smeulders et al., "Content based image retrieval at the end of the early years," IEEE Trans. Pattern Anal. Machine Intell, vol. 22, no. 12, pp. 1349-1380, Dec. 2000.
2. M. Lew, Proc. Int. Conf. Image and Video Retrieval. 2002, London, U.K., 2002.
3. J. R. Smith, S. Basu, C.Ling-Y. Lin, M. Naphade, and B. Tseng, "Interactive content-based retrieval of video," in Proc. IEEE Int. Conf. Image Processing (ICIP), Rochester, NY, 2002.
4. H. D. Wactlar et al., "Lessons learned from the creation and deployment of a terabyte digital video library," IEEE Computer, vol. 32, no. 2, pp. 66-73, Feb. 1999.
5. M. Markkula and E. Sormunen, "End-user searching challenges: Indexing practices in the digital newspaper photo archive," Inform. Retrieval, vol. 1, no. 4, pp. 259-285, 2000.
6. K. Rodden et al., "Does organisation by similarity assist image browsing?" in Proc. SIGCHI Conf. Human Factors in Computing Systems, New York, 2001.
7. S. F. Chang, R. Manmatha, and T. S. Chua, "Combining text and audio-visual features in video indexing," Proc. IEEE ICASSP'05, 2005.
8. K. Barnard et al., "Matching words and pictures," J. Machine Learning Res., vol. 3, 2002.
9. M. R. Naphade et al., "Probabilistic multimedia objects (multijects): A novel approach to video indexing and retrieval in multimedia systems," in Proc ICIP, 1998.
10. C.-Y. Lin, B. L. Tseng, and M. Naphade, "VideoAL: A novel end-to-end MPEG-7 automatic labeling system," in Proc. IEEE Int. Conf. Image Processing, Barcelona, 2003.
11. J. Yang, A. Hauptmann, and M.-Y. Chen, "Finding person X: Correlating names with visual appearances, " in Proc. Int. Conf. Image and Video Retrieval (CIVR'04), U.K., 2004, Dublin City Univ.
12. W. Lin and A. Hauptmann, "News video classification using SVM-based multimodalClassifiers and combination strategies," in Proc. ACM Multimedia 2002, Juan-les-Pins, France, 2002.
13. J. Jeon, V. Lavrenko, and R. Manmatha, "Automatic image annotation and retrieval using cross-media relevance models," in Proc. 26th Annu. Int. ACM SIGIR Conf. Research Development in Information Retrieval, 2003.
14. Y. Wu et al., "Optimal multimodal fusion for multimedia data analysis," in Proc. 12th Annu. ACM Int. Conf. Multimedia, 2004.
15. A. Hauptmann et al., "Informedia at TRECVID 2003: Analyzing and searching broadcast news video," in Proc. (VIDEO) TREC 2003 (Twelfth Text Retrieval Conf.), Gaithersburg, MD, 2003.
16. A. P. Natsev, M. R. Naphade, and J. Teísi'c, "Learning the semantics of multimedia queries and concepts from a small number of examples," in Proc. 13th ACM Int. Conf. Multimedia, 2005.
17. P. Over et al., "TRECVID 2006-An overview," in Proc. TRECVTD'06 Video Retrieval Evaluation, Gaithersburg, MD, 2006.
18. A. Smeaton et al., "The TREC2001 video track: Information retrieval on digital video information," in Proc 6th Eur. Conf. Digital Libraries, Rome, Italy, 2002.
19. S. Shatford, "Analyzing the subject of a picture: A theoretical approach," Cataloging Classification Quart., vol. 6, no. 3, pp. 39-62, 1986.
20. A. Alexander and T. Meehleib, "The thesaurus for graphic materials: Its history, use, and future," Cataloging Classification Quart., vol. 31, no. 3/4, pp. 89-212, 2001.
21. T. Petersen and P. J. Barnett, Art & Architecture Thesaurus; Guide to Indexing and Cataloging. New York: Oxford Univ. Press, 1994.
22. H. Schneiderman, "Learning statistical structure for object detection," in Computer Analysis of Images and Patterns (CAIP). New York: Springer-Verlag, 2003.
23. P. Over. (2007). TRECVID: TREC Video Retrieval Evaluation. [Online]. Available: http://www-nlpir.nist.gov/projects/t01v/
24. M. R. Naphade et al, "Large-scale concept ontology for multimedia," IEEE MultiMedia, vol. 13, no. 3, pp. 86-91, Mar. 2006.
25. C. G. Snoek et al., "The challenge problem for automated detection of 101 semantic concepts in multimedia," in Proc 14th Annu. ACM Int. Conf. Multimedia-Multimedia '06, Santa Barbara, CA, 2006.
26. T. S. Chua et al., "Trecvid 2004 search and feature extraction task by NUS PRIS," in Proc. NIST TRECVID, Gaithersburg, MD, 2004.
27. T.-S. Chua et al., "Trecvid 2005 by NUS PRIS," in Proc. TRECVID-2005, Gaithersburg, MD, 2005.
28. B. Huurnink, Proc AutoSeefe: Towards a Fully Automated Video Search System, Amsterdam, The Netherlands, 2005.
29. J. Yuan et al., "Tsinghua Univ. at TRECVID 2005," in Proc. TRECVID 2005, 2005.
30. L. Kennedy et al, "Automatic discovery of query class dependent models for multimodal search," in Proc. ACM Multimedia, Singapore, 2005.
31. J. J. Rocchio, "Relevance feedback in information retrieval," in The SMART Retrieval System: Experiments in Automatic Document Processing. Englewood Cliffs, NJ: Prentice-Hall, 1971.
32. S. E. Robertson and K. S. Jones, "Relevance weighting of search terms," J. Amer. Soc. Inform. Sci, vol. 27, 1977.
33. J. Lafferty and C. Zhai, "Probabilistic relevance models based on document and query generation," in Language Modeling for Information Retrieval. New York: Springer, 2003.
34. T. Tao and C. Zhai, "Regularized estimation of mixture models for robust pseudo-relevance feedback," in Proc. 29th Annu. Int. ACM SIGIR Conf. Research Development in Information Retrieval, Seattle, WA, 2006.
35. V. Lavrenko and W. B. Croft, "Relevance based language models," in Proc. 24th Annu. Int. ACM SIGIR Conf. Research Development in Information Retrieval, New Orleans, LA, 2001.
36. J. Xu and W. B. Croft, "Improving the effectiveness of information retrieval with local context analysis," ACM Trans. Inform. Syst., vol. 18, no. 1, pp. 79-112, 2000.
37. P. Over et al., "TRECVID 2005-An overview," in Proc. TRECVID 2005 Video Retrieval Evaluation Online, Gaithersburg, MD, 2005.
38. N. Fuhr, "Probabilistic models in information retrieval," Computer J., vol. 35, no. 3, pp. 243-255,1992.
39. R. Nallapati, "Discriminative models for information retrieval," in Proc. 27th SIGIR Conf. Research Development in Information Retrieval, 2004.
40. W. B. Croft and D. J. Harper, "Using probabilistic models of document retrieval without relevance information," in Document Retrieval Syst., 1988, pp. 161-171.
41. J. Lafferty, A. McCallum, and F. Pereira, "Conditional random fields: Probabilistic models for segmenting and labeling sequence data," in Proc. 28th Int. Conf. Machine Learning (ICML), San Francisco, CA, 2001.
42. C. Peterson and J. Anderson, "A mean field theory learning algorithm for neural networks," Complex Syst., vol. 1, pp. 995-1019, 1987.
43. M. Collins, "Discriminative reranking for natural language parsing," in Proc. 17th Int. Conf. Machine Learning, 2000.
44. K. Tieu and P. Viola, "Boosting image retrieval," in Proc. Int. Conf. Computer Vision, 2001.
45. Y. Freund et al., "An efficient boosting algorithm for combining preferences," in Proc. 25th Int. Conf. Machine Learning, San Francisco, CA, 1998.
46. B. Taskar, M. F. Wong, and D. Koller, "Learning on the test data: Leveraging unseen features," in Proc. 20th Int. Conf. Machine Learning, 2003.
47. A. G. Hauptmann et al., "CMU Informedia's TRECVID 2005 skirmishes," in Proc. 2005 TREC Video Retrieval Evaluation (TRECVID), Gaithersburg, MD, 2005.
48. Y. Yang and J. O. Pederson, "A comparative study on feature selection in statistical learning of text categorization," in Proc. 14th Int. Conf. Machine Learning (ICML'97), Nashville, TN, 1997.
49. R. Yan, J. Yang, and A. G. Hauptmann, "Learning query-class dependent weights in automatic video retrieval," in Proc. 22th Annu. ACM Int. Conf. Multimedia, New York, 2004.
50. R. Yan, "Probabilistic models for combining diverse knowledge sources in multimedia retrieval," in Language Technologies institute, School of Computer Science, Carnegie Mellon Univ., Pittsburgh, PA, 2006.
51. L. Boldareva and D. Hiemstra, "Interactive content-based retrieval using pre-computed object-object similarities," in Proc. Int. Conf. Image and Video Retrieval, Dublin, U.K., 2004.
52. M. Christel and N. Moraveji, "Finding the right shots: Assessing usability and performance of a digital video library interface," in Proc. ACM Multimedia, New York, 2004.
53. L. Hollink et al., "Assessing user behaviour in news video retrieval," IEE Proc. Vision, Image, Signal Processing, vol. 152, no. 6, pp. 911-918, 2005.
54. M. Yang, B. Wildemuth, and G. Marchionini, "The relative effectiveness of concept-based versus content-based video retrieval," in Proc. ACM Multimedia, New York, 2004.
55. M. Christel and R. Conescu, "Addressing the challenge of visual information access from digital image and video libraries, " in Proc. Joint Conf. Digital Libraries (JCDL'05), Denver, CO, 2005.
56. C G. M. Snoek et al., "The MediaMill TRECVID 2004 semantic video search engine," in Proc. TRECVID, Gaithersburg, MD, 2004.
57. J. Nielsen, "Heuristic evaluation," in Usability Inspection Methods, J. Nielsen and R. L. Mack, Eds. New York: Wiley, 1994.
58. B. Shneiderman, D. Byrd, and W. B. Croft, "Clarifying search: A user-interface framework for text searches," D-Lib Mag., vol. 3, no. 1,1997.
59. M. Christel and R. Conescu, "Mining novice user activity with TRECVID interactive retrieval tasks," in Proc. int. Conf. Image and Video Retrieval (CIVR), Tempe, AZ, 2006.
60. J. Yang and A. G. Hauptmann, "Naming every individual in news video monologues," in Proc. ACM Multimedia, New York, 2004.
61. C G. M. Snoek et al., "The mediaMill TRECVID 2006 semantic video search engine," in Proc. TRECVID 2006 Online, Gaithersburg, MD.
62. C G. M. Snoek et al., "A learned lexicon-driven paradigm for interactive video retrieval," IEEE Trans. Multimedia, vol. 9, no. 2, pp. 280-292, Feb. 2007.
63. C G. M. Snoek et al., "Learned lexicon-driven interactive video retrieval" in Proc. Int. Conf. Image and Video Retrieval (CIVR), Tempe, AZ, 2006.
64. M. G. Christel et al., "Assessing the filtering and browsing utility of automatic semantic concepts for multimedia retrieval," in Proc. Conf Computer Vision and Pattern Recognition Workshop, New York, 2006.
65. A. G. Hauptmann et al., "Extreme video retrieval: Joint maximization of human and computer performance," in Proc 14th ACM Int. Conf. Multimedia, Santa Barbara, CA, 2006.
66. M. G. Christel and R. Yan, "Merging storyboard strategies and automatic retrieval for improving interactive video search," in Proc. Image and Video Retrieval (CIVR), Amsterdam, The Netherlands, 2007.
67. R. Yan and A. Hauptmann, "The combination limit of video retrieval," in Proc. ACM Multimedia 2003 (MM'03), Berkeley, CA, 2003.
68. A. Huyer and A. Neumaier, "Global optimization by multilevel coordinate search," J. Global Optimization, vol. 14, pp. 331-355, 1999.
69. S. M. Beitzel et al., "Surrogate scoring for improved metasearch precision," in Proc. 28th Annu. int. ACM SIGIR Conf. Research Development in Information Retrieval, New York, 2005.
70. S. Reed and D. Lenat, "Mapping ontologies into Cyc.," in Proc. AAAI Conf. Workshop Ontologies for the Semantic Web (AAAI'02), Edmonton, Canada, 2002.
71. H. Wang, S. Liu, and L.-T. Chia, "Does ontology help in image retrieval?" in Proc. 14th Annu. ACM Int. Conf. Multimedia, Santa Barbara, CA, 2006.
72. M. Srikanth et al., "Exploiting ontologies for automatic image annotation," in Proc. 28th Annu. Int. ACM SIGIR 2005), 2005.
73. M. G. Christel and A. Hauptmann, "The use and utility of high-level semantic features in video retrieval," in Proc. Int. Conf. Image and Video Retrieval (CIVR'05), Singapore, 2005.