Semantic indexing of multimedia content using visual, audio, and text cues

Eurasip Journal on Applied Signal Processing

Volumn 2003, Issue 2, 2003, Pages 170-185

  Adams, W H ^a   Iyengar, Giridharan ^a   Lin, Ching Yung ^a   Naphade, Milind Ramesh ^a   Neti, Chalapathy ^a   Nock, Harriet J ^a   Smith, John R ^a  

^a IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

GMM; HMM; Multimodal information fusion; Query by keywords; Spoken document retrieval; Statistical modeling of multimedia; SVM; Video event detection; Video indexing and retrieval; Video TREC

Indexed keywords

INFORMATION ANALYSIS; LEARNING SYSTEMS; MARKOV PROCESSES; SEMANTICS; STATISTICAL METHODS;

MULTIMODAL INFORMATION FUSION;

MULTIMEDIA SYSTEMS;

EID: 0037299467     PISSN: 11108657     EISSN: None     Source Type: Journal    
DOI: 10.1155/S1110865703211173     Document Type: Review

References (33)

1. J. R. Smith and S.-F. Chang, "VisualSEEk: a fully automated content-based image query system," in Proc. 4th ACM International Conference on Multimedia, pp. 87-98, ACM, Boston, Mass, USA, November 1996.
2. M. Naphade, T. Kristjansson, B. Frey, and T. S. Huang, "Probabilistic multimedia objects (multijects): a novel approach to video indexing and retrieval in multimedia systems," in Proc. IEEE International Conference on Image Processing, vol. 3, pp. 536-540, IEEE, Chicago, 111, USA, October 1998.
3. S. F. Chang, W. Chen, and H. Sundaram, "Semantic visual templates - linking features to semantics," in Proc, IEEE International Conference on Image Processing, vol. 3, pp. 531-535, IEEE, Chicago, 111, USA, October 1998.
4. R. Qian, N. Hearing, and I. Sezan, "A computational approach to semantic event detection," in Proc. Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 200-206, IEEE, Fort Collins, Colo, USA, June 1999.
5. T. Zhang and C. Kuo, "An integrated approach to multimodal media content analysis," in Storage and Retrieval for Media Databases 2000, vol. 3972 of SPIE Proceedings, pp. 506-517, SPIE, San Jose, Calif, USA, January 2000.
6. D. Ellis, Prediction-driven computational auditory scene analysis, Ph.D. thesis, MIT Department of Electrical Engineering and Computer Science, Cambridge, Mass, USA, 1996.
7. K. Barnard and D. Forsyth, "Learning the semantics of words and pictures," in Proc. International Conf. on Computer Vision, vol. 2, pp. 408-415, IEEE, Vancouver, Canada, July 2001.
8. M. A. Casey, "Reduced-rank spectra and minimum-entropy priors as consistent and reliable cues for generalized sound recognition," in Proc. Eurospeech, Aalborg, Denmark, September 2001.
9. W. Wolf, "Hidden Markov model parsing of video programs," in Proc. IEEE Int. Conf. Acoustics, Speech, Signal Processing, vol. 4, pp. 2609-2611, IEEE, Munich, Germany, April 1997.
10. G. Iyengar and A. B. Lippman, "Models for automatic classification of video sequences," in Storage and Retrieval for Image and Video Databases VI, vol. 3312 of SPIE Proceedings, pp. 216-227, SPIE, San Jose, Calif, USA, January 1998.
11. A. M. Ferman and A. M. Tekalp, "Probabilistic analysis and extraction of video content," in Proc. IEEE International Conference on Image Processing, vol. 2, pp. 91-95, IEEE, Kobe, Japan, October 1999.
12. N. Vasconcelos and A. Lippman, "Bayesian modeling of video editing and structure: semantic features for video summarization and browsing," in Proc. IEEE International Conference on Image Processing, vol. 3, pp. 153-157, IEEE, Chicago, Ill, USA, 1998.
13. E. Scheirer and M. Slaney, "Construction and evaluation of a robust multifeature speech/music discriminator," in Proc. IEEE Int. Conf. Acoustics, Speech, Signal Processing, vol. 2, pp. 1331-1334, IEEE, Munich, Germany, April 1997.
14. B. Adams, C. Dorai, and S. Venkatesh, "Towards automatic extraction of expressive elements from motion pictures: Tempo," in Proc. IEEE International Conference on Multimedia and Expo, vol. II, pp. 641-645, IEEE, New York, NY, USA, July 2000.
15. E. M. Voorhees and D. K. Harman, Eds., The 10th Text REtrieval Conference (TREC 2001), vol. 500-250 of NIST Special Publication, NIST, Gaithersburg, Md, USA, 2001.
16. M. Davis, "Media Streams: an iconic visual language for video annotation," Telektronikk, vol. 89, no. 4, pp. 59-71, 1993.
17. C. Neti, G. Potamianos, J. Luettin, et al., "Audio-visual speech recognition," Final workshop 2000 report, Center for Language and Speech Processing, The Johns Hopkins University, Baltimore, Md, USA, October 2000.
18. G. Iyengar and C. Neti, "Speaker change detection using joint audio-visual statistics," in Proc. RIAO, Paris, France, April 2000.
19. C. M. Bishop, Neural Networks for Pattern Recognition, Oxford University Press, Oxford, UK, 1995.
20. L. R. Rabiner and B.-H. Juang, Fundamentals of Speech Recognition, Prentice-Hall, Englewood Cliffs, NJ, USA, 1st edition, 1993.
21. A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum likelihood from incomplete data via the EM algorithm," Journal of Royal Statistical Society, Series B, vol. 39, no. 1, pp. 1-38, 1977.
22. V. Vapnik, The Nature of Statistical Learning Theory, Springer-Verlag, New York, NY, USA, 1995.
23. B. Kingsbury, G. Saon, L. Mangu, M. Padmanabhan, and R. Sarikaya, "Robust speech recognition in noisy environments: The IBM SPINE-2 evaluation system," in Proc. IEEE Int. Conf. Acoustics, Speech, Signal Processing, Orlando, Fla, USA, May 2002.
24. M. Franz and S. Roukos, "TREC-6 Ad-hoc retrieval," in Proc. 6th Text REtrieval Conference (TREC-6), vol. 500-240 of NIST Special Publication, pp. 511-516, NIST, Gaithersburg, Md, USA, 1998.
25. C. Fellbaum, Ed., WordNet: An Electronic Lexical Database, MIT Press, Cambridge, Mass, USA, 1998.
26. S. E. Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, and M. Gatford, "Okapi at TREC-3," in The 3rd Text REtrieval Conference (TREC-3), vol. 500-225 of NIST Special Publication, pp. 109-126, NIST, Gaithersburg, Md, USA, 1995.
27. IBM Almaden Research Center, "The IBM cuevideo project," 1997, www.almaden.ibm.com/projects/cuevideo.shtml.
28. J. R. Smith, S. Srinivasan, A. Amir, et al., "Integrating features, models, and semantics for TREC video retrieval," in Proc. 10th Text REtrieval Conference (TREC 2001), vol. 500-250 of NIST Special Publication, pp. 240-249, NIST, Gaithersburg, Md, USA, 2001.
29. A. K. Jain and A. Vailaya, "Shape-based retrieval: A case study with trademark image databases," Pattern Recognition, vol. 31, no. 9, pp. 1369-1390, 1998.
30. A. K. Jain, A. Vailaya, and W. Xiong, "Query by video clip," Multimedia Systems: Special Issue on Video Libraries, vol. 7, no, 5, pp. 369-384, 1999.
31. S. Dudani, K. Breeding, and R. McGhee, "Aircraft identification by moment invariants," IEEE Trans. on Computers, vol. 26, no. 1, pp. 39-45, 1977.
32. R. Bakis, S. Sehen, P. Gopalakrishnan, R. Gopinath, S. Maes, and L. Polymenakos, "Transcription of broadcast news - system robustness issues and adaptation techniques," in Proc. IEEE Int. Conf. Acoustics, Speech, Signal Processing, vol. 2, pp. 711-714, IEEE, Munich, Germany, April 1997.
33. K. Murphy, "Bayes net toolbox for matlab," 2001, http://www.ai.mit.edu/̃murphyk/Software/BNT/bnt.html.