Fast semantic diffusion for large-scale context-based image and video annotation

IEEE Transactions on Image Processing

Volumn 21, Issue 6, 2012, Pages 3080-3091

  Jiang, Yu Gang ^a   Dai, Qi ^a   Wang, Jun ^b   Ngo, Chong Wah ^c   Xue, Xiangyang ^a   Chang, Shih Fu ^d  

^a FUDAN UNIVERSITY   (China)

^b IBM T J WATSON RESEARCH CENTER   (United States)

^c CITY UNIVERSITY OF HONG KONG   (Hong Kong)

^d Columbia University ^*  (United States)

Author keywords

Context; Image and video annotation; Semantic concept; Semantic diffusion (SD)

Indexed keywords

CONTEXT; CONTEXT INFORMATION; CONTEXT-BASED; DATA SAMPLE; GRAPH-BASED LEARNING; PERFORMANCE GAIN; SEMANTIC CONCEPT; SEMANTIC CONTEXT; SEMANTIC GRAPHS; SOURCE CODES; TEST DATA; TV PROGRAMS; VIDEO ANNOTATIONS; VIDEO DATA; VISUAL RECOGNITION;

ALGORITHMS; DIFFUSION; GRAPHIC METHODS; LEARNING SYSTEMS;

SEMANTICS;

EID: 84861159415     PISSN: 10577149     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIP.2012.2188038     Document Type: Article

References (48)

1. Fast Semantic Diffusion for Context-Based Image and Video Annotation: Project Page and Source Codes [Online]. Available: http://www.ee.columbia.edu/ ln/dvmm/researchProjects/MultimediaIndexing/DASD/dasd.htm
2. L. Fei-Fei, R. Fergus, and P. Perona, "Learning generative visual models from few training examples: An incremental Bayesian approach tested on 101 object categories," in Proc. CVPR Workshop Generative-Model Based Vis., 2004, p. 178.
3. C. G. M. Snoek, M.Worring, J. C. van Gemert, J.-M. Geusebroek, and A. W.M. Smeulders, "The challenge problem for automated detection of 101 semantic concepts in multimedia," in Proc. ACM Int. Conf.Multimedia, Oct. 2006, pp. 421-430.
4. M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, The PASCAL visual object classes (VOC) challenge, vol. 88, no. 2, pp. 303-338, Jun. 2010.
5. J. Fan, Y. Gao, and H. Luo, "Integrating concept ontology and multitask learning to achieve more effective classifier training for multilevel image annotation," IEEE Trans. Image Process., vol. 17, no. 3, pp. 407-426, Mar. 2008.
6. Y.-G. Jiang, J. Yang, C.-W. Ngo, and A. G. Hauptmann, "Representations of keypoint-based semantic concept detection: A comprehensive study," IEEE Trans. Multimedia, vol. 12, no. 1, pp. 42-53, Jan. 2010.
7. Z. Lu, H. Ip, and Y. Peng, "Contextual kernel and spectral methods for learning the semantics of images," IEEE Trans. Image Process., vol. 20, no. 6, pp. 1739-1750, Jun. 2011.
8. A. Hauptmann, R. Yan, W.-H. Lin, M. Christel, and H. Wactlar, "Can high-level concepts fill the semantic gap in video retrieval? A case study with broadcast news," IEEE Trans. Multimedia, vol. 9, no. 5, pp. 958-966, Aug. 2007.
9. K. Grauman and T. Darrell, "The pyramid match kernel: Discriminative classification with sets of image features," in Proc. IEEE Int. Conf. Comput. Vis., 2005, pp. 1458-1465.
10. S. Lazebnik, C. Schmid, and J. Ponce, "Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2006, pp. 2169-2178.
11. A. Rabinovich, A. Vedaldi, C. Galleguillos, E. Wiewiora, and S. Belongie, "Objects in context," in Proc. IEEE Int. Conf. Comput. Vis., 2007, pp. 1-8.
12. G. J. Qi, X. S. Hua, Y. Rui, J. Tang, T. Mei, and H. J. Zhang, "Correlative multi-label video annotation," in Proc. ACM Multimedia, 2007, pp. 17-26.
13. A. Oliva and A. Torralba, "Modeling the shape of the scene: A holistic representation of the spatial envelope," Int. J. Comput. Vis., vol. 42, no. 3, pp. 145-175, May/Jun. 2001.
14. D. Lowe, "Distinctive image features from scale-invariant keypoints," Int. J. Comput. Vis., vol. 60, no. 2, pp. 91-110, Nov. 2004.
15. A. C. Berg and J. Malik, "Geometric blur for template matching," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2001, pp. I-607-I-614.
16. K. Yu, T. Zhang, and Y. Gong, "Nonlinear learning using local coordinate coding," in Proc. Adv. Neural Inf. Process. Syst., 2009, pp. 2223-2231.
17. R. Fergus, P. Perona, and A. Zisserman, "Object class recognition by unsupervised scale-invariant learning," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2003, pp. II-264-II-271.
18. M. Varma and D. Ray, "Learning the discriminative power-invariance trade-off," in Proc. IEEE Int. Conf. Comput. Vis., 2007, pp. 1-8.
19. L. Cao, J. Luo, F. Liang, and T. Huang, "Heterogeneous feature machines for visual recognition," in Proc. IEEE Int. Conf. Comput. Vis., 2009, pp. 1095-1102.
20. A. Torralba, "Contextual priming for object detection," Int. J. Comput. Vis., vol. 53, no. 2, pp. 169-191, Jul. 2003.
21. K.Murphy, A. Torralba, and W. Freeman, "Using the forest to see the tree: A graphical model relating features, objects and the scenes," Adv. Neural Inf. Process. Syst., pp. 1-8, 2003.
22. J. Shotton, J. Winn, C. Rother, and A. Criminisi, "Textonboost for image understanding: Multi-class object recognition and segmentation by jointly modeling appearance, shape and context," Int. J. Comput. Vis., vol. 81, no. 1, pp. 2-23, Jan. 2009.
23. A. Torralba, K. Murphy, W. Freeman, and M. Rubin, "Context-based vision system for place and object recognition," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2003, pp. 273-280.
24. L. Wolf and S. Bileschi, "A critical view of context," Int. J. Comput. Vis., vol. 69, no. 2, pp. 251-261, Aug. 2006.
25. P. Carbonetto, N. de Freitas, O. de Freitas, and K. Barnard, "A statistical model for general contextual object recognition," in Proc. Eur. Conf. Comput. Vis., 2004, pp. 350-362.
26. T.Malisiewicz andA. A. Efros, "Beyond categories:The visualmemex model for reasoning about object relationships," in Proc. Adv. Neural Inf. Process. Syst., 2009, pp. 1222-1230.
27. C. Galleguillos, A. Rabinovich, and S. Belongie, "Object categorization using co-occurrence, location and appearance," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2008, pp. 1-8.
28. Y. J. Lee and K. Grauman, "Object-graphs for context-aware category discovery," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2010, pp. 1-8.
29. J. J. Lim, P. Arbeláez, C. Gu, and J. Malik, "Context by region ancestry," in Proc. IEEE Int. Conf. Comput. Vis., 2009, pp. 1978-1985.
30. D. Lin, A. Kapoor, G. Hua, and S. Baker, "Joint people, event, and location recognition in personal photo collections using cross-domain context," in Proc. Eur. Conf. Comput. Vis., 2010, pp. 243-256.
31. N. Rasiwasia and N. Vasconcelos, "Holistic context modeling using semantic co-occurrences," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2009, pp. 1889-1895.
32. J. Tang, X.-S. Hua, M.Wang, Z. Gu, G.-J. Qi, and X.Wu, "Correlative linear neighborhood propagation for video annotation," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 39, no. 2, pp. 409-416, Apr. 2009.
33. M.-F. Weng and Y.-Y. Chuang, "Multi-cue fusion for semantic video indexing," in Proc. ACM Multimedia, 2008, pp. 71-80.
34. W. Jiang, S. F. Chang, and A. Loui, "Context-based concept fusion with boosted conditional random fields," in Proc. ICASSP, 2007, pp. I-949-I-952.
35. Z. Song, Q. Chen, Z. Huang, Y. Hua, and S. Yan, "Contextualizing object detection and classification," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2011, pp. 1585-1592.
36. Y.-G. Jiang, J. Wang, S.-F. Chang, and C.-W. Ngo, "Domain adaptive semantic diffusion for large scale context-based video annotation," in Proc. IEEE Int. Conf. Comput. Vis., 2009, pp. 1420-1427.
37. P. Perona and J. Malik, "Scale-space and edge detection using anisotropic diffusion," IEEE Trans. Pattern Anal. Mach. Intell., vol. 12, no. 7, pp. 629-639, Jul. 1990.
38. M. Proesmans, L. J. van Gool, E. Pauwels, and A. Oosterlinck, "Determination of optical flow and its discontinuities using nonlinear diffusion," in Proc. Eur. Conf. Comput. Vis., 1994, pp. 295-304.
39. D. Zhou, O. Bousquet, T. Lal, J. Weston, and B. Schölkopf, "Learning with local and global consistency," in Proc. Adv. Neural Inf. Process. Syst., 2004, pp. 321-328.
40. X. Zhu, Z. Ghahramani, and J. Lafferty, "Semi-supervised learning using Gaussian fields and harmonic functions," in Proc. Int. Conf. Mach. Learn., 2003, pp. 912-919.
41. A. D. Szlam, M. Maggioni, and R. R. Coifman, "Regularization on graphs with function-adapted diffusion processes," J. Mach. Learn. Res., vol. 9, pp. 1711-1739, Jun. 2008.
42. T.-S. Chua, J. Tang, R. Hong, H. Li, Z. Luo, and Y.-T. Zheng, "NUSWIDE: A real-world web image database from National University of Singapore," in Proc. ACM CIVR, 2009, pp. 1-9.
43. A. F. Smeaton, P. Over, and W. Kraaij, "Evaluation campaigns and TRECVID," in Proc. ACM MIR, 2006, pp. 321-330.
44. M. Naphade, J. R. Smith, J. Tesic, S.-F. Chang, W. Hsu, L. Kennedy, A. Hauptmann, and J. Curtis, "Large scale concept ontology for multimedia," IEEE Multimedia, vol. 13, no. 3, pp. 86-91, Jul.-Sep. 2006.
45. Y.-G. Jiang, A. Yanagawa, S.-F. Chang, and C.-W. Ngo, CU-VIREO374: Fusing Columbia374 and VIREO374 for large scale semantic concept detection Columbia Univ., Manhattan, NY, ADVENT Tech. Rep. 223-2008-1, Aug. 2008.
46. T. Jebara, J. Wang, and S.-F. Chang, "Graph construction and b-matching for semi-supervised learning," in Proc. Int. Conf. Mach. Learn., 2009, pp. 441-448.
47. E. Yilmaz and J. A. Aslam, "Estimating average precision with incomplete and imperfect judgments," in Proc. ACM Int. Conf. Inf. Knowl. Manage., 2006, pp. 102-111.
48. Y. Aytar, O. B. Orhan, and M. Shah, "Improving semantic concept detection and retrieval using contextual estimates," in Proc. ICME, 2007, pp. 536-539.