Joint Intermodal and Intramodal Label Transfers for Extremely Rare or Unseen Classes

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 39, Issue 7, 2017, Pages 1360-1373

  Qi, Guo Jun ^a   Liu, Wei ^b   Aggarwal, Charu ^b   Huang, Thomas ^c  

^a University of Central Florida   (United States)

^b IBM T J WATSON RESEARCH CENTER   (United States)

^c University of Illinois at Urbana Champaign   (United States)

Author keywords

image classification; intermodal and intramodal label transfers (I2LT); Multimodal analysis; zero shot learning

Indexed keywords

CLASSIFICATION (OF INFORMATION); MODAL ANALYSIS; SEMANTICS; TEXT PROCESSING;

CLASSIFICATION TASKS; FUNCTIONAL RELATIONSHIP; INTERMODAL AND INTRAMODAL LABEL TRANSFERS (I2LT); LEARNING SCENARIOS; MULTIMODAL ANALYSIS; NATURAL SEMANTICS; TEXT CLASSIFICATION; ZERO-SHOT LEARNING;

IMAGE CLASSIFICATION;

EID: 85020294607     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2016.2587643     Document Type: Article

References (49)

1. Z. Akata, F. Perronnin, Z. Harchaoui, and C. Schmid, "Labelembedding for attribute-based classification," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2013, pp. 819-826.
2. Z. Akata, S. Reed, D. Walter, H. Lee, and B. Schiele, "Evaluation of output embeddings for fine-grained image classification," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 2927-2936.
3. Y. Amit, M. Fink, N. Srebro, and S. Ullman, "Uncovering shared structures in multiclass classification," in Proc. 24th Int. Conf. Mach. Learn., 2007, pp. 17-24.
4. F. Bach, G. R. G. Lanckriet, and M. I. Jordan, "Multiple kernel learning, conic duality, and the SMO algorithm," in Proc. 21st Int. Conf. Mach. Learn., 2004, Art. no. 6.
5. D. P. Bertsekas, Nonlinear Programming. Belmont, MA, USA: Athena Scientific, 1999.
6. D. M. Blei, A. Y. Ng, and M. I. Jordan, "Latent Dirichlet allocation," J. Mach. Learn. Res., vol. 3, pp. 993-1022, Jan. 2003.
7. A. Blum and T. Mitchell, "Combining labeled and unlabeled data with co-training," in Proc. 11th Annu. Conf. Comput. Learn. Theory, 1998, pp. 92-100.
8. L. Bo and C. Sminchisescu, "Twin Gaussian processes for structured prediction," Int. J. Comput. Vis., vol. 87, no. 1/2, pp. 28-52, 2010.
9. S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge, U.K.: Cambridge Univ. Press, 2004.
10. J.-F. Cai, E. Candés, and Z. Shen, "A singular value thresholding algorithm for matrix completion," SIAM J. Optimization, vol. 20, pp. 1956-1982, Sep. 2010.
11. T.-S. Chua, J. Tang, R. Hong, H. Li, Z. Luo, and Y.-T. Zheng, "NUS-WIDE: A real-world web image database from national university of singapore," in Proc. ACM Conf. Image Video Retrieval, Jul. 8-10, 2009, Art. no. 48.
12. N. Cristianini and J. Shawe-Taylor, An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods. Cambridge, U.K.: Cambridge Univ. Press, 2000.
13. Y. Chen, T. V. Nguyen, M. S. Kankanhalli, J. Yuan, S. Yan, and M. Wang, "Audio matters in visual attention," IEEE Trans. Circuits Syst. Video Technol., vol. 24, no. 11, pp. 1992-2003, 2014.
14. L. Duan, I. W. Tsang, D. Xu, and S. J. Maybank, "Domain transfer SVM for video concept detection," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2009, pp. 1375-1381.
15. M. Elhoseiny, B. Saleh, and A. Elgammal, "Write a classifier: Zeroshot learning using purely textual descriptions," in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 2584-2591.
16. Y. Fu, T. M. Hospedales, T. Xiang, and S. Gong, "Learning multimodal latent attributes," IEEE Trans. Pattern Anal. Mach. Intell., vol. 36, no. 2, pp. 303-316, Feb. 2014.
17. Y. Gong, Q. Ke, M. Isard, and S. Lazebnik, "A multi-view embedding space for modeling internet images, tags, and their semantics," Int. J. Comput. Vis., vol. 106, no. 2, pp. 210-233, 2013.
18. S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Comput., vol. 9, no. 8, pp. 1735-1780, 1997.
19. T. Hofmann, "Probabilistic latent semantic analysis," in Proc. Uncertainty Artificial Intell., 1999, pp. 289-296.
20. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet classification with deep convolutional neural networks," in Proc. Neural Inf. Process. Syst., 2012, pp. 1097-1105.
21. B. Kulis, K. Saenko, and T. Darrell, "What you saw is not what you get: Domain adaptation using asymmetric kernel transforms," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2011, pp. 1785-1792.
22. C. H. Lampert, H. Nickisch, and S. Harmeling, "Learning to detect unseen object classes by between-class attribute transfer," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2009, pp. 951-958.
23. T. K. Landauer, P.W. Foltz, and D. Laham, "An introduction to latent semantic analysis," Discourse Processes, vol. 25, pp. 259-284, 1998.
24. W. Li, L. Duan, D. Xu, and I. W. Tsang, "Learning with augmented features for supervised and semi-supervised heterogeneous domain adaptation," IEEE Trans. Pattern Anal. Mach. Intell., vol. 36, no. 6, pp. 1134-1148, Jun. 2014.
25. W. Li, L. Niu, and D. Xu, "Exploiting privileged information from Web data for image categorization," in Proc. 13th Eur. Conf. Comput. Vis., Sep. 2014, pp. 437-452.
26. D. G. Lowe, "Distinctive image features from scale-invariant keypoints," Int. J. Comput. Vis., vol. 60, no. 2, pp. 91-110, 2004.
27. J. Masci, M. M. Bronstein, A. M. Bronstein, and J. Schmidhuber, "Multimodal similarity-preserving hashing," IEEE Trans. Pattern Anal. Mach. Intell., vol. 36, no. 4, pp. 824-830, Apr. 2014.
28. S. Moon, S. Kim, and H. Wang, "Multimodal transfer deep learning for audio visual recognition," arXiv:1412.3121, 2014.
29. J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng, "Multimodal deep learning," in Proc. 28th Int. Conf. Mach. Learn., 2011, pp. 689-696.
30. M.-E. Nilsback and A. Zisserman, "Automated flower classification over a large number of classes," in Proc. 6th Indian Conf. Comput. Vis. Graph. Image Process., 2008, pp. 722-729.
31. L. Niu, W. Li, and D. Xu, "Exploiting privileged information from Web data for action and event recognition," Int. J. Comput. Vis., vol. 118, no. 2, pp. 1-21, Nov. 2015.
32. V. Ordonez, G. Kulkarni, and T. L. Berg, "Im2Text: Describing images using 1 million captioned photographs," in Proc. Advances Neural Inf. Process. Syst., 2011, pp. 1143-1151.
33. M. Palatucci, D. Pomerleau, G. E. Hinton, and T. M. Mitchell, "Zero-shot learning with semantic output codes," in Proc. Advances Neural Inf. Process. Syst., 2009, pp. 1410-1418.
34. G.-J. Qi, C. Aggarwal, and T. Huang, "Towards semantic knowledge propagation from text corpus to Web images," in Proc. 20th Int. World Wide Web Conf., 2011, pp. 297-306.
35. G.-J. Qi, C. Aggarwal, Q. Tian, H. Ji, and T. S. Huang, "Exploring context and content links in socialmedia: A latent spacemethod," IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 5, pp. 850-862, May 2012.
36. G.-J. Qi, X.-S. Hua, and H.-J. Zhang, "Learning semantic distance from community-tagged media collection," in Proc. 17th ACM Int. Conf. Multimedia, 2009, pp. 243-252.
37. G.-J. Qi, Q. Tian, C. Aggarwal, Y. Rui, S. Chang, and T. Huang, "Towards cross-category knowledge propagation for learning visual concepts," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2011, pp. 897-904.
38. R. Raina, A. Battle, H. Lee, B. Packer, and A. Ng, "Self-taught learning: Transfer learning from unlabeled data," in Proc. 24th Int. Conf. Mach. Learn., 2007, pp. 759-766.
39. R. Raina, A. Ng, and D. Koller, "Constructing informative priors using transfer learning," in Proc. 23rd Int. Conf. Mach. Learn., 2006, pp. 713-720.
40. C. E. Rasmussen and C. K. I. Williams, Gaussian Processes for Machine Learning. Cambridge, MA, USA: MIT Press, 2006.
41. M. T. Rosenstein, Z. Marx, L. P. Kaelbling, and T. G. Dietterich, "To transfer or not to transfer," in Proc. NIPS Workshop Inductive Transfer: 10 Years Later, 2005, vol. 2, Art. no. 7.
42. N. Srebro, J. Rennie, and T. Jaakkola, "Maximum margin matrix factorization," in Proc. Advances Neural Inf. Process. Syst., 2005, pp. 1329-1336.
43. K. C. Toh and S. Yun, "An accelerated proximal gradient algorithm for nuclear norm regularized least squares problems," Preprint on Optimization Online, Apr. 2009.
44. S. Venugopalan, H. Xu, J. Donahue, M. Rohrbach, R. Mooney, and K. Saenko, "Translating videos to natural language using deep recurrent neural networks," arXiv:1412.4729, 2014.
45. P. Welinder, et al., "Caltech-UCSD Birds 200," California Inst. Technol., Pasadena, CA, USA, Tech. Rep. CNS-TR-2010-001, 2010.
46. P. Wu and T. Dietterich, "Improving SVM accuracy by training on auxiliary data sources," in Proc. 21st Int. Conf. Mach. Learn., 2004, Art. no. 110.
47. Q. Yang, Y. Chen, G. R. Xue, W. Dai, and Y. Yu, "Heterogeneous transfer learning for image clustering via the social Web," in Proc. 47th Annu. Meeting ACL 4th IJCNLP AFNLP, Aug. 2009, pp. 1-9.
48. M. Zhu, "Recall, precision and average precision," 2004.
49. Y. Zhu, et al., "Heterogeneous transfer learning for image classification," inProc. 25th AAAI Conf. Artificial Intell., 2011, pp. 1304-1309.