Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 41, Issue 9, 2019, Pages 2251-2265

  Xian, Yongqin ^a   Lampert, Christoph H ^b   Schiele, Bernt ^a   Akata, Zeynep ^c  

^a MAX PLANCK INSTITUTE FOR INFORMATICS   (Germany)

^b INSTITUTE OF SCIENCE AND TECHNOLOGY AUSTRIA   (Austria)

^c UNIVERSITY OF AMSTERDAM   (Netherlands)

Author keywords

Generalized zero shot learning; Image classification; Transductive learning; Weakly supervised learning

Indexed keywords

CLASSIFICATION (OF INFORMATION); FISH; FLOW VISUALIZATION; IMAGE CLASSIFICATION; JOB ANALYSIS; NETWORK PROTOCOLS; PERSONNEL TRAINING; SEMANTICS;

COMPARE AND ANALYZE; COMPREHENSIVE EVALUATION; GENERALIZED ZERO-SHOT LEARNING; LABELED TRAINING DATA; STATE-OF-THE-ART METHODS; TASK ANALYSIS; TRANSDUCTIVE LEARNING; WEAKLY SUPERVISED LEARNING;

LEARNING SYSTEMS;

EID: 85050180490     PISSN: 01628828     EISSN: 19393539     Source Type: Journal    
DOI: 10.1109/TPAMI.2018.2857768     Document Type: Article

References (76)

1. C. Lampert, H. Nickisch, and S. Harmeling, "Attribute-based classification for zero-shot visual object categorization," IEEE Trans. Pattern Anal. Mach. Intell., vol. 36, no. 3, pp. 453-465, Mar. 2014.
2. H. Larochelle, D. Erhan, and Y. Bengio, "Zero-data learning of new tasks," in Proc. 23rd Nat. Conf. Artif. Intell., 2008, pp. 646-651.
3. M. Rohrbach, M. Stark, and B. Schiele, "Evaluating knowledge transfer and zero-shot learning in a large-scale setting," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2011, pp. 1641-1648.
4. X. Yu and Y. Aloimonos, "Attribute-based transfer learning for object categorization with zero or one training example," in Proc. 11th Eur. Conf. Comput. Vis., 2010, pp. 127-140.
5. X. Xu, Y. Yang, D. Zhang, H. T. Shen, and J. Song, "Matrix trifactorization with manifold regularizations for zero-shot learning," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 2007-2016.
6. Z. Ding, M. Shao, and Y. Fu, "Low-rank embedded ensemble semantic dictionary for zero-shot learning," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 6005-6013.
7. A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. A. Ranzato, and T. Mikolov, "Devise: A deep visual-semantic embedding model," inProc. AdvancesNeural Inf.Process. Syst., 2013, pp. 2121-2129.
8. Z. Akata, F. Perronnin, Z. Harchaoui, and C. Schmid, "Label embedding for attribute-based classification," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2013, pp. 819-826.
9. Z. Akata, S. Reed, D. Walter, H. Lee, and B. Schiele, "Evaluation of output embeddings for fine-grained image classification," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2015, pp. 2927-2936.
10. B. Romera-Paredes and P. H. Torr, "An embarrassingly simple approach to zero-shot learning," Proc. Int. Conf. Mach. Learn., 2015, pp. 2152-2161.
11. Y. Xian, Z. Akata, G. Sharma, Q. Nguyen, M. Hein, and B. Schiele, "Latent embeddings for zero-shot classification," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 69-77.
12. R. Socher, M. Ganjoo, C. D. Manning, and A. Ng, "Zero-shot learning through cross-modal transfer," in Proc. Int. Conf. Neural Inf. Process. Syst., 2013, pp. 935-943.
13. Z. Zhang and V. Saligrama, "Zero-shot learning via semantic similarity embedding," in Proc. IEEE Int. Conf. Comput. Vis., 2015, pp. 4166-4174.
14. S. Changpinyo, W.-L. Chao, B. Gong, and F. Sha, "Synthesized classifiers for zero-shot learning," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 5327-5336.
15. M. Norouzi, T.Mikolov, S. Bengio, Y. Singer, J. Shlens, A. Frome, G. Corrado, and J.Dean, "Zero-shot learning by convex combination of semantic embeddings," in Proc. Int. Conf. Learn. Representation, 2014.
16. G. Patterson and J. Hays, "Sun attribute database: Discovering, annotating, and recognizing scene attributes," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2012, pp. 2751-2758.
17. P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona, "Caltech-UCSD Birds 200," California Institute Technol., Pasadena, CA, Tech. Rep. CNS-TR-2010-001, 2010.
18. A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth, "Describing objects by their attributes," Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2009, pp. 1778-1785.
19. J.Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "ImageNet: A large-scale hierarchical image database," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2009, pp. 248-255.
20. D. G. Lowe, "Distinctive image features from scale-invariant keypoints," Int. J. Comput. Vis., vol. 60, no. 2, pp. 91-110, 2004.
21. J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell, "DeCAF: A deep convolutional activation feature for generic visual recognition," in Proc. Int. Conf. Mach. Learn., 2014, pp. I-647-I-655.
22. K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," arXiv:1409.1556, 2014.
23. K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 770-778.
24. Z. Al-Halah, M. Tapaswi, and R. Stiefelhagen, "Recovering the missing link: Predicting class-attribute associations for unsupervised zero-shot learning," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 5975-5984.
25. D. Jayaraman and K. Grauman, "Zero-shot recognition with unreliable attributes," in Proc. 27th Int. Conf. Neural Inf. Process. Syst., 2014, pp. 3464-3472.
26. P. Kankuekul, A. Kawewong, S. Tangruamsub, and O. Hasegawa, "Online incremental attribute-based zero-shot learning," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2012, pp. 3657-3664.
27. T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, "Distributed representations of words and phrases and their compositionality," in Proc. Int. Conf. Neural Inf. Process. Syst., 2013, pp. 3111-3119.
28. Y. Fu, T. M. Hospedales, T. Xiang, and S. Gong, "Transductive multi-view zero-shot learning," IEEE Trans. Pattern Anal. Mach. Intell., vol. 37, no. 11, pp. 2332-2345, Nov. 2015.
29. M. Palatucci, D. Pomerleau, G. E. Hinton, and T. M. Mitchell, "Zero-shot learning with semantic output codes," in Proc. Int. Conf. Neural Inf. Process. Syst., 2009, pp. 1410-1418.
30. Z. Akata, F. Perronnin, Z. Harchaoui, and C. Schmid, "Labelembedding for image classification," IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 7, pp. 1425-1438, Jul. 2016.
31. R. Qiao, L. Liu, C. Shen, and A. van den Hengel, "Less is more: Zero-shot learning from online textual documents with noise suppression," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 2249-2257.
32. M. Bucher, S. Herbin, and F. Jurie, "Improving semantic embedding consistency by metric learning for zero-shot classiffication," in Proc. 11th Eur. Conf. Comput. Vis., 2016, pp. 730-746.
33. E. Kodirov, T. Xiang, and S. Gong, "Semantic autoencoder for zero-shot learning," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 4447-4456.
34. J. Lei Ba, K. Swersky, S. Fidler, et al., "Predicting deep zero-shot convolutional neural networks using textual descriptions," in Proc. IEEE Int. Conf. Comput. Vis., 2015, pp. 4247-4255.
35. L. Zhang, T. Xiang, and S. Gong, "Learning a deep embedding model for zero-shot learning," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 3010-3019.
36. S. Changpinyo, W.-L. Chao, and F. Sha, "Predicting visual exemplars of unseen classes for zero-shot learning," ICCV, pp. 3496-3505, 2017.
37. Z. Zhang and V. Saligrama, "Zero-shot learning via joint semantic similarity embedding," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 6034-6042.
38. Z. Fu, T. Xiang, E. Kodirov, and S. Gong, "Zero-shot object recognition by semantic manifold distance," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2015, pp. 2635-2644.
39. Z. Akata, M. Malinowski, M. Fritz, and B. Schiele, "Multi-cue zero-shot learning with strong supervision," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 59-68.
40. Y. Long, L. Liu, L. Shao, F. Shen, G. Ding, and J. Han, "From zeroshot learning to conventional supervised classification: Unseen visual data synthesis," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 6165-6174.
41. V. K. Verm and P. Rai, "A simple exponential family framework for zero-shot learning," in Proc. Eur. Conf. Mach. Learn., 2017, pp. 792-808.
42. Y. Li and D. Wang, "Zero-shot learning with generative latent prototype model," arXiv:1705.09474, 2017.
43. T. Mukherjee and T. Hospedales, "Gaussian visual-linguistic embedding for zero-shot recognition," in Proc. Conf. Empirical Methods Natural Language Process., 2016, pp. 912-918.
44. M. Rohrbach, S. Ebert, and B. Schiele, "Transfer learning in a transductive setting," in Proc. Int. Conf. Neural Inf. Process. Syst., 2013, pp. 46-54.
45. E. Kodirov, T. Xiang, Z. Fu, and S. Gong, "Unsupervised domain adaptation for zero-shot learning," in Proc. IEEE Int. Conf. Comput. Vis., 2015, pp. 2452-2460.
46. X. Li, Y. Guo, and D. Schuurmans, "Semi-supervised zero-shot classification with label representation learning," in Proc. IEEE Int. Conf. Comput. Vis., 2015, pp. 4211-4219.
47. X. Li and Y. Guo, "Max-margin zero-shot learning for multi-class classification," in Proc. 18th Int. Conf. Artif. Int. Statistics, 2015, pp. 626-634.
48. Y. Fu and L. Sigal, "Semi-supervised vocabulary-informed learning," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 5337-5346.
49. Z. Zhang and V. Saligrama, "Zero-shot recognition via structured prediction," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 2007-2016.
50. T. Mensink, E. Gavves, and C. G. Snoek, "Costa: Co-occurrence statistics for zero-shot classification," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2014, pp. 2441-2448.
51. M. Rohrbach, M. Stark, G. Szarvas, I. Gurevych, and B. Schiele, "What helps where-and why? semantic relatedness for knowledge transfer," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2010, pp. 910-917.
52. S. Reed, Z. Akata, H. Lee, and B. Schiele, "Learning deep representations of fine-grained visual descriptions," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 49-58.
53. M. Elhoseiny, B. Saleh, and A. Elgammal, "Write a classifier: Zeroshot learning using purely textual descriptions," in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 2584-2591.
54. S. Antol, C. L. Zitnick, and D. Parikh, "Zero-shot learning via visual abstraction," in Proc. Eur. Conf. Comput.Vis., 2014, pp. 401-416.
55. T. Mensink, J. Verbeek, F. Perronnin, and G. Csurka, "Metric learning for large scale image classification: Generalizing to new classes at near-zero cost," in Proc. Eur. Conf. Comput. Vis., pp. 488-501, 2012.
56. J. Pennington, R. Socher, and C. D. Manning, "Glove: Global vectors for word representation," in Proc. Conf. Empirical Methods Natural Language Process., 2014, pp. 1532-1543.
57. G. A. Miller, "Wordnet: A lexical database for English," CACM, vol. 38, pp. 39-41, 1995. [Online]. Available: http://doi.acm.org/ 10.1145/219717.219748
58. N. Karessli, Z. Akata, B. Schiele, and A. Bulling, "Gaze embeddings for zero-shot image classification," in Proc. IEEE Comput. Vis. Pattern Recognit., 2017, pp. 6412-6421.
59. W. J. Scheirer, A. Rocha, A. Sapkota, and T. E. Boult, "Towards open set recognition," IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 7, pp. 1757-1772, Jul. 2013.
60. L. Jain, W. Scheirer, and T. Boult, "Multi-class open set recognition using probability of inclusion," in Proc. 11th Eur. Conf. Comput. Vis., 2014, pp. 393-409.
61. H. Zhang, X. Shang, W. Yang, H. Xu, H. Luan, and T.-S. Chua, "Online collaborative learning for open-vocabulary visual classifiers," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 2809-2817.
62. A. Bendale and T. E. Boult, "Towards open set deep networks," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 1563-1572.
63. W.-L. Chao, S. Changpinyo, B. Gong, and F. Sha, "An empirical study and analysis of generalized zero-shot learning for object recognition in the wild," in Proc. 11th Eur. Conf. Comput. Vis., 2016, pp. 52-68.
64. T. Joachims, "Optimizing search engines using clickthrough data," in Proc. 8th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2002, pp. 133-142.
65. N. Usunier, D. Buffoni, and P. Gallinari, "Ranking with ordered weighted pairwise classification," in Proc. Int. Conf. Mach. Learn., 2009, pp. 1057-1064.
66. J. Weston, S. Bengio, and N. Usunier, "Wsabie: Scaling up to large vocabulary image annotation," in Proc. 22nd Int. Joint Conf. Artif. Intell., 2011, pp. 2764-2770.
67. I. Tsochantaridis, T. Joachims, T. Hofmann, and Y. Altun, "Large margin methods for structured and interdependent output variables," J. Mach. Learn. Res., 2005, pp. 1453-1484.
68. R. H. Bartels and G. Stewart, "Solution of the matrix equation ax+ xb= c [f4]," Commun. ACM, vol. 15, no. 9, pp. 820-826, 1972.
69. O. Chapelle, B. Scholkopf, and A. Zien, "Semi-supervised learning," IEEE Trans. Neural Netw., vol. 20, no. 3, p. 542, 2009.
70. D. Zhou, O. Bousquet, T. N. Lal, J. Weston, and B. Schölkopf, "Learning with local and global consistency," in Proc. Int. Conf. Neural Inf. Process. Syst., 2004, pp. 321-328.
71. M. Ye and Y. Guo, "Zero-shot classification with discriminative semantic representation learning," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 5103-5111.
72. Y. Fujiwara and G. Irie, "Efficient label propagation," in Proc. Int. Conf. Mach. Learn., 2014, pp. 784-792.
73. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, "Going deeper with convolutions," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2015, pp. 1-5.
74. B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva, "Learning deep features for scene recognition using places database," in Proc. Int. Conf. Neural Inf. Process. Syst., 2014, pp. 487-495.
75. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, "Going deeper with convolutions," Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2015, pp. 1-9.
76. S. Garcia and F. Herrera, "An extension on"statistical comparisons of classifiers over multiple data sets"for all pairwise comparisons," J. Mach. Learn. Res., vol. 9, pp. 2677-2694, 2008