Training highly multiclass classifiers

Journal of Machine Learning Research

Volumn 15, Issue , 2014, Pages 1461-1492

  Gupta, Maya R ^a   Bengio, Samy ^a   Weston, Jason ^a  

^a GOOGLE INC   (United States)

Author keywords

Classification; Large scale; Multiclass; Online learning; Stochastic gradient

Indexed keywords

IMAGE RECOGNITION;

CLASSIFICATION ACCURACY; DIMENSIONALITY REDUCTION; LARGE-SCALE; MULTI-CLASS CLASSIFIER; MULTICLASS; ONLINE LEARNING; STOCHASTIC GRADIENT; STOCHASTIC GRADIENT DESCENT;

CLASSIFICATION (OF INFORMATION);

EID: 84901640061     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (51)

1. E. L. Allwein, R. E. Schapire, and Y. Singer. Reducing multiclass to binary: a unifying approach for margin classifiers. Journal Machine Learning Research, 1:113-141, 2000. (Pubitemid 33738778)
2. A. Barla, F. Odone, and A. Verri. Histogram intersection kernel for image classification. Intl. Conf. Image Processing (ICIP), 3:513-516, 2003.
3. S. Bengio, J. Weston, and D. Grangier. Label embedding trees for large multi-class tasks. In NIPS, 2010.
4. Y. Bengio. Learning deep architectures for AI. Foundations and Trends in Machine Learn- ing, 2(1):1-127, 2009.
5. Y. Bengio, J. Laouradour, R. Collobert, and J. Weston. Curriculum learning. In ICML, 2009.
6. E. J. Bredensteiner and K. P. Bennet. Multicategory classification by support vector machines. Computational Optimization and Applications, 12:53-79, 1999. (Pubitemid 129574925)
7. L. D. Brown, T. T. Cai, and A. DasGupta. Interval estimation for a binomial proportion. Statistical Science, 16(2):101-117, 2001.
8. K. Crammer and Y. Singer. On the algorithmic implementation of multiclass kernel-based vector machines. Journal Machine Learning Research, 2001.
9. K. Crammer and Y. Singer. On the learnability and design of output codes for multiclass problems. Machine Learning, 47(2):201-233, 2002. (Pubitemid 34126110)
10. K. Crammer, O. Dekel, J. Keshet, S. Shalev-Shwartz, and Y. Singer. Online passive aggressive algorithms. Journal Machine Learning Research, 7:551-585, 2006.
11. A. Daniely, S. Sabato, and S. Shalev-Shwartz. Multiclass learning approaches: A theoretical comparison with implications. In NIPS, 2012.
12. J. Dean, G. S. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M. Ranzato, A. Senoir, P. Tucker, K. Yang, and A. Ng. Large scale distributed deep networks. In NIPS, 2012.
13. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In IEEE Conf. Computer Vision Pattern Recognition (CVPR), 2009.
14. J. Deng, A. Berg, K. Li, and L. Fei-Fei. What does classifying more than 10,000 image categories tell us? In European Conf. Computer Vision (ECCV), 2010.
15. J. Deng, S. Satheesh, A. C. Berg, and L. Fei-Fei. Fast and balanced: Eficient label tree learning for large scale object recognition. In NIPS, 2011.
16. T. G. Dietterich and G. Bakiri. Solving multiclass learning problems via error-correcting output codes. Journal Artificial Intelligence Research, 2:264-286, 1995.
17. J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal Machine Learning Research, 12:2121-2159, 2011.
18. D. Eck. Personal communication from Google music expert Douglas Eck. 2013.
19. C. Fellbaum, editor. WordNet: An Electronic Lexical Database. MIT Press, 1998.
20. E. K. Garcia, S. Feldman, M. R. Gupta, and S. Srivastava. Completely lazy learning. IEEE Trans. Knowledge and Data Engineering, 22(9):1274-1285, 2010.
21. D. Grangier and S. Bengio. A discriminative kernel-based model to rank images from text queries. IEEE Trans. Pattern Analysis and Machine Intelligence, 30:1371-1384, 2008.
22. K. Grauman and T. Darrell. The pyramid match kernel: Eficient learning with sets of features. Journal Machine Learning Research, 8:725-760, April 2007. (Pubitemid 46677049)
23. G. Grifin and P. Perona. Learning and using taxonomies for fast visual categorization. In CVPR, 2008.
24. Y. Guermeur. Combining discriminant models with new multi-class SVMs. Pattern Analysis and Applications, 5:168-179, 2002. (Pubitemid 40830130)
25. P. Hall and J. S. Marron. Geometric representation of high dimension, low sample size data. J. R. Statst. Soc. B, 67(3):427-444, 2005.
26. T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning. Springer- Verlag, New York, 2001.
27. R. Herbrich, T. Graepel, and K. Obermayer. Large margin rank boundaries for ordinal regression. In Advances in Large Margin Classifiers, pages 115-132. MIT Press, 2000.
28. A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS, 2012.
29. Y. Lee, Y. Lin, and G. Wahba. Multicategory support vector machines. Journal American Statistical Association, 99(465):67-81, 2004.
30. T. Leung and J. Malik. Recognizing surface using three-dimensional textons. Intl. Conf. Computer Vision (ICCV), 1999.
31. Y. Lin, F. Lv, S. Zhu, M. Yang, T. Cour, K. Yu, L. Cao, and T. Huang. Large-scale image classification: Fast feature extraction and SVM training. In CVPR, 2011.
32. M. Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of Machine Learning. MIT Press, Cambridge, MA, 2012.
33. Y. Mroueh, T. Poggio, L. Rosasco, and J-J E. Slotine. Multiclass learning with simplex coding. In NIPS, 2012.
34. D. Nister and H. Stewenius. Scalable recognition with a vocabulary tree. In CVPR, 2006.
35. A. Paton, N. Brummitt, R. Govaerts, K. Harman, S. Hinchclifie, B. Allkin, and E. Lughadha. Target 1 of the global strategy for plant conservation: a working list of all known plant speciesprogress and prospects. Taxon, 57:602-611, 2008. (Pubitemid 351934178)
36. F. Perronnin, Z. Akata, Z. Harchaoui, and C. Schmid. Towards good practice in large-scale learning for image classification. In CVPR, 2012.
37. R. Rifkin and A. Klatau. In defense of one-vs-all classification. Journal Machine Learning Research, 5:101-141, 2004.
38. J. Sanchez and F. Perronnin. High-dimensional signature compression for large-scale image classification. In CVPR, 2011.
39. B. Schoelkopf, A. J. Smola, and K. R. Muller. Kernel principal component analysis. Ad- vances in kernel methods: support vector learning, pages 327-352, 1999.
40. O. Shamir and O. Dekel. Multiclass-multilabel classication with more classes than examples. In Proc. AISTATS, 2010.
41. A. Statnikov, C. F. Aliferis, I. Tsamardinos, D. Hardin, and S. Levy. A comprehensive evaluation of multicategory classification methods for microarray gene expression cancer diagnosis. Bioinformatics, 21(5):631-643, 2005. (Pubitemid 40424789)
42. A. Tewari and P. L. Bartlett. On the consistency of multiclass classification methods. Journal Machine Learning Research, 2007.
43. N. Usunier, D. Bufioni, and P. Gallinari. Ranking with ordered weighted pairwise classifi- cation. In ICML, 2009.
44. V. Vapnik. Statistical Learning Theory. 1998.
45. J. Weston and C. Watkins. Multi-class support vector machines. Technical Report CSD- TR-98-04 Dept. Computer Science, Royal Holloway, University London, 1998.
46. J. Weston and C. Watkins. Support vector machines for multi-class pattern recognition. In Proc. European Symposium on Artificial Neural Networks, 1999.
47. J. Weston, S. Bengio, and N. Usunier. Wsabie: Scaling up to large vocabulary image annotation. In Intl. Joint Conf. Artificial Intelligence, (IJCAI), pages 2764-2770, 2011.
48. J. Weston, A. Makadia, and H. Yee. Label partitioning for sublinear ranking. In ICML, 2013a.
49. J. Weston, R. Weiss, and H. Yee. Afinity weighted embedding. In Proc. International Conf. Learning Representations, 2013b.
50. L. Xiao. Dual averaging methods for regularized stochastic learning and online optimization. Microsoft Research Technical Report MSR-TR-2010-23, 2010.
51. T. Zhang. Statistical analysis of some multi-category large margin classification methods. Journal Machine Learning Research, 5:1225-1251, 2004.