Breaking the curse of kernelization: Budgeted stochastic gradient descent for large-scale SVM training

Journal of Machine Learning Research

Volumn 13, Issue , 2012, Pages 3103-3131

  Wang, Zhuang ^a,c   Crammer, Koby ^b   Vucetic, Slobodan ^c  

^a SIEMENS CORPORATE RESEARCH   (United States)

^b TECHNION ISRAEL INSTITUTE OF TECHNOLOGY   (Israel)

^c Temple University   (United States)

Author keywords

Kernel methods; Large scale learning; Online learning; Stochastic gradient descent; SVM

Indexed keywords

KERNEL METHODS; LARGE-SCALE LEARNING; ONLINE LEARNING; STOCHASTIC GRADIENT DESCENT; SVM;

BUDGET CONTROL; LEARNING ALGORITHMS; MAINTENANCE; TIME VARYING NETWORKS;

SUPPORT VECTOR MACHINES;

EID: 84869463516     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (55)

1. A. Bordes, S. Ertekin, J.Weston, and L. Bottou. Fast kernel classifiers for online and active learning. Journal of Machine Learning Research, 2005.
2. A. Bordes, L. Bottou, and P. Gallinari. Sgd-qn: careful quasi-newton stochastic gradient descent. Journal of Machine Learning Research, 2009.
3. C. J. C. Burges. Simplified support vector decision rules. In Advances in Neural Information Processing Systems, 1996.
4. G. Cauwenberghs and T. Poggio. Incremental and decremental support vector machine learning. In Advances in Neural Information Processing Systems, 2000.
5. N. Cesa-Bianchi and C. Gentile. Tracking the best hyperplane with a simple budget perceptron. In Annual Conference on Learning Theory, 2006.
6. C.-C. Chang and C.-J. Lin. Libsvm: a library for support vector machines, http://www.csie.ntu.edu. tw/cjlin/libsvm. 2001.
7. F. Chang, C.-Y. Guo, X.-R. Lin, and C.-J. Lu. Tree decomposition for large-scale svm problems. Journal of Machine Learning Research, 2010a.
8. Y.-W. Chang, C.-J. Hsie, K.-W. Chang, M. Ringgaard, and C.-J. Lin. Training and testing low-degree polynomial data mappings via linear svm. Journal of Machine Learning Research, 2010b.
9. L. Cheng, S. V. N. Vishwanathan, D. Schuurmans, S. Wang, and T. Caelli. Implicit online earning with kernels. In Advances in Neural Information Processing Systems, 2007.
10. R. Collobert, F. Sinz, J.Weston, and L. Bottou. Trading convexity for scalability. In International Conference on Machine Learning, 2006.
11. C. Cortes and V. Vapnik. Support-vector networks. Machine Learning, 1995.
12. K. Crammer and Y. Singer. On the algorithmic implementation of multiclass kernel-based vector machines. Journal of Machine Learning Research, 2001.
13. K. Crammer and Y. Singer. Ultraconservative online algorithms for multiclass problems. Journal of Machine Learning Research, 2003.
14. K. Crammer, J. Kandola, and Y. Singer. Online classification on a budget. In Advances in Neural Information Processing Systems, 2004.
15. K. Crammer, O. Dekel, J. Keshet, S. Shalev-Shwartz, and Y. Singer. Online passive-aggressive algorithms. Journal of Machine Learning Research, 2006.
16. L. Csat'o and M. Opper. Sparse representation for gaussian process models. In Advances in Neural Information Processing Systems, 2001.
17. O. Dekel and Y. Singer. Support vector machines on a budget. In Advances in Neural Information Processing Systems, 2006.
18. O. Dekel, S. Shalev-Shwartz, and Y. Singer. The forgetron: a kernel-based perceptron on a budget. SIAM Journal on Computing, 2008.
19. R. Duda and P. Hart. Pattern Classification and Scene Analysis. New York: Wiley, 1973.
20. Y. Engel, S. Mannor, and R. Meir. Sparse online greedy support vector regression. In European Conference on Machine Learning, 2002.
21. C. Gentile. A new approximate maximal margin classification algorithm. Journal of Machine Learning Research, 2001.
22. H.-P. Graf, E. Cosatto, L. Bottou, I. Dourdanovic, and V. Vapnik. Parallel support vector machines: the cascade svm. In Advances in Neural Information Processing Systems, 2005.
23. C.-J. Hsieh, K.-W. Chang, C.-J. Lin, S. S. Keerthi, and S. Sundararajan. A dual coordinate descent method for large-scale linear svm. In International Conference on Machine Learning, 2008.
24. T. Joachims. Training linear svms in linear time. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2006.
25. S. S. Keerthi, O. Chapelle,, and D. DeCoste. Building support vector machines with reduced classifier complexity. Journal of Machine Learning Research, 2006.
26. J. Kivinen, A. J. Smola, and R. C. Williamson. Online learning with kernels. IEEE Transactions on Signal Processing, 2002.
27. Y.-J. Lee and O. L. Mangasarian. Rsvm: reduced support vector machines. In SIAM Conference on Data Mining, 2001.
28. B. Li, M. Chi, J. Fan, and X. Xue. Support cluster machine. In International Conference on Machine Learning, 2007.
29. Y. Li and P. Long. The relaxed online maximum margin algorithm. Machine Learning, 2002.
30. D. Nguyen and T. Ho. An efficient method for simplifying support vector machines. In International Conference on Machine Learning, 2005.
31. F. Orabona, J. Keshet, and B. Caputo. Bounded kernel-based online learning. Journal of Machine Learning Research, 2009.
32. J. Platt. Fast training of support vector machines using sequential minimal optimization. Advances in Kernel Methods -Support Vector Learning, MIT Press, 1998.
33. A. Rahimi and B. Rahimi. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems, 2007.
34. F. Rosenblatt. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological Review, 1958.
35. B. Schölkopf, S. Mika, C. J. C. Burges, P. Knirsch, K. Müller, G. Rätsch, and A. J. Smola. Input space versus feature space in kernel-based methods. IEEE Transactions on Neural Networks, 1999.
36. S. Shalev-Shwartz and Y. Singer. Logarithmic regret algorithms for strongly convex repeated games (technical report). The Hebrew University, 2007.
37. S. Shalev-Shwartz, Y. Singer, N. Srebro, and A. Cotter. Pegasos: primal estimated sub-gradient solver for svm. Mathematical Programming, 2011.
38. S. Sonnenburg and V. Franc. Coffin: a computational framework for linear svms. In International Conference on Machine Learning, 2010.
39. I. Steinwart. Sparseness of support vector machines. Journal of Machine Learning Research, 2003.
40. I. Sutskever. A simpler unified analysis of budget perceptrons. In International Conference on Machine Learning, 2009.
41. C.H. Teo, S. V. N. Vishwanathan, A. J. Smola, and Q. V. Le. Bundle methods for regularized risk minimization. Journal of Machine Learning Research, 2010.
42. I. W. Tsang, J. T. Kwok, and P.-M. Cheung. Core vector machines: fast svm training on very large data sets. Journal of Machine Learning Research, 2005.
43. I. W. Tsang, A. Kocsor, and J. T. Kwok. Simpler core vector machines with enclosing balls. In International Conference on Machine Learning, 2007.
44. S. V. N. Vishwanathan, A. J. Smola, and M. N.Murty. Simplesvm. In International Conference on Machine Learning, 2003.
45. Z. Wang and S. Vucetic. Tighter perceptron with improved dual use of cached data for model representation and validation. In International Joint Conference on Neutral Netweok, 2009.
46. Z. Wang and S. Vucetic. Online passive-aggressive algorithms on a budget. In International Conference on Artificial Intelligence and Statistics, 2010a.
47. Z. Wang and S. Vucetic. Online training on a budget of support vector machines using twin prototypes. Statisitcal Analysis and Data Mining Journal, 2010b.
48. Z. Wang, N. Djuric, K. Crammer, and S. Vucetic. Trading representability for scalability: adaptive multihyperplane machine for nonlinear classification. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2011.
49. J. Weston, A. Bordes, and L. Bottou. Online (and offline) on an even tighter budget. In International Workshop on Artificial Intelligence and Statistics, 2005.
50. M.-R.Wu, B. Schölkopf, and G. Barik. Building sparse large margin classifiers. In International Conference on Machine Learning, 2005.
51. H.-F. Yu, C.-J. Hsieh, K.-W. Chang, and C.-J. Lin. Large linear classification when data cannot fit in memory. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2010.
52. K. Zhang, L. Lan, Z. Wang, and F. Moerchen. Scaling up kernel svm on limited resources: a low-rank linearization approach. In International Conference on Artificial Intelligence and Statistics, 2012.
53. T. Zhang. Solving large scale linear prediction problems using stochastic gradient descent. In International Conference on Machine Learning, 2004.
54. Z. A. Zhu,W. Chen, G.Wang, C. Zhu, and Z. Chen. P-packsvm: parallel primal gradient descent kernel svm. In IEEE International Conference on Data Mining, 2009.
55. M. Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In International Conference on Machine Learning, 2003.