Pegasos: Primal estimated sub-gradient solver for SVM

Mathematical Programming

Volumn 127, Issue 1, 2011, Pages 3-30

  Shalev Shwartz, Shai ^a   Singer, Yoram ^b   Srebro, Nathan ^c   Cotter, Andrew ^c  

^a HEBREW UNIVERSITY OF JERUSALEM   (Israel)

^b GOOGLE INC   (United States)

^c TOYOTA TECHNOLOGICAL INSTITUTE AT CHICAGO   (United States)

Author keywords

Stochastic gradient descent; SVM

Indexed keywords

GRADIENT DESCENT ALGORITHMS; LARGE DATASETS; LINEAR KERNEL; NON-LINEAR; NUMBER OF ITERATIONS; OBJECTIVE FUNCTIONS; OPTIMIZATION PROBLEMS; ORDER OF MAGNITUDE; REGULARIZATION PARAMETERS; RUNTIMES; STOCHASTIC GRADIENT DESCENT; STOCHASTIC GRADIENT DESCENT METHOD; SVM; SVM LEARNING METHODS; SVM SOLVERS; TEXT CLASSIFICATION; TRAINING EXAMPLE; TRAINING SETS;

ITERATIVE METHODS; LEARNING ALGORITHMS; STOCHASTIC SYSTEMS; TEXT PROCESSING; TIME VARYING NETWORKS;

SUPPORT VECTOR MACHINES;

EID: 79952748054     PISSN: 00255610     EISSN: 14364646     Source Type: Journal    
DOI: 10.1007/s10107-010-0420-4     Document Type: Article

References (37)

1. S. Amari 1998 Natural gradient works efficiently in learning Neural Comput. 10 251 276 10.1162/089976698300017746 (Pubitemid 128463152)
2. A. Bordes S. Ertekin J. Weston L. Bottou 2005 Fast kernel classifiers with online and active learning J. Mach. Learn. Res. 6 1579 1619 2249866
3. Bottou L.: Online Algorithms and Stochastic Approximations. In: Saad, D. (eds) Online learning and neural networks, Cambridge University Press, Cambridge (1998)
4. Bottou, L., Bousquet, O.: The tradeoffs of large scale learning. In: Advances in Neural Information Processing Systems 20, pp. 161-168 (2008)
5. Bottou L., LeCun Y.: Large scale online learning. In: Thrun, S., Saul, L., Schölkopf, B. (eds) Advances in Neural Information Processing Systems 16, MIT Press, Cambridge (2004)
6. Bottou L., Murata N.: Stochastic approximations and efficient learning. In: Arbib, M.A. (eds) The Handbook of Brain Theory and Neural Networks, The MIT Press, Cambridge (2002)
7. Boyd S., Vandenberghe L.: Convex Optimization, 2nd edn. Cambridge University Press, Cambridge (2004)
8. Censor Y., Zenios S.: Parallel Optimization: Theory, Algorithms, and Applications. Oxford University Press, New York (1997)
9. N. Cesa-Bianchi A. Conconi C. Gentile 2004 On the generalization ability of on-line learning algorithms IEEE Trans. Inf, Theory 50 9 2050 2057 10.1109/TIT.2004.833339 2097190
10. Chapelle, O.: Training a support vector machine in the primal. Neural Comput. 19(5), 1155-1178 (2007). doi: 10.1162/neco.2007.19.5.1155. http://www.mitpressjournals.org/doi/abs/10.1162/neco.2007.19.5.1155
11. K. Crammer O. Dekel J. Keshet S. Shalev-Shwartz Y. Singer 2006 Online passive aggressive algorithms J. Mach. Learn. Res. 7 551 585 2274378
12. Cristianini N., Shawe-Taylor J.: An Introduction to Support Vector Machines. Cambridge University Press, Cambridge (2000)
13. Do, C., Le, Q., Foo, C.: Proximal regularization for online and batch learning. In: Proceedings of the 26th International Conference on Machine Learning (2009)
14. Duda R.O., Hart P.E.: Pattern Classification and Scene Analysis. Wiley, New York (1973)
15. S. Fine K. Scheinberg 2001 Efficient SVM training using low-rank kernel representations J. Mach. Lear. Res. 2 242 264
16. Y. Freund R.E. Schapire 1999 Large margin classification using the perceptron algorithm Mach. Learn. 37 3 277 296 0944.68535 10.1023/A: 1007662407062 (Pubitemid 32210619)
17. Hazan, E., Kalai, A., Kale, S., Agarwal, A.: Logarithmic regret algorithms for online convex optimization. In: Proceedings of the Nineteenth Annual Conference on Computational Learning Theory (2006)
18. Hsieh, C., Chang, K., Lin, C., Keerthi, S., Sundararajan, S.: A dual coordinate descent method for large-scale linear SVM. In: ICML, pp. 408-415 (2008)
19. Hush, D., Kelly, P., Scovel, C., Steinwart, I.: Qp algorithms with guaranteed accuracy and run time for support vector machines. J. Mach. Learn. Res. (2006)
20. Joachims T.: Making large-scale support vector machine learning practical. In: Schölkopf, B., Burges, C., Smola, A. (eds) Advances in Kernel Methods-Support Vector Learning., MIT Press, Cambridge (1998)
21. Joachims, T.: Training linear SVMs in linear time. In: Proceedings of the ACM Conference on Knowledge Discovery and Data Mining (KDD), pp. 216-226 (2006)
22. Kakade, S., Tewari, A.: On the generalization ability of online strongly convex programming algorithms. In: Advances in Neural Information Processing Systems 22 (2009)
23. G. Kimeldorf G. Wahba 1971 Some results on tchebycheffian spline functions J. Math. Anal. Appl. 33 82 95 0201.39702 10.1016/0022-247X(71)90184-3 290013
24. J. Kivinen A.J. Smola R.C. Williamson 2002 Online learning with kernels IEEE Trans. Signal Process. 52 8 2165 2176 10.1109/TSP.2004.830991 2085578
25. Kushner H., Yin G.: Stochastic Approximation Algorithms and Applications. Springer, New York (1997)
26. Murata N.: A statistical study of on-line learning. In: Saad, D. (eds) Online Learning and Neural Networks, Cambridge University Press, Cambridge (1998)
27. N. Murata S. Amari 1999 Statistical analysis of learning dynamics Signal Process. 74 1 3 28 0922.68094 10.1016/S0165-1684(98)00206-0
28. Nesterov, Y.: Primal-dual subgradient methods for convex problems. Tech. rep., Center for Operations Research and Econometrics (CORE), Catholic University of Louvain (UCL) (2005)
29. Platt J.C.: Fast training of Support Vector Machines using sequential minimal optimization. In: Schölkopf, B., Burges, C., Smola, A. (eds) Advances in Kernel Methods-Support Vector Learning, MIT Press, Cambridge (1998)
30. Rockafellar R.: Convex Analysis. Princeton University Press, Princeton (1970)
31. Shalev-Shwartz, S., Singer, Y., Srebro, N.: Pegasos: Primal Estimated sub-GrAdient SOlver for SVM. In: Proceedings of the 24th International Conference on Machine Learning, pp. 807-814 (2007)
32. Shalev-Shwartz, S., Srebro, N.: SVM optimization: inverse dependence on training set size. In: Proceedings of the 25th International Conference on Machine Learning, pp. 928-935 (2008)
33. Smola, A., Vishwanathan, S., Le, Q.: Bundle methods for machine learning. In: Advances in Neural Information Processing Systems 21 (2007)
34. Spall J.C.: Introduction to Stochastic Search and Optimization. Wiley, New York (2003)
35. Sridharan, K., Srebro, N., Shalev-Shwartz, S.: Fast rates for regularized objectives. In: Advances in Neural Information Processing Systems 22 (2009)
36. Vapnik V.N.: Statistical Learning Theory. Wiley, New York (1998)
37. Zhang, T.: Solving large scale linear prediction problems using stochastic gradient descent algorithms. In: Proceedings of the Twenty-First International Conference on Machine Learning (2004)