A novel Frank-Wolfe algorithm. Analysis and applications to large-scale SVM training

Information Sciences

Volumn 285, Issue 1, 2014, Pages 66-99

  Ñanculef, Ricardo ^a   Frandi, Emanuele ^b   Sartori, Claudio ^c   Allende, Héctor ^a  

^a DEPARTAMENTO DE FÍSICA   (Chile)

^b UNIVERSITY OF INSUBRIA   (Italy)

^c UNIVERSITY OF BOLOGNA   (Italy)

Author keywords

Concave optimization; Frank Wolfe methods; Large scale support vector machines; Learning from massive datasets; Quadratic programming

Indexed keywords

ALGORITHMS; ARTIFICIAL INTELLIGENCE; CLASSIFICATION (OF INFORMATION); COMPUTATIONAL GEOMETRY; LEARNING SYSTEMS; NUMBER THEORY; QUADRATIC PROGRAMMING;

CLASSIFICATION DATASETS; CONCAVE OPTIMIZATION; CONVERGENCE ANALYSIS; FRANK-WOLFE ALGORITHMS; FRANK-WOLFE METHODS; LINEAR RATE OF CONVERGENCES; MACHINE LEARNING COMMUNITIES; MASSIVE DATA SETS;

SUPPORT VECTOR MACHINES;

EID: 84926204705     PISSN: 00200255     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ins.2014.03.059     Document Type: Article

References (66)

1. S. Damla Ahipasaoglu, Sun Peng, Michael Todd, Linear convergence of a modified Frank-Wolfe algorithm for computing minimum volume enclosing ellipsoids, Optim. Methods Softw. 23 (1) (2008) 5-19.
2. Héctor Allende, Emanuele Frandi, Ricardo Ñanculef, Claudio Sartori, Novel Frank-Wolfe methods for SVM learning, Technical report, 2013. .
3. Gükhan Bakir, Thomas Hofmann, Bernhard Schölkopf, Alexander Smola, Ben Taskar, S.V.N. Vishwanathan (Eds.), Predicting Structured Data (Neural Information Processing), The MIT Press, 2007.
4. Amir Beck, Marc Teboulle, A conditional gradient method with linear rate of convergence for solving convex linear systems, Math. Methods Oper. Res. 59 (2) (2004) 235-247.
5. Kristin P. Bennett, Erin J. Bredensteiner, Geometry in learning, in: Geometry at Work, 1997.
6. Kristin P. Bennett, Erin J. Bredensteiner, Duality and geometry in SVM classifiers, in: Proc. 17th International Conf. on Machine Learning, Morgan Kaufmann, 2000, pp. 57-64.
7. Léon Bottou, Stochastic learning, in: Olivier Bousquet, Ulrike von Luxburg (Eds.), Advanced Lectures on Machine Learning, Lecture Notes in Artificial Intelligence, LNAI, vol. 3176, Springer Verlag, 2004, pp. 146-168.
8. Léon Bottou, Olivier Bousquet, The tradeoffs of large scale learning, in: NIPS, 2007.
9. Mihai Bədoiu, Kenneth Clarkson, Smaller core-sets for balls, in: Proceedings of the SODA'03, SIAM, 2003, pp. 801-802.
10. Mihai Bədoiu, Kenneth Clarkson, Optimal core-sets for balls, Comput. Geomet. 40 (1) (2008) 14-22.
11. Cristopher J.C. Burges, David J. Crisp, A geometric interpretation of nu-SVM classifiers, in: Advances in Neural Information Processing Systems, MIT Press, Cambridge, MA, 2000, pp. 244-250.
12. Chih-Chung Chang, Chih-Jen Lin, LIBSVM: A Library for Support Vector Machines, 2011.
13. Kenneth Clarkson, Coresets, sparse greedy approximation, and the Frank-Wolfe algorithm, in: Proceedings of SODA'08, SIAM, 2008, pp. 922-931.
14. Kenneth Clarkson, Elad Hazan, David Woodruff, Sublinear optimization for machine learning, J. ACM 59 (5) (2012).
15. Janez Demsar, Statistical comparison of classifiers over multiple data sets, J. Mach. Learn. Res. 7 (2006) 1-30.
16. Rong-En Fan, Pai-Hsuen Chen, Chih-Jen Lin, Working set selection using second order information for training support vector machines, J. Mach. Learn. Res. 6 (2005) 1889-1918.
17. Anthony V. Fiacco, Garth P. McCormick, Nonlinear Programming: Sequential Unconstrained Minimization Techniques, SIAM, 1968 (reprinted 1990).
18. Shai Fine, Katya Scheinberg, Efficient SVM training using low-rank kernel representations, J. Mach. Learn. Res. 2 (2002) 243-264.
19. Emanuele Frandi, Maria Grazia Gasparo, Stefano Lodi, Ricardo Ñanculef, Claudio Sartori, A new algorithm for training SVMs using approximate minimal enclosing balls, in: Proceedings of the 15th Iberoamerican Congress on Pattern Recognition, Lecture Notes in Computer Science, Springer, 2010, pp. 87-95.
20. Emanuele Frandi, Maria Grazia Gasparo, Stefano Lodi, Ricardo Ñanculef, Claudio Sartori, Training support vector machines using Frank-Wolfe methods, International Journal of Pattern Recognition and Artificial Intelligence 27 (3) (2011).
21. Emanuele Frandi, Maria Grazia Gasparo, Ricardo Ñanculef, Alessandra Papini, Solution of classification problems via computational geometry methods, Recent Advances in Nonlinear Optimization and Equilibrium Problems: A Tribute to Marco D'Apuzzo, Quaderni di Matematica 27 (2012) 201-226.
22. Andrew Frank and Arthur Asuncion, The UCI KDD Archive, 2010. .
23. Marguerite Frank, Philip Wolfe, An algorithm for quadratic programming, Naval Res. Logist. 0-. 1 (1956) 95-110.
24. Thilo-Thomas Friess, Nello Cristianini, Colin Campbell, The Kernel-Adatron algorithm: a fast and simple learning procedure for support vector machines, in: Proceedings of the Fifteenth International Conference on Machine Learning, ICML '98, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1998, pp. 188-196.
25. Dan Garber, Elad Hazan, A polynomial time conditional gradient algorithm with applications to online and stochastic optimization. CoRR, abs/1301.4666, 2013.
26. Bernd Gärtner, Martin Jaggi, Coresets for polytope distance, in: John Hershberger, Efi Fogel (Eds.), Symposium on Computational Geometry, ACM, 2009, pp. 33-42.
27. Elmer Gilbert, An iterative procedure for computing the minimum of a quadratic form on a convex set, SIAM J. Control 4 (1) (1966) 61-80.
28. Jacques Guélat, Patrice Marcotte, Some comments on Wolfe's "away step", Math. Programming 35 (1986) 110-119.
29. Elad Hazan, Satyen Kale, Projection-free online learning, in: International Conference on Machine Learning, 2012.
30. Thomas Hofmann, Bernhard Schölkopf, Alexander Smola, Kernel methods in machine learning, Ann. Statist. 36 (3) (2008) 1171-1220.
31. Cho-Jui Hsieh, Kai-Wei Chang, Chih-Jen Lin, S Sathiya Keerthi, Sellamanickam Sundararajan, A dual coordinate descent method for large-scale linear SVM, in: Proceedings of the 25th international conference on Machine learning, ACM, 2008, pp. 408-415.
32. Chih-Wei Hsu, Chih-Jen Lin, A comparison of methods for multiclass support vector machines, IEEE Trans. Neural Networks 13 (2) (2002) 415-425.
33. Martin Jaggi, Revisiting Frank-Wolfe: projection-free sparse convex optimization, in: International Conference on Machine Learning (to appear), 2013.
34. Thorsten Joachims, Making large-scale support vector machine learning practical, in: Advances in Kernel Methods: Support Vector Learning, MIT Press, 1999, pp. 169-184.
35. Thorsten Joachims, SVM-light Support Vector Machine, 2011.
36. S.S. Keerthi, S.K. Shevade, C. Bhattacharyya, K.R.K. Murthy, A fast iterative nearest point algorithm for support vector machine classifier design, IEEE Trans. Neural Networks 11 (1) (2000) 124-136.
37. Sathiya Keerthi, Elmer Gilbert, Convergence of a generalized SMO algorithm for SVM classifier design, Mach. Learn. 46 (1-3) (2002) 351-360.
38. Piyush Kumar, AlperYildirim, A linearly convergent linear-time first-order algorithm for support vector classification with a core set result, INFORMS J. Comput. 23 (3) (2011) 377-391.
39. Simon Lacoste-Julien, Martin Jaggi, Mark Schmidt, Patrick Pletscher, Block-coordinate Frank-Wolfe optimization for structural SVMs, in: International Conference on Machine Learning (to appear), 2013.
40. Jorge López, Álvaro Barbero, José R. Dorronsoro, On the equivalence of the SMO and MDM algorithms for SVM training, in: Proceedings of the 2008 European Conference on Machine Learning and Knowledge Discovery in Databases - Part I, ECML PKDD '08, Springer-Verlag, Berlin, Heidelberg, 2008, pp. 288-300.
41. Jorge Lopez, José R Dorronsoro, The convergence rate of the MDM algorithm, in: The 2012 International Joint Conference on Neural Networks (IJCNN), IEEE, 2012, pp. 1-7.
42. B.F. Mitchell, V.F. Dem'yanov, V.N. Malozemov, Finding the point of a polyhedron closest to the origin, SIAM J. Control 12 (1) (1974) 19-26.
43. Jorge Nocedal, Stephen Wright, Numerical Optimization, second ed., Springer, 2006.
44. Hua Ouyang, Alexander Gray, Fast stochastic Frank-Wolfe algorithms for nonlinear SVMs, in: SDM, 2010, pp. 245-256.
45. John Platt, Fast training of support vector machines using sequential minimal optimization, in: Advances in Kernel Methods: Support Vector Learning, MIT Press, 1999, pp. 185-208.
46. R Core Team, R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria, 2012, ISBN 3-900051-07-0.
47. Piyush Rai, Hal Daumé, Suresh Venkatasubramanian, Streamed learning: one-pass SVMs, in: IJCAI'09: Proceedings of the 21st International Jont Conference on Artifical Intelligence, Morgan Kaufmann Publishers, 2009, pp. 1211-1216.
48. Stephen Robinson, Generalized equations and their solutions, part II: applications to nonlinear programming, Optimality and Stability in Mathematical Programming, Mathematical Programming Studies, vol. 19, Springer, Berlin, Heidelberg, 1982, pp. 200-221.
49. Katya Scheinberg, An efficient implementation of an active set method for SVMs, J. Mach. Learn. Res. 7 (2006) 2237-2257.
50. Bernard Schölkopf, Christopher Burges, Alexander Smola (Eds.), Advances in Kernel Methods: Support Vector Learning, MIT Press, 1999.
51. Bernard Schölkopf, Alexander Smola, Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond, MIT Press, Cambridge, MA, USA, 2001.
52. Bernhard Schölkopf, Alexander J. Smola, Sparse greedy matrix approximation for machine learning, in: Proceedings of the 17th International Conference on Machine Learning, 2001, pp. 911-918.
53. Shai Shalev-Shwartz, Online learning and online convex optimization, Found. Trends Mach. Learn. 4 (2) (2012) 107-194.
54. Shai Shalev-Shwartz, Yoram Singer, Nathan Srebro, Andrew Cotter, Pegasos: primal estimated sub-gradient solver for SVM, Math. Program. 127 (1) (2011) 3-30.
55. Shai Shalev-Shwartz, Tong Zhang, Stochastic dual coordinate ascent methods for regularized loss minimization, J. Mach. Learn. Res. 14 (2013) 567-599.
56. Ingo Steinwart, Sparseness of support vector machines, J. Mach. Learn. Res. 4 (2003) 1071-1105.
57. Ivor Tsang, Andras Kocsor, James Kwok, LibCVM Toolkit, 2011. .
58. Ivor Tsang, James Kwok, Pak-Ming Cheung, Core vector machines: fast SVM training on very large data sets, J. Mach. Learn. Res. 6 (2005) 363-392.
59. Ivor Tsang, James Kwok, Jacek Zurada, Generalized core vector machines, IEEE Trans. Neural Networks 17 (5) (2006) 1126-1140.
60. Ioannis Tsochantaridis, Thorsten Joachims, Thomas Hofmann, Yasemin Altun, Large margin methods for structured and interdependent output variables, J. Mach. Learn. Res. 6 (2005) 1453-1484.
61. Di Wang, Bo Zhang, Peng Zhang, Hong Qiao, An online core vector machine with adaptive MEB adjustment, Pattern Recogni. 43 (2010) 3468-3482.
62. Philip Wolfe, Convergence theory in nonlinear programming, in: J. Abadie (Ed.), Integer and Nonlinear Programming, North-Holland, Amsterdam, 1970, pp. 1-36.
63. Kristian Woodsend, Jacek Gondzio, Exploiting separability in large scale linear support vector machine training, Comput. Optim. Appl. 29 (2011) 241-269.
64. Emre Alper Yildirim, Two algorithms for the minimum enclosing ball problem, SIAM J. Optim. 19 (3) (2008) 1368-1391.
65. Guo-Xun Yuan, Chia-Hua Ho, Chih-Jen Lin, Recent advances of large-scale linear classification, Proc. IEEE 100 (9) (2012) 2584-2603.
66. Tong Zhang, Sequential greedy approximation for certain convex optimization problems, IEEE Trans. Inf. Theory 49 (3) (2003) 682-691.