A coordinate gradient descent method for linearly constrained smooth optimization and support vector machines training

Computational Optimization and Applications

Volumn 47, Issue 2, 2010, Pages 179-206

  Tseng, Paul ^a   Yun, Sangwoon ^a  

^a UNIVERSITY OF WASHINGTON   (United States)

Author keywords

Conformal realization; Continuous quadratic knapsack problem; Coordinate gradient descent; Error bound; Global convergence; Linear constraints; Linear convergence rate; Quadratic program; Support vector machine

Indexed keywords

CONFORMAL REALIZATION; ERROR BOUND; GLOBAL CONVERGENCE; GRADIENT DESCENT; LINEAR CONSTRAINTS; LINEAR CONVERGENCE RATE; QUADRATIC KNAPSACK PROBLEMS; QUADRATIC PROGRAMS;

APPROXIMATION THEORY; CONSTRAINED OPTIMIZATION; GEARS; INTEGER PROGRAMMING; OPERATIONS RESEARCH; PROBLEM SOLVING; VECTORS;

SUPPORT VECTOR MACHINES;

EID: 77956736675     PISSN: 09266003     EISSN: 15732894     Source Type: Journal    
DOI: 10.1007/s10589-008-9215-4     Document Type: Article

References (45)

1. Berman, P., Kovoor, N., Pardalos, P.M.: Algorithms for the least distance problem. In: Pardalos, P.M. (ed.) Complexity in Numerical Optimization, pp. 33-56. World Scientific, Singapore (1993)
2. Bertsekas, D.P.: Nonlinear Programming, 2nd edn. Athena Scientific, Belmont (1999)
3. P. Brucker 1984 An O(n) algorithm for quadratic knapsack problems Oper. Res. Lett. 3 163 166 0544.90086 10.1016/0167-6377(84)90010-5 761510 (Pubitemid 14671810)
4. Chang, C.-C., Lin, C.-J.: LIBSVM: a library for support vector machines (2001). Available from http://www.csie.ntu.edu.tw/~cjlin/libsvm
5. C.-C. Chang C.-W. Hsu C.-J. Lin 2000 The analysis of decomposition methods for support vector machines IEEE Trans. Neural Netw. 11 1003 1008 10.1109/72.857780
6. P.-H. Chen R.-E. Fan C.-J. Lin 2006 A study on SMO-type decomposition methods for support vector machines IEEE Trans. Neural Netw. 17 893 908 10.1109/TNN.2006.875973 (Pubitemid 44194150)
7. Cristianini, N., Shawe-Taylor, J.: An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods. Cambridge University Press, Cambridge (2000)
8. R.-E. Fan P.-H. Chen C.-J. Lin 2005 Working set selection using second order information for training support vector machines J. Mach. Learn. Res. 6 1889 1918 2249875
9. M.C. Ferris T.S. Munson 2003 Interior-point methods for massive support vector machines SIAM J. Optim. 13 783 804 1039.90092 10.1137/S1052623400374379 1972216 (Pubitemid 36971112)
10. M.C. Ferris T.S. Munson 2004 Semismooth support vector machines Math. Program. 101 185 204 1076.90041 2085263
11. S. Fine K. Scheinberg 2001 Efficient SVM training using low-rank kernel representations J. Mach. Learn. Res. 2 243 264 10.1162/15324430260185619
12. Fletcher, R.: Practical Methods of Optimization, 2nd edn. Wiley, New York (1987)
13. T. Glasmachers C. Igel 2006 Maximum-gain working set selection for SVMs J. Mach. Learn. Res. 7 1437 1466 2274412
14. D. Hush C. Scovel 2003 Polynomial-time decomposition algorithms for support vector machines Mach. Learn. 51 51 71 1056.68118 10.1023/A:1021877911972
15. D. Hush P. Kelly C. Scovel I. Steinwart 2006 QP algorithms with guaranteed accuracy and run time for support vector machines J. Mach. Learn. Res. 7 733 769 2274385
16. Joachims, T.: Making large-scale SVM learning practical. In: Schölkopf, B., Burges, C.J.C., Smola, A.J. (eds.) Advances in Kernel Methods-Support Vector Learning, pp. 169-184. MIT Press, Cambridge (1999)
17. S.S. Keerthi E.G. Gilbert 2002 Convergence of a generalized SMO algorithm for SVM classifier design Mach. Learn. 46 351 360 0998.68109 10.1023/A:1012431217818 (Pubitemid 34129975)
18. Keerthi, S.S., Ong, C.J.: On the role of the threshold parameter in SVM training algorithm. Technical Report CD-00-09, Department of Mathematical and Production Engineering, National University of Singapore, Singapore (2000)
19. S.S. Keerthi S.K. Shevade 2003 SMO algorithm for least-squares SVM formulations Neural Comput. 15 487 507 1047.68100 10.1162/089976603762553013 (Pubitemid 37049831)
20. S.S. Keerthi S.K. Shevade C. Bhattacharyya K.R.K. Murthy 2001 Improvements to Platt's SMO algorithm for SVM classifier design Neural Comput. 13 637 649 1085.68629 10.1162/089976601300014493 (Pubitemid 33595014)
21. K.C. Kiwiel 2007 On linear time algorithms for the continuous quadratic knapsack problem J. Optim. Theory Appl. 134 549 554 1145.90077 10.1007/s10957-007-9259-0 2332481
22. C.-J. Lin 2001 On the convergence of the decomposition method for support vector machines IEEE Trans. Neural Netw. 12 1288 1298 10.1109/72.963765 (Pubitemid 33106034)
23. Lin, C.-J.: Linear convergence of a decomposition method for support vector machines. Technical Report, Department of Computer Science and Information Engineering, Taiwan University, Taipei, Taiwan (2001)
24. C.-J. Lin 2002 Asymptotic convergence of an SMO algorithm without any assumptions IEEE Trans. Neural Netw. 13 248 250 10.1109/72.977319 (Pubitemid 34236853)
25. Lin, C.-J., Lucidi, S., Palagi, L., Risi, A., Sciandrone, M.: A decomposition algorithm model for singly linearly constrained problems subject to lower and upper bounds. Technical Report, DIS-Università di Roma "La Sapienza", Rome, January (2007). To appear in J. Optim. Theory Appl.
26. List, N., Simon, H.U.: A general convergence theorem for the decomposition method. In: Proceedings of the 17th Annual Conference on Learning Theory, pp. 363-377 (2004)
27. List, N., Simon, H.U.: General polynomial time decomposition algorithms. In: Lecture Notes in Computer Science, vol. 3559, pp. 308-322. Springer, Berlin (2005)
28. Lucidi, S., Palagi, L., Risi, A., Sciandrone, M.: On the convergence of hybrid decomposition methods for SVM training. Technical Report, DIS-Università di Roma "La Sapienza", Rome, July 2006. Submitted to IEEE Trans. Neural Netw.
29. Z.-Q. Luo P. Tseng 1992 Error bounds and the convergence analysis of matrix splitting algorithms for the affine variational inequality problem SIAM J. Optim. 2 43 54 0777.49010 10.1137/0802004 1147882
30. Z.-Q. Luo P. Tseng 1993 Error bounds and convergence analysis of feasible descent methods: a general approach Ann. Oper. Res. 46 157 178 10.1007/BF02096261 1260016
31. O.L. Mangasarian D.R. Musicant 1999 Successive overrelaxation for support vector machines IEEE Trans. Neural Netw. 10 1032 1037 10.1109/72.788643
32. Nocedal, J., Wright, S.J.: Numerical Optimization. Springer, New York (1999)
33. Osuna, E., Freund, R., Girosi, F.: Improved training algorithm for support vector machines. In: Proc. IEEE NNSP'97 (1997)
34. light algorithm Optim. Methods Softw. 20 317 334 1072.90042 10.1080/10556780512331318209 2141836 (Pubitemid 40146548)
35. Platt, J.: Sequential minimal optimization: A fast algorithm for training support vector machines. In: Schölkopf, B., Burges, C.J.C., Smola, A.J. (eds.) Advances in Kernel Methods-Support Vector Learning, pp. 185-208. MIT Press, Cambridge (1999)
36. N. In: Bose, R.C., Dowling, T.A. (eds.) Combinatorial Mathematics and Its Applications, Proc. of the Chapel Hill Conference 1967, pp. 104-127. Univ. North Carolina Press, Chapel Hill (1969)
37. Rockafellar, R.T.: Network Flows and Monotropic Optimization. Wiley, New York, 1984. Republished by Athena Scientific, Belmont (1998)
38. Rockafellar, R.T., Wets, R.J.-B.: Variational Analysis. Springer, New York (1998)
39. Saunders, C., Stitson, M.O., Weston, J., Bottou, L., Schölkopf., B., Smola, A.J.: Support vector machine-reference manual. Report CSD-TR-98-03, Department of Computer Science, Royal Holloway, University of London, Egham, UK (1998)
40. K. Scheinberg 2006 An efficient implementation of an active set method for SVM J. Mach. Learn. Res. 7 2237 2257 2274439
41. B. Schölkopf A.J. Smola R.C. Williamson P.L. Bartlett 2000 New support vector algorithms Neural Comput. 12 1207 1245 10.1162/089976600300015565
42. Simon, H.U.: On the complexity of working set selection. In: Proceedings of the 15th International Conference on Algorithmic Learning Theory, pp. 324-337 (2004)
43. Suykens, J.A.K., Van Gestel, T., De Brabanter, J., De Moor, B., Vandewalle, J.: Least Squares Support Vector Machines. World Scientific, Singapore (2002)
44. P. Tseng S. Yun 2009 A coordinate gradient descent method for nonsmooth separable minimization Math. Program. B 117 387 423 1166.90016 10.1007/s10107-007-0170-0 2421312
45. Vapnik, V.: Estimation of Dependences Based on Empirical Data. Springer, New York (1982)