An affine-scaling interior-point method for continuous knapsack constraints with application to support vector machines

SIAM Journal on Optimization

Volumn 21, Issue 1, 2011, Pages 361-390

  Gonzalez Lima, Maria D ^a   Hager, William W ^b   Zhang, Hongchao ^c  

^a UNIVERSIDAD SIMÓN BOLÍVAR   (Venezuela)

^b UNIVERSITY OF FLORIDA   (United States)

^c LOUISIANA STATE UNIVERSITY   (United States)

Author keywords

Affine scaling; Cyclic Barzilai Borwein methods; Global convergence; Interior point; Linear convergence; Support vector machines

Indexed keywords

AFFINE-SCALING; CYCLIC BARZILAI-BORWEIN METHODS; GLOBAL CONVERGENCE; INTERIOR-POINT; LINEAR CONVERGENCE;

APPROXIMATION ALGORITHMS; CONVERGENCE OF NUMERICAL METHODS; GEARS; NEWTON-RAPHSON METHOD; OPTIMIZATION;

SUPPORT VECTOR MACHINES;

EID: 79957510063     PISSN: 10526234     EISSN: None     Source Type: Journal    
DOI: 10.1137/090766255     Document Type: Article

References (68)

1. L. ARMIJO, Minimization of functions having Lipschitz continuous first partial derivatives, Pacific J. Math., 16 (1966), pp. 1-3.
2. K. E. ATKINSON, An Introduction to Numerical Analysis, John Wiley, New York, 1978.
3. J. BARZILAI AND J. M. BORWEIN, Two point step size gradient methods, IMA J. Numer. Anal., 8 (1988), pp. 141-148.
4. E. G. BIRGIN AND J. M. MARTÍNEZ, Large-scale active-set box-constrained optimization method with spectral projected gradients, Comput. Optim. Appl., 23 (2002), pp. 101-125. (Pubitemid 35167001)
5. E. G. BIRGIN, J. M. MARTÍNEZ, AND M. RAYDAN, Nonmonotone spectral projected gradient methods for convex sets, SIAM J. Optim., 10 (2000), pp. 1196-1211.
6. E. G. BIRGIN, J. M. MARTÍNEZ, AND M. RAYDAN, Algorithm 813: SPG-software for convexconstrained optimization, ACM Trans. Math. Software, 27 (2001), pp. 340-349. (Pubitemid 33654982)
7. E. G. BIRGIN, J. M. MARTÍNEZ, AND M. RAYDAN, Inexact spectral projected gradient methods on convex sets, IMA J. Numer. Anal., 23 (2003), pp. 539-559. (Pubitemid 37445929)
8. V. BLANZ, B. SCHÖLKOPF, H. BÜLTHOFFAND, C. BURGES, V. N. VAPNIK, AND T. VETTER, Comparison of view-based object recognition algorithms using realistic 3d models, in Artificial Neural Networks, ICIANN '96, Lecture Notes in Comput. Sci. 1112 J. V. C. von der Malsburg, W. von Seelen, and B. Sendhoff, eds., Springer, Berlin, 1996, pp. 251-256. (Pubitemid 126098540)
9. A. BORDES, S. ERTEKIN, J. WESTON, AND L. BOTTOU, Fast kernel classifiers with online and active learning, 2005. MNIST data set available at http://leon. bottou.org/papers/bordesertekin-weston-bottou-2005.
10. B. BOSER, I. GUYON, AND V. VAPNIK, A training algorithm for optimal margin classifier, in Proceedings of the 5th ACM Workshop on Computational Learning Theory, Pittsburgh, PA, 1992, pp. 144-152. (Pubitemid 23615470)
11. C. BURGES, A tutorial on support vector machines for pattern recognition, Data Min. Knowl. Disc., 2 (1998), pp. 121-167. (Pubitemid 128695475)
12. C. BURGES AND B. SCHÖLKOPF, Improving the accuracy and speed of support vector machines, in Advances in Neural Information Processing Systems 9 M. Mozer, M. Jordan, and T. Petsche, eds., MIT Press, Cambridge, MA, 1997.
13. C.-C. CHANG, C.-W. HSU, AND C.-J. LIN, The analysis of decomposition methods for support vector machines, IEEE Trans. Neural Networks, 11 (2000), pp. 1003-1008.
14. C.-C. CHANG AND C.-J. LIN, LIBSVM: A library for support vector machines, 2001. Software available at http://www.csie.ntu.edu.tw/-cjlin/libsvm.
15. C. CORTES AND V. VAPNIK, Support vector networks, Mach. Learn., 20 (1995), pp. 1-25.
16. Y. H. DAI, Alternate stepsize gradient method, Optimization, 52 (2003), pp. 395-415.
17. Y. H. DAI AND R. FLETCHER, New algorithms for singly linearly constrained quadratic programs subject to lower and upper bounds, Math. Program., 106 (2006), pp. 403-421.
18. Y. H. DAI, W. W. HAGER, K. SCHITTKOWSKI, AND H. ZHANG, The cyclic Barzilai-Borwein method for unconstrained optimization, IMA J. Numer. Anal., 26 (2006), pp. 604-627. (Pubitemid 44048227)
19. Y. H. DAI AND H. ZHANG, An adaptive two-point stepsize gradient algorithm, Numer. Algorithms, 27 (2001), pp. 377-385. (Pubitemid 33643870)
20. E. D. DOLAN AND J. J. MORÉ, Benchmarking optimization software with performance profiles, Math. Program., 91 (2002), pp. 201-213. (Pubitemid 44744777)
21. J. X. DONG, A. KRZYZAK, AND C. Y. SUEN, Fast SVM training algorithm with decomposition on very large data sets, IEEE Trans. Pattern Anal. Mach. Intell., 27 (2005), pp. 603-618. (Pubitemid 40556114)
22. R.-E. FAN, K.-W. CHANG, C.-J. HSIEH, X.-R. WANG, AND C.-J. LIN, LIBLINEAR: A library for large linear classification, J. Mach. Learn. Res., 9 (2008), pp. 1871-1874.
23. R.-E. FAN, P.-H. CHEN, AND C.-J. LIN, Working set selection using second order information for training support vector machines, J. Mach. Learn. Res., 6 (2005), pp. 1889-1918. (Pubitemid 41798130)
24. M. C. FERRIS AND T. S. MUNSON, Interior point methods for massive support vector machines, SIAM J. Optim., 13 (2003), pp. 783-804. (Pubitemid 36971112)
25. M. C. FERRIS AND T. S. MUNSON, Semismooth support vector machines, Math. Program., 101 (2004), pp. 185-204. (Pubitemid 41201323)
26. S. FINE AND K. SCHEINBERG, Efficient SVM training using low-rank kernel representations, J. Mach. Learn. Res., 2 (2001), pp. 243-264.
27. A. FRIEDLANDER, J. M. MARTÍNEZ, B. MOLINA, AND M. RAYDAN, Gradient method with retards and generalizations, SIAM J. Numer. Anal., 36 (1999), pp. 275-289.
28. E. M. GERTZ AND J. D. GRIFFIN, Using an iterative linear solver in an interior-point method for generating support vector machines, Comput. Optim. Appl., (2008), DOI 10.1007/s10589-008-9228-z.
29. L. E. GIBBONS, D. W. HEARN, P. M. PARDALOS, AND M. V. RAMANA, Continuous characterizations of the maximum clique problem, Math. Oper. Res., 22 (1997), pp. 754-768. (Pubitemid 127653360)
30. T. GLASMACHERS AND C. IGEL, Maximum-gain working set selection for SVMs, J. Mach. Learn. Res., 7 (2006), pp. 1437-1466. (Pubitemid 44024593)
31. M. D. GONZALEZ-LIMA, W. W. HAGER, AND H. ZHANG, An affine-scaling interior-point method for continuous knapsack constraints, 2010. Software available at http://www.math.-ufl.edu/-hager/papers/SVM.
32. L. GRIPPO, F. LAMPARIELLO, AND S. LUCIDI, A nonmonotone line search technique for Newton's method, SIAM J. Numer. Anal., 23 (1986), pp. 707-716. (Pubitemid 16604645)
33. W. W. HAGER AND Y. KRYLYUK, Graph partitioning and continuous quadratic programming, SIAM J. Discrete Math., 12 (1999), pp. 500-523. (Pubitemid 129643179)
34. W. W. HAGER, B. A. MAIR, AND H. ZHANG, An affine-scaling interior-point CBB method for box-constrained optimization, Math. Program., 119 (2009), pp. 1-32.
35. W. W. HAGER AND H. ZHANG, A new conjugate gradient method with guaranteed descent and an efficient line search, SIAM J. Optim., 16 (2005), pp. 170-192.
36. W. W. HAGER AND H. ZHANG, A new active set algorithm for box constrained optimization, SIAM J. Optim., 17 (2006), pp. 526-557. (Pubitemid 46798079)
37. D. HUSH, P. KELLY, C. SCOVEL, AND I. STEINWART, QP algorithms with guaranteed accuracy and run time for support vector machines, J. Mach. Learn. Res., 7 (2006), pp. 733-769.
38. J. JOACHIMS, Text Categorization with Support Vector Machine, Technical report LS VIII Number 23, ftp://ftp-ai.informatik.uni-dortmund.de/pub/Reports/ report23.ps. Z, University of Dortmund, 1997.
39. T. JOACHIMS, Making large-scale support vector machine learning practical, in Advances in Kernel Methods-Support Vector Learning, B. Schölkopf, C. Burges, and A. Smola, eds., MIT Press, Cambridge, MA, 1998, pp. 169-184.
40. T. JOACHIMS, Training linear SVMs in linear time, in Proceedings of the ACM Conference on Knowledge Discovery and Data Mining, 2006.
41. S. KEERTHI, S. SHEVADE, C. BHATTACHARYYA, AND K. MURTHY, Improvements to Platts SMO, Neural Comput., 13 (2001), pp. 637-649.
42. S. S. KEERTHI AND E. G. GILBERT, Convergence of a generalized SMO algorithm for SVM classifier design, Mach. Learn., 46 (2002), pp. 351-360. (Pubitemid 34129975)
43. S. S. KEERTHI AND S. K. SHEVADE, SMO algorithm for least-squares SVM formulations, Neural Comput., 15 (2003), pp. 487-507. (Pubitemid 37049831)
44. C.-J. LIN, Linear Convergence of a Decomposition Method for Support Vector Machines, Technical report, Department of Computater Science and Information Engineering, National Taiwan University, Taipei, Taiwan, 2001.
45. C.-J. LIN, On the convergence of a decomposition method for support vector machines, IEEE Trans. Neural Networks, 12 (2001), pp. 1288-1298. (Pubitemid 33106034)
46. C.-J. LIN, Asymptotic convergence of an SMO algorithm without any assumptions, IEEE Trans. Neural Networks, 13 (2002), pp. 248-250. (Pubitemid 34236853)
47. O. L. MANGASARIAN AND D. R. MUSICANT, Lagrangian support vector machines, J. Mach. Learn. Res., 1 (2001), pp. 161-177. (Pubitemid 33687201)
48. T. S. MOTZKIN AND E. G. STRAUSS, Maxima for graphs and a new proof of a theorem of Turan, Canad. J. Math., 17 (1965), pp. 533-540.
49. E. OSUNA, R. FREUND, AND F. GIROSI, Improved training algorithm for support vector machines, in Proceedings of the IEEE Workshop on Neural Networks for Signal Processing, 1997, pp. 276-285.
50. E. OSUNA, R. FREUND, AND F. GIROSI, Support Vector Machines: Training and Applications, Technical report AIM-1602, C. B. C. L. Paper No. 144, MIT, Cambridge, MA, 1997.
51. E. OSUNA, R. FREUND, AND F. GIROSI, Training support vector machines: An application to face detection, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1997, pp. 130-136.
52. light Algorithm, Technical report, Istituto di Analisi dei Sistemi ed Informatica, Rome, 2002.
53. J. PLATT, Fast training of support vector machines using sequential minimal optimization, in Advances in Kernel Methods-Support Vector Learning, B. Schölkopf, C. Burges, and A. Smola, eds., MIT Press, Cambridge, MA, 1998, pp. 41-65.
54. M. RAYDAN, The Barzilai and Borwein gradient method for the large scale unconstrained minimization problem, SIAM J. Optim., 7 (1997), pp. 26-33.
55. M. RAYDAN AND B. F. SVAITER, Relaxed steepest descent and Cauchy-Barzilai-Borwein method, Comput. Optim. Appl., 21 (2002), pp. 155-167. (Pubitemid 34168103)
56. K. SCHEINBERG, An efficient implementation of an active set method for SVMs, J. Mach. Learn. Res., 7 (2006), pp. 2237-2257. (Pubitemid 44611764)
57. M. SCHMIDT, Identifying speaker with support vector networks, in Proceedings of the 28th Symposium on the Interface, Sydney, Australia, 1996.
58. B. SCHÖLKOPF, C. BURGES, AND V. N. VAPNIK, Extracting support data for a given task, in Proceedings of the First International Conference on Knowledge Discovery and Data Mining, U. M. Fayyad and R. Uthurusamy, eds., AAAI Press, Menlo Park, CA, 1995.
59. B. SCHÖLKOPF, Incorporating invariances in support vector learning machines, in Artificial Neural Networks, J. V. C. von der Malsburg, W. von Seelen, and B. Sendhoff, eds., Springer, Berlin, 1996, pp. 47-52. (Pubitemid 126098507)
60. T. SERAFINI, G. ZANGHIRATI, AND L. ZANNI, Gradient projection methods for quadratic programs and applications in training support vector machines, Optim. Methods Softw., 20 (2005), pp. 353-378.
61. T. SERAFINI AND L. ZANNI, On the working set selection in gradient-based decomposition techniques for support vector machines, Optim. Methods Softw., 20 (2005), pp. 583-596. (Pubitemid 41602158)
62. T. SERAFINI, L. ZANNI, AND G. ZANGHIRATI, GPDT: Gradient projection-based decomposition technique, 2004. Software available at http://dm.unife.it/gpdt.
63. P. TSENG AND S. YUN, A coordinate gradient descent method for linearly constrained smooth optimization and support vector machines training, Comput. Optim. Appl., (2008), DOI 10.1007/s10589-008-9215-4.
64. V. VAPNIK, The Nature of Statistical Learning Theory, Springer, New York, 1995.
65. J. WESTON, A. GAMMERMAN, M. STITSON, V. VAPNIK, V. VORK, AND C. WATKINS, Density estimation using support vector machines, Technical report CSD-TR-97-23, Royal Holloway College, 1997.
66. K. WOODSEND AND J. GONDZIO, Exploiting separability in large-scale linear support vector machine training, Comput. Optim. Appl., (2009), DOI 10.1007/s10589-009-9296-8.
67. L. ZANNI, An improved gradient projection-based decomposition technique for support vector machines, Comput. Manag. Sci., 3 (2006), pp. 131-145. (Pubitemid 43414563)
68. L. ZANNI, T. SERAFINI, AND G. ZANGHIRATI, Parallel software for training large scale support vector machines on multiprocessor systems, J. Mach. Learn. Res., 7 (2006), pp. 1467-1492. (Pubitemid 44024592)