A quasi-newton approach to nonsmooth convex optimization problems in machine learning

Journal of Machine Learning Research

Volumn 11, Issue , 2010, Pages 1145-1200

  Yu, Jin ^a   Vishwanathan, S V N ^b   Günter, Simon ^c   Schraudolph, Nicol N ^d  

^a UNIVERSITY OF ADELAIDE   (Australia)

^b PURDUE UNIVERSITY   (United States)

^c DV Bern AG   (Switzerland)

^d Adaptive Tools AG   (Australia)

Author keywords

BFGS; BMRM; Bundle methods; Hinge loss; Multiclass; Multilabel; Ocas; Owl QN; Risk minimization; Subgradient; Variable metric methods; Wolfe conditions

Indexed keywords

BFGS; BMRM; BUNDLE METHODS; MULTI-CLASS; MULTI-LABEL; RISK MINIMIZATION; SUBGRADIENT; VARIABLE METRIC METHODS; WOLFE CONDITIONS;

CONVEX OPTIMIZATION; LEARNING ALGORITHMS; NEWTON-RAPHSON METHOD;

OPTIMIZATION;

EID: 77951160087     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (37)

1. N. Abe, J. Takeuchi, and M. K. Warmuth. Polynomial Learnability of Stochastic Rules with Respect to the KL-Divergence and Quadratic Distance. IEICE Transactions on Information and Systems, 84(3):299-316, 2001.
2. P. K. Agarwal and M. Sharir. Davenport-Schinzel sequences and their geometric applications. In J. Sack and J. Urrutia, editors, Handbook of Computational Geometry, pages 1-47. NorthHolland, New York, 2000.
3. 1-regularized log-linear models. In Proc. Intl. Conf. Machine Learning, pages 33-40, New York, NY, USA, 2007. ACM.
4. J. Basch. Kinetic Data Structures. PhD thesis, Stanford University, June 1999.
5. D. P. Bertsekas. Nonlinear Programming. Athena Scientific, Belmont, MA, 1999.
6. J. R. Birge, L. Qi, and Z. Wei. A general approach-convergence properties of some methods for nonsmooth convex optimization. Applied Mathematics and Optimization, 38(2):141-158,1998. (Pubitemid128621131)
7. A. Bordes, L. Bottou, P. Gallinari, and J. Weston. Solving multiclass support vector machines with LaRank. In Proc. Intl. Conf. Machine Learning, pages 89-96, New York, NY, USA, 2007. ACM.
8. S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, Cambridge, England, 2004.
9. K. Crammer and Y. Singer. Ultraconservative online algorithms for multiclass problems. Journal of Machine Learning Research, 3:951-991, January 2003a.
10. K. Crammer and Y. Singer. A family of additive online algorithms for category ranking. J. Mach. Learn. Res., 3:1025-1058, February 2003b.
11. V. Franc and S. Sonnenburg. Optimized cutting plane algorithm for support vector machines. In A. McCallum and S. Roweis, editors, ICML, pages 320-327. Omnipress, 2008.
12. V. Franc and S. Sonnenburg. Optimized cutting plane algorithm for large-scale risk minimization. Journal of Machine Learning Research, 10:2157-2192, 2009.
13. M. Haarala. Large-Scale Nonsmooth Optimization. PhD thesis, University of Jyväskylä, 2004.
14. J. Hershberger. Finding the upper envelope of n line segments in O(n log n) time. Information Processing Letters, 33(4):169-174, December 1989.
15. J. B. Hiriart-Urruty and C. Lemaréchal. Convex Analysis and Minimization Algorithms, I and II, volume 305 and 306. Springer-Verlag, 1993.
16. T. Joachims. Training linear SVMs in linear time. In Proc. ACM Conf. Knowledge Discovery and Data Mining (KDD). ACM, 2006.
17. Y. J. Lee and O. L. Mangasarian. SSVM: A smooth support vector machine for classification. Computational optimization and Applications, 20(1):5-22, 2001. (Pubitemid 32797935)
18. C. Lemarechal. Numerical experiments in nonsmooth optimization. Progress in Nondifferentiable Optimization, 82:61-84, 1982.
19. A. S. Lewis and M. L. Overton. Nonsmooth optimization via BFGS. Technical report, Optimization Online, 2008a. URL http://www.optimization-online.org/DB- FILE/2008/12/ 2172.pdf. Submitted to SIAM J. Optimization.
20. A. S. Lewis and M. L. Overton. Behavior of BFGS with an exact line search on nonsmooth examples. Technical report, Optimization Online, 2008b. URL http://www. optimization-online.org/DB-FILE/2008/12/2173.pdf. Submitted to SIAM J. Optimization.
21. D. C. Liu and J. Nocedal. On the limited memory BFGS method for large scale optimization. Mathematical Programming, 45(3):503-528, 1989.
22. L. Lukšan and J. Vlček. Globally convergent variable metric method for convex nonsmooth unconstrained minimization. Journal of Optimization Theory and Applications, 102(3):593-613, 1999.
23. F. Maes, L. Denoyer, and P. Gallinari. XML structure mapping application to the PASCAL/INEX 2006 XML document mining track. In Advances in XML Information Retrieval and Evaluation: Fifth Workshop of the INitiative for the Evaluation of XML Retrieval (INEX'06), Dagstuhl, Germany, 2007.
24. A. Nedić and D. P. Bertsekas. Convergence rate of incremental subgradient algorithms. In S. Uryasev and P. M. Pardalos, editors, Stochastic Optimization: Algorithms and Applications, pages 263-304. Kluwer Academic Publishers, 2000.
25. A. Nemirovski. Prox-method with rate of convergence O(1/t) for variational inequalities with Lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM J. on Optimization, 15(1):229-251, 2005. ISSN 1052-6234.
26. Y. Nesterov. Smooth minimization of non-smooth functions. Math. Program., 103(1): 127-152, 2005.
27. J. Nocedal and S. J. Wright. Numerical Optimization. Springer Series in Operations Research. Springer, 1999.
28. S. Shalev-Shwartz and Y. Singer. On the equivalence of weak learnability and linear separability: New relaxations and efficient boosting algorithms. In Proceedings of COLT, 2008.
29. A. J. Smola, S. V. N. Vishwanathan, and Q. V. Le. Bundle methods for machine learning. In D. Koller and Y. Singer, editors, Advances in Neural Information Processing Systems 20, Cambridge MA, 2007. MIT Press.
30. B. Taskar, C. Guestrin, and D. Koller. Max-margin Markov networks. In S. Thrun, L. Saul, and B. Schölkopf, editors, Advances in Neural Information Processing Systems 16, pages 25-32, Cambridge, MA, 2004. MIT Press.
31. C.-H. Teo, S. V. N. Vishwanthan, A. J. Smola, and Q. V. Le. Bundle methods for regularized risk minimization. Journal of Machine Learning Research, 11:311-365, 2010.
32. I. Tsochantaridis, T. Joachims, T. Hofmann, and Y. Altun. Large margin methods for structured and interdependent output variables. Journal of Machine Learning Research, 6:1453-1484, 2005.
33. M. K. Warmuth, K. A. Glocer, and S. V. N. Vishwanathan. Entropy regularized LPBoost. In Y. Freund, Y. Làszlò Györfi, and G. Turàn, editors, Proc. Intl. Conf. Algorithmic Learning Theory, number 5254 in Lecture Notes in Artificial Intelligence, pages 256 - 271, Budapest, October 2008. Springer-Verlag.
34. P. Wolfe. Convergence conditions for ascent methods. SIAM Review, 11 (2): 226-235, 1969.
35. P. Wolfe. A method of conjugate subgradients for minimizing nondifferentiable functions. Mathematical Programming Study, 3:145-173, 1975.
36. J. Yu, S. V. N. Vishwanathan, S. Gunter, and N. N. Schraudolph. A quasi-Newton approach to nonsmooth convex optimization. In A. McCallum and S. Roweis, editors, ICML, pages 1216-1223. Omnipress, 2008.
37. T. Zhang and F. J. Oles. Text categorization based on regularized linear classification methods. Information Retrieval, 4:5-31, 2001.