Nonlinear regularization path for quadratic loss support vector machines

IEEE Transactions on Neural Networks

Volumn 22, Issue 10, 2011, Pages 1613-1625

  Karasuyama, Masayuki ^a   Takeuchi, Ichiro ^a  

^a NAGOYA INSTITUTE OF TECHNOLOGY   (Japan)

Author keywords

Parametric programming; rational approximation; support vector machines

Indexed keywords

ALGORITHM PERFORMANCE; BREAK-POINTS; LEARNING MACHINES; MACHINE-LEARNING; MODEL SELECTION PROBLEM; PARAMETRIC PROGRAMMING; PATH ALGORITHM; PENALTY TERM; PIECE-WISE; PIECEWISE LINEAR; PREDICTOR CORRECTOR; QUADRATIC CONVERGENCE; QUADRATIC LOSS; RATIONAL APPROXIMATIONS; RATIONAL EQUATIONS; REAL DATA SETS; REGULARIZATION PARAMETERS; SOLUTION PATH; SUPPORT VECTOR;

APPROXIMATION ALGORITHMS; LEARNING ALGORITHMS; NONLINEAR EQUATIONS; PIECEWISE LINEAR TECHNIQUES; RATIONAL FUNCTIONS;

SUPPORT VECTOR MACHINES;

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; COMPUTER; COMPUTER PROGRAM; HUMAN; NONLINEAR SYSTEM;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTERS; HUMANS; NEURAL NETWORKS (COMPUTER); NONLINEAR DYNAMICS; SOFTWARE DESIGN;

EID: 80053646089     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2011.2164265     Document Type: Article

References (43)

1. T. Hastie, S. Rosset, R. Tibshirani, and J. Zhu, "The entire regularization path for the support vector machine," J. Mach. Learn. Res., vol. 5, pp. 1391-1415, Oct. 2004.
2. E. L. Allgower and K. Georg, "Continuation and path following," Acta Numer., vol. 2, pp. 1-64, 1993.
3. T. Gal, Postoptimal Analysis, Parametric Programming, and Related Topics. Berlin, Germany: Walter de Gruyter, 1995.
4. F. Bach, D. Heckerman, and E. Horvitz, "Considering cost asymmetry in learning classifiers," J. Mach. Learn. Res., vol. 7, pp. 1713-1741, Aug. 2006.
5. G. Cauwenberghs and T. Poggio, "Incremental and decremental support vector machine learning," in Advances in Neural Information Processing Systems, vol. 13, T. K. Leen, T. G. Dietterich, and V. Tresp, Eds. Cambridge, MA: MIT Press, 2001, pp. 409-415.
6. L. Gunter and J. Zhu, "Efficient computation and model selection for the support vector regression," Neural Comput., vol. 19, no. 6, pp. 1633-1655, Jun. 2007.
7. M. Karasuyama and I. Takeuchi, "Multiple incremental decremental learning of support vector machines," in Advances in Neural Information Processing Systems 22, Y. Bengio, D. Schuurmans, J. Lafferty, C. K. I. Williams, and A. Culotta, Eds. Cambridge, MA: MIT Press, 2009, pp. 907-915.
8. P. Laskov, C. Gehl, S. Kruger, and K.-R. Muller, "Incremental support vector learning: Analysis, implementation and applications," J. Mach. Learn. Res., vol. 7, pp. 1909-1936, Sep. 2006. (Pubitemid 44477117)
9. J. Ma and J. Theiler, "Accurate on-line support vector regression," Neural Comput., vol. 15, no. 11, pp. 2683-2703, Nov. 2003. (Pubitemid 37206931)
10. M. Martin, "On-line support vector machines for function approximation," Softw. Dept., Univ. Politecnica de Catalunya, Barcelona, Spain, Tech. Rep. LSI-02-11-R, 2002.
11. C.-J. Ong, S. Shao, and J. Yang, "An improved algorithm for the solution of the regularization path of support vector machine," IEEE Trans. Neural Netw., vol. 21, no. 3, pp. 451-462, Mar. 2010.
12. I. Takeuchi, K. Nomura, and T. Kanamori, "Nonparametric conditional density estimation using piecewise-linear solution path of kernel quantile regression," Neural Comput., vol. 21, no. 2, pp. 533-559, Feb. 2009.
13. G. Wang, D.-Y. Yeung, F. H. Lochovsky, "A new solution path algorithm in support vector regression," IEEE Trans. Neural Netw., vol. 19, no. 10, pp. 1753-1767, Oct. 2008.
14. V. N. Vapnik, The Nature of Statistical Learning Theory. New York: Springer-Verlag, 1995.
15. S. Rosset and J. Zhu, "Piecewise linear regularized solution paths," Ann. Stat., vol. 35, no. 3, pp. 1012-1030, 2007.
16. R. Tibshirani, "Regression shrinkage and selection via the lasso," J. Royal Stat. Soc. Ser. B: Methodol., vol. 58, no. 1, pp. 267-288, 1996.
17. K. Ritter, "On parametric linear and quadratic programming problems," in Proc. Int. Congr. Math. Program., 1984, pp. 307-335.
18. F. R. Bach, R. Thibaux, and M. I. Jordan, "Computing regularization paths for learning multiple kernels," in Advances in Neural Information Processing Systems 17, L. K. Saul, Y. Weiss, and L. Bottou, Eds. Cambridge, MA: MIT Press, 2005, pp. 73-80.
19. V. Krishnamurthy, S. D. Ahipasaoglu, and A. d'Aspremont, "A pathwise algorithm for covariance selection," in Proc. Neural Inf. Process. Syst. Workshop Optim. Mach. Learn., 2009, pp. 1-12.
20. M. Park and T. Hastie, "L1-regularization path algorithm for generalized linear models," J. Royal Stat. Soc.: Ser. B: Stat. Methodol., vol. 69, no. 4, pp. 659-677, 2007.
21. S. Rosset, "Following curved regularized optimization solution paths," in Advances in Neural Information Processing Systems 17, L. K. Saul, Y. Weiss, and L. Bottou, Eds. Cambridge, MA: MIT Press, 2005, pp. 1153-1160.
22. G. Wang, D.-Y. Yeung, and F. H. Lochovsky, "A kernel path algorithm for support vector machines," in Proc. 24th Int. Conf. Mach. Learn., 2007, pp. 951-958.
23. T. Zhang, "Statistical behavior and consistency of classification methods based on convex risk minimization," Ann. Stat., vol. 32, no. 1, pp. 56-134, 2004. (Pubitemid 41449305)
24. O. Chapelle, "Training a support vector machine in the primal," Neural Comput., vol. 19, no. 5, pp. 1155-1178, May 2007.
25. P. L. Bartlett and A. Tewari, "Sparseness versus estimating conditional probabilities: Some asymptotic results," J. Mach. Learn. Res., vol. 8, pp. 775-790, Apr. 2007. (Pubitemid 46677047)
26. L. Bo, L. Wang, and L. Jiao, "Recursive finite newton algorithm for support vector regression in the primal," Neural Comput., vol. 19, no. 4, pp. 1082-1096, Apr. 2007.
27. S. S. Keerthi and D. DeCoste, "A modified finite Newton method for fast solution of large scale linear SVMs," J. Mach. Learn. Res., vol. 6, pp. 341-361, Mar. 2005.
28. J. Bunch, C. Nielsen, and D. Sorensen, "Rank-one modification of the symmetric eigenproblem," Numer. Math., vol. 31, no. 1, pp. 31-48, 1979.
29. W. Chu, S. S. Keerthi, and C. J. Ong, "Bayesian support vector regression using a unified loss function," IEEE Trans. Neural Netw., vol. 15, no. 1, pp. 29-44, Jan. 2004.
30. G. H. Golub, "Some modified eigenvalue problems," Dept. Comput. Sci., Stanford Univ., Stanford, CA, Tech. Rep. SU326 P30-11, Aug. 1971.
31. G. H. Golub and C. F. V. Loan, Matrix Computations. Baltimore, MD: Johns Hopkins Univ. Press, 1996.
32. J. C. Platt, "Fast training of support vector machines using sequential minimal optimization," in Advances in Kernel Methods: Support Vector Learning, B. Schölkopf, C. J. C. Burges, and A. J. Smola, Eds. Cambridge, MA: MIT Press, 1999, pp. 185-208.
33. M. Vogt, "SMO algorithms for support vector machines without bias term," Inst. Automatic Control, Technische Universität Darmstadt, Darmstadt, Germany, Tech. Rep., 2002.
34. C.-C. Chang and C.-J. Lin. (2001). LIBSVM: A Library for Support Vector Machines [Online]. Available: http://www.csie.ntu.edu.tw/~cjlin/libsvm
35. D. DeCoste and K. Wagstaff, "Alpha seeding for support vector machines," in Proc. Int. Conf. Knowl. Discov. Data Min., 2000, pp. 345-359.
36. E. Anderson, Z. Bai, C. Bischof, L. S. Blackford, J. Demmel, J. J. Dongarra, J. Du Croz, S. Hammarling, A. Greenbaum, A. McKenney, and D. Sorensen, LAPACK Users' Guide, 3rd ed. Philadelphia, PA: SIAM, 1999.
37. L. Bottou and C.-J. Lin, "Support vector machine solvers," in Large Scale Kernel Machines, L. Bottou, O. Chapelle, D. DeCoste, and J. Weston, Eds. Cambridge, MA: MIT Press, 2007, pp. 301-320.
38. T. Fawcett, "An introduction to ROC analysis," Pattern Recognit. Lett., vol. 27, no. 8, pp. 861-874, Jun. 2006.
39. K. Duan, S. S. Keerthi, and A. N. Poo, "Evaluation of simple performance measures for tuning SVM hyperparameters," Neurocomputing, vol. 51, pp. 41-59, Apr. 2003. (Pubitemid 36367224)
40. V. N. Vapnik and O. Chapelle, "Bounds on error expectation for support vector machines," Neural Comput., vol. 12, no. 9, pp. 2013-2036, Sep. 2000.
41. P. Gutiérrez, C. Hervás-Martines, and F. Martinez-Estudillo, "Logistic regression by means of evolutionary radial basis function neural networks," IEEE Trans. Neural Netw., vol. 22, no. 2, pp. 246-263, Feb. 2011.
42. R. Mahdi and E. Rouchka, "Reduced hyperBF networks: Regularization by explicit complexity reduction and scaled rprop-based training," IEEE Trans. Neural Netw., vol. 22, no. 5, pp. 673-686, May 2011.
43. Y. Washizawa, "Feature extraction using constrained approximation and suppression," IEEE Trans. Neural Netw., vol. 21, no. 2, pp. 201-210, Feb. 2010.