Sparse conjugate directions pursuit with application to fixed-size kernel models

Machine Learning

Volumn 85, Issue 1-2, 2011, Pages 109-148

  Karsmakers, Peter ^a,c   Pelckmans, Kristiaan ^b   De Brabanter, Kris ^a   Van Hamme, Hugo ^a   Suykens, Johan A K ^a  

^a UNIVERSITY OF LEUVEN   (Belgium)

^b UPPSALA UNIVERSITY   (Sweden)

^c Associatie KULeuven   (Belgium)

Author keywords

Greedy heuristic; Kernel methods; Linear system; Matching pursuit; Sparse approximation

Indexed keywords

ALGORITHMIC ISSUES; APPROXIMATE SOLUTION; CARDINALITIES; COMPONENT SELECTION; COMPUTATIONAL ANALYSIS; CONJUGATE BASIS; CONJUGATE GRADIENT ALGORITHMS; GENERALIZATION PERFORMANCE; GREEDY HEURISTICS; KERNEL METHODS; KERNEL MODELS; LEAST SQUARES SUPPORT VECTOR MACHINES; LINEAR SUBSYSTEMS; MACHINE-LEARNING; MATCHING PURSUIT; MODEL SIZE; OPTIMIZATION SCHEME; SCALABLE ALGORITHMS; SPARSE APPROXIMATIONS; STOPPING CRITERIA; WORK STUDY;

APPROXIMATION ALGORITHMS; COMPUTATIONAL COMPLEXITY; CONJUGATE GRADIENT METHOD; LEARNING SYSTEMS; LINEAR SYSTEMS;

COMPUTATIONAL EFFICIENCY;

EID: 80053087394     PISSN: 08856125     EISSN: 15730565     Source Type: Journal    
DOI: 10.1007/s10994-011-5253-8     Document Type: Article

References (60)

1. D. Allan 1974 The relationship between variable selection and prediction Technometrics 16 125 127 343481 10.2307/1267500
2. T. Blumensath M. E. Davies 2008 Gradient pursuits IEEE Transactions on Signal Processing 56 6 2370 2382 2516640 10.1109/TSP.2007.916124 (Pubitemid 351795890)
3. A. M. Bruckstein D. L. Donoho M. Elad 2009 From sparse solutions of systems of equations to sparse modeling of signals and images SIAM Review 51 1 34 81 2481111 1178.68619 10.1137/060657704
4. P. Bühlmann B. Yu 2003 Boosting with the l2-loss: Regression and classification Journal of the American Statistical Association 98 324 339 1995709 1041.62029 10.1198/016214503000125 (Pubitemid 36931557)
5. Candès, E. (2006). Compressive sampling. In Proc. of the international congress of mathematicians, Madrid, Spain.
6. Cawley, G. C. (2006). Leave-one-out cross-validation based model selection criteria for weighted LS-SVMs. In Proc. of the international joint conference on neural networks (IJCNN-2006), Vancouver, Canada (Vol. 2415, pp. 1661-1668). (Pubitemid 351369522)
7. Cawley, G. C., & Talbot, N. L. C. (2002). A greedy training algorithm for sparse least-squares support vector machines. In Proc. of the international conference on artificial neural networks, Madrid, Spain (Vol. 2415, pp. 681-686).
8. Chang, C. C., & Lin, C. J. (2001). Libsvm: a library for support vector machines. http://www.csie.ntu.edu.tw/cjlin/libsvm.
9. S. Chen S. A. Billings W. Luo 1989 Orthogonal least squares methods and their application to non-linear system identification International Journal of Control 50 5 1873 1896 1032440 0686.93093 10.1080/00207178908953472 (Pubitemid 20636409)
10. S. Chen C. Cowan P. Grant 1991 Orthogonal least squares learning algorithm for radial basis function networks IEEE Transactions on Neural Networks 2 2 302 309 10.1109/72.80341
11. S. Chen S. A. Billings M. Grant 1992 Recursive hybrid algorithm for non-linear system identification using radial basis function networks International Journal of Control 55 5 1051 1070 1164955 0757.93015 10.1080/00207179208934272
12. S. Chen X. Hong B. L. Luk C. J. Harris 2009 Orthogonal-least-squares regression: A unified approach for data modelling Neurocomputing 72 10-12 2670 2681 10.1016/j.neucom.2008.10.002
13. S. S. Chen D. L. Donoho M. A. Saunders 1999 Atomic decomposition by basis pursuit SIAM Journal on Scientific Computing 20 1 33 61 1639094 0919.94002 10.1137/S1064827596304010 (Pubitemid 128689501)
14. W. Chu S. S. Keerthi C. Ong 2005 An improved conjugate gradient scheme to the solution of least squares SVM IEEE Transactions on Neural Networks 16 2 498 501 10.1109/TNN.2004.841785 (Pubitemid 40390833)
15. Daubechies, I., Vore, R. D., Fornasier, M., & Gunturk, S. (2008). Iteratively re-weighted least squares minimization: Proof of faster than linear rate for sparse recovery. In Proc. of the information sciences and systems, Princeton, NJ (pp. 26-29).
16. K. De Brabanter J. De Brabanter J. A. K. Suykens B. De Moor 2010 Optimized fixed-size kernel models for large data sets Computational Statistics & Data Analysis 54 6 1484 1504 10.1016/j.csda.2010.01.024
17. B. J. de Kruif T. J. A. de Vries 2003 Pruning error minimization in least squares support vector machines IEEE Transactions on Neural Networks 14 3 696 702 10.1109/TNN.2003.810597
18. D. L. Donoho 2006 Compressed sensing IEEE Transactions on Information Theory 52 4 1289 1306 2241189 10.1109/TIT.2006.871582
19. Donoho, D. L., & Tsaig, Y. (2006). Fast solution of l1-norm minimization problems when the solution may be sparse (Tech. rep.). Stanford University.
20. Donoho, D. L., Tsaig, Y., Drori, I., & Starck, J. (2006). Sparse solution of underdetermined linear equations by stagewise orthogonal matching pursuit (Tech. rep.). Stanford University.
21. B. Efron T. Hastie I. Johnstone R. Tibshirani 2004 Least angle regression (with discussion) Annals of Statistics 32 2 407 499 2060166 1091.62054 10.1214/009053604000000067 (Pubitemid 41250302)
22. M. A. T. Figueiredo R. D. Nowak S. J. Wright 2007 Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems IEEE Journal of Selected Topics in Signal Processing 1 4 586 597 10.1109/JSTSP.2007.910281 (Pubitemid 351276802)
23. S. Floyd M. Warmuth 1995 Sample compression, learnability, and the Vapnik-Chervonenkis dimension Machine Learning Journal 21 269 304
24. T. Gonzalez 1985 Clustering to minimize the maximum intercluster distance Theoretical Computer Science 38 293 306 807927 0567.62048 10.1016/0304-3975(85) 90224-5 (Pubitemid 15607838)
25. Hastie, T., Tibshirani, R., & Friedman, J. (2001). The elements of statistical learning. Berlin: Springer.
26. M. R. Hestenes E. Stiefel 1952 Methods of conjugate gradients for solving linear systems Journal of Research of the National Bureau of Standards 49 409 436 60307 0048.09901
27. L. Jiao L. Bo L. Wang 2007 Fast sparse approximation for least squares support vector machine IEEE Transactions on Neural Networks 18 3 685 697 10.1109/TNN.2006.889500 (Pubitemid 46773736)
28. T. Joachims 1999 Making large-scale SVM learning practical B. Schlkopf C. Burges A. Smola (eds) Advances in kernel methods-support vector learning MIT Press Cambridge Chap. 11. URL http://www.cs.cornell.edu/People/tj/publications/ joachims-99a.pdf
29. S. S. Keerthi O. Chapelle D. Decoste P. Bennett E. Parrado- hernández 2006 Building support vector machines with reduced classifier complexity Journal of Machine Learning Research 7 1493 1515 (Pubitemid 44024591)
30. S. J. Kim K. Koh M. Lustig S. Boyd D. Gorinevsky 2007 An interior-point method for large-scale l1-regularized least squares IEEE Journal on Selected Topics in Signal Processing 1 4 606 617 10.1109/JSTSP.2007.910971 (Pubitemid 351266626)
31. T. Kohonen 1990 The self-organizing map Proceedings of the IEEE 78 9 1464 1480 10.1109/5.58325
32. R. W. Lutz P. Bühlmann 2006 Conjugate direction boosting Journal of Computational and Graphical Statistics 15 2 287 311 2252528 10.1198/ 106186006X113548 (Pubitemid 43935979)
33. Mallat, S. (1999). A wavelet tour of signal processing. New York: Academic Press.
34. S. Mika G. Rätsch J. Weston B. Schölkopf A. Smola K. R. Müller 2000 Invariant feature extraction and classification in feature spaces Advances in Neural Information Processing Systems 12 526 532
35. B. Moghaddam Y. Weiss S. Avidan 2006 Spectral bounds for sparse PCA: exact and greedy algorithms Advances in Neural Information Processing Systems 18 915 922
36. M. F. Möller 1993 A scaled conjugate gradient algorithm for fast supervised learning Neural Networks 6 525 533 10.1016/S0893-6080(05)80056-5
37. B. K. Natarajan 1995 Sparse approximate solutions to linear systems SIAM Journal on Computing 24 227 234 1320206 0827.68054 10.1137/S0097539792240406
38. J. A. Nelder R. Mead 1965 A simplex method for function minimization Computer Journal 7 308 313 0229.65053
39. Nocedal, J., & Wright, S. J. (2006). Numerical optimization (2nd ed.). Berlin: Springer.
40. M. R. Osborne B. Presnell B. A. Turlach 2000 A new approach to variable selection in least squares problems IMA Journal of Numerical Analysis 20 389 403 1773265 0962.65036 10.1093/imanum/20.3.389
41. Pati, Y. C., Rezaifar, R., & Krishnaprasad, P. S. (1993). Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition. In Proc. of the 27-th annual Asilomar conference on signals, systems, and computers (pp. 40-44).
42. V. Popovici S. Bengio J. P. Thiran 2005 Kernel matching pursuit for large datasets Pattern Recognition 38 12 2385 2390 10.1016/j.patcog.2005.01.021 (Pubitemid 41336707)
43. Press, W. H., Cornell, B. P. F., Teukolsky, S. A., & Vetterling, W. T. (1993). Numerical recipes in C: The art of scientific computing (2nd ed.). Cambridge: Harvard University Press.
44. S. Rajasekaran 2000 On simulated annealing and nested annealing Journal of Global Optimization 16 43 56 1770521 1112.90383 10.1023/A:1008307523936
45. Saunders, C., Gammerman, A., & Vovk, V. (1998). Ridge regression learning algorithm in dual variables. In Proc. of the 15th int. conf. on machine learning (ICML-98) (pp. 515-521).
46. Smola, A. J., & Bartlett, P. L. (2001). Sparse greedy Gaussian process regression. In Proc. neural information processing systems (Vol. 13, pp. 619-625).
47. J. A. K. Suykens J. Vandewalle 1999 Least squares support vector machine classifiers Neural Processing Letters 9 3 293 300 1721843 10.1023/A: 1018628609742
48. Suykens, J. A. K., Lukas, L., & Vandewalle, J. (2000). Sparse approximation using least squares support vector machines. In IEEE Proc. int. symp. circuits syst., Geneva, Switzerland (pp. 757-760).
49. J. A. K. Suykens J. Vandewalle B. De Moor 2001 Intelligence and cooperative search by coupled local minimizers International Journal of Bifurcation and Chaos 11 8 2133 2144 10.1142/S0218127401003371
50. J. A. K. Suykens J. De Brabanter L. Lukas J. Vandewalle 2002 Weighted least squares support vector machines: robustness and sparse approximation Neurocomputing 48 1-4 85 105 1006.68799 10.1016/S0925-2312(01)00644-0 (Pubitemid 36221490)
51. Suykens, J. A. K., Van Gestel, T., De Brabanter, J., De Moor, B., & Vandewalle, J. (2002b). Least squares support vector machines. Singapore: World Scientific.
52. R. Tibshirani 1996 Regression shrinkage and selection via the LASSO Journal of the Royal Statistical Society. Series B 58 1 267 289 1379242 0850.62538
53. J. A. Tropp 2004 Greed is good: algorithmic results for sparse approximation IEEE Transactions on Information Theory 50 10 2231 2242 2097044 10.1109/TIT.2004.834793
54. Vapnik, V. N. (1998). Statistical learning theory. New York: Wiley.
55. P. Vincent Y. Bengio 2002 Kernel matching pursuit Machine Learning 48 165 187 0998.68120 10.1023/A:1013955821559 (Pubitemid 34247577)
56. C. K. I. Williams M. Seeger 2001 Using the Nyström method to speed up kernel machines Advances in Neural Information Processing Systems 13 682 688
57. S. Xavier de Souza J. A. K. Suykens J. Vandewalle D. Bollé 2010 Coupled simulated annealing IEEE Transactions on Systems, Man and Cybernetics. Part B 40 2 320 336 10.1109/TSMCB.2009.2020435
58. Young, D. (2003). Iterative solution of large linear systems. New York: Courier Dover Publications.
59. X. Y. Zeng X. W. Chen 2005 Smo-based pruning methods for sparse least squares support vector machines IEEE Transactions on Neural Networks 16 6 1541 1546 10.1109/TNN.2005.852239 (Pubitemid 41709650)
60. Zhang, K., Tsang, I. W., & Kwok, J. T. (2008). Improved Nyström low-rank approximation and error analysis. In Proc. of the 25th international conference on machine learning, Helsinki, Finland (pp. 1232-1239).