Recent advances and trends in large-scale kernel methods

IEICE Transactions on Information and Systems

Volumn E92-D, Issue 7, 2009, Pages 1338-1353

  Kashima, Hisashi ^a   Id́e, Tsuyoshi ^a   Kato, Tsuyoshi ^b   Sugiyama, Masashi ^c  

^a IBM RESEARCH   (United States)

^b OCHANOMIZU UNIVERSITY   (Japan)

^c TOKYO INSTITUTE OF TECHNOLOGY   (Japan)

Author keywords

Kernel methods; Kernel trick; Lowrank approximation; Optimization; Structured data; Support vector machines

Indexed keywords

OPTIMIZATION; SUPPORT VECTOR MACHINES;

KERNEL METHODS; KERNEL TRICK; LINEAR ALGORITHMS; LOW RANK APPROXIMATIONS; MODERN MACHINES; NON LINEAR; STRUCTURED DATA; TRAINING SAMPLE;

LEARNING SYSTEMS;

EID: 77950308459     PISSN: 09168532     EISSN: 17451361     Source Type: Journal    
DOI: 10.1587/transinf.E92.D.1338     Document Type: Article

References (98)

1. V. Vapnik, The Nature of Statistical Learning Theory, Springer, 2000.
2. C. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.
3. J. Shawe-Taylor and N. Cristianini, Kernel Methods for Pattern Analysis, Cambridge University Press, New York, NY, USA, 2004.
4. L. Bottou, O. Chapelle, D. Decoste, and J. Weston, eds., Large-scale Kernel Machines, MIT Press, Cambridge, MA, 2007.
5. G. Wahba, "Spline models for observational data," volume 59 of CBMS-NSF Regional Conference Series in Applied Mathematics, SIAM, 1987.
6. J. Mercer, "Functions of positive and negative type, and their connection with the theory of integral equations," Proc. Royal Society of London. Series A, vol.83, no.559, pp.69-70, 1909.
7. N. Aronszajn, "Theory of reproducing kernels," Transactions of the American Mathematical Society, vol.68, pp.337-404, 1950.
8. G. Strang, Linear Algebra and its Applications, Academic Press, 1976.
9. B. Schölkopf, A. Smola, and K.R. Müller, "Nonlinear component analysis as a kernel eigenvalue problem," Neural Comput., vol.10, no.5, pp.1299-1319, 1998.
10. H. Press, B.P. Flannery, S.A. Teukolsky, and W.T. Vetterling, Numerical Recipes in C, 2nd ed., Cambridge University Press, 1992.
11. S. Fine and K. Scheinberg, "Efficient SVM training using low-rank kernel representations," Journal of Machine Learning Research, vol.2, pp.243-264, 2002.
12. F. Bach and M. Jordan, "Kernel independent component analysis," Journal of Machine Learning Research, vol.3, pp.1-48, 2003.
13. G.H. Golub and C.F.V. Loan, Matrix Computations, 3rd ed., Johns Hopkins University Press, Baltimore, MD, 1996.
14. T. Idé and K. Tsuda, "Change-point detection using Krylov subspace learning," Proc. 2007 SIAM International Conference on Data Mining, pp.515-520, 2007.
15. S. Mahadevan, "Fast spectral learning using Lanczos eigenspace projections," Proc. AAAI Conference on Artificial Intelligence, pp.1472-1475, 2008.
16. C.E. Rasmussen and C. Williams, Gaussian Processes for Machine Learning, MIT Press, 2006.
17. C.K.I. Williams and M. Seeger, "Using the Nyström method to speed up kernel machines," Advances in Neural Information Processing Systems 13, pp.682-688, 2001.
18. C.K.I. Williams, C.E. Rasmussen, A. Schwaighofer, and V. Tresp, "Observations on the Nyström method for Gaussian process prediction," Tech. Rep., University of Edinburgh, 2002.
19. N.D. Freitas, Y. Wang, M. Mahdaviani, and D. Lang, "Fast Krylov methods for N-body learning," Advances in Neural Information Processing Systems 18, pp.251-258, 2006.
20. C. Yang, R. Duraiswami, and L. Davis, "Efficient kernel machines using the improved fast Gauss transform," Advances in Neural Information Processing Systems 17, pp.1561-1568, 2005.
21. D. Lee, A.G. Gray, and A.W. Moore, "Dual-tree fast Gauss transforms," Advances in Neural Information Processing Systems 18, pp.1561-1568, 2006.
22. J. Shawe-Taylor and N. Cristianini, Kernel Methods for Pattern Analysis, Cambridge University Press, 2004.
23. R. Rosipal and L.J. Trejo, "Kernel partial least squares regression in reproducing kernel Hilbert space," Journal of Machine Learning Research, vol.2, pp.97-123, 2001.
24. R. Tibshirani, "Regression shrinkage and selection via the lasso," Journal of the Royal Statistical Society, B, vol.58, no.1, pp.267-288, 1996.
25. M.E. Tipping, "Sparse Bayesian learning and the relevance vector machine," Journal of Machine Learning Research, vol.1, pp.211-244, 2001.
26. G.C. Cawley and N.L. Talbot, "Fast exact leave-one-out cross-validation of sparse least-squares support vector machines," Neural Netw., vol.17, no.10, pp.1467-1475, 2004.
27. N. Krämer and M.L. Braun, "Kernelizing PLS, degrees of freedom, and efficient model selection," Proc. International Conference on Machine Learning, pp.441-448, 2007.
28. N. Krämer, M. Sugiyama, and M. Braun, "Lanczos approximations for the speedup of kernel partial least squares regression," Proc. Twelfth International Conference on Artificial Intelligence and Statistics (AISTATS2009), 2009.
29. M. Momma, "Efficient computations via scalable sparse kernel partial least squares and boosted latent features," Proc. ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp.654-659, 2005.
30. J. Arenas-Garcia, K.B. Petersen, and L.K. Hansen, "Sparse kernel orthogonalized PLS for feature extraction in large data sets," Advances in Neural Information Processing Systems 18, pp.33-40, 2006.
31. H. Saigo, N. Krämer, and K. Tsuda, "Partial least squares regression for graph mining," Proc. ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp.578-586, 2008.
32. B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, "Least angle regression," Annals of Statistics, vol.32, no.2, pp.407-499, 2004.
33. M.J. Best, "An algorithm for the solution of the parametric quadratic programming problem," CORR Report 82-124, Faculty of Mathematics, University of Waterloo, 1982.
34. J. Friedman, T. Hastie, H. Höfling, and R. Tibshirani, "Pathwise coordinate optimization," Annals of Applied Statistics, vol.1, no.2, pp.302-332, 2007.
35. N. Meinshausen and P. Bühlmann, "High-dimensional graphs and variable selection with the lasso," Annals of Statistics, vol.34, no.3, pp.1436-1462, 2006.
36. J. Friedman, T. Hastie, and R. Tibshirani, "Sparse inverse covari-ance estimation with the graphical lasso," Biostatistics, vol.9, no.3, pp.432-441, 2008.
37. T. Idé, A.C. Lozano, N. Abe, and Y. Liu, "Proximity-based anomaly detection using sparse structure learning," Proc. 2009 SIAM International Conference on Data Mining, 2009.
38. V. Roth, "Sparse kernel regressors," Proc. International Conference on Artificial Neural Networks, pp.339-346, 2001.
39. V. Roth, "The generalized lasso," IEEE Trans. Neural Netw., vol.15, no.1, pp.16-28, 2004.
40. G. Wang, D.Y. Yeung, and F.H. Lochovsky, "The kernel path in ker-nelized lasso," Proc. Eleventh International Workshop on Artificial Intelligence and Statistics, pp.580-587, 2007.
41. M.N. Gibbs, Bayesian Gaussian Processes for Regression and Classification, Ph.D. Thesis, Department of Physics, University of Cambridge, 1997.
42. J. Quiñonero-Candela and C.E. Rasmussen, "A unifying view of sparse approximate Gaussian process regression," Journal of Machine Learning Research, vol.6, pp.1939-1959, 2005.
43. E. Snelson and Z. Ghahramani, "Sparse Gaussian processes using pseudo-inputs," Advances in Neural Information Processing Systems 18, ed. Y. Weiss, B. Schölkopf, and J. Platt, pp.1257-1264, Cambridge, MA, MIT Press, 2006.
44. M.E. Tipping and A.C. Faul, "Fast marginal likelihood maximisation for sparse Bayesian models," Proc. Ninth International Workshop on Artificial Intelligence and Statistics, pp.3-6, 2003.
45. C.H. Teo, Q. Le, A.J. Smola, and S.V.N. Vishwanathan, "A scalable modular convex solver for regularized risk minimization," Proc. 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp.727-736, ACM, New York, NY, 2007.
46. C.J. Hsieh, K.W. Chang, C.J. Lin, S.S. Keerthi, and S. Sundararajan, "A dual coordinate descent method for large-scale linear SVM," Proc. 25nd International Conference on Machine Learning, pp.408-415, ACM, New York, NY, 2008.
47. T. Joachims, "Training linear SVMs in linear time," Proc. 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp.217-226, ACM, New York, NY, 2006.
48. V. Franc and S. Sonnenburg, "Optimized cutting plane algorithm for support vector machines," Proc. 25nd International Conference on Machine Learning, pp.320-327, ACM, New York, NY, 2008.
49. J.C. Platt, "Fast training of support vector machines using sequential minimal optimization," Advances in Kernel Methods-Support Vector Learning, ed. B. Schölkopf, C. Burges, and A. Smola, pp.185-208, MIT Press, Cambridge, MA, 1999.
50. S.S. Keerthi, S.K. Shevade, C. Bhattacharyya, and K.R.K. Murthy, "Improvements to Platt's SMO algorithm for SVM classier design," Neural Comput., vol.13, pp.637-649, March 2001.
51. I.T. Jolliffe, Principal Component Analysis, Springer-Verlag, New York, 1986.
52. E. Oja, "A simplified neuron model as a principal component analyzer," Journal of Mathematical Biology, vol.15, pp.267-273, 1982.
53. V.N. Vapnik, Statistical Learning Theory, Wiley, New York, 1998.
54. J.B. Tenenbaum, V. de Silva, and J.C. Langford, "A global geometric framework for nonlinear dimensionality reduction," Science, vol.290, no.5500, pp.2319-2323, 2000.
55. S. Roweis and L. Saul, "Nonlinear dimensionality reduction by locally linear embedding," Science, vol.290, no.5500, pp.2323-2326, 2000.
56. M. Belkin and P. Niyogi, "Laplacian eigenmaps for dimensionality reduction and data representation," Neural Comput., vol.15, no.6, pp.1373-1396, 2003.
57. D.L. Donoho and C.E. Grimes, "Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data," Proc. National Academy of Arts and Sciences, pp.5591-5596, 2003.
58. J. Ham, D.D. Lee, S. Mika, and B. Schölkopf, "A kernel view of the dimensionality reduction of manifolds," Proc. Twenty-First International Conference on Machine Learning, ACM Press, New York, NY, 2004.
59. K.Q. Weinberger and L.K. Saul, "Unsupervised learning of image manifolds by semidefinite programming," Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2004.
60. X. He and P. Niyogi, "Locality preserving projections," Advances in Neural Information Processing Systems 16, ed. S. Thrun, L. Saul, and B. Schölkopf, MIT Press, Cambridge, MA, 2004.
61. F.R.K. Chung, Spectral Graph Theory, American Mathematical Society, Providence, R.I., 1997.
62. I.S. Dhillon, Y. Guan, and B. Kulis, "Kernel k-means, spectral clustering and normalized cuts," Proc. Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp.551-556, ACM Press, New York, NY, USA, 2004.
63. J. Shi and J. Malik, "Normalized cuts and image segmentation," IEEE Trans. Pattern Anal. Mach. Intell., vol.22, no.8, pp.888-905, 2000.
64. A.Y. Ng, M.I. Jordan, and Y. Weiss, "On spectral clustering: Analysis and an algorithm," Advances in Neural Information Processing Systems 14, ed. T.G. Dietterich, S. Becker, and Z. Ghahramani, MIT Press, Cambridge, MA, 2002.
65. R.I. Kondor and J. Lafferty, "Diffusion kernels on graphs and other discrete input spaces," Proc. 19th International Conference on Machine Learning, pp.315-322, 2002.
66. S. Oyama and C.D. Manning, "Using feature conjunctions across examples for learning pairwise classifiers," Proc. 15th European Conference on Machine Learning, pp.322-333, 2004.
67. J. Basilico and T. Hofmann, "Unifying collaborative and content-based filtering," Proc. 21st International Conference on Machine Learning, 2004.
68. A. Ben-Hur and W.S. Noble, "Kernel methods for predicting protein-protein interactions," Bioinformatics, vol.21, no.Suppl. 1, pp.i38-i46, 2005.
69. H. Kashima, S. Oyama, Y. Yamanishi, and K. Tsuda, "On pairwise kernels: An efficient alternative and generalization analysis," Proc. 13th Pacific-Asia Conference on Knowledge Discovery and Data Mining, 2009.
70. H. Kashima, T. Kato, Y. Yamanishi, M. Sugiyama, and K. Tsuda, "Link propagation: A fast semi-supervised algorithm for link prediction," Proc. 2009 SIAM International Conference on Data Mining, 2009.
71. D. Haussler, "Convolution kernels on discrete structures," Tech. Rep. UCSC-CRL-99-110, University of California in Santa Cruz, 1999.
72. H. Lodhi, C. Saunders, J. Shawe-Taylor, N. Cristianini, and C. Watkins, "Text classification using string kernels," Journal of Machine Learning Research, vol.2, pp.419-444, 2002.
73. C. Leslie, E. Eskin, and W. Noble, "The spectrum kernel: A string kernel for SVM protein classification," Proc. Pacific Symposium on Biocomputing, ed. R.B. Altman, A.K. Dunker, L. Hunter, K. Lauerdale, and T.E. Klein, pp.566-575, World Scientific, 2002.
74. S. Vishwanathan and A. Smola, "Fast kernels for string and tree matching," Advances in Neural Information Processing Systems 15, ed. S. Becker, S. Thrun, and K. Obermayer, MIT Press, Cambridge, MA, 2003.
75. C. Leslie, E. Eskin, J. Weston, and W. Noble, "Mismatch string kernels for svm protein classification," Advances in Neural Information Processing Systems 15, ed. S. Becker, S. Thrun, and K. Obermayer, MIT Press, Cambridge, MA, 2003.
76. M. Collins and N. Duffy, "Convolution kernels for natural language," Advances in Neural Information Processing Systems 14, MIT Press, Cambridge, MA, 2002.
77. H. Kashima and T. Koyanagi, "Kernels for semi-structured date," Proc. 19th International Conference on Machine Learning, pp.291-298, Morgan Kaufmann, San Francisco, CA, 2002.
78. H. Kuboyama, K. Hirata, K.F. Aoki-Kinoshita, H. Kashima, and H. Yasuda, "A gram distribution kernel applied to glycan classification and motif extraction," Proc. 17th International Conference on Genome Informatics, 2006.
79. H. Kashima, K. Tsuda, and A. Inokuchi, "Marginalized kernels between labeled graphs," Proc. 20th International Conference on Machine Learning, Morgan Kaufmann, San Francisco, CA, 2003.
80. T. Gärtner, P. Flach, and S. Wrobel, "On graph kernels: Hardness results and efficient alternatives," Proc. 16th Annual Conference on Computational Learning Theory, 2003.
81. S.V.N. Vishwanathan, K. Borgwardt, and N. Schraudolph, "Fast computation of graph kernels," Advances in Neural Information Processing Systems 19, 2007.
82. P. Pavlidis, J. Weston, J. Cai, and W.S. Noble, "Learning gene functional classifications from multiple data types," Journal of Computational Biology, vol.9, no.2, pp.401-411, 2002.
83. G.R.G. Lanckriet, N. Cristianini, P. Bartlett, L.E. Ghaoui, and M.I. Jordan, "Learning the kernel matrix with semidefinite programming," Journal of Machine Learning Research, vol.5, pp.27-72, 2004.
84. T. Kato, K. Tsuda, and K. Asai, "Selective integration of multiple biological data for supervised network inference," Bioinformatics, vol.21, pp.2488-2495, 2005.
85. F.R. Bach, G.R.G. Lanckriet, and M.I. Jordan, "Multiple kernel learning, conic duality, and the SMO algorithm," Proc. 21st International Conference on Machine Learning, 2004.
86. S. Sonnenburg, G. Rätsch, C. Schäfer, and B. Schölkopf, "Large scale multiple kernel learning," Journal of Machine Learning Research, vol.7, pp.1531-1565, July 2006.
87. A. Rakotomamonjy, F. Bach, S. Canu, and Y. Grandvalet, "Sim-plemkl," Journal of Machine Learning Research, vol.9, pp.2491-2521, 2008.
88. A. Zien and C.S. Ong, "Multiclass multiple kernel learning," Proc. 24th International Conference on Machine Learning, pp.1191-1198, ACM, New York, NY, 2007.
89. T. Kato, H. Kashima, and M. Sugiyama, "Robust label propagation on multiple networks," IEEE Trans. Neural Netw. vol.20, no.1, pp.35-44, 2009.
90. T. Kato, Y. Murata, K. Miura, K. Asai, P.B. Horton, K. Tsuda, and W. Fujibuchi, "Network-based de-noising improves prediction from microarray data," BMC Bioinformatics, vol.7 (Suppl 1), March 2006.
91. K. Fukumizu, F.R. Bach, and M.I. Jordan, "Dimensionality reduction for supervised learning with reproducing kernel Hilbert spaces," Journal of Machine Learning Research, vol.5, no.Jan, pp.73-99, 2004.
92. L. Song, A. Smola, A. Gretton, K.M. Borgwardt, and J. Bedo, "Supervised feature selection via dependence estimation," Proc. 24th International Conference on Machine learning, pp.823-830, 2007.
93. J. Huang, A. Smola, A. Gretton, K.M. Borgwardt, and B. Schölkopf, "Correcting sample selection bias by unlabeled data," Advances in Neural Information Processing Systems 19, pp.601-608, 2007.
94. Y. Freund and R.E. Schapire, "Experiments with a new boosting algorithm," International Conference on Machine Learning, 1996.
95. Y. Freund and R.E. Schapire, "A decision-theoretic generalization of on-line learning and an application to boosting," J. Comput. Syst. Sci., vol.55, pp.119-139, 1997.
96. R. Schapire, Y. Freund, P. Bartlett, and W. Lee, "Boosting the margin: A new explanation for the effectiveness of voting methods," Annals of Statistics, vol.26, pp.1651-1686, 1998.
97. A. Takeda and M. Sugiyama, "Nu-support vector machine as conditional value-at-risk minimization," Proc. 25th Annual International Conference on Machine Learning, ed. A. McCallum and S. Roweis, pp.1056-1063, 2008.
98. L. Wang, M. Sugiyama, C. Yang, Z.H. Zhou, and J. Feng, "On the margin explanation of boosting algorithms," Proc. 21st International Conference on Learning Theory, ed. R. Servedio and T. Zhang, pp.479-490, 2008.