Computational complexity of kernel-based density-ratio estimation: A condition number analysis

Machine Learning

Volumn 90, Issue 3, 2013, Pages 431-460

  Kanamori, Takafumi ^a   Suzuki, Taiji ^b   Sugiyama, Masashi ^c  

^a NAGOYA UNIVERSITY   (Japan)

^b UNIVERSITY OF TOKYO   (Japan)

^c TOKYO INSTITUTE OF TECHNOLOGY   (Japan)

Author keywords

Condition number; Density ratio; Kernel; Smoothed analysis

Indexed keywords

COMPUTATIONAL PROPERTIES; CONDITION NUMBERS; CONVERGENCE RATES; DENSITY-RATIO; DENSITY-RATIO ESTIMATIONS; HESSIAN MATRICES; KERNEL; LEAST SQUARE; LEAST SQUARES METHODS; LEAST-SQUARES ESTIMATOR; LOSS FUNCTIONS; M-ESTIMATORS; NUMERICAL EXPERIMENTS; SMOOTHED ANALYSIS;

ESTIMATION;

NUMBER THEORY;

EID: 84874701688     PISSN: 08856125     EISSN: 15730565     Source Type: Journal    
DOI: 10.1007/s10994-012-5323-6     Document Type: Article

References (92)

1. Ali, S. M., & Silvey, S. D. (1966). A general class of coefficients of divergence of one distribution from another. Journal of the Royal Statistical Society. Series B. Methodological, 28, 131-142.
2. Aronszajn, N. (1950). Theory of reproducing kernels. Transactions of the American Mathematical Society, 68, 337-404.
3. Axelsson, O., & Neytcheva, M. (2002). Robust preconditioners for saddle point problems. In Numerical methods and application (pp. 158-166).
4. Badia, S., Nobile, F., & Vergara, C. (2009). Robin-robin preconditioned Krylov methods for fluid-structure interaction problems. Computer Methods in Applied Mechanics and Engineering, 198, 2768-2784.
5. Becchetti, L., Leonardi, S., Marchetti-Spaccamela, A., Schafer, G., & Vredeveld, T. (2006). Average-case and smoothed competitive analysis of the multilevel feedback algorithm. Open Access publications from Maastricht University (urn:nbn:nl:ui:27-17093). Maastricht University.
6. Beltran, C., & Pardo, L. M. (2006). Estimates on the distribution of the condition number of singular matrices. Foundations of Computational Mathematics, 7, 87-134.
7. Benzi, M., Haber, E., & Taralli, L. (2011). A preconditioning technique for a class of PDE-constrained optimization problems. Advances in Computational Mathematics, 35, 149-173.
8. Bickel, S., Bogojeska, J., Lengauer, T., & Scheffer, T. (2008). Multi-task learning for HIV therapy screening. In Proceedings of 25th annual international conference on machine learning (ICML2008) (pp. 56-63). Helsinki: Omnipress.
9. Bickel, S., Brückner, M., & Scheffer, T. (2009). Discriminative learning under covariate shift. Journal of Machine Learning Research, 10, 2137-2155.
10. Blum, A., & Dunagan, J. (2002). Smoothed analysis of the perceptron algorithm for linear programming. In Proc. of the 13th annual ACM-SIAM symp. on discrete algorithms (pp. 905-914).
11. Blum, L., & Shub, M. (1986). Evaluating rational functions: infinite precision is finite cost and tractable on average. SIAM Journal on Computing, 15, 384-398.
12. Bürgisser, P., & Cucker (2010). Smoothed analysis of Moore-Penrose inversion. SIAM Journal on Matrix Analysis and Applications, 31, 2769-2783.
13. Bürgisser, P., Cucker, F., & de Naurois, P. (2006a). The complexity of semilinear problems in succinct representation. Computational Complexity, 15, 197-235.
14. Bürgisser, P., Cucker, F., & Lotz, M. (2006b). General formulas for the smoothed analysis of condition numbers. Comptes Rendus de L'Académie des Sciences. Series 1, Mathematics, 343, 145-150.
15. Bürgisser, P., Cucker, F., & Lotz, M. (2006c). Smoothed analysis of complex conic condition numbers. Journal de Mathématiques Pures et Appliquées, 86, 293-309.
16. Bürgisser, P., Cucker, F., & Lotz, M. (2010). Coverage processes on spheres and condition numbers for linear programming. Annals of Probability, 38, 570-604.
17. Caputo, B., Sim, K., Furesjo, F., & Smola, A. (2002). Appearance-based object recognition using SVMs: which kernel should I use? In Proceedings of NIPS workshop on statistical methods for computational experiments in visual processing and computer vision.
18. Chapelle, O. (2007). Training a support vector machine in the primal. Neural Computation, 19, 1155-1178.
19. Cheung, D., & Cucker, F. (2002). Probabilistic analysis of condition numbers for linear programming. Journal of Optimization Theory and Applications, 114, 55-67.
20. Csiszár, I. (1967). Information-type measures of difference of probability distributions and indirect observation. Studia Scientiarum Mathematicarum Hungarica, 2, 229-318.
21. Cucker, F., & Wschebor, M. (2002). On the expected condition number of linear programming problems. Numerische Mathematik, 94, 94-419.
22. Demmel, J. (1988). The probability that a numerical analysis problem is difficult. Mathematics of Computation, 50, 449-480.
23. Demmel, J. W. (1997). Applied numerical linear algebra. Philadelphia: SIAM.
24. Eckart, C., & Young, G. (1936). The approximation of one matrix by another of lower rank. Psychometrika, 1, 211-218.
25. Edelman, A. (1988). Eigenvalues and condition numbers of random matrices. SIAM Journal on Matrix Analysis and Applications, 9, 543-560.
26. Edelman, A. (1992). On the distribution of a scaled condition number. Mathematics of Computation, 58, 185-190.
27. Edelman, A., & Sutton, B. D. (2005). Tails of condition number distributions. SIAM Journal on Matrix Analysis and Applications, 27, 547-560.
28. Gretton, A., Smola, A., Huang, J., Schmittfull, M., Borgwardt, K., & Schölkopf, B. (2009). Covariate shift by kernel mean matching. In J. Quiñonero-Candela, M. Sugiyama, A. Schwaighofer, & N. Lawrence (Eds.), Dataset shift in machine learning (pp. 131-160). Cambridge: MIT Press.
29. Hager, W. W., & Zhang, H. (2006). A survey of the nonlinear conjugate gradient methods. Pacific Journal of Optimization, 2, 35-58.
30. Hido, S., Tsuboi, Y., Kashima, H., Sugiyama, M., & Kanamori, T. (2008). Inlier-based outlier detection via direct density ratio estimation. In Proceedings of IEEE international conference on data mining (ICDM2008), Pisa, Italy (pp. 223-232).
31. Hido, S., Tsuboi, Y., Kashima, H., Sugiyama, M., & Kanamori, T. (2011). Statistical outlier detection using direct density ratio estimation. Knowledge and Information Systems, 26, 309-336.
32. Horn, R., & Johnson, C. (1985). Matrix analysis. Cambridge: Cambridge University Press.
33. Kanamori, T., Hido, S., & Sugiyama, M. (2009). A least-squares approach to direct importance estimation. Journal of Machine Learning Research, 10, 1391-1445.
34. Kanamori, T., Suzuki, T., & Sugiyama, M. (2012). Statistical analysis of kernel-based least-squares density-ratio estimation. Machine Learning, 86, 335-367.
35. Karatzoglou, A., Smola, A., Hornik, K., & Zeileis, A. (2004). Kernlab - an S4 package for kernel methods in R. Journal of Statistical Software, 11, 1-20.
36. Kawahara, Y., & Sugiyama, M. (2011). Sequential change-point detection based on direct density-ratio estimation. Statistical Analysis and Data Mining, 5, 114-127.
37. Kimeldorf, G. S., & Wahba, G. (1971). Some results on Tchebycheffian spline functions. Journal of Mathematical Analysis and Applications, 33, 82-95.
38. Kimura, M., & Sugiyama, M. (2011). Dependence-maximization clustering with least-squares mutual information. Journal of Advanced Computational Intelligence and Intelligent Informatics, 15, 800-805.
39. Kostlan, E. (1988). Complexity theory of numerical linear algebra. Journal of Computational and Applied Mathematics, 22, 219-230.
40. Luenberger, D., & Ye, Y. (2008). Linear and nonlinear programming. Berlin: Springer.
41. Manthey, B., & Röglin, H. (2009). Worst-case and smoothed analysis of k-means clustering with Bregman divergences. In ISAAC (pp. 1024-1033).
42. Mika, S., Schölkopf, B., Smola, A., Müller, K.-R., Scholz, M., & Rätsch, G. (1999). Kernel PCA and de-noising in feature spaces. In Proceedings of the 1998 conference on advances in neural information processing systems II (pp. 536-542). Cambridge: MIT Press.
43. Moré, J. J., & Sorensen, D. C. (1984). Newton's method. In G. H. Golub (Ed.), Studies in numerical analysis. pub-MATH-ASSOC-AMER.
44. Nakahara, M. (2003). Geometry, topology and physics (2nd ed.). London: Taylor & Francis.
45. Nguyen, X., Wainwright, M. J., & Jordan, M. I. (2010). Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56, 5847-5861.
46. Nocedal, J., & Wright, S. J. (1999). Numerical optimization. Berlin: Springer.
47. Quiñonero-Candela, J., Sugiyama, M., Schwaighofer, A., & Lawrence, N. (Eds.) (2009). Dataset shift in machine learning. Cambridge: MIT Press.
48. R Development Core Team (2009). R: a language and environment for statistical computing. Vienna: R Foundation for Statistical Computing. ISBN 3-900051-07-0.
49. Ratliff, N., & Bagnell, J. D. (2007). Kernel conjugate gradient for fast kernel machines. In International joint conference on artificial intelligence.
50. Rätsch, G., Onoda, T., & Müller, K.-R. (2001). Soft margins for adaboost. Machine Learning, 42, 287-320.
51. Renegar, J. (1987). On the efficiency of newton's method in approximating all zeros of a system of complex polynomials. Mathematics of Operations Research, 12, 121-148.
52. Renegar, J. (1995). Incorporating condition measures into the complexity theory of linear programming. SIAM Journal on Optimization, 5.
53. Rockafellar, R. T. (1970). Convex analysis. Princeton: Princeton University Press.
54. Röglin, H., & Vöcking, B. (2007). Smoothed analysis of integer programming. Mathematical Programming, 110, 21-56.
55. Sankar, A., Spielman, D. A., & Teng, S.-H. (2006). Smoothed analysis of the condition numbers and growth factors of matrices. SIAM Journal on Matrix Analysis and Applications, 28, 446-476.
56. Schmidt, M., Le Roux, N., & Bach, F. (2011). Convergence rates of inexact proximal-gradient methods for convex optimization. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, & K. Weinberger (Eds.), Advances in neural information processing systems (Vol. 24, pp. 1458-1466).
57. Schölkopf, B., & Smola, A. J. (2002). Learning with kernels. Cambridge: MIT Press.
58. Shimodaira, H. (2000). Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference, 90, 227-244.
59. Shub, M. (1993). Some remarks on Bézout's theorem and complexity theory. In From topology to computation: proceedings of the smalefest (pp. 443-455). Berlin: Springer.
60. Shub, M., & Smale, S. (1994). Complexity of Bézout's theorem. V: polynomial time. Theoretical Computer Science, 133.
61. Shub, M., & Smale, S. (1996). Complexity of Bézout's theorem. IV: probability of success; extensions. SIAM Journal on Numerical Analysis, 33, 128-148.
62. Simm, J., Sugiyama, M., & Kato, T. (2011). Computationally efficient multi-task learning with least-squares probabilistic classifiers. IPSJ Transactions on Computer Vision and Applications, 3, 1-8.
63. Smale, S. (1981). The fundamental theorem of algebra and complexity theory. Bulletin of the American Mathematical Society, 4, 1-36.
64. Smola, A., Song, L., & Teo, C. H. (2009). Relative novelty detection. In Twelfth international conference on artificial intelligence and statistics (pp. 536-543).
65. Spielman, D. A., & Teng, S.-H. (2004). Smoothed analysis of algorithms: why the simplex algorithm usually takes polynomial time. Journal of the ACM, 51, 385-463.
66. Spivak, M. (1979). A comprehensive introduction to differential geometry (Vol. I) (2nd ed.). Berkley: Publish or Perish.
67. Steinwart, I. (2001). On the influence of the kernel on the consistency of support vector machines. Journal of Machine Learning Research, 2, 67-93.
68. Sugiyama, M. (2010). Superfast-trainable multi-class probabilistic classifier by least-squares posterior fitting. IEICE Transactions on Information and Systems, E93-D, 2690-2701.
69. Sugiyama, M., & Kawanabe, M. (2012). Machine learning in non-stationary environments: Introduction to covariate shift adaptation. Cambridge: MIT Press.
70. Sugiyama, M., & Müller, K.-R. (2005). Input-dependent estimation of generalization error under covariate shift. Statistics & Decisions, 23, 249-279.
71. Sugiyama, M., & Suzuki, T. (2011). Least-squares independence test. IEICE Transactions on Information and Systems, E94-D, 1333-1336.
72. Sugiyama, M., Krauledat, M., & Müller, K.-R. (2007). Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 8, 985-1005.
73. Sugiyama, M., Nakajima, S., Kashima, H., von Bünau, P., & Kawanabe, M. (2008a). Direct importance estimation with model selection and its application to covariate shift adaptation. In Advances in neural information processing systems (Vol. 20, pp. 1433-1440). Cambridge: MIT Press.
74. Sugiyama, M., Suzuki, T., Nakajima, S., Kashima, H., von Bünau, P., Kawanabe, M., & Nakajima, S. (2008b). Direct importance estimation for covariate shift adaptation. Annals of the Institute of Statistical Mathematics, 60, 699-746.
75. Sugiyama, M., Kanamori, T., Suzuki, T., Hido, S., Sese, J., Takeuchi, I., & Wang, L. (2009). A density-ratio framework for statistical data processing. IPSJ Transactions on Computer Vision and Applications, 1, 183-208.
76. Sugiyama, M., Takeuchi, I., Kanamori, T., Suzuki, T., Hachiya, H., & Okanohara, D. (2010a). Conditional density estimation via least-squares density ratio estimation. In Proceedings of the thirteenth international conference on artificial intelligence and statistics (AISTATS2010), Sardinia, Italy (pp. 781-788).
77. Sugiyama, M., Takeuchi, I., Suzuki, T., Kanamori, T., Hachiya, H., & Okanohara, D. (2010b). Least-squares conditional density estimation. IEICE Transactions on Information and Systems, E93-D, 583-594.
78. Sugiyama, M., Suzuki, T., Itoh, Y., Kanamori, T., & Kimura, M. (2011). Least-squares two-sample test. Neural Networks, 24, 735-751.
79. Sugiyama, M., Suzuki, T., & Kanamori, T. (2012a). Density ratio matching under the Bregman divergence: A unified framework of density ratio estimation. Annals of the Institute of Statistical Mathematics, 64, 1009-1044.
80. Sugiyama, M., Suzuki, T., & Kanamori, T. (2012b). Density ratio estimation in machine learning. Cambridge: Cambridge University Press.
81. Suzuki, T., & Sugiyama, M. (2011). Least-squares independent component analysis. Neural Computation, 23, 284-301.
82. Suzuki, T., Sugiyama, M., Sese, J., & Kanamori, T. (2008). Approximating mutual information by maximum likelihood density ratio estimation. In JMLR workshop and conference proceedings (pp. 5-20).
83. Suzuki, T., Sugiyama, M., Kanamori, T., & Sese, J. (2009a). Mutual information estimation reveals global associations between stimuli and biological processes. BMC Bioinformatics, 10, S52.
84. Suzuki, T., Sugiyama, M., & Tanaka, T. (2009b). Mutual information approximation via maximum likelihood estimation of density ratio. In Proceedings of 2009 IEEE international symposium on information theory (ISIT2009), Seoul, Korea (pp. 463-467).
85. Tao, T., & Vu, V. H. (2007). The condition number of a randomly perturbed matrix. In Proceedings of the thirty-ninth annual ACM symposium on theory of computing (pp. 248-255). New York: ACM.
86. Todd, M. J., Tunçel, L., & Ye, Y. (2001). Characterizations, bounds, and probabilistic analysis of two complexity measures for linear programming problems. Mathematical Programming, 90, 59-69.
87. Turing, A. M. (1948). Rounding-off errors in matrix processes. Quarterly Journal of Mechanics and Applied Mathematics, 1, 287-308.
88. Vershynin, R. (2006). Beyond Hirsch conjecture: walks on random polytopes and smoothed complexity of the simplex method. In FOCS 2006 (47th annual symposium on foundations of computer science (pp. 133-142).
89. von Neumann, J., & Goldstine, H. (1947). Numerical inverting of matrices of high order. Bulletin of the American Mathematical Society, 53, 1021-1099.
90. Yamada, M., & Sugiyama, M. (2010). Dependence minimizing regression with model selection for non-linear causal inference under non-Gaussian noise. In Proceedings of the twenty-fourth AAAI conference on artificial intelligence (AAAI2010) (pp. 643-648). Atlanta: AAAI Press.
91. Yamada, M., & Sugiyama, M. (2011). Cross-domain object matching with model selection. In Proceedings of the fourteenth international conference on artificial intelligence and statistics (AISTATS2011), Fort Lauderdale, Florida, USA (pp. 807-815).
92. Zadrozny, B. (2004). Learning and evaluating classifiers under sample selection bias. In Proceedings of the Twenty-First international conference on machine learning. New York: ACM.