Least-squares conditional density estimation

IEICE Transactions on Information and Systems

Volumn E93-D, Issue 3, 2010, Pages 583-594

  Sugiyama, Masashi ^a,b   Takeuchi, Ichiro ^c   Suzuki, Taiji ^d   Kanamori, Takafumi ^e   Hachiya, Hirotaka ^a   Okanohara, Daisuke ^d  

^a TOKYO INSTITUTE OF TECHNOLOGY   (Japan)

^b JAPAN SCIENCE AND TECHNOLOGY AGENCY   (Japan)

^c NAGOYA INSTITUTE OF TECHNOLOGY   (Japan)

^d UNIVERSITY OF TOKYO   (Japan)

^e NAGOYA UNIVERSITY   (Japan)

Author keywords

Conditional density estimation; Direct density ratio estimation; Heteroscedastic noise; Multimodality; Transition estimation

Indexed keywords

INFORMATION SCIENCE; SOFTWARE ENGINEERING;

CONDITIONAL DENSITY; CONDITIONAL DISTRIBUTION; CONTINUOUS VARIABLES; DENSITY ESTIMATION; DENSITY-RATIO ESTIMATIONS; HETEROSCEDASTIC NOISE; INPUT-OUTPUT RELATIONS; MULTI-MODALITY;

REGRESSION ANALYSIS;

EID: 77950208664     PISSN: 09168532     EISSN: 17451361     Source Type: Journal    
DOI: 10.1587/transinf.E93.D.583     Document Type: Article

References (49)

1. S.M. Ali and S.D. Silvey, "A general class of coefficients of divergence of one distribution from another," J. Royal Statistical Society, Series B, vol.28, no.1, pp.131-142, 1966.
2. A. Basu, I.R. Harris, N.L. Hjort, and M.C. Jones, "Robust and efficient estimation by minimising a density power divergence," Biometrika, vol.85, no.3, pp.540-559, 1998.
3. S. Bickel, M. Brückner, and T. Scheffer, "Discriminative learning for differing training and test distributions," Proc. 24th International Conference on Machine Learning, pp.81-88, 2007.
4. C.M. Bishop, Pattern Recognition and Machine Learning, Springer, New York, NY, USA, 2006.
5. O. Chapelle, B. Schölkopf, and A. Zien, eds., Semi-Supervised Learning, MIT Press, Cambridge, 2006.
6. K.F. Cheng and C.K. Chu, "Semiparametric density estimation under a two-sample density ratio model," Bernoulli, vol.10, no.4, pp.583-604, 2004.
7. I. Csiszár, "Information-type measures of difference of probability distributions and indirect observation," Studia Scientiarum Mathematicarum Hungarica, vol.2, pp.229-318, 1967.
8. A.P. Dempster, N.M. Laird, and D.B. Rubin, "Maximum likelihood from incomplete data via the EM algorithm," J. Royal Statistical Society, Series B, vol.39, no.1, pp.1-38, 1977.
9. D. Edmunds and H. Triebel, eds., Function Spaces, Entropy Numbers, Differential Operators, Cambridge Univ Press, 1996.
10. J. Fan, Q. Yao, and H. Tong, "Estimation of conditional densities and sensitivity measures in nonlinear dynamical systems," Biometrika, vol.83, no.1, pp.189-206, 1996.
11. S. Geman and C.R. Hwang, "Nonparametric maximum likelihood estimation by the method of sieves," The Annals of Statistics, vol.10, no.2, pp.401-414, 1982.
12. W. Härdle, M. Müller, S. Sperlich, and A. Werwatz, Nonparametric and Semiparametric Models, Springer, Berlin, 2004.
13. T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Springer, New York, 2001.
14. J. Huang, A. Smola, A. Gretton, K.M. Borgwardt, and B. Schölkopf, "Correcting sample selection bias by unlabeled data," in Advances in Neural Information Processing Systems 19, ed. B. Schölkopf, J. Platt, and T. Hoffman, pp.601-608, MIT Press, Cambridge, MA, 2007.
15. R.J. Hyndman, D.M. Bashtannyk, and G.K. Grunwald, "Estimating and visualizing conditional densities," J. Computational and Graphical Statistics, vol.5, no.4, pp.315-336, 1996.
16. E.T. Jaynes, "Information theory and statistical mechanics," Phys. Rev., vol.106, no.4, pp.620-630, 1957.
17. T. Kanamori, S. Hido, and M. Sugiyama, "Efficient direct density ratio estimation for non-stationarity adaptation and outlier detection," Advances in Neural Information Processing Systems 21, ed. D. Koller, D. Schuurmans, Y. Bengio, and L. Botton, pp.809-816, MIT Press, Cambridge, MA, 2009.
18. T. Kanamori, S. Hido, and M. Sugiyama, "A least-squares approach to direct importance estimation," J. Machine Learning Research, vol.10, pp.1391-1445, July 2009.
19. T. Kanamori, T. Suzuki, and M. Sugiyama, "Condition number analysis of kernel-based density ratio estimation," Tech. Rep. TR09-0006, Department of Computer Science, Tokyo Institute of Technology, Feb. 2009.
20. G.S. Kimeldorf and G. Wahba, "Some results on Tchebycheffian spline functions," J. Mathematical Analysis and Applications, vol.33, no.1, pp.82-95, 1971.
21. A.N. Kolmogorov and V.M. Tikhomirov, "ε-entropy and ε-capacity of sets in function spaces," American Mathematical Society Translations, vol.17, no.2, pp.277-364, 1961.
22. S. Kullback and R.A. Leibler, "On information and sufficiency," Annals of Mathematical Statistics, vol.22, pp.79-86, 1951.
23. J. Lafferty, A. McCallum, and F. Pereira, "Conditional random fields: Probabilistic models for segmenting and labeling sequence data," Proc. 18th International Conference on Machine Learning, pp.282-289, 2001.
24. M.G. Lagoudakis and R. Parr, "Least-squares policy iteration," J. Machine Learning Research, vol.4, pp.1107-1149, 2003.
25. Y. Li, Y. Liu, and J. Zhu, "Quantile regression in reproducing kernel Hilbert spaces," J. American Statistical Association, vol.102, no.477, pp.255-268, 2007.
26. A. Luntz and V. Brailovsky, "On estimation of characters obtained in statistical procedure of recognition," Technicheskaya Kibernetica, vol.3, Russia, 1969.
27. W.K. Newey, "Convergence rates and asymptotic normality for series estimators," J. Econometrics, vol.70, no.1, pp.147-168, 1997.
28. X. Nguyen, M. Wainwright, and M. Jordan, "Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization," in Advances in Neural Information Processing Systems 20, ed. J.C. Platt, D. Koller, Y. Singer, and S. Roweis, pp.1089-1096, MIT Press, Cambridge, MA, 2008.
29. X. Nguyen, M.J. Wainwright, and M.I. Jordan, "Nonparametric estimation of the likelihood ratio and divergence functionals," Proc. IEEE International Symposium on Information Theory, pp.2016-2020, Nice, France, 2007.
30. J. Qin, "Inferences for case-control and semiparametric two-sample density ratio models," Biometrika, vol.85, no.3, pp.619-639, 1998.
31. R Development Core Team, The R Manuals, 2008. http://www.r-project.org
32. A. Sankar, D.A. Spielman, and S.H. Teng, "Smoothed analysis of the condition numbers and growth factors of matrices," SIAM J. Matrix Anal. Appl., vol.28, no.2, pp.446-476, 2006.
33. B. Schölkopf and A.J. Smola, Learning with Kernels, MIT Press, Cambridge, MA, 2002.
34. D.W. Scott, "Remarks on fitting and interpreting mixture models," Computing Science and Statistics, vol.31, pp.104-109, 1999.
35. D.A. Spielman and S.H. Teng, "Smoothed analysis of algorithms: Why the simplex algorithm usually takes polynomial time," J. ACM, vol.51, no.3, pp.385-463, 2004.
36. M. Sugiyama, M. Kawanabe, and P.L. Chui, "Dimensionality reduction for density ratio estimation in high-dimensional spaces," Neural Netw., vol.23, no.1, pp.44-59, 2010.
37. M. Sugiyama, S. Nakajima, H. Kashima, P. von Bünau, and M. Kawanabe, "Direct importance estimation with model selection and its application to covariate shift adaptation," Advances in Neural Information Processing Systems 20, ed. J.C. Platt, D. Koller, Y. Singer, and S. Roweis, pp.1433-1440, MIT Press, Cambridge, MA, 2008.
38. M. Sugiyama, T. Suzuki, S. Nakajima, H. Kashima, P. von Bünau, and M. Kawanabe, "Direct importance estimation for covariate shift adaptation," Annals of the Institute of Statistical Mathematics, vol.60, no.4, pp.699-746, 2008.
39. R.S. Sutton and G.A. Barto, Reinforcement Learning: An Introduction, MIT Press, Cambridge, MA, 1998.
40. I. Takeuchi, Q.V. Le, T.D. Sears, and A.J. Smola, "Nonparametric quantile estimation," J.Machine Learning Research, vol.7, pp.1231-1264, 2006.
41. I. Takeuchi, K. Nomura, and T. Kanamori, "Nonparametric conditional density estimation using piecewise-linear solution path of kernel quantile regression," Neural Comput., vol.21, no.2, pp.533-559, 2009.
42. V. Tresp, "Mixtures of gaussian processes," Advances in Neural Information Processing Systems 13, ed. T.K. Leen, T.G. Dietterich, and V. Tresp, pp.654-660, MIT Press, 2001.
43. Y. Tsuboi, H. Kashima, S. Hido, S. Bickel, and M. Sugiyama, "Direct density ratio estimation for large-scale covariate shift adaptation," Proc. Eighth SIAM International Conference on Data Mining (SDM2008), ed. M.J. Zaki, K.Wang, C. Apte, and H. Park, pp.443-454, Atlanta, Georgia, USA, April 2008.
44. Y. Tsuboi, H. Kashima, S. Hido, S. Bickel, and M. Sugiyama, "Direct density ratio estimation for large-scale covariate shift adaptation," J. Information Processing, vol.17, pp.138-155, 2009.
45. A.W. van der Vaart and J.A.Wellner,Weak Convergence and Empirical Processes with Applications to Statistics, Springer, New York, NY, USA, 1996.
46. S. Weisberg, Applied Linear Regression, John Wiley, New York, NY, USA, 1985.
47. R.C.L. Wolff, Q. Yao, and P. Hall, "Methods for estimating a conditional distribution function," J. American Statistical Association, vol.94, no.445, pp.154-163, 1999.
48. M. Yamada and M. Sugiyama, "Direct importance estimation with Gaussian mixture models," IEICE Trans. Inf. & Syst., vol.E92-D, no.10, pp.2159-2162, Oct. 2009.
49. D.X. Zhou, "The covering number in learning theory," J. Complexity Archive, vol.18, no.3, pp.739-767, 2002.