Dimensionality reduction for density ratio estimation in high-dimensional spaces

Neural Networks

Volumn 23, Issue 1, 2010, Pages 44-59

  Sugiyama, Masashi ^a,b   Kawanabe, Motoaki ^b,c   Chui, Pui Ling ^a  

^a TOKYO INSTITUTE OF TECHNOLOGY   (Japan)

^b Mathematisches Forschungsinstitut Oberwolfach   (Germany)

^c FRAUNHOFER FIRST   (Germany)

Author keywords

Density ratio estimation; Dimensionality reduction; Local Fisher discriminant analysis; Unconstrained least squares importance fitting

Indexed keywords

DENSITY RATIO; DENSITY RATIO ESTIMATION; DIMENSIONALITY REDUCTION; FISHER DISCRIMINANT ANALYSIS; LEAST SQUARE;

DATA PROCESSING; DISCRIMINANT ANALYSIS; FISHER INFORMATION MATRIX; MAXIMUM LIKELIHOOD ESTIMATION; PROBABILITY DENSITY FUNCTION;

ESTIMATION;

ACCURACY; AREA UNDER THE CURVE; ARTICLE; DATA MINING; DENSITY FUNCTIONAL THEORY; DISCRIMINANT ANALYSIS; INFORMATION PROCESSING; MACHINE LEARNING; MATHEMATICAL ANALYSIS; MATHEMATICAL COMPUTING; PRINCIPAL COMPONENT ANALYSIS; PRIORITY JOURNAL; PROBABILITY;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; DISCRIMINANT ANALYSIS; GENETIC TESTING; HUMANS; INFORMATION STORAGE AND RETRIEVAL; MODELS, STATISTICAL; NEURAL NETWORKS (COMPUTER); REGRESSION ANALYSIS; SIGNAL PROCESSING, COMPUTER-ASSISTED;

EID: 70649085072     PISSN: 08936080     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neunet.2009.07.007     Document Type: Article

References (74)

1. Albert A. Regression and the Moore-Penrose pseudoinverse (1972), Academic Press, New York and London
2. Anderson N., Hall P., and Titterington D. Two-sample test statistics for measuring discrepancies between two multivariate probability density functions using kernel-based density estimates. Journal of Multivariate Analysis 50 (1994) 41-54
3. Best, M. J. (1982). An algorithm for the solution of the parametric quadratic programming problem. Technical report 82-24. Faculty of Mathematics, University of Waterloo
4. Biau G., and Gyorfi L. On the asymptotic properties of a nonparametric l1-test statistic of homogeneity. IEEE Transactions on Information Theory 51 (2005) 3965-3973
5. Bickel P. A distribution free version of the Smirnov two sample test in the p-variate case. The Annals of Mathematical Statistics 40 (1969) 1-23
6. Bickel S., Bogojeska J., Lengauer T., and Scheffer T. Multi-task learning for HIV therapy screening. Proceedings of 25th annual international conference on machine learning (2008), Omnipress, Helsinki, Finland 56-63
7. Bickel, S., Brückner, M., & Scheffer, T. (2007). Discriminative learning for differing training and test distributions. In Proceedings of the 24th international conference on machine learning (pp. 81-88)
8. Bishop C.M. Pattern recognition and machine learning (2006), Springer, New York
9. Borgwardt K.M., Gretton A., Rasch M.J., Kriegel H.-P., Schölkopf B., and Smola A.J. Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics 22 (2006) e49-e57
10. Bradley A.P. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition 30 (1997) 1145-1159
11. Breunig, M. M., Kriegel, H.-P., Ng, R. T., & Sander, J. (2000). LOF: Identifying density-based local outliers. In Proceedings of the ACM SIGMOD international conference on management of data
12. Brodsky B., and Darkhovsky B. Nonparametric methods in change-point problems (1993), Kluwer Academic Publishers, Dordrecht
13. Cheng K.F., and Chu C.K. Semiparametric density estimation under a two-sample density ratio model. Bernoulli 10 (2004) 583-604
14. Cover T.M., and Thomas J.A. Elements of information theory (1991), John Wiley & Sons, Inc, New York, NY, USA
15. Duda R.O., Hart P.E., and Stor D.G. Pattern classification (2001), Wiley, New York
16. Efron B., Hastie T., Tibshirani R., and Johnstone I. Least angle regression. The Annals of Statistics 32 (2004) 407-499
17. Fisher R.A. The use of multiple measurements in taxonomic problems. Annals of Eugenics 7 (1936) 179-188
18. Fishman G.S. Monte Carlo: Concepts, algorithms, and applications (1996), Springer-Verlag, Berlin
19. Friedman J., and Rafsky L. Multivariate generalizations of the Wald-Wolfowitz and Smirnov two-sample tests. The Annals of Statistics 7 (1979) 697-717
20. Fukumizu K., Bach F.R., and Jordan M.I. Dimensionality reduction for supervised learning with reproducing kernel Hilbert spaces. Journal of Machine Learning Research 5 (2004) 73-99
21. Fukunaga K. Introduction to statistical pattern recognition. second ed. (1990), Academic Press, Inc, Boston
22. Globerson A., and Roweis S. Metric learning by collapsing classes. Advances in neural information processing systems Vol. 18 (2006), MIT Press, Cambridge, MA 451-458
23. Goldberger J., Roweis S., Hinton G., and Salakhutdinov R. Neighbourhood components analysis. Advances in neural information processing systems Vol. 17 (2005), MIT Press, Cambridge, MA 513-520
24. Golub G.H., and Loan C.F.V. Matrix computations (1996), Johns Hopkins University Press, Baltimore, MD
25. Gretton A., Borgwardt K.M., Rasch M., Schölkopf B., and Smola A.J. A kernel method for the two-sample-problem. In: Schölkopf B., Platt J., and Hoffman T. (Eds). Advances in neural information processing systems Vol. 19 (2007), MIT Press, Cambridge, MA 513-520
26. Hachiya, H., Akiyama, T., Sugiyama, M., & Peters, J. (2009). Adaptive importance sampling for value function approximation in off-policy reinforcement learning. Neural Networks, in press (doi:10.1016/j.neunet.2009.01.002)
27. Hall P., and Tajvidi N. Permutation tests for equality of distributions in highdimensional settings. Biometrika 89 (2002) 359-374
28. Härdle W., Müller M., Sperlich S., and Werwatz A. Nonparametric and semiparametric models (2004), Springer, Berlin
29. Hastie T., Rosset S., Tibshirani R., and Zhu J. The entire regularization path for the support vector machine. Journal of Machine Learning Research 5 (2004) 1391-1415
30. Hastie T., and Tibshirani R. Discriminant adaptive nearest neighbor classification. IEEE Transactions on Pattern Analysis and Machine Intelligence 18 (1996) 607-615
31. Hastie T., and Tibshirani R. Discriminant analysis by Gaussian mixtures. Journal of the Royal Statistical Society, Series B 58 (1996) 155-176
32. Henkel R.E. Tests of significance (1979), SAGE Publication, Beverly Hills
33. Hido, S., Tsuboi, Y., Kashima, H., Sugiyama, M., & Kanamori, T. (2008). Inlier-based outlier detection via direct density ratio estimation. In Proceedings of IEEE international conference on data mining (pp. 223-232)
34. Hotelling H. A generalized t test and measure of multivariate dispersion. Proceedings of the 2nd Berkeley symposium on mathematical statistics and probability (1951), University of California Press, Berkeley, CA, USA 23-41
35. Huang J., Smola A., Gretton A., Borgwardt K.M., and Schölkopf B. Correcting sample selection bias by unlabeled data. In: Schölkopf B., Platt J., and Hoffman T. (Eds). Advances in neural information processing systems 19 (2007), MIT Press, Cambridge, MA 601-608
36. Hulle M.M.V. Edgeworth approximation of multivariate differential entropy. Neural Computation 17 (2005) 1903-1910
37. Hyvärinen A., Karhunen J., and Oja E. Independent component analysis (2001), Wiley, New York
38. Jolliffe I.T. Principal component analysis (1986), Springer-Verlag, New York
39. Kanamori, T., Hido, S., & Sugiyama, M. (2008). A least-squares approach to direct importance estimation. Technical report TR08-0003. Department of Computer Science, Tokyo Institute of Technology
40. Kanamori T., Hido S., and Sugiyama M. Efficient direct density ratio estimation for non-stationarity adaptation and outlier detection. Advances in neural information processing systems Vol. 21 (2009), MIT Press, Cambridge, MA 809-816
41. Kanamori T., and Shimodaira H. Active learning algorithm using the maximum weighted log-likelihood estimator. Journal of Statistical Planning and Inference 116 (2003) 149-162
42. Kanamori, T., Suzuki, T., & Sugiyama, M. (2009b). Condition number analysis of kernel-based density ratio estimation. Technical report TR09-0006. Department of Computer Science, Tokyo Institute of Technology
43. Kawahara, Y., & Sugiyama, M. (2009). Change-point detection in time-series data by direct density-ratio estimation. In Proceedings of 2009 SIAM international conference on data mining (pp. 389-400)
44. Kawanabe M., Sugiyama M., Blanchard G., and Müller K.-R. A new algorithm of non-Gaussian component analysis with radial kernel functions. Annals of the Institute of Statistical Mathematics 59 (2007) 57-75
45. 1-regularized logistic regression. Journal of Machine Learning Research 8 (2007) 1519-1555
46. Kraskov A., Stögbauer H., and Grassberger P. Estimating mutual information. Physical Review E 69 (2004) 066138
47. Kullback S., and Leibler R.A. On information and sufficiency. Annals of Mathematical Statistics 22 (1951) 79-86
48. Minka, T. P. (2007). A comparison of numerical optimizers for logistic regression. Technical report. Microsoft Research
49. Nguyen X., Wainwright M., and Jordan M. Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization. In: Platt J.C., Koller D., Singer Y., and Roweis S. (Eds). Advances in neural information processing systems Vol. 20 (2008), MIT Press, Cambridge, MA 1089-1096
50. Qin J. Inferences for case-control and semiparametric two-sample density ratio models. Biometrika 85 (1998) 619-639
51. In: Quiñonero-Candela J., Sugiyama M., Schwaighofer A., and Lawrence N. (Eds). Dataset shift in machine learning (2009), MIT Press, Cambridge, MA
52. Rätsch G., Onoda T., and Müller K.-R. Soft margins for adaboost. Machine Learning 42 (2001) 287-320
53. Schölkopf B., Platt J.C., Shawe-Taylor J., Smola A.J., and Williamson R.C. Estimating the support of a high-dimensional distribution. Neural Computation 13 (2001) 1443-1471
54. Shimodaira H. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference 90 (2000) 227-244
55. Steinwart I. On the influence of the kernel on the consistency of support vector machines. Journal of Machine Learning Research 2 (2001) 67-93
56. Storkey A., and Sugiyama M. Mixture regression for covariate shift. Advances in neural information processing systems Vol. 19 (2007), MIT Press, Cambridge, MA 1337-1344
57. Sugiyama M. Active learning in approximately linear regression based on conditional expectation of generalization error. Journal of Machine Learning Research 7 (2006) 141-166
58. Sugiyama M. Dimensionality reduction of multimodal labeled data by local Fisher discriminant analysis. Journal of Machine Learning Research 8 (2007) 1027-1061
59. Sugiyama M., Krauledat M., and Müller K.-R. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research 8 (2007) 985-1005
60. Sugiyama M., and Müller K.-R. Input-dependent estimation of generalization error under covariate shift. Statistics & Decisions 23 (2005) 249-279
61. Sugiyama M., and Nakajima S. Pool-based active learning in approximate linear regression. Machine Learning 75 (2009) 249-274
62. Sugiyama M., Suzuki T., Nakajima S., Kashima H., von Bünau P., and Kawanabe M. Direct importance estimation for covariate shift adaptation. Annals of the Institute of Statistical Mathematics 60 (2008) 699-746
63. Sugiyama, M., Takeuchi, I., Suzuki, T., Kanamori, T., & Hachiya, H. (2009). Least-squares conditional density estimation. Technical report TR09-0004. Department of Computer Science, Tokyo Institute of Technology.
64. Suzuki T., and Sugiyama M. Estimating squared-loss mutual information for independent component analysis. Independent component analysis and signal separation (2009), Springer, Berlin 130-137
65. Suzuki, T., & Sugiyama, M. (2009b). Sufficient dimension reduction via squared-loss mutual information estimation. Technical report TR09-0005. Department of Computer Science, Tokyo Institute of Technology
66. Suzuki T., Sugiyama M., Kanamori T., and Sese J. Mutual information estimation reveals global associations between stimuli and biological processes. BMC Bioinformatics 10 (2009) S52
67. Suzuki, T., Sugiyama, M., Sese, J., & Kanamori, T. (2008). Approximating mutual information by maximum likelihood density ratio estimation. In JMLR workshop and conference proceedings (pp. 5-20)
68. Takeuchi I., Nomura K., and Kanamori T. Nonparametric conditional density estimation using piecewise-linear solution path of kernel quantile regression. Neural Computation 21 (2009) 533-559
69. Tsuboi Y., Kashima H., Hido S., Bickel S., and Sugiyama M. Direct density ratio estimation for large-scale covariate shift adaptation. Journal of Information Processing 17 (2009) 138-155
70. Vapnik V.N. Statistical learning theory (1998), Wiley, New York
71. Wiens D.P. Robust weights and designs for biased regression models: Least squares and generalized M-estimation. Journal of Statistical Planning and Inference 83 (2000) 395-412
72. Yamada, M., Sugiyama, M., & Matsui, T. (2009). Covariate shift adaptation for semi-supervised speaker identification. In Proceedings of 2009 IEEE international conference on acoustics, speech, and signal processing (pp. 1661-1664)
73. Zadrozny B. Learning and evaluating classifiers under sample selection bias. Proceedings of the twenty-first international conference on machine learning (2004), ACM Press, New York, NY 903-910
74. Zelnik-Manor L., and Perona P. Self-tuning spectral clustering. Advances in neural information processing systems Vol. 17 (2005), MIT Press, Cambridge, MA 1601-1608