A kernel two-sample test

Journal of Machine Learning Research

Volumn 13, Issue , 2012, Pages 723-773

  Gretton, Arthur ^a,e   Borgwardt, Karsten M ^b   Rasch, Malte J ^c   Schölkopf, Bernhard ^a   Smola, Alexander ^d,f  

^a MAX PLANCK INSTITUTE FOR INTELLIGENT SYSTEMS   (Germany)

^b Machine Learning and Computational Biology Research Group   (Germany)

^c BEIJING NORMAL UNIVERSITY   (China)

^d YAHOO RESEARCH   (United States)

^e UNIVERSITY COLLEGE LONDON   (United Kingdom)

^f AUSTRALIAN NATIONAL UNIVERSITY   (Australia)

Author keywords

Hypothesis testing; Integral probability metric; Kernel methods; Schema matching; Two sample test; Uniform convergence bounds

Indexed keywords

HYPOTHESIS TESTING; INTEGRAL PROBABILITY METRIC; KERNEL METHODS; SCHEMA MATCHING; TWO-SAMPLE TESTS; UNIFORM CONVERGENCE;

STATISTICAL TESTS;

PROBABILITY DISTRIBUTIONS;

EID: 84859477054     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Review

References (91)

1. Y. Altun and A.J. Smola. Unifying divergence minimization and statistical inference via convex duality. InProc. Annual Conf. Computational Learning Theory, LNCS, pages 139-153. Springer, 2006. (Pubitemid 44072191)
2. N. Anderson, P. Hall, and D. Titterington. Two-sample test statistics for measuring discrepancies between two multivariate probability density functions using kernel-based density estimates. Journal of Multivariate Analysis, 50:41-54, 1994.
3. S. Andrews, I. Tsochantaridis, and T. Hofmann. Support vector machines for multiple-instance learning. In Advances in Neural Information Processing Systems 15, Cambridge, MA, 2003. MIT Press.
4. M. Arcones and E. Giné. On the bootstrap of u and v statistics. The Annals of Statistics, 20(2): 655-674, 1992.
5. F. R. Bach and M. I. Jordan. Kernel independent component analysis. Journal of Machine Learning Research, 3:1-48, 2002.
6. P. L. Bartlett and S. Mendelson. Rademacher and Gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3:463-482, 2002.
7. S. Ben-David, J. Blitzer, K. Crammer, and F. Pereira. Analysis of representations for domain adaptation. In Advances in Neural Information Processing Systems 19, pages 137-144. MIT Press, 2007.
8. A. Berlinet and C. Thomas-Agnan. Reproducing Kernel Hilbert Spaces in Probability and Statistics. Kluwer, 2004.
9. 1-test statistic of homo geneity. IEEE Transactions on Information Theory, 51(11):3965-3973, 2005. (Pubitemid 41622594)
10. P. Bickel. A distribution free version of the Smirnov two sample test in the p-variate case. The Annals of Mathematical Statistics, 40(1):1-23, 1969.
11. C. L. Blake and C. J. Merz. UCI repository of machine learning databases, 1998. URL http://www.ics.uci.edu/~mlearn/MLRepository.html.
12. K. M. Borgwardt, C. S. Ong, S. Schonauer, S. V. N. Vishwanathan, A. J. Smola, and H. P. Kriegel. Protein function prediction via graph kernels. Bioinformatics (ISMB), 21(Suppl I):i47-i56, Jun 2005. (Pubitemid 41794474)
13. K. M. Borgwardt, A. Gretton, M. J. Rasch, H.-P. Kriegel, B. Schölkopf, and A. J. Smola. Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics (ISMB), 22(14): e49-e57, 2006. (Pubitemid 44288272)
14. O. Bousquet, S. Boucheron, and G. Lugosi. Theory of classification: a survey of recent advances. ESAIM: Probability and Statistics, 9:323-375, 2005.
15. R. Caruana and T. Joachims. KDD cup. 2004. URL http ://kodiak.cs.Cornell. edu/kddcup/index.html.
16. G. Casella and R. Berger. Statistical Inference. Duxbury, Pacific Grove, CA, 2nd edition, 2002.
17. B. Chazelle. A minimum spanning tree algorithm with inverse-Ackermann type complexity. Journal of the ACM, 47:1028-1047, 2000.
18. P. Comon. Independent component analysis, a new concept? Signal Processing, 36:287-314, 1994.
19. C. Cortes, M. Mohri, M. Riley, and A. Rostamizadeh. Sample selection bias correction theory. In Proceedings of the International Conference on Algorithmic Learning Theory, volume 5254 of Lecture Notes in Computer Science, pages 38-53. Springer, 2008.
20. M. Davy, A. Gretton, A. Doucet, and P. J. W. Rayner. Optimized support vector machines for nonstationary signal classification. IEEE Signal Processing Letters, 9(12): 442-445, 2002.
21. V. de la Pena and E. Giné. Decoupling: from Dependence to Independence. Springer, New York, 1999.
22. M. Dudik and R. E. Schapire. Maximum entropy distribution estimation with generalized regu-larization. In Proceedings of the Annual Conference on Computational Learning Theory, pages 123-138. Springer Verlag, 2006. (Pubitemid 44072190)
23. M. Dudik, S. Phillips, and R.E. Schapire. Performance guarantees for regularized maximum entropy density estimation. In Proceedings of the Annual Conference on Computational Learning Theory, pages 472-486. Springer Verlag, 2004. (Pubitemid 38940354)
24. R. M. Dudley. Real Analysis and Probability. Cambridge University Press, Cambridge, UK, 2002.
25. W. Feller. An Introduction to Probability Theory and its Applications. John Wiley and Sons, New York, 2nd edition, 1971.
26. A. Feuerverger. A consistent test for bivariate dependence. International Statistical Review, 61(3): 419-133, 1993.
27. S. Fine and K. Scheinberg. Efficient SVM training using low-rank kernel representations. Journal of Machine Learning Research, 2:243-264, 2001.
28. R. Fortet and E. Mourier. Convergence de la réparation empirique vers la réparation théorique. Ann. Scient. Ecole Norm. Sup., 70:266-285, 1953.
29. J. Friedman. On multivariate goodness-of-fit and two-sample testing. Technical Report SLAC-PUB-10325, University of Stanford Statistics Department, 2003.
30. J. Friedman and L. Rafsky. Multivariate generalizations of the Wald-Wolfowitz and Smirnov two-sample tests. The Annals of Statistics, 7(4):697-717, 1979.
31. K. Fukumizu, F. R. Bach, and M. I. Jordan. Dimensionality reduction for supervised learning with reproducing kernel Hilbert spaces. Journal of Machine Learning Research, 5:73-99, 2004.
32. K. Fukumizu, A. Gretton, X. Sun, and B. Schölkopf. Kernel measures of conditional dependence. In Advances in Neural Information Processing Systems 20, pages 489-496, Cambridge, MA, 2008. MIT Press.
33. T. Gärtner, P. A. Flach, A. Kowalczyk, and A. J. Smola. Multi-instance kernels. In Proceedings of the International Conference on Machine Learning, pages 179-186. Morgan Kaufmann Publishers Inc., 2002.
34. E. Gokcay and J.C. Principe. Information theoretic clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(2):158-171, 2002. (Pubitemid 34198206)
35. A. Gretton, O. Bousquet, A.J. Smola, and B. Schölkopf. Measuring statistical dependence with Hilbert-Schmidt norms. In Proceedings of the International Conference on Algorithmic Learning Theory, pages 63-77. Springer-Verlag, 2005a.
36. A. Gretton, R. Herbrich, A. Smola, O. Bousquet, and B. Schölkopf. Kernel methods for measuring independence. Journal of Machine Learning Research, 6:2075-2129, 2005b. (Pubitemid 41798124)
37. A. Gretton, K. Borgwardt, M. Rasch, B. Schölkopf, and A. Smola. A kernel method for the two-sample problem. In Advances in Neural Information Processing Systems 15, pages 513-520, Cambridge, MA, 2007a. MIT Press.
38. A. Gretton, K. Borgwardt, M. Rasch, B. Schlkopf, and A. Smola. A kernel approach to comparing distributions. Proceedings of the 22nd Conference on Artificial Intelligence (AAAI-07), pages 1637-1641, 2007b. (Pubitemid 350149802)
39. A. Gretton, K. Borgwardt, M. Rasch, B. Schölkopf, and A. Smola. A kernel method for the two sample problem. Technical Report 157, MPI for Biological Cybernetics, 2008a.
40. A. Gretton, K. Fukumizu, C.-H. Teo, L. Song, B. Schölkopf, and A. Smola. A kernel statistical test of independence. In Advances in Neural Information Processing Systems 20, pages 585-592, Cambridge, MA, 2008b. MIT Press.
41. A. Gretton, K. Fukumizu, Z. Harchaoui, and B. Sriperumbudur. A fast, consistent kernel two-sample test. In Advances in Neural Information Processing Systems 22, Red Hook, NY, 2009. Curran Associates Inc.
42. G. R. Grimmet and D. R. Stirzaker. Probability and Random Processes. Oxford University Press, Oxford, third edition, 2001.
43. P. Hall and N. Tajvidi. Permutation tests for equality of distributions in high-dimensional settings. Biometrika, 89(2):359-374, 2002. (Pubitemid 41311986)
44. Z. Harchaoui, F. Bach, and E. Moulines. Testing for homogeneity with kernel Fisher discriminant analysis. In Advances in Neural Information Processing Systems 20, pages 609-616. MIT Press, Cambridge, MA, 2008.
45. M. Hein, T.N. Lal, and O. Bousquet. Hilbertian metrics on probability measures and their application in SVMs. In Proceedings of the 26th DAGM Symposium, pages 270-277, Berlin, 2004. Springer.
46. N. Henze and M. Penrose. On the multivariate runs test. The Annals of Statistics, 27(1):290-298, 1999.
47. W. Hoeffding. A class of statistics with asymptotically normal distribution. The Annals of Mathematical Statistics, 19(3):293-325, 1948.
48. W. Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58:13-30, 1963.
49. T. Jebara and R. Kondor. Bhattacharyya and expected likelihood kernels. In Proceedings of the Annual Conference on Computational Learning Theory, volume 2777 of LNCS, pages 57-71, Heidelberg, Germany, 2003. Springer-Verlag. (Pubitemid 37053195)
50. N. L. Johnson, S. Kotz, and N. Balakrishnan. Continuous Univariate Distributions. Volume 1. John Wiley and Sons, 2nd edition, 1994.
51. A. Kankainen. Consistent Testing of Total Independence Based on the Empirical Characteristic Function. PhD thesis, University of Jyväskylä, 1995.
52. D. Kifer, S. Ben-David, and J. Gehrke. Detecting change in data streams. In Proceedings of the International Conference on Very Large Data Bases, pages 180-191. VLDB Endowment, 2004.
53. H.W. Kuhn. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2:83-97, 1955.
54. E. L. Lehmann and J. P. Romano. Testing Statistical Hypotheses. Springer, 3rd edition, 2005.
55. C. McDiarmid. On the method of bounded differences. In Survey in Combinatorics, pages 148-188. Cambridge University Press, 1989.
56. C. Micchelli, Y. Xu, and H. Zhang. Universal kernels. Journal of Machine Learning Research, 7: 2651-2667, 2006. (Pubitemid 44927394)
57. A. Müller. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 29(2):429-443, 1997. (Pubitemid 127554412)
58. X.L. Nguyen, M. Wainwright, and M. Jordan. Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization. In Advances in Neural Information Processing Systems 20, pages 1089-1096. MIT Press, Cambridge, MA, 2008.
59. W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery. Numerical Recipes in C. The Art of Scientific Computation. Cambridge University Press, Cambridge, UK, 1994.
60. M. Rasch, A. Gretton, Y. Murayama, W. Maass, and N. K. Logothetis. Predicting spiking activity from local field potentials. Journal of Neurophysiology, 99:1461-1476, 2008. (Pubitemid 351413722)
61. M. Reed and B. Simon. Methods of modern mathematical physics. Vol. 1: Functional Analysis. Academic Press, San Diego, 1980.
62. M. Reid and R. Williamson. Information, divergence and risk for binary experiments. Journal of Machine Learning Research, 12:731-817, 2011.
63. P. Rosenbaum. An exact distribution-free test comparing two multivariate distributions based on adjacency. Journal of the Royal Statistical Society B, 67(4):515-530, 2005. (Pubitemid 41351403)
64. Y. Rubner, C. Tomasi, and L.J. Guibas. The earth mover's distance as a metric for image retrieval. International Journal of Computer Vision, 40(2):99-121, 2000. (Pubitemid 32136368)
65. B. Schölkopf. Support Vector Learning. R. Oldenbourg Verlag, Munich, 1997. Download: http://www.kernel-machines.org.
66. B. Schölkopf and A. Smola. Learning with Kernels. MIT Press, Cambridge, MA, 2002.
67. B. Schölkopf, J. Platt, J. Shawe-Taylor, A. J. Smola, and R. C. Williamson. Estimating the support of a high-dimensional distribution. Neural Computation, 13(7):1443-1471, 2001. (Pubitemid 33595028)
68. B. Schölkopf, K. Tsuda, and J.-P. Vert. Kernel Methods in Computational Biology. MIT Press, Cambridge, MA, 2004.
69. R. Serfling. Approximation Theorems of Mathematical Statistics. Wiley, New York, 1980.
70. J. Shawe-Taylor and N. Cristianini. Kernel Methods for Pattern Analysis. Cambridge University Press, Cambridge, UK, 2004.
71. J. Shawe-Taylor and A. Dolia. A framework for probability density estimation. In Proceedings of the International Conference on Artificial Intelligence and Statistics, pages 468-475, 2007.
72. H. Shen, S. Jegelka, and A. Gretton. Fast kernel-based independent component analysis. IEEE Transactions on Signal Processing, 57:3498-3511, 2009.
73. B. W. Silverman. Density Estimation for Statistical and Data Analysis. Monographs on statistics and applied probability. Chapman and Hall, London, 1986.
74. N.V. Smirnov. On the estimation of the discrepancy between empirical curves of distribution for two independent samples. Moscow University Mathematics Bulletin, 2:3-26, 1939. University of Moscow.
75. A. J. Smola and B. Schölkopf. Sparse greedy matrix approximation for machine learning. In Proceedings of the International Conference on Machine Learning, pages 911-918, San Francisco, 2000. Morgan Kaufmann Publishers.
76. A. J. Smola, A. Gretton, L. Song, and B. Schölkopf. A Hilbert space embedding for distributions. In Proceedings of the International Conference on Algorithmic Learning Theory, volume 4754, pages 13-31. Springer, 2007.
77. L. Song, X. Zhang, A. Smola, A. Gretton, and B. Schölkopf. Tailoring density estimation via reproducing kernel moment matching. In Proceedings of the International Conference on Machine Learning, pages 992-999. ACM, 2008.
78. B. Sriperumbudur, A. Gretton, K. Fukumizu, G. Lanckriet, and B. Schölkopf. Injective Hilbert space embeddings of probability measures. In Proceedings of the Annual Conference on Computational Learning Theory, pages 111-122, 2008.
79. B. Sriperumbudur, K. Fukumizu, A. Gretton, G. Lanckriet, and B. Schoelkopf. Kernel choice and classifiability for RKHS embeddings of probability distributions. In Advances in Neural Information Processing Systems 22, Red Hook, NY, 2009. Curran Associates Inc.
80. B. Sriperumbudur, K. Fukumizu, A. Gretton, B. Schölkopf, and G. Lanckriet. Non-parametric estimation of integral probability metrics. In International Symposium on Information Theory, pages 1428-1432, 2010a.
81. B. Sriperumbudur, A. Gretton, K. Fukumizu, G. Lanckriet, and B. Schölkopf. Hilbert space em-beddings and metrics on probability measures. Journal of Machine Learning Research, 11:1517-1561, 2010b.
82. B. Sriperumbudur, K. Fukumizu, and G. Lanckriet. Universality, characteristic kernels and RKHS embedding of measures. Journal of Machine Learning Research, 12:2389-2410, 2011a.
83. B. Sriperumbudur, K. Fukumizu, and G. Lanckriet. Learning in Hilbert vs. Banach spaces: A measure embedding viewpoint. In Advances in Neural Information Processing Systems 24. Curran Associates Inc., Red Hook, NY, 2011b.
84. I. Steinwart. On the influence of the kernel on the consistency of support vector machines. Journal of Machine Learning Research, 2:67-93, 2001.
85. I. Steinwart and A. Christmann. Support Vector Machines. Information Science and Statistics. Springer, 2008.
86. I. Takeuchi, Q. V. Le, T. Sears, and A. J. Smola. Nonparametric quantile estimation. Journal of Machine Learning Research, 7, 2006.
87. D. M. J. Tax and R. P. W. Duin. Data domain description by support vectors. In Proceedings ESANN, pages 251-256, Brussels, 1999. D Facto.
88. A. W. van der Vaart and J. A. Wellner. Weak Convergence and Empirical Processes. Springer, 1996.
89. L. Wasserman. All of Nonparametric Statistics. Springer, 2006.
90. J. E. Wilkins. A note on skewness and kurtosis. The Annals of Mathematical Statistics, 15(3): 333-335, 1944.
91. C. K. I. Williams and M. Seeger. Using the Nystrom method to speed up kernel machines. In Advances in Neural Information Processing Systems 13, pages 682-688, Cambridge, MA, 2001. MIT Press.