Learning theory for distribution regression

Journal of Machine Learning Research

Volumn 17, Issue , 2016, Pages

  Szabó, Zoltán ^a,d   Sriperumbudur, Bharath K ^b   Póczos, Barnabás ^c   Gretton, Arthur ^a  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

^b PENNSYLVANIA STATE UNIVERSITY   (United States)

^c Carnegie Mellon University ^*  (United States)

^d UNIVERSITÉ PARIS SACLAY   (France)

Author keywords

Kernel Ridge regression; Mean embedding; Minimax optimality; Multi instance learning; Two stage sampled distribution regression

Indexed keywords

ARTIFICIAL INTELLIGENCE; COMPUTATION THEORY; COMPUTATIONAL EFFICIENCY; ECONOMIC AND SOCIAL EFFECTS; LEARNING ALGORITHMS; LEARNING SYSTEMS; PROBABILITY DISTRIBUTIONS;

KERNEL RIDGE REGRESSIONS; MEAN EMBEDDING; MULTI-INSTANCE LEARNING; OPTIMALITY; TWO-STAGE SAMPLED DISTRIBUTION REGRESSION;

REGRESSION ANALYSIS;

EID: 84995467512     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (106)

1. Ahmed Alaoui and Michael W. Mahoney. Fast randomized kernel ridge regression with statistical guarantees. In Advances in Neural Information Processing Systems (NIPS), pages 775-783, 2015.
2. Charalambos D. Aliprantis and Kim C. Border. Infinite Dimensional Analysis: A Hitchhiker's Guide. Springer, 2006.
3. Yasemin Altun and Alexander Smola. Unifying divergence minimization and statistical inference via convex duality. In Conference on Learning Theory (COLT), pages 139-153, 2006.
4. Mauricio A. Alvarez, Lorenzo Rosasco, and Neil D. Lawrence. Kernels for vector-valued functions: A review. Foundations and Trends in Machine Learning, 4: 195-266, 2011.
5. Jaume Amores. Multiple instance classification: Review, taxonomy and comparative study. Artificial Intelligence, 201: 81-105, 2013.
6. Peter Auer. Approximating hyper-rectangles: Learning and pseudorandom sets. Journal of Computer and System Sciences, 57: 376-388, 1998.
7. Boris Babenko. Multiple instance learning: Algorithms and applications. Technical report, Department of Computer Science and Engineering, University of California, San Diego, 2004. (http://cms.brookes.ac.uk/research/visiongroup/talks/rg-dec-11-09/bbabenkoffre.pdf).
8. Boris Babenko, Nakul Verma, Piotr Dollár, and Serge Belongie. Multiple instance learning with manifold bags. In International Conference on Machine Learning (ICML), pages 81-88, 2011.
9. Charles Bergeron, Jed Zaretzki, Curt Breneman, and Kristin P. Bennett. Multiple instance ranking. In International Conference on Machine Learning (ICML), pages 48-55, 2008.
10. Charles Bergeron, Gregory Moore, Jed Zaretzki, Curt M. Breneman, and Kristin P. Bennett. Fast bundle algorithm for multiple-instance learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34: 1068-1079, 2012.
11. Alain Berlinet and Christine Thomas-Agnan. Reproducing Kernel Hilbert Spaces in Probability and Statistics. Kluwer, 2004.
12. Avrim Blum and Adam Kalai. A note on learning from multiple-instance examples. Machine Learning, 30: 23-29, 1998.
13. Hanen Borchani, Gherardo Varando, Concha Bielza, and Pedro Larranaga. A survey on multi-output regression. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 5: 216-233, 2015.
14. Celine Brouard, Florence d'Alche Buc, and Marie Szafranski. Semi-supervised penalized output kernel regression for link prediction. In International Conference on Machine Learning (ICML), pages 593-600, 2011.
15. Andrea Caponnetto. Optimal rates for regularization operators in learning theory. Technical report, Massachusetts Institute of Technology, 2006. (http://www6.cityu.edu.hk/ma/doc/people/caponnettoa/regopTR%28TR11%29.pdf).
16. Andrea Caponnetto and Ernesto De Vito. Optimal rates for regularized least-squares algorithm. Foundations of Computational Mathematics, 7: 331-368, 2007.
17. Claudio Carmeli, Ernesto De Vito, and Alessandro Toigo. Vector valued reproducing kernel Hilbert spaces of integrable functions and Mercer theorem. Analysis and Applications, 4: 377-408, 2006.
18. Claudio Carmeli, Ernesto De Vito, Alessandro Toigo, and Veronica Umanitá. Vector valued reproducing kernel Hilbert spaces and universality. Analysis and Applications, 8: 19-61, 2010.
19. Kevin M. Carter, Raviv Raich, William G. Finn, and Alfred O. Hero. Information-geometric dimensionality reduction. IEEE Signal Processing Magazine, 28: 89-99, 2011.
20. Jing Chai, Hongtao Chen, Lixia Huang, and Fanhua Shang. Maximum margin multipleinstance feature weighting. Pattern Recognition, 47: 2091-2103, 2014a.
21. Jing Chai, Xinghao Ding, Hongtao Chen, and Tingyu Li. Multiple-instance discriminant analysis. Pattern Recognition, 47: 2517-2531, 2014b.
22. Ying Chen and Ou Wu. Contextual Hausdor dissimilarity for multi-instance clustering. In International Conference on Fuzzy Systems and Knowledge Discovery (FSKD), pages 870-873, 2012.
23. Andreas Christmann and Ingo Steinwart. Universal kernels on non-standard input spaces. In Advances in Neural Information Processing Systems (NIPS), pages 406-414, 2010.
24. Donald L. Cohn. Measure Theory: Second Edition. Birkhäuser Basel, 2013.
25. Marco Cuturi, Kenji Fukumizu, and Jean-Philippe Vert. Semigroup kernels on measures. Journal of Machine Learning Research, 6: 11691198, 2005.
26. Ernesto de Vito, Lorenzo Rosasco, and Andrea Caponnetto. Discretization error analysis for Tikhonov regularization. Analysis and Applications, 4: 81-99, 2006.
27. Thomas G. Dietterich, Richard H. Lathrop, and Tomás Lozano-Perez. Solving the multiple instance problem with axis-parallel rectangles. Artificial Intelligence, 89: 31-71, 1997.
28. Jiu Ding and Aihui Zhou. A spectrum theorem for perturbed bounded linear operators. Applied Mathematics and Computation, 201: 723-728, 2008.
29. Daniel R. Dooly, Qi Zhang, Sally A. Goldman, and Robert A. Amar. Multiple-instance learning of real-valued data. Journal of Machine Learning Research, 3: 651-678, 2002.
30. Gerald Edgar. Measure, Topology and Fractal Geometry. Springer-Verlag, 1995.
31. Thomas Eiter and Heikki Mannila. Distance measures for point sets and their computation. Acta Informatica, 34: 109-133, 1997.
32. Theodoros Evgeniou, Charles A. Micchelli, and Massimiliano Pontil. Learning multiple tasks with kernel methods. Journal of Machine Learning Research, 6: 615-637, 2005.
33. Seth Flaxman, Yu-XiangWang, and Alex Smola. Who supported Obama in 2012 ecological inference through distribution regression. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pages 289-298, 2015.
34. James Foulds and Eibe Frank. A review of multi-instance learning assumptions. The Knowledge Engineering Review, 25: 1-25, 2010.
35. Kenji Fukumizu, Francis Bach, and Michael Jordan. Dimensionality reduction for supervised learning with reproducing kernel Hilbert spaces. Journal of Machine Learning Research, 5: 73-99, 2004.
36. Thomas Gärtner, Peter A. Flach, Adam Kowalczyk, and Alexander Smola. Multi-instance kernels. In International Conference on Machine Learning (ICML), pages 179-186, 2002.
37. Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13: 723-773, 2012.
38. Lászlo Györ, Michael Kohler, Adam Krzyzak, and HarroWalk. A Distribution-Free Theory of Nonparametric Regression. Springer, New-york, 2002.
39. David Haussler. Convolution kernels on discrete structures. Technical report, Department of Computer Science, University of California at Santa Cruz, 1999. (http://cbse.soe. ucsc.edu/sites/default/files/convolutions.pdf).
40. Matthias Hein and Olivier Bousquet. Hilbertian metrics and positive definite kernels on probability measures. In International Conference on Artificial Intelligence and Statistics (AISTATS), pages 136-143, 2005.
41. Yang Hu, Mingjing Li, and Nenghai Yu. Multiple-instance ranking: Learning to rank images for image retrieval. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1-8, 2008.
42. Tony Jebara, Risi Kondor, and Andrew Howard. Probability product kernels. Journal of Machine Learning Research, 5: 819-844, 2004.
43. Hachem Kadri, Emmanuel Duos, Philippe Preux, Stephane Canu, and Manuel Davy. Nonlinear functional regression: A functional RKHS approach. International Conference on Artificial Intelligence and Statistics (AISTATS; JMLR W& CP), 9: 374-380, 2010.
44. Hachem Kadri, Alain Rakotomamonjy, Francis Bach, and Philippe Preux. Multiple operator-valued kernel learning. In Advances in Neural Information Processing Systems (NIPS), pages 2429-2437, 2012.
45. Hachem Kadri, Mohammad Ghavamzadeh, and Philippe Preux. A generalized kernel approach to structured output learning. International Conference on Machine Learning (ICML; JMLR W& CP), 28: 471-479, 2013.
46. Hachem Kadri, Emmanuel Duos, Philippe Preux, Stephane Canu, Alain Rakotomamonjy, and Julien Audiren. Operator-valued kernels for learning from functional response data. Journal of Machine Learning Research, 17: 1-54, 2016.
47. Risi Kondor and Tony Jebara. A kernel between sets of vectors. In International Conference on Machine Learning (ICML), pages 361-368, 2003.
48. Samory Kpotufe. k-NN regression adapts to local intrinsic dimension. Technical report, Max Planck Institute for Intelligent Systems, 2011. (http://arxiv.org/abs/1110.4300).
49. James T. Kwok and Pak-Ming Cheung. Marginalized multi-instance kernels. In International Joint Conferences on Artificial Intelligence (IJCAI), pages 901-906, 2007.
50. Philip M. Long and Lei Tan. PAC learning of axis-aligned rectangles with respect to product distributions from multiple-instance examples. Machine Learning, 30: 7-21, 1998.
51. David Lopez-Paz, Krikamol Muandet, Bernhard Schölkopf, and Iliya Tolstikhin. Towards a learning theory of cause-effect inference. International Conference on Machine Learning (ICML; JMLR W& CP), 37: 1452-1461, 2015.
52. Andre F. T. Martins, Noah A. Smith, Eric P. Xing, Pedro M. Q. Aguiar, and Mário A. T. Figueiredo. Nonextensive information theoretical kernels on measures. Journal of Machine Learning Research, 10: 935-975, 2009.
53. Shahar Mendelson and Joseph Neeman. Regularization in kernel learning. The Annals of Statistics, 38: 526-565, 2010.
54. Charles A. Micchelli and Massimiliano Pontil. On learning vector-valued functions. Neural Computation, 17: 177-204, 2005.
55. Krikamol Muandet, Kenji Fukumizu, Francesco Dinuzzo, and Bernhard Schölkopf. Learning from distributions via support measure machines. In Advances in Neural Information Processing Systems (NIPS), pages 10-18, 2012.
56. Hans-Georg Müller. Functional modelling and classification of longitudinal data. Scandinavian Journal of Statistics, 32: 223-240, 2005.
57. Frank Nielsen and Richard Nock. A closed-form expression for the Sharma-Mittal entropy of exponential families. Journal of Physics A: Mathematical and Theoretical, 45: 032003, 2012.
58. Junier Oliva, Barnabás Poczos, and Je Schneider. Distribution to distribution regression. International Conference on Machine Learning (ICML; JMLR W& CP), 28: 1049-1057, 2013.
59. Junier Oliva, William Neiswanger, Barnabás Poczos, Eric Xing, Hy Trac, Shirley Ho, and Je Schneider. Fast function to function regression. International Conference on Artificial Intelligence and Statistics (AISTATS; JMLR W& CP), 38: 717-725, 2015.
60. Junier B. Oliva, Willie Neiswanger, Barnabás Poczos, Je Schneider, and Eric Xing. Fast distribution to real regression. International Conference on Artificial Intelligence and Statistics (AISTATS; JMLR W& CP), 33: 706-714, 2014.
61. Kalyanapuram R. Parthasarathy. Probability Measures on Metric Spaces. Academic Press, 1967.
62. George Pedrick. Theory of reproducing kernels for Hilbert spaces of vector valued functions. Technical report, 1957.
63. Wei Ping, Ye Xu, Kexin Ren, Chi-Hung Chi, and Shen Furao. Non-I.I.D. multi-instance dimensionality reduction by learning a maximum bag margin subspace. In AAAI Conference on Artificial Intelligence, pages 551-556, 2010.
64. Barnabás Poczos, Liang Xiong, and Je Schneider. Nonparametric divergence estimation with applications to machine learning on distributions. In Uncertainty in Artificial Intelligence (UAI), pages 599-608, 2011.
65. Barnabás Poczos, Liang Xiong, Dougal Sutherland, and Je Schneider. Support distribution machines. Technical report, Carnegie Mellon University, 2012. (http://arxiv.org/abs/1202.0302).
66. Barnabás Poczos, Alessandro Rinaldo, Aarti Singh, and LarryWasserman. Distribution-free distribution regression. International Conference on Artificial Intelligence and Statistics (AISTATS; JMLR W& CP), 31: 507-515, 2013.
67. Jan Ramon and Maurice Bruynooghe. A polynomial time computable metric between point sets. Acta Informatica, 37: 765-780, 2001.
68. James O. Ramsay and Bernard W. Silverman. Applied Functional Data Analysis. Springer Verlag, New York, 2002.
69. James O. Ramsay and Bernard W. Silverman. Functional Data Analysis. Springer Verlag, New York, 2005.
70. Soumya Ray and David Page. Multiple instance regression. In International Conference on Machine Learning (ICML), pages 425-432, 2001.
71. Vikas C. Raykar, Balaji Krishnapuram, Jinbo Bi, Murat Dundar, and R. Bharat Rao. Bayesian multiple instance learning: Automatic feature selection and inductive transfer. In International Conference on Machine Learning (ICML), pages 808-815, 2008.
72. Peter Richtárik and Martin Takác. Distributed coordinate descent method for learning with big data. Journal of Machine Learning Research, 17: 1-25, 2016.
73. R. Tyrrell Rockafellar and Roger J-B Wets. Variational Analysis. Springer, 2008.
74. Alessandro Rudi, Raáello Camoriano, and Lorenzo Rosasco. Less is more: Nyström computational regularization. In Advances in Neural Information Processing Systems (NIPS), pages 1648-1656, 2015.
75. Sivan Sabato and Naftali Tishby. Multi-instance learning with any hypothesis class. Journal of Machine Learning Research, 13: 2999-3039, 2012.
76. Laurent Schwartz. Analyse III, Calcul Integrel. Hermann, 1998.
77. Steve Smale and Ding-Xuan Zhou. Estimating the approximation error in learning theory. Analysis and Applications, 1: 17-41, 2003.
78. Bharath Sriperumbudur, Arthur Gretton, Kenji Fukumizu, Gert Lanckriet, and Bernhard Schölkopf. Hilbert space embeddings and metrics on probability measures. Journal of Machine Learning Research, 11: 1517-1561, 2010.
79. Bharath K. Sriperumbudur, Kenji Fukumizu, and Gert R. G. Lanckriet. Universality, characteristic kernels and RKHS embedding of measures. Journal of Machine Learning Research, 12: 2389-2410, 2011.
80. Bharath K. Sriperumbudur, Kenji Fukumizu, Revant Kumar, Arthur Gretton, and Aapo Hyvärinen. Density estimation in infinite dimensional exponential families. Technical report, 2014. (http://arxiv.org/pdf/1312.3516).
81. Ingo Steinwart and Andres Christmann. Support Vector Machines. Springer, 2008.
82. Ingo Steinwart and Clint Scovel. Mercer's theorem on general domains: On the interaction between measures, kernels, and RKHSs. Constructive Approximation, 35: 363-417, 2012.
83. Ingo Steinwart, Don R. Hush, and Clint Scovel. Optimal rates for regularized least squares regression. In Conference on Learning Theory (COLT), 2009.
84. Hongwei Sun and Qiang Wu. Application of integral operator for regularized least-square regression. Mathematical and Computer Modelling, 49: 276-285, 2009a.
85. Hongwei Sun and Qiang Wu. A note on application of integral operator in learning theory. Applied and Computational Harmonic Analysis, 26: 416-421, 2009b.
86. Xu Sun, Hisashi Kashima, and Naonori Ueda. Large-scale personalized human activity recognition using online multitask learning. IEEE Transactions on Knowledge and Data Engine, 25: 2551-2563, 2013.
87. Yu-Yin Sun, Michael K. Ng, and Zhi-Hua Zhou. Multi-instance dimensionality reduction. In AAAI Conference on Artificial Intelligence, pages 587-592, 2010.
88. Dougal J. Sutherland, Junier B. Oliva, Barnabás Poczos, and Je Schneider. Linear-time learning on distributions with approximate kernel embeddings. In AAAI Conference on Artifical Intelligence (AAAI), pages 2073-2079, 2016.
89. Zoltán Szabo, Arthur Gretton, Barnabás Poczos, and Bharath Sriperumbudur. Two-stage sampled learning theory on distributions. In International Conference on Artificial Intelligence and Statistics (AISTATS), pages 948-957, 2015.
90. Leslie Valiant. A theory of the learnable. Communications of the ACM, 27: 1134-1142, 1984.
91. Fei Wang, Tanveer Syeda-Mahmood, Baba C. Vemuri, David Beymer, and Anand Rangarajan. Closed-form Jensen-Renyi divergence for mixture of Gaussians and applications to group-wise shape registration. Medical Image Computing and Computer-Assisted Intervention, 12: 648-655, 2009.
92. Jun Wang and Jean-Daniel Zucker. Solving the multiple-instance problem: A lazy learning approach. In International Conference on Machine Learning (ICML), pages 1119-1126, 2000.
93. Larry Wasserman. All of Nonparametric Statistics. Springer, 2006.
94. Ou Wu, Jun Gao, Weiming Hu, Bing Li, and Mingliang Zhu. Identifying multi-instance outliers. In SIAM International Conference on Data Mining (SDM), pages 430-441, 2010.
95. Yun Yang, Mert Pilanci, and Martin J. Wainwright. Randomized sketches for kernels: Fast and optimal non-parametric regression. Annals of Statistics, 2016. (to appear; arXiv: Http://arxiv.org/abs/1501.06195).
96. Amelia Zafra, Mykola Pechenizkiy, and Sebastián Ventura. HyDR-MI: A hybrid algorithm to reduce dimensionality in multiple instance learning. Information Sciences, 222: 282-301, 2013.
97. Dan Zhang and Luo Si. Multiple instance transfer learing. In IEEE International Conference on Data Mining Workshops (ICDMW), pages 406-411, 2009.
98. Dan Zhang, Fei Wang, Luo Si, and Tao Li. M3IC: Maximum margin multiple instance clustering. In International Joint Conferences on Artificial Intelligence (IJCAI), pages 1339-1344, 2009.
99. Dan Zhang, Fei Wang, Luo Si, and Tao Li. Maximum margin multiple instance clustering with applications to image and text clustering. IEEE Transactions on Neural Networks, 22: 739-751, 2011.
100. Dan Zhang, Jingrui He, Luo Si, and Richard D. Lawrence. MILEAGE: Multiple Instance LEArning with Global Embedding. International Conference on Machine Learning (ICML; JMLR W& CP), 28: 82-90, 2013.
101. Min-Ling Zhang and Zhi-Hua Zhou. Multi-instance clustering with applications to multiinstance prediction. Applied Intelligence, 31: 47-68, 2009.
102. Yuchen Zhang, John C. Duchi, and Martin J. Wainwright. Divide and conquer kernel ridge regression: A distributed algorithm with minimax optimal rates. 16: 3299-3340, 2015.
103. Jiayu Zhou, Jun Liu, Vaibhav A. Narayan, and Jieping Ye. Modeling disease progression via multi-task learning. NeuroImage, 78: 233-248, 2013.
104. Shaohua Kevin Zhou and Rama Chellappa. From sample similarity to ensemble similarity: Probabilistic distance measures in reproducing kernel Hilbert space. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28: 917-929, 2006.
105. Zhi-Hua Zhou. Multi-instance learning: A survey. Technical report, AI Lab, Department of Computer Science & Technology, Nanjing University, Nanjing, China, 2004. (http://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/techrep04.pdf).
106. Zhi-Hua Zhou, Yu-Yin Sun, and Yu-Feng Li. Multi-instance learning by treating instances as non-I.I.D. samples. In International Conference on Machine Learning (ICML), pages 1249-1256, 2009.