Fast rates in statistical and online learning

Journal of Machine Learning Research

Volumn 16, Issue , 2015, Pages 1793-1861

  Van Erven, Tim ^a   Grünwald, Peter D ^a   Mehta, Nishant A ^b   Reid, Mark D ^c   Williamson, Robert C ^c  

^a LEIDEN UNIVERSITY   (Netherlands)

^b CWI   (Netherlands)

^c AUSTRALIAN NATIONAL UNIVERSITY   (Australia)

Author keywords

Exp concavity; Fast rates; Mixability; Statistical learning theory; Tsybakov margin condition

Indexed keywords

ALGORITHMS; ARTIFICIAL INTELLIGENCE; E-LEARNING; LEARNING SYSTEMS; STATISTICS; STOCHASTIC MODELS; STOCHASTIC SYSTEMS;

EXP-CONCAVITY; FAST RATE; MIXABILITY; STATISTICAL LEARNING THEORY; TSYBAKOV MARGIN CONDITION;

LEARNING ALGORITHMS;

EID: 84962208880     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (78)

1. Misha Alekhnovich, Mark Braverman, Vitaly Feldman, Adam R Klivans, and Toniann Pitassi. Learnability and automatizability. In Foundations of Computer Science, 2004. Proceedings. 45th Annual IEEE Symposium on, pages 621-630. IEEE, 2004.
2. Sylvain Arlot and Peter L. Bartlett. Margin-adaptive model selection in statistical learning. Bernoulli, 17(2):687-713, 2011.
3. Jean-Yves Audibert. PAC-Bayesian statistical learning theory. PhD thesis, Université Paris 6, 2004.
4. Jean-Yves Audibert. Progressive mixture rules are deviation suboptimal. In J. C. Platt, D. Koller, Y. Singer, and S. T. Roweis, editors, Advances in Neural Information Processing Systems 20, pages 41-48, 2007.
5. Jean-Yves Audibert. Fast learning rates in statistical inference through aggregation. The Annals of Statistics, 37(4):1591-1646, 2009.
6. Andrew R. Barron. Are Bayes rules consistent in information? Open Problems in Communication and Computation, pages 85-91, 1987.
7. Andrew R. Barron. Personal Communication, 2001.
8. Andrew R. Barron and Thomas M. Cover. Minimum complexity density estimation. Information Theory, IEEE Transactions on, 37(4):1034-1054, 1991.
9. Peter L. Bartlett and Shahar Mendelson. Empirical minimization. Probability Theory and Related Fields, 135(3):311-334, 2006.
10. Peter L. Bartlett, Michael I. Jordan, and Jon D. McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):138-156, 2006.
11. Patrick Billingsley. Convergence of Probability Measures. Wiley, 1968.
12. Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration Inequalities: A Nonasymptotic Theory of Independence. Oxford University Press, 2013.
13. Olivier Catoni. Pac-Bayesian Supervised Classification: The Thermodynamics of Statistical Learning. IMS, 2007.
14. Nicolò Cesa-Bianchi and Gábor Lugosi. Prediction, Learning, and Games. Cambridge University Press, 2006.
15. Nicolò Cesa-Bianchi, Alex Conconi, and Claudio Gentile. On the generalization ability of on-line learning algorithms. IEEE Transactions on Information Theory, 50(9):2050-2057, 2004.
16. Alexey Chernov, Yuri Kalnishkan, Fedor Zhdanov, and Vladimir Vovk. Supermartingales in prediction with expert advice. Theoretical Computer Science, 411:2647-2669, 2010.
17. Arnak S. Dalalyan and Alexandre B. Tsybakov. Mirror averaging with sparsity priors. Bernoulli, 18(3):914-944, 2012.
18. Pierpaolo De Blasi and Stephen G Walker. Bayesian asymptotics with misspcified models. Statistica Sinica, 23:169-187, 2013.
19. Steven de Rooij, Tim van Erven, Peter D. Grünwald, and Wouter M. Koolen. Follow the leader if you can, hedge if you must. Journal of Machine Learning Research, 15:1281-1316, 2014.
20. Joseph L. Doob. Application of the theory of martingales. Actes du Colloque International Le Calcul des Probabilités et ses Applications, pages 23-27, 1949.
21. Thomas S. Ferguson. Mathematical Statistics: A Decision Theoretic Approach. Academic Press, 1967.
22. David Freedman. On tail probabilities for martingales. Annals of Probability, 3:100-118, 1975.
23. Subhashis Ghosal, Jayanta K. Ghosh, and Aad W. van der Vaart. Convergence rates of posterior distributions. Annals of Statistics, 28(2):500-531, 2000.
24. P. D. Grünwald. Viewing all models as "probabilistic". In Proceedings of the Twelfth ACM Conference on Computational Learning Theory (COLT' 99), pages 171-182, 1999.
25. Peter D. Grünwald. The Minimum Description Length Principle. MIT Press, 2007.
26. Peter D. Grünwald. That simple device already used by Gauss. In P. D. Grünwald, P. Myllymäki, I. Tabus, M. Weinberger, and B. Yu, editors, Festschrift in Honor of Jorma Rissanen on the Occasion of his 75th Birthday, pages 293-304. Tampere University Press, Tampere, Finland, 2008.
27. Peter D. Grünwald. Safe learning: Bridging the gap between Bayes, MDL and statistical learning theory via empirical convexity. In Proceedings of the 24th Conference on Learning Theory, pages 397-419, 2011.
28. Peter D. Grünwald. The safe Bayesian. In Proceedings of the 23rd International Conference on Algorithmic Learning Theory (ALT 2012), pages 169-183. Springer, 2012.
29. Peter D. Grünwald and A. Philip Dawid. Game theory, maximum entropy, minimum discrepancy and robust bayesian decision theory. The Annals of Statistics, 32(4):1367-1433, 2004.
30. Peter D. Grünwald and John Langford. Suboptimality of MDL and Bayes in classification under misspecification. In Proceedings of the 17th Annual Conference on Learning Theory (COLT 2004), New York, 2004. Springer-Verlag.
31. Peter D. Grünwald and Thijs van Ommen. Inconsistency of Bayesian inference for misspecified linear models, and a proposal for repairing it. arXiv preprint arXiv:1412.3730, 2014.
32. Elad Hazan, Amit Agarwal, and Satyen Kale. Logarithmic regret algorithms for online convex optimization. Machine Learning, 69(2-3):169-192, 2007.
33. Elad Hazan, Alexander Rakhlin, and Peter L. Bartlett. Adaptive online gradient descent. In J. C. Platt, D. Koller, Y. Singer, and S. T. Roweis, editors, Advances in Neural Information Processing Systems (NIPS) 20, pages 65-72, 2008.
34. Anatoli Juditsky, Philippe Rigollet, and Alexandre B. Tsybakov. Learning by mirror averaging. The Annals of Statistics, 36(5):2183-2206, 2008.
35. Parmeswaran Kamalaruban, Robert C. Williamson, and Xinhua Zhang. Exp-Concavity of Proper Composite Losses. In JMLR Workshop and Conference Proceedings (Proceedings COLT 2015), volume 40, 2015.
36. Samuel Karlin and William J. Studden. Tchebycheff Systems: With Applications in Analysis and Statistics. Interscience Publishers, 1966.
37. Johannes H. B. Kemperman. The general moment problem, a geometric approach. The Annals of Mathematical Statistics, 39(1):93-122, 1968.
38. Johannes H. B. Kemperman. Geometry of the moment problem. In Proceedings of Synmposia in Applied Mathematics, volume 37, pages 16-53, 1987.
39. Jyrki Kivinen and Manfred Warmuth. Averaging expert predictions. In Proceedings of the Annual Conference on Learning Theory (COLT), pages 153-167, 1999.
40. Bas J. K. Kleijn and Aad W. van der Vaart. Misspecification in infinite-dimensional Bayesian statistics. The Annals of Statistics, 34(2):837-877, 2006.
41. Vladimir Koltchinskii. Local Rademacher complexities and oracle inequalities in risk minimization. The Annals of Statistics, 34(6):2593-2656, 2006.
42. Wouter M. Koolen, Tim van Erven, and Peter D. Grünwald. Learning the learning rate for prediction with expert advice. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27(NIPS), pages 2294-2302, 2014.
43. Guillaume Lecué. Interplay between concentration, complexity and geometry in learning theory with applications to high dimensional data analysis. Habilitation à diriger des recherches, Université Paris-Est, 2011.
44. Wee Sun Lee, Peter L Bartlett, and Robert C Williamson. Efficient agnostic learning of neural networks with bounded fan-in. IEEE Transactions on Information Theory, 42(6):2118-2132, 1996.
45. Wee Sun Lee, Peter L. Bartlett, and Robert C. Williamson. The importance of convexity in learning with squared loss. IEEE Transactions on Information Theory, 44(5):1974-1980, 1998.
46. Correction 54(9), 4395 (2008).
47. Jonathan Qiang Li. Estimation of mixture models. PhD thesis, Yale University, 1999.
48. Nishant A. Mehta and Robert C. Williamson. From stochastic mixability to fast rates. In Advances in Neural Information Processing Systems, pages 1197-1205, 2014.
49. Shahar Mendelson. Lower bounds for the empirical minimization algorithm. IEEE Transactions on Information Theory, 54(8):3797-3803, 2008a.
50. Shahar Mendelson. Obtaining fast error rates in nonconvex situations. Journal of Complexity, 24(3):380-397, 2008b.
51. Shahar Mendelson. Learning without concentration. In Proceedings of The 27th Conference on Learning Theory, pages 25-39, 2014.
52. Shahar Mendelson and Robert C. Williamson. Agnostic learning of nonconvex function classes. In Proceedings of the 15th Annual Conference on Computational Learning Theory (COLT 2002), pages 1-13. Springer, 2002.
53. Yuri V. Prokhorov. Convergence of random processes and limit theorems in probability theory. Theory of Probability and Its Applications, I (2): 157-214, 1956.
54. R. V. Ramamoorthi, Karthik Sriram, and Ryan Martin. On posterior concentration in misspecified models. arXiv preprint arXiv:1312.4620, 2013.
55. Hans Richter. Parameterfreie abschätzung und realisierung von erwartungswerten. Blätter der DGVFM, 3(2):147-162, 1957.
56. Shai Shalev-Shwartz and Yoram Singer. Logarithmic regret algorithms for strongly convex repeated games. Technical report, The Hebrew University, 2007.
57. Claude E. Shannon. Bounds on the tails of martingales and related questions (seminar notes on information theory, Massachusetts Institute of Technology). In Neil J. A. Sloane and Aaron D. Wyner, editors, Claude Elwood Shannon Miscellaneous Writings, pages 621-639. Mathematical Sciences Research Centre, AT&T Bell Laboratories, 1956. URL https://archive.org/details/ShannonMiscellaneousWritings.
58. Albert N. Shiryaev. Probability. Springer-Verlag, New York, 1996.
59. Alexander B. Tsybakov. Optimal aggregation of classifiers in statistical learning. The Annals of Statistics, 32(1):135-166, 2004.
60. Ruth Urner and Shai Ben-David. The sample complexity of agnostic learning under deterministic labels. In Proceedings of the 27th Annual Conference on Learning Theory (COLT 2014), 2014.
61. Aad W. Van der Vaart and Jon A. Wellner. Weak Convergence and Empirical Processes. Springer, 1996.
62. Tim van Erven. From exp-concavity to mixability. Tim van Erven's Blog, 2012.
63. Tim van Erven, Peter D. Grünwald, Mark D. Reid, and Robert C. Williamson. Mixability in statistical learning. In Advances in Neural Information Processing Systems 25(NIPS 2012), pages 1700-1708, 2012a.
64. Tim van Erven, Mark D. Reid, and Robert C. Williamson. Mixability is Bayes risk curvature relative to log loss. Journal of Machine Learning Research, 13:1639-1663, 2012b.
65. Vladimir N. Vapnik. Statistical Learning Theory. John Wiley and Sons, 1998.
66. Vladimir N. Vapnik and Alexey Ya. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and its Applications, 16(2):264-280, 1971.
67. Vladimir N. Vapnik and Alexey Ya. Chervonenkis. Theory of Pattern Recognition (in Russian). Nauka, Moscow, 1974. German translation: Theorie der Zeichenerkennung, Akademie Verlag, Berlin, 1979.
68. Vladimir N. Vapnik and Alexey Ya. Chervonenkis. Necessary and sufficient conditions for the uniform convergence of means to their expectations. Theory of Probability and its Applications, 26(3):532-553, 1981.
69. Vladimir N. Vapnik and Alexey Ya. Chervonenkis. The necessary and sufficient conditions for consistency of the method of empirical risk minimization. Pattern Recognition and Image Analysis, 1(3):284-305, 1991.
70. Elodie Vernet, Mark D. Reid, and Robert C. Williamson. Composite multiclass losses. In Advances in Neural Information Processing Systems, pages 1224-1232, 2011.
71. Mathukumalli Vidyasagar. Learning and Generalization with Applications to Neural Networks. Springer, 2002.
72. Vladimir Vovk. Aggregating strategies. In Proceedings of the third annual workshop on Computational learning theory, pages 371-383. Morgan Kaufmann Publishers Inc., 1990.
73. Vladimir Vovk. A game of prediction with expert advice. Journal of Computer and System Sciences, 56(2):153-173, 1998.
74. Vladimir Vovk. Competitive on-line statistics. International Statistical Review, 69(2):213-248, 2001.
75. Vladimir Vovk and Fedor Zhdanov. Prediction with expert advice for the Brier game. Journal of Machine Learning Research, 10:2445-2471, 2009.
76. Yuhong Yang and Andrew R. Barron. Information-theoretic determination of minimax rates of convergence. The Annals of Statistics, 27(5):1564-1599, 1999.
77. Tong Zhang. From ε-entropy to KL-entropy: Analysis of minimum information complexity density estimation. The Annals of Statistics, 34(5):2180-2210, 2006a.
78. Tong Zhang. Information-theoretic upper and lower bounds for statistical estimation. IEEE Transactions on Information Theory, 52(4):1307-1321, 2006b.