Global Rademacher Complexity Bounds: From Slow to Fast Convergence Rates

Neural Processing Letters

Volumn 43, Issue 2, 2016, Pages 567-602

  Oneto, Luca ^a   Ghio, Alessandro ^a   Ridella, Sandro ^a   Anguita, Davide ^a  

^a UNIVERSITY OF GENOA   (Italy)

Author keywords

Fast rates; Performance estimation; Rademacher complexity; Statistical learning theory

Indexed keywords

ARTIFICIAL INTELLIGENCE; SOFTWARE ENGINEERING;

CONCENTRATION INEQUALITY; FAST CONVERGENCE RATE; FAST RATE; LEARNING PROCEDURES; PERFORMANCE ESTIMATION; RADEMACHER COMPLEXITY; SEMI- SUPERVISED LEARNING; STATISTICAL LEARNING THEORY;

SUPERVISED LEARNING;

EID: 84929104006     PISSN: 13704621     EISSN: 1573773X     Source Type: Journal    
DOI: 10.1007/s11063-015-9429-2     Document Type: Article

References (72)

1. Abney SP (2008) Semisupervised learning in computational linguistics. CRC Press, Boca Raton
2. Adamczak R (2007) A tail inequality for suprema of unbounded empirical processes with applications to markov chains. Electron J Probab 13:1000–1034
3. Anava O, Hazan E, Mannor S, Shamir O (2013) Online learning for time series prediction. In: The 26th annual conference on learning theory, Princeton University, NJ, USA
4. Anguita D, Ghio A, Oneto L, Ridella S (2011) The impact of unlabeled patterns in rademacher complexity theory for kernel classifiers. In: Proceedings of neural information processing systems
5. Anguita D, Ghio A, Oneto L, Ridella S (2012) In-sample and out-of-sample model selection and error estimation for support vector machines. IEEE Trans Neural Netw Learn Syst 23(9):1390–1406
6. Anguita D, Ghio A, Oneto L, Ridella S (2013) A learning machine with a bit-based hypothesis space. In: European symposium on artificial neural networks
7. Anguita D, Ghio A, Oneto L, Ridella S (2014) Unlabeled patterns to tighten Rademacher complexity error bounds for kernel classifiers. Pattern Recognit Lett 37:210–219
8. Anguita D, Ghio A, Oneto L, Xavier P, Reyes-Ortiz JL (2013) Energy efficient smartphone-based activity recognition using fixed-point arithmetic. J Univers Comput Sci 19:1295–1314
9. Anguita D, Ghio A, Ridella S (2011) Maximal discrepancy for support vector machines. Neurocomputing 74(9):1436–1443
10. Anguita S, Ghio A, Oneto L, Ridella S (2011) Maximal discrepancy vs. rademacher complexity for error estimation. In: European symposium on artificial neural networks
11. Audibert JY (2009) Fast learning rates in statistical inference through aggregation. Ann Stat 37(4):1591–1646
12. Audibert JY, Tsybakov AB (2007) Fast learning rates for plug-in classifiers. Ann Stat 35(2):608–633
13. Bartlett PL, Boucheron S, Lugosi G (2002) Model selection and error estimation. Mach Learn 48(1–3):85–113
14. Bartlett PL, Bousquet O, Mendelson S (2002) Localized rademacher complexities. In: Proceedings of the 15th annual conference on computational learning theory
15. Bartlett PL, Bousquet O, Mendelson S (2005) Local rademacher complexities. Ann Stat 33(4):1497–1537
16. Bartlett PL, Mendelson S (2003) Rademacher and gaussian complexities: Risk bounds and structural results. J Mach Learn Res 3:463–482
17. Belkin M, Matveeva I, Niyogi P (2004) Regularization and semi-supervised learning on large graphs. In: Conference on learning theory
18. Belkin M, Niyogi P (2004) Semi-supervised learning on riemannian manifolds. Mach Learn 56(1–3):209–239
19. Ben-David S, Lu T, Pál D (2008) Does unlabeled data provably help? worst-case analysis of the sample complexity of semi-supervised learning. In: Conference on computational learning theory
20. Bennett K, Demiriz A (1999) Semi-supervised support vector machines. In: Advances in neural information processing systems 11: proceedings of the 1998 conference, p. 368. The MIT Press, Cambridge
21. Bentkus V (2008) An extension of the hoeffding inequality to unbounded random variables. Lith Math J 48(2):137–157
22. Blumer A, Ehrenfeucht A, Haussler D, Warmuth MK (1987) Occam’s razor. Inf Process Lett 24(6):377–380
23. Boucheron S, Lugosi G, Massart P (2000) A sharp concentration inequality with applications. Random Struct Algorithms 16(3):277–292
24. Bousquet O (2002) A bennett concentration inequality and its application to suprema of empirical processes. Comptes Rendus Math 334(6):495–500
25. Bousquet O, Elisseeff A (2002) Stability and generalization. J Mach Learn Res 2:499–526
26. Chapelle O, Schölkopf B, Zien A (2006) Semi-supervised learning. MIT press, Cambridge
27. Cherkassky V (2002) Model complexity control and statistical learning theory. Nat Comput 1(1):109–133
28. Cherkassky V, Mulier FM (2007) Learning from data: concepts, theory, and methods. Wiley, New York
29. Clopper CJ, Pearson ES (1934) The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26(4):404
30. Cohen I, Cozman FG, Sebe N, Cirelo MC, Huang TS (2004) Semisupervised learning of classifiers: theory, algorithms, and their application to human-computer interaction. IEEE Trans Pattern Anal Mach Intell 26(12):1553–1566
31. Corless RM, Gonnet GH, Hare DEG, Jeffrey DJ, Knuth DE (1996) On the Lambert w function. Adv Comput Math 5(1):329–359
32. Cortes C, Kloft M, Mohri M (2013) Learning kernels using local rademacher complexity. In: Conference on neural information processing systems
33. El-Yaniv R, Pechyony D (2009) Transductive rademacher complexity and its applications. J Artif Intell Res 35(1):193
34. Floyd S, Warmuth M (1995) Sample compression, learnability, and the vapnik-chervonenkis dimension. Mach Learn 21(3):269–304
35. Friedland L, Jensen D, Lavine M (2013) Copy or coincidence? a model for detecting social influence and duplication events. In: International conference on machine learning
36. Gopal S, Bai B, Yang Y, Niculescu-Mizil A (2013) Bayesian models for large-scale hierarchical classification. In: Neural information processing systems
37. Kääriäinen M (2005) Generalization error bounds using unlabeled data. In: Conference on computational learning theory
38. Katrenko S, Van Zaanen M (2010) Rademacher complexity and grammar induction algorithms: what it may (not) tell us. In: Grammatical inference: theoretical results and applications
39. Klein T (2002) Une inégalité de concentration à gauche pour les processus empiriques. Comptes Rendus Math 334(6):501–504
40. Klein T, Rio E (2005) Concentration around the mean for maxima of empirical processes. Ann Probab 33(3):1060–1077
41. Kloft M, Blanchard G (2011) The local rademacher complexity of lp-norm multiple kernel learning. In: Neural information processing systems
42. Koltchinskii V (2001) Rademacher penalties and structural risk minimization. IEEE Trans Inf Theory 47(5):1902–1914
43. Koltchinskii V (2006) Local rademacher complexities and oracle inequalities in risk minimization. Ann Stat 34(6):2593–2656
44. Kumar A, Saha A, Daume H (2010) Co-regularization based semi-supervised domain adaptation. In: Advances in neural information processing systems
45. Kutin S (2002) Extensions to mcdiarmid’s inequality when differences are bounded with high probability. In: Department Computer Science, University of Chicago, Chicago, IL. Technical report TR-2002-04
46. Langford J (2006) Tutorial on practical prediction theory for classification. J Mach Learn Res 6(1):273
47. Langford J, McAllester D (2000) Computable shell decomposition bounds. In: Conference on computational learning theory
48. Ledoux M, Talagrand M (2011) Probability in banach spaces: isoperimetry and processes. Springer, Berlin
49. Li M, Tromp J, Vitányi P (2002) Sharpening Occam razor. Springer, Berlin
50. Li M, Vitanyi P (1993) An introduction to Kolmogorov complexity and its applications. Springer, New York
51. Mammen E, Tsybakov AB (1999) Smooth discrimination analysis. Ann Stat 27(6):1808–1829
52. Massart P (2000) About the constants in talagrand’s concentration inequalities for empirical processes. Ann Probab 28(2):863–884 (English summary)
53. McAllester DA (1999) Pac-bayesian model averaging. In: Conference on computational learning theory
54. McDiarmid C (1989) On the method of bounded differences. Surv Comb 141(1):148–188
55. Oneto L, Ghio A, Anguita D, Ridella S (2013) An improved analysis of the rademacher data-dependent bound using its self bounding property. Neural Netw 44:107–111
56. Panchenko D (2002) Some extensions of an inequality of vapnik and chervonenkis. Electron Commun Probab 7:55–65
57. Parrado-Hernández E, Ambroladze A, Shawe-Taylor J, Sun S (2012) Pac-bayes bounds with data dependent priors. J Mach Learn Res 13:3507–3531
58. Parrondo JMR, Van den Broeck C (1993) Vapnik-chervonenkis bounds for generalization. J Phys A 26(9):2211
59. Paulo JL, Azzam FGT (2006) The use of artificial neural networks in decision support in cancer: a systematic review. Neural Netw 19(4):408–415
60. Poggio T, Rifkin R, Mukherjee S, Niyogi P (2004) General conditions for predictivity in learning theory. Nature 428(6981):419–422
61. Rosenberg DS, Bartlett PL (2007) The rademacher complexity of co-regularized kernel classes. In: International conference on artificial intelligence and statistics, pp 396–403
62. Shalev-Shwartz S, Shamir O, Srebro N, Sridharan K (2010) Learnability, stability and uniform convergence. J Mach Learn Res 11:2635–2670
63. Shao X, Cherkassky V, Li W (2000) Measuring the VC-dimension using optimized experimental design. Neural Comput 12(8):1969–1986
64. Shawe-Taylor J, Bartlett PL, Williamson RC, Anthony M (1998) Structural risk minimization over data-dependent hierarchies. IEEE Trans Inf Theory 44(5):1926–1940
65. Srebro N, Sridharan K, Tewari A (2010) Smoothness, low-noise and fast rates. In: Neural information processing systems
66. Statnikov A, Henaff M, Narendra V, Konganti K, Li Z, Yang L, Pei Z, Blaser MJ, Aliferis CF, Alekseyenko AV (2013) A comprehensive evaluation of multicategory classification methods for microbiomic data. Microbiome 1(1):11
67. Sun S, Shawe-Taylor J (2010) Sparse semi-supervised learning using conjugate functions. J Mach Learn Res 11:2423–2455
68. Talagrand M (1996) A new look at independence. Ann Probab 24(1):1–34
69. Vapnik VN (1998) Statistical learning theory. Wiley, New York
70. Vapnik VN, Červonenkis AJ (1979) Theorie der Zeichenerkennung. Akademie-Verlag, Berlin
71. Weisstein EW (2013) Lambert W-Function. From MathWorld-A Wolfram Web Resource. http://mathworld.wolfram.com/LambertW-Function.html
72. Zhu X, Goldberg AB (2009) Introduction to semi-supervised learning. Synth Lect Artif Intell Mach Learn 3(1):1–130