Convexity, classification, and risk bounds

Journal of the American Statistical Association

Volumn 101, Issue 473, 2006, Pages 138-156

  Bartlett, Peter L ^a   Jordan, Michael I ^a   McAuliffe, Jon D ^b,c  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

Boosting; Convex optimization; Empirical process theory; Machine learning; Rademacher complexity; Support vector machine

Indexed keywords

EID: 33645505792     PISSN: 01621459     EISSN: None     Source Type: Journal    
DOI: 10.1198/016214505000000907     Document Type: Review

References (53)

1. Arora, S., Babai, L., Stern, J., and Sweedyk, Z. (1997), "The Hardness of Approximate Optima in Lattices, Codes, and Systems of Linear Equations," Journal of Computer and System Sciences, 54, 317-331.
2. Bartlett, P. L. (1998), "The Sample Complexity of Pattern Classification With Neural Networks: The Size of the Weights Is More Important Than the Size of the Network," IEEE Transactions on Information Theory, 44, 525-536.
3. Bartlett, P. L., Bousquet, O., and Mendelson, S. (2005), "Local Rademacher Complexities," The Annals of Statistics, 33, 1497-1537.
4. Bartlett, P. L., and Mendelson, S. (2005), "Empirical Minimization," Probability Theory and Related Fields, to appear.
5. Boser, B. E., Guyon, I. M., and Vapnik, V. N. (1992), "A Training Algorithm for Optimal Margin Classifiers," in Proceedings of the 5th Annual Workshop on Computational Learning Theory, New York: ACM Press, pp. 144-152.
6. Bousquet, O. (2002), "A Bennett Concentration Inequality and Its Application to Suprema of Empirical Processes," Comptes Rendus de l'Académie des Sciences, Série I, 334, 495-500.
7. Boyd, S., and Vandenberghe, L. (2004), Convex Optimization, Cambridge, U.K.: Cambridge University Press.
8. Breiman, L. (1999), "Prediction Games and Arcing Algorithms," Neural Computation, 11, 1493-1517.
9. _ (2004), "Population Theory for Boosting Ensembles," The Annals of Statistics, 32, 1-11.
10. Brown, L. D. (1986), Fundamentals of Statistical Exponential Families, Hayward, CA: Institute of Mathematical Statistics.
11. Collins, M., Schapire, R. E., and Singer, Y. (2002), "Logistic Regression, Adaboost and Bregman Distances," Machine Learning, 48, 253-285.
12. Cortes, C., and Vapnik, V. (1995), "Support-Vector Networks," Machine Learning, 20, 273-297.
13. Cristianini, N., and Shawe-Taylor, J. (2000), An Introduction to Support Vector Methods, Cambridge, U.K.: Cambridge University Press.
14. Devroye, L., Györfi, L., and Lugosi, G. (1996), A Probabilistic Theory of Pattern Recognition, New York: Springer-Verlag.
15. Dudley, R. M. (1999), Uniform Central Limit Theorems, Cambridge, U.K.: Cambridge University Press.
16. Feder, M., Figueiredo, M. A. T., Hero, A. O., Lee, C.-H., Loeliger, H.-A., Nowak, R., Singer, A. C., and Yu, B. (2004), "Special Issue on Machine Learning Methods in Signal Processing," IEEE Transactions on Signal Processing, 52, 2152.
17. Freund, Y., and Schapire, R. E. (1997), "A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting," Journal of Computer and System Sciences, 55, 119-139.
18. Friedman, J., Hastie, T., and Tibshirani, R. (2000), "Additive Logistic Regression: A Statistical View of Boosting," The Annals of Statistics, 28, 337-374.
19. Jiang, W. (2004), "Process Consistency for Adaboost," The Annals of Statistics, 32, 13-29.
20. Joachims, T. (2002), Learning to Classify Text Using Support Vector Machines, Dordrecht: Kluwer Academic.
21. Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., and Saul, L. K. (1999), "Introduction to Variational Methods for Graphical Models," Machine Learning, 37, 183-233.
22. Klein, T. (2002), "Une Inégalité de Concentration à Gauche Pour les Processus Empiriques" [A Left Concentration Inequality for Empirical Processes], Comptes Rendus de l'Académie des Sciences, Série I, 334, 501-504.
23. Koltchinskii, V. I., and Panchenko, D. (2000), "Rademacher Processes and Bounding the Risk of Function Learning," in High Dimensional Probability II, Vol. 47, eds. E. Giné, D. M. Mason, and J. A. Wellner, Boston, MA: Birkhäuser, pp. 443-459.
24. _ (2002), "Empirical Margin Distributions and Bounding the Generalization Error of Combined Classifiers," The Annals of Statistics, 30, 1-50.
25. Lebanon, G., and Lafferty, J. (2002), "Boosting and Maximum Likelihood for Exponential Models," in Advances in Neural Information Processing Systems 14, eds. T. Dietterich, S. Becker, and Z. Ghahramani, Cambridge, MA: MIT Press, pp. 447-454.
26. Ledoux, M. (2001), The Concentration of Measure Phenomenon, Providence, RI: American Mathematical Society.
27. Ledoux, M., and Talagrand, M. (1991), Probability in Banach Spaces: Isoperimetry and Processes, New York: Springer-Verlag.
28. Lee, W. S., Bartlett, P. L., and Williamson, R. C. (1996), "Efficient Agnostic Learning of Neural Networks With Bounded Fan-in," IEEE Transactions on Information Theory, 42, 2118-2132.
29. Lin, Y. (2004), "A Note on Margin-Based Loss Functions in Classification," Statistics and Probability Letters, 68, 73-82.
30. Lugosi, G., and Vayatis, N. (2004), "On the Bayes Risk Consistency of Regularized Boosting Methods," The Annals of Statistics, 32, 30-55.
31. Lugosi, G., and Wegkamp, M. (2004), "Complexity Regularization via Localized Random Penalties," The Annals of Statistics, 32, 1679-1697.
32. Mammen, E., and Tsybakov, A. B. (1999), "Smooth Discrimination Analysis," The Annals of Statistics, 27, 1808-1829.
33. Mannor, S., and Meir, R. (2001), "Geometric Bounds for Generalization in Boosting," in Proceedings of the Fourteenth Annual Conference on Computational Learning Theory, eds. D. Helmbold and R. Williamson, Springer-Verlag, pp. 461-472.
34. Mannor, S., Meir, R., and Zhang, T. (2002), "The Consistency of Greedy Algorithms for Classification," in Proceedings of the Annual Conference on Computational Learning Theory, eds. J. Kivinen and R. Sloan, Springer-Verlag, pp. 319-333.
35. Marron, J. S., and Todd, M. (2002), "Distance-Weighted Discrimination," Technical Report 1339, Cornell University, School of Operations Research and Industrial Engineering.
36. Massart, P. (2000a), "About the Constants in Talagrand's Concentration Inequality for Empirical Processes," The Annals of Probability, 28, 863-884.
37. _ (2000b), "Some Applications of Concentration Inequalities to Statistics," Annales de la Faculté des Sciences de Toulouse, IX, 245-303.
38. Mendelson, S. (2002), "Improving the Sample Complexity Using Global Data," IEEE Transactions on Information Theory, 48, 1977-1991.
39. Nesterov, Y., and Nemirovskii, A. (1994), Interior-Point Polynomial Algorithms in Convex Programming, Philadelphia: SIAM Publications.
40. Rio, E. (2001), "Inégalités de Concentration Pour les Processus Empiriques de Classes de Parties" [Concentration Inequalities for Set-Indexed Empirical Processes], Probability Theory and Related Fields, 119, 163-175.
41. Rockafellar, R. T. (1997), Convex Analysis, Princeton, NJ: Princeton University Press.
42. Schapire, R. E., Freund, Y., Bartlett, P., and Lee, W. S. (1998), "Boosting the Margin: A New Explanation for the Effectiveness of Voting Methods," The Annals of Statistics, 26, 1651-1686.
43. Schölkopf, B., and Smola, A. (2002), Learning With Kernels, Cambridge, MA: MIT Press.
44. Schölkopf, B., Tsuda, K., and Vert, J.-P. (2003), Kernel Methods in Computational Biology, Cambridge, MA: MIT Press.
45. Shawe-Taylor, J., Bartlett, P. L., Williamson, R. C., and Anthony, M. (1998), "Structural Risk Minimization Over Data-Dependent Hierarchies," IEEE Transactions on Information Theory, 44, 1926-1940.
46. Shen, X., Tseng, G. C., Zhang, X., and Wong, W. H. (2003), "On Psi-Learning," Journal of the American Statistical Association, 98, 724-734.
47. Steinwart, I. (2005), "Consistency of Support Vector Machines and Other Regularized Kernel Classifiers," IEEE Transactions on Information Theory, 51, 128-142.
48. Talagrand, M. (1994), "Sharper Bounds for Gaussian and Empirical Processes," The Annals of Probability, 22, 28-76.
49. Tsybakov, A. (2004), "Optimal Aggregation of Classifiers in Statistical Learning," The Annals of Statistics, 32, 135-166.
50. Van der Vaart, A. W., and Wellner, J. A. (1996), Weak Convergence and Empirical Processes, New York: Springer-Verlag.
51. Zhang, T. (2003), "Sequential Greedy Approximation for Certain Convex Optimization Problems," IEEE Transactions on Information Theory, 49, 682-691.
52. _ (2004), "Statistical Behavior and Consistency of Classification Methods Based on Convex Risk Minimization," The Annals of Statistics, 32, 56-85.
53. Zhang, T., and Yu, B. (2005), "Boosting With Early Stopping: Convergence and Consistency," The Annals of Statistics, 33, 1538-1579.