Structural risk minimization over data-dependent hierarchies

IEEE Transactions on Information Theory

Volumn 44, Issue 5, 1998, Pages 1926-1940

  Shawe Taylor, John ^a,b   Bartlett, Peter L ^a,b   Williamson, Robert C ^a,b   Anthony, Martin ^a,b  

^a UNIVERSITY OF LONDON   (United Kingdom)

^b AUSTRALIAN NATIONAL UNIVERSITY   (Australia)

Author keywords

Computational learning theory, fat shattering dimension, learning machines, maximal margin, probable smooth luckiness, probably approximately correct learning, support vector machines, uniform convergence, vapnik chervonenkis dimension

Indexed keywords

COMPUTATIONAL METHODS; CONVERGENCE OF NUMERICAL METHODS; ERROR ANALYSIS; LEARNING ALGORITHMS;

STRUCTURAL RISK MINIMIZATION (SRM); VAPNIK-CHERVONENSKI DIMENSION;

ARTIFICIAL INTELLIGENCE;

EID: 0032166068     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/18.705570     Document Type: Article

References (51)

1. N. Alon, S. Ben-David, N. Cesa-Bianchi, and D. Haussler, "Scalesensitive dimensions, uniform convergence, and learnability," in Proc. Conf. Foundations of Computer Science (FOCS), 1993.
2. Also, J. Assoc. Comput. Mach, vol. 44, no.4, pp. 615-631, 1997.
3. M. Anthony and P. Bartlett (1995), "Function learning from interpolation," NeuroCOLT Tech. Rep. NC-TR-94-013 [Online] Available FTP: ftp.dcs.rhbnc.ac.uk/pub/neurocolt/tech_reports, (an extended abstract appeared in Computational Learning Theory, Proc. 2nd European Conf., EuroCOLT'95 (Lecture Notes in Artificial Intelligence, vol. 904, P. Vitanyi, Ed.) Berlin, Germany: Springer-Verlag, 1995, pp. 211-221.
4. M. Anthony, N. Biggs, and J. Shawe-Taylor, "The learnability of formal concepts," in Proc. 3rdAnnu. Workshop Computational Learning Theory (Rochester) San Mateo, CA: Morgan Kaufmann, 1990, pp. 246-257.
5. M. Anthony and J. Shawe-Taylor, "A result of Vapnik with applications," Discr. Appl Math., vol. 47, pp. 207-217, 1993.
6. _, "A sufficient condition for polynomial distribution-dependent learnability," Discr. Appl. Math., vol. 77, pp. 1-12, 1997.
7. A. R. Barron, "Approximation and estimation bounds for artificial neural networks," Mach. Learning, vol. 14, pp. 115-133, 1994.
8. _, "Complexity regularization with applications to artificial neural networks," in Nonparametric Functional Estimation and Related Topics, G. Roussas, Ed. Boston, MA: Kluwer, 1991, pp. 561-576.
9. A. R. Barron and T. M. Cover, "Minimum complexity density estimation," IEEE Trans. Inform. Theory, vol. 37, pp. 1034-1054, 1738, 1991.
10. P. L. Bartlett, "The sample complexity of pattern classification with neural networks: The size of the weights is more important than the size of the network," IEEE Trans. Inform. Theory, 1998, to be published.
11. P. L. Bartlett and P. M. Long, "Prediction, learning, uniform convergence, and scale-sensitive dimensions," submitted to J. Comp. Syst. Scie. 1998.
12. P. L. Bartlett, P. M. Long, and R. C. Williamson, "Fat-shattering and the learnability of real-valued functions," J. Comp. Syst. Sci., vol. 52, no. 3, pp. 434-452, 1996.
13. G. M. Benedek and A. Itai, "Nonuniform learnability," J. Comp. Syst. Scie., vol. 48, pp. 311-323, 1994.
14. M. Biehl and M. Opper, "Perceptron learning: The largest version space," in Neural Networks: The Statistical Mechanics Perspective, Proc. CTP-PBSRI Workshop on Theoretical /Vrysics.World Scientific, also available at: http://brain.postech.ac.kr/nnsmp/compressed/biehl.ps.Z-l-.
15. B. E. Böser, I. M. Guyon, and V. N. Vapnik, "A training algorithm for optimal margin classifiers," in Proc. 5th Anna. Workshop Computational Learning Theory. Pittsburgh, PA: Assoc. Comput. Mach., 1992, pp. 144-152.
16. K. L. Buescher and P. R. Kumar, "Learning by canonical smooth estimation, Part I: Simultaneous estimation," IEEE Trans. Automat. Contr., vol. 41, no. 4, p. 545, 1996.
17. C. Cortes and V. Vapnik, "Support-vector networks," Mach. Learning, vol. 20, pp. 273-297, 1995.
18. T. M. Cover and J. Thomas, Elements of Information Theory. New York: Wiley, 1994.
19. R. O. Duda and P. E. Hart, Pattern Classification and Scene Analysis. New York: Wiley, 1973.
20. S. Floyd and M. Warmuth, "Sample compression, learnability, and the Vapnik-Chervonenkis Dimension," Mach. Learning, vol. 21, pp. 269-304, 1995.
21. F. Girosi, M. Jones, and T. Poggio, "Regularization theory and neural networks architecture," Neural Comp., vol. 7, pp. 219-269, 1995.
22. L. Gurvits and P. Koiran, "Approximation and learning of convex superpositions," in Computational Learning Theory, Proc. 2nd European Conf., EuroCOLT'95 (Lecture Notes in Artificial Intelligence, vol. 904, P. Vitânyi, Ed.) Berlin, Germany: Springer-Verlag, 1995, pp. 222-236.
23. I. Guyon, V. N. Vapnik, B. E. Böser, L. Bottou, and S. A. Sofia, "Structural risk minimization for character recognition," in Advances in Neural Information Processing Systems 4, J. E. Moody et al., Eds. San Mateo, CA: Morgan Kaufmann, 1992, pp. 471-479.
24. M. H. Hassoun, Fundamentals of Artificial Neural Networks. Cambridge, MA: MIT Press, 1995.
25. D. Haussler, "Decision theoretic generalizations of the PAC model for neural net and other learning applications," Inform. Comp., vol. 100, pp. 78-150, 1992.
26. D. Home, The Lucky Country: Australia in the Sixties. Victoria: Penguin Books, Ringwood, 1964.
27. M. J. Kearns and R. E. Schapire, "Efficient distribution-free learning of probabilistic concepts," in Proc. 31st Symp. Foundations of Computer Science. Los Alamitos, CA: IEEE Computer Soc., 1990, pp. 382-391.
28. P. Koiran and E. D. Sontag, "Neural networks with quadratic VC dimension," in Neural Information Processing Systems 7 (NIPS95) and to appear in J. Comp. Syst. Sci.; also available as a NeuroCOLT Tech. Rep. NC-TR-95-044 [Online]: Available FTP: ftp.dcs.rhbnc.ac.uk/pub/neurocolt/tech_reports.
29. P. R. Kumar and K. L. Buescher, "Learning by canonical smooth estimation, Part 2: Learning and choice of model complexity," IEEE Trans. Automat. Contr., vol. 41, no. 4, p. 557, 1996.
30. N. Linial, Y. Mansour, and R. L. Rivest, "Results on learnability and the Vapnik-Chervonenkis dimension," Inform. Comp., vol. 90, pp. 33-49, 1991.
31. N. Littlestone, "Learning quickly when irrelevant attributes abound: A new linear threshold algorithm," Mach. Learning, vol. 2, pp. 285-318, 1988.
32. _, "Mistake-driven bayes sports: Bounds for symmetric Apobayesian learning algorithms," Tech. Rep., NEC Res. Ctr.,1996.
33. N. Littlestone and C. Mesterham, "An Apobayesian relative of winnow," NEC Res. Ctr., 1996, preprint.
34. N. Littlestone and M. Warmuth, "Relating data compression and learnability," Univ. Calif., Santa Cruz, 1986, unpublished manuscript.
35. L. Ljung, System Identification: Theory for the User. Upper Saddle River, NJ: Prentice-Hall PTR, 1987.
36. G. Lugosi and A. B. Nobel, "Adaptive model selection using empirical complexities," Dept. Math. Comp. Sci., Tech. Univ. Budapest, Budapest, Hungary, 1996, preprint.
37. G. Lugosi and M. Pintér, "A data-dependent skeleton estimate for learning," in Proc. 9th Annu. Workshop Computational Learning Theory. New York: Assoc. Comput. Mach., 1996, pp. 51-56.
38. G. Lugosi and K. Zeger, "Nonparametric estimation via empirical risk minimization," IEEE Trans. Inform. Theory, vol. 41, pp. 677-687, May 1995.
39. _, "Concept learning using complexity regularization," IEEE Trans. Inform. Theory, vol. 42, pp. 48-54, Jan. 1996.
40. D. J. C. MacKay, "Bayesian model comparison and backprop nets," in Adv. Neural Inform. Processing Syst. 4, J. E. Moody et al., Eds. San Mateo, CA: Morgan Kaufmann, 1992, pp. 839-846.
41. _, "Probable networks and plausible predictions-A review of practical Bayesian methods for supervised neural networks," Cavendish Laboratory, Cambridge Univ., Cambridge, U.K., 1996, preprint.
42. D. Pollard, Convergence of Stochastic Processes. New York: Springer, 1984.
43. J. Shawe-Taylor, M. Anthony, and N. Biggs, "Bounding sample size with the Vapnik-Chervonenkis dimension," Discr. Appl. Math., vol. 42, pp. 65-73, 1993.
44. J. Shawe-Taylor, P. Bartlett, R. Williamson, and M. Anthony, "A framework for structural risk minimization," in Proc. 9th Annu. Conf. Computational Learning Theory. New York: Assoc. Comput. Mach., 1996, pp. 68-76.
45. E. D. Sontag, "Shattering all sets of k points in 'general position' requires (k -1)/2 parameters," Rutgers Center for Systems and Control (SYCON), Rep. 96-01; also NeuroCOLT Tech. Rep. NC-TR-96-042. [Online] Available FTP: ftp.dcs.rhbnc.ac.uk/pub/neurocolt/tech_reports).
46. A. W. van der Vaart and J. A. Wellner, Weak Convergence and Empirical Processes. New York: Springer-Verlag, 1996.
47. V. N. Vapnik, Estimation of Dependences Based on Empirical Data. New York: Springer-Verlag, 1982.
48. _, "Principles of risk minimization for learning theory," in Advances in Neural Information Processing Systems 4, J. E. Moody et al., Eds. San Mateo, CA: Morgan Kaufmann, 1992, pp. 831-838.
49. _, The Nature of Statistical Learning Theory. New York: Springer-Verlag, 1995.
50. V. N. Vapnik and A. J. Chervonenkis, "On the uniform convergence of relative frequencies of events to their probabilities," Theory of Probability and Applications, vol. 16, pp. 264-280, 1971.
51. _, "Ordered risk minimization (I and II)," Automat. Remote Contr., vol. 34, pp. 1226-1235 and 1403-1412, 1974.