Improved lower bounds for learning from noisy examples: An information-theoretic approach

Information and Computation

Volumn 166, Issue 2, 2001, Pages 133-155

  Gentile, Claudio ^a   Helmbold, David P ^b  

^a UNIVERSITY OF MILAN   (Italy)

^b University of California Santa Cruz   (United States)

Author keywords

Entropy; Lower bounds; Mutual information; Noisy examples; PAC learning

Indexed keywords

ALGORITHMS; COMPUTER SIMULATION; ENTROPY; THEOREM PROVING;

PROBABLY APPROXIMATELY CORRECT (PAC);

LEARNING SYSTEMS;

EID: 0035354041     PISSN: 08905401     EISSN: None     Source Type: Journal    
DOI: 10.1006/inco.2000.2919     Document Type: Article

References (40)

1. Angluin, D. (1987), Learning regular sets from queries and counterexamples, Inform. and Comput. 75(2), 87-106.
2. Angluin, D. (1988), Queries and concept learning, Machine Learning 2(4), 319-342.
3. Angluin D., Krikis, M., Sloan, R., and Turán, G. (1997), Malicious omissions and errors in answers to membership queries, Machine Learning 28(2/3), 211-255.
4. Angluin, D., and Laird, P. D. (1988), Learning from noisy examples, Machine Learning 2(2), 343-370.
5. Angluin, D., and Valiant, L. (1979), Fast probabilistic algorithms for Hamiltonian circuits and matchings, J. Comput. System Sci. 18, 155-193.
6. Apolloni, B., and Gentile, C. (1998), Sample size lower bounds in PAC learning by algorithmic complexity theory, Theoret. Comput. Sci. 209, 141-162.
7. Aslam, J. A., and Decatur S. E. (1996), On the sample complexity of noise-tolerant learning, Inform. Process. Lett. 57, 189-195.
8. Bartlett, P. (1992), Learning with a slowly changing distribution, in "Proceedings, 5th Workshop on Computational Learning Theory," pp. 243-252.
9. Bartlett, P., and Helmbold, D. (1998), "Tracking a Drifting Concept in a Changing Environment," Technical Report UCSC-CRL-98-12, Baskin School of Engineering, University of California, Santa Cruz.
10. Barve, R. D., and Long, P. M. (1997), On the complexity of learning from drifting distributions, Inform. and Comput. 138 (2), 170-193.
11. Benedek, G., and Itai, A. (1991), Learnability by fixed distributions, Theoret. Comput. Sci. 86(2), 377-389.
12. Blumer, A., Ehrenfeucht, A., Haussler, D., and Warmuth, M. (1989), Learnability and the Vapnik-Chervonenkis dimension, J. Assoc. Comput. Mach. 36, 929-965.
13. Cesa-Bianchi, N., Dichterman, E., Fischer, P., Shamir, E., and Simon, H. U. (1997), Sample-efficient strategies for learning in the presence of noise, eCOLT Teca. Rep. 97-003 [www: http://ecolt.informatik.uni-dortmund.de/]; preliminary versions appeared in "Proceedings, 28th Symposium on Theory of Computing," pp. 141-150, and "Proceedings, 3rd European Conference on Computational Learning Theory," pp. 119-133.
14. Cesa-Bianchi, N., Dichterman, E., Fischer, P., Shamir, E., and Simon, H. U. (1997), Sample-efficient strategies for learning in the presence of noise, eCOLT Teca. Rep. 97-003 [www: http://ecolt.informatik.uni-dortmund.de/]; preliminary versions appeared in "Proceedings, 28th Symposium on Theory of Computing," pp. 141-150, and "Proceedings, 3rd European Conference on Computational Learning Theory," pp. 119-133.
15. Cover, T. M., and Thomas, J. A. (1991), "Elements of Information Theory", Wiley, New York.
16. Dobrushin, R. L. (1959), General formulation of Shannon's main theorem in information theory, Uspekhi Mat. Nauk, 14, 3-104. [Russian]; (1963), Ameri. Math. Soc. Transl. Ser. 2 33, 323-438 [Engl. transl.].
17. Dobrushin, R. L. (1959), General formulation of Shannon's main theorem in information theory, Uspekhi Mat. Nauk, 14, 3-104. [Russian]; (1963), Ameri. Math. Soc. Transl. Ser. 2 33, 323-438 [Engl. transl.].
18. Dudley, R. M. (1984), "A Course on Empirical Processes," Lecture Notes in Mathematics, Vol. 1097, Springer-Verlag, Berlin/New York.
19. Ehrenfeucht, A., Haussler D., Kearns, M., and Valiant, L. (1989), A general lower bound on the number of examples needed for learning, Inform. and Comput. 82(3), 247-261.
20. Eisenberg, B., and Rivest, R. L. (1990), On the sample complexity of Pac-learning using random and chosen examples, in "Proceedings, 3th Workshop on Computational Learning Theory," pp. 154-162.
21. Gentile, C. (1997), A note on sample size lower bounds for PAC-learning, manuscript.
22. Haussler, D., and Barron, A. (1992), How well do Bayes' methods work for on-line prediction of {-1, +1} values?, in "Proceedings, 3rd NEC Symposium on Computation and Cognition," pp. 74-100.
23. Haussler, D., Kearns, M., and Schapire, R. (1994), Bounds on the sample complexity of Bayesian learning using information theory and the VC dimension, Machine Learning 14, 84-114.
24. Haussler, D., Littlestone, N., and Warmuth, M. (1994), Predicting {0, 1}-functions on randomly drawn points, Inform. and Comput. 115(2), 248-293.
25. Haussler, D., and Opper, M. (1997), Mutual information, metric entropy, and cumulative relative entropy risk, Ann. Statist. 25(6), 2451-2492.
26. Helmbold, D., Littlestone, N., and Long, P. (2000), Apple tasting, Inform. and Comput., to appear.
27. Ihara, S. (1993), "Information Theory for Continuous Systems," World Scientific, River Edge, NJ.
28. Kearns, M., and Li, M. (1993), Learning in the presence of malicious errors, SIAM J. Comput. 22, 807-837.
29. Kolmogorov, A. N., and Tihomirov, V. M. (1961), ε-entropy and ε-capacity of sets in functional spaces, Amer. Math. Soc. Transl. Ser. 17, 277-364.
30. Laird, P. (1988), "Learning from Good and Bad Data," Kluwer International Series in Engineering and Computer Science, Kluwer Academic, Boston, MA.
31. Maass, W., and Turán, G. (1990), On the complexity of learning from counterexamples and membership queries, in "Proceedings, 31th Symposium on the Foundations of Computer Science," pp. 203-210.
32. Sakakibara, K. (1991), On learning from queries and counterexamples in the presence of noise, Inform. Process. Lett. 37(5), 279-284.
33. Simon, H. U. (1996), General bounds on the number of examples needed for learning probabilistic concepts, J. Comput. System Sci. 52(2), 239-254.
34. Sloan, R. (1995), Four types of noise in data for PAC learning, Inform. Process. Lett. 54, 157-162.
35. Shackelford, G., and Volper, D. (1988), Learning k-DNF with noise in the attributes, in "Proceedings, 1988 Workshop on Computational Learning Theory," pp. 97-103.
36. Turán, G. (1993), Lower bounds for PAC learning with queries, in "Proceedings, 6th Workshop on Computational Learning Theory," pp. 384-391.
37. Valiant, L. (1984), A theory of the learnable, Comm. Assoc. Comput. Math. 27(11), 1134-1142.
38. Vapnik, V. N. (1982), "Estimation of Dependencies Based on Empirical Data," Springer-Verlag, New York.
39. Vapnik, V. N. (1995), "The Nature of Statistical Learning Theory," Springer-Verlag, New York.
40. Yu, B. (1996), Lower bounds on expected redundancy for nonparametric classes, IEEE Trans. Inform. Theory 42(1), 272-275.