The Perceptron algorithm versus Winnow: Linear versus logarithmic mistake bounds when few input variables are relevant

Artificial Intelligence

Volumn 97, Issue 1-2, 1997, Pages 325-343

  Kivinen, J ^a   Warmuth, M K ^b   Auer, P ^c  

^a UNIVERSITY OF HELSINKI   (Finland)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c GRAZ UNIVERSITY OF TECHNOLOGY   (Austria)

Author keywords

Linear threshold functions; Mistake bounds; Multiplicative updates; Perceptron algorithm; Relevant variables

Indexed keywords

ALGORITHMS; COMPUTATIONAL COMPLEXITY; VECTORS;

LINEAR THRESHOLD FUNCTIONS; PERCEPTRON ALGORITHMS; RELEVANT VARIABLES;

NEURAL NETWORKS;

EID: 0031375503     PISSN: 00043702     EISSN: None     Source Type: Journal    
DOI: 10.1016/s0004-3702(97)00039-8     Document Type: Article

References (25)

1. P. Auer and M.K. Warmuth, Tracking the best disjunction, in: Proceedings 36th Symposium on the Foundations of Computer Science, Milwaukee, WI (IEEE Computer Society Press, Los Alamitos, CA, 1995) 312-321.
2. A. Blumer, A. Ehrenfeucht, D. Haussler and M.K. Warmuth, Learnability and the Vapnik-Chervonenkis dimension, J. ACM 36 (1989) 929-965.
3. N. Cesa-Bianchi, Y. Freund, D.P. Helmbold, D. Haussler, R.E. Schapire and M.K. Warmuth, How to use expert advice, Report UCSC-CRL-94-33, University of California, Santa Cruz, CA (1994); extended abstract in: Proceedings 25th Annual ACM Symposium on Theory of Computing (1993) 382-391.
4. N. Cesa-Bianchi, Y. Freund, D.P. Helmbold, D. Haussler, R.E. Schapire and M.K. Warmuth, How to use expert advice, Report UCSC-CRL-94-33, University of California, Santa Cruz, CA (1994); extended abstract in: Proceedings 25th Annual ACM Symposium on Theory of Computing (1993) 382-391.
5. R.O. Duda and P.E. Hart, Pattern Classification and Scene Analysis (Wiley, New York, 1973).
6. D.P. Helmbold and R.E. Schapire, Predicting nearly as well as the best pruning of a decision tree, in: Proceedings 8th Annual Conference on Computational Learning Theory (ACM Press, New York, 1995) 61-68.
7. L.G. Khachiyan, A polynomial algorithm in linear programming, Dokl Akad. Nauk SSSR 244 (1979) 1093-1096 (in Russian); English translation: Soviet Math. Dokl. 20 (1979) 191-194.
8. L.G. Khachiyan, A polynomial algorithm in linear programming, Dokl Akad. Nauk SSSR 244 (1979) 1093-1096 (in Russian); English translation: Soviet Math. Dokl. 20 (1979) 191-194.
9. J. Kivinen and M.K. Warmuth, Exponentiated gradient versus gradient descent for linear predictors, Inform. and Comput. 132 (1997) 1-63.
10. N. Littlestone, Learning quickly when irrelevant attributes abound: a new linear-threshold algorithm, Machine Learning 2 (1988) 285-318.
11. N. Littlestone, Mistake bounds and logarithmic linear-threshold learning algorithms, Ph.D. Thesis, Report UCSC-CRL-89-11, University of California, Santa Cruz, CA (1989).
12. N. Littlestone, Redundant noisy attributes, attribute errors and linear threshold learning using Winnow, in: Proceedings 4th Annual Workshop on Computational Learning Theory (Morgan Kaufmann, San Mateo, CA, 1991) 147-156.
13. N. Littlestone, Comparing several linear-threshold learning algorithms on tasks involving superfluous attributes, in: Proceedings 12th International Conference on Machine Learning, Lake Tahoe, CA (Morgan Kaufmann, San Francisco, CA, 1995) 353-361.
14. N. Littlestone, P.M. Long and M.K. Warmuth, On-line learning of linear functions, J. Comput. Complexity 5 (1995) 1-23.
15. W. Maass and G. Turán, How fast can a threshold gate learn, in: Computational Learning Theory and Natural Learning Systems, Vol. I (MIT Press, Cambridge, MA, 1994) 381-414.
16. W. Maass and M.K. Warmuth, Efficient learning with virtual threshold gates, in: Proceedings 12th International Conference on Machine Learning, Lake Tahoe, CA (Morgan Kaufmann, San Francisco, 1995) 378-386.
17. F. Rosenblatt, The Perceptron: a probabilistic model for information storage and organization in the brain, Psych. Rev. 65 (1958) 386-407; reprinted in: Neurocomputing (MIT Press, Cambridge, MA, 1988).
18. F. Rosenblatt, The Perceptron: a probabilistic model for information storage and organization in the brain, Psych. Rev. 65 (1958) 386-407; reprinted in: Neurocomputing (MIT Press, Cambridge, MA, 1988).
19. Y. Singer, Adaptive mixture of probabilistic transducers, in: Advances in Neural Information Processing Systems, Vol. 8 (MIT Press, Cambridge, MA, 1996) 381-387.
20. H. Sompolinsky, H.S. Seung and N. Tishby, Learning curves in large neural networks, in: Proceedings 4th Annual Workshop on Computational Learning Theory (Morgan Kaufmann, San Mateo, CA, 1991) 112-127.
21. P.M. Vaidya, A new algorithm for minimizing convex functions over convex sets, in: Proceedings 30th Symposium on Foundations of Computer Science (IEEE Computer Society Press, Los Alamitos, CA, 1989) 338-343.
22. L.G. Valiant, A theory of the learnable, Comm. ACM 27 (1984) 1134-1142.
23. V.N. Vapnik and A.Y. Chervonenkis, On the uniform convergence of relative frequencies of events to their probabilities, Theory Probab. Appl. 16 (1971) 264-280.
24. V. Vovk, Aggregating strategies, in: Proceedings 3rd Annual Workshop on Computational Learning Theory (Morgan Kaufmann, San Mateo, CA, 1990) 371-383.
25. T.L.H. Watkin, A. Rau and M. Biehl, The statistical mechanics of learning a rule, Rev. Mod. Phys. 65 (1993) 499-556.