Exponentiated Gradient versus Gradient Descent for Linear Predictors

Information and Computation

Volumn 132, Issue 1, 1997, Pages 1-63

  Kivinen, Jyrki ^a   Warmuth, Manfred K ^b  

^a UNIVERSITY OF HELSINKI   (Finland)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0008815681     PISSN: 08905401     EISSN: None     Source Type: Journal    
DOI: 10.1006/inco.1996.2612     Document Type: Article

References (30)

1. Amari, S. (1994), "Information Geometry of the EM and em Algorithms for Neural Networks," Technical Report METR 94-4, Univ. of Tokyo.
2. Amari, S. (1995), The EM algorithm and information geometry in neural network learning, Neural Comput. 7, 13-18.
3. Boser, B. E., Guyon, I. M., and Vapnik, V. N. (1992), A training algorithm for optimal margin classifiers in "Proceedings, 5th Annual Workshop on Computational Learning Theory," pp. 144-152, ACM Press, New York.
4. Cesa-Bianchi, N., Freund, Y., Haussler, D., Helmbold, D. P., Schapire, R. E., and Warmuth, M. K. (1994), "How to Use Expert Advice," Technical Report UCSC-CRL-94-33, Univ. of California, Santa Cruz, Computer Research Laboratory. An extended abstract appeared in "Proceedings, 25th Annual ACM Symposium on the Theory of Computing," pp. 382-381, ACM Press, New York.
5. Cesa-Bianchi, N., Long, P., and Warmuth, M. K. (1996), Worst-case quadratic loss bounds for on-line prediction of linear functions by gradient descent, IEEE Trans. Neural Networks 7, 604-619.
6. Cormen, T. H., Leiserson, C. E., and Rivest, R. L. (1990), "Introduction to Algorithms," MIT Press, Cambridge, MA.
7. Dempster, A. P., Laird, N. M., and Rubin, D. B. (1977), Maximum-likelihood from incomplete data via the EM algorithm, J. Roy. Statist. Soc. Ser. B 39, 1-38.
8. Haussler, D., Kivinen, J., and Warmuth, M. K. (1994), "Tight Worst-Case loss bounds for Predicting with Expert Advice," Technical Report UCSC-CRL-94-36, Univ. California, Santa Cruz, Computer Research Laboratory. Partial results appeared in "EuroCOLT '93," pp. 109-120, Clarendon Press, Oxford, and in "EuroCOLT '95," pp. 69-83, Springer, Berlin. To appear in IEEE Transactions on Information Theory.
9. Haussler, D., Kivinen, J., and Warmuth, M. K. (1994), "Tight Worst-Case loss bounds for Predicting with Expert Advice," Technical Report UCSC-CRL-94-36, Univ. California, Santa Cruz, Computer Research Laboratory. Partial results appeared in "EuroCOLT '93," pp. 109-120, Clarendon Press, Oxford, and in "EuroCOLT '95," pp. 69-83, Springer, Berlin. To appear in IEEE Transactions on Information Theory.
10. Haykin, S. (1991), "Adaptive Filter Theory," 2nd ed., Prentice-Hall, Englewood Cliffs, NJ.
11. Haykin, S. (1993), "Neural Networks: A Comprehensive Foundation," Macmillan, New York.
12. Helmbold, D. P., Kivinen, J., and Warmuth, M. K. (1996a), Worst-case loss bounds for sigmoided linear neurons, in "Advances in Neural Information Processing Systems 8," MIT Press, Cambridge, MA, pp. 309-315.
13. Helmbold, D. P., Schapire, R. E., Singer, Y., and Warmuth, M. K. (1996b), A comparison of new and old algorithms for a mixture estimation problem, Mach. Learning, to appear.
14. Helmbold, D. P., Schapire, R. E., Singer, Y., and Warmuth, M. K. (1996c), On-line portfolio selection using multiplicative updates in "Proceedings, 13th International Conference on Machine Learning," Morgan Kaufmann, San Francisco, pp. 243-251.
15. Helmbold, D. P., and Warmuth, M. K. (1995), On weak learning, J. Comput. System Sci. 50, 551-573.
16. Hinton, G. E. (1986), Learning distributed representations of concepts, in "Proceedings, 8th Annual Conference of the Cognitive Science Society," pp. 1-12, Erlbaum, Hillsdale, NJ.
17. Jumarie, G. (1990), "Relative Information," Springer, New York.
18. Kapur, J. N., and Kesavan, H. K. (1992), "Entropy Optimization Principles with Applications," Academic Press, New York.
19. Kearns, M. J., Schapire, R. E., and Sellie, L. M. (1994), Toward efficient agnostic learning, Mach. Learning 17, 115-142.
20. Littlestone, N. (1988), Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm, Mach. Learning 2, 285-318.
21. Littlestone, N. (1989), From on-line to batch learning in "Procedings, 2nd Annual Workshop on Computational Learning Theory," pp. 269-284, Morgan Kaufmann, San Mateo, CA.
22. Littlestone, N. (1989), "Mistake Bounds and Logarithmic Linear-threshold Learning Algorithms," Ph.D. thesis, Technical Report UCSC-CRL-89-11, Univ. of California, Santa Cruz, Computer Research Laboratory.
23. Littlestone, N. (1991), Redundant noisy attributes, attribute errors, and linear threshold learning using Winnow, in "Proceedings, 4th Annual Workshop on Computational Learning Theory," pp. 147-156, Morgan Kaufmann, San Mateo, CA.
24. Littlestone, N., Long, P. M., and Warmuth, M. K. (1995), On-line learning of linear functions, J. Comput. Complexity 5, 1-23.
25. Littlestone, N., and Warmuth, M. K. (1994), The weighted majority algorithm, Inform, and Comput. 108, 212-261.
26. Luenberger, D. G. (1984), "Linear and Nonlinear Programming," Addison-Wesley, Reading, MA.
27. Royden, H. (1963), "Real Analysis," Macmillan, New York.
28. Schapire, R. E., and Warmuth, M. K. (1994), On the worst-case analysis of temporal-difference learing algorithms, in "Proceedings, 11th International Conference on Machine Learning," pp. 266-274, Morgan Kaufmann, San Francisco; Mach. Learning, to appear.
29. Vovk, V. (1990), Aggregating strategies, in "Proceedings, 3rd Annual Workshop on Computational Learning Theory," pp. 371-383, Morgan Kaufmann, San Mateo, CA.
30. Widrow, B., and Stearns, S. (1985), "Adaptive Signal Processing," Prentice-Hall, Englewood Cliffs, NJ.