Sequential procedures for aggregating arbitrary estimators of a conditional mean

IEEE Transactions on Information Theory

Volumn 54, Issue 4, 2008, Pages 1725-1735

  Bunea, Florentina ^a   Nobel, Andrew ^b  

^a FLORIDA STATE UNIVERSITY   (United States)

^b The University of North Carolina at Chapel Hill   (United States)

Author keywords

Aggregation; Bayesian model averaging; Individual sequence; Oracle inequality; Prediction; Regression

Indexed keywords

APPROXIMATION THEORY; BAYESIAN NETWORKS; COMPUTATIONAL COMPLEXITY; RANDOM PROCESSES;

AGGREGATE ESTIMATOR; EXPONENTIAL WEIGHTING; ORACLE INEQUALITY; SEQUENTIAL PREDICTION;

LINEAR REGRESSION;

EID: 41949142938     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2008.917657     Document Type: Article

References (37)

1. J.-Y. Audibert, "Aggregated estimators and empirical complexity for least square regression," Annal. l'Institut Henri Poincaré (B), Probab. Statist., vol. 40, pp. 685-736, 2003.
2. S. Azoury and M. Warmuth, "Relative loss bounds for on-line density estimation with the exponential family of distributions," Mach. Learn., vol. 43, pp. 211-245, 2001.
3. A. R. Barron, "Universal approximation bounds for superpositions of a sigmoidal function," IEEE Trans. Inf. Theory, vol. 39, no. 3, pp. 930-945, May 1993.
4. L. Birgé, "Model selection via testing: An alternative to (penalized) maximum likelihood estimators," Anna. l'Institut Henri Poincare (B) Prob. Statist., vol. 42, pp. 273-325, 2006.
5. F. Bunea, A. B. Tsybakov, and M. H. Wegkamp, "Aggregation for Gaussian regression," Anna. Statist., vol. 35, pp. 1674-1697, 2007.
6. N. Cesa-Bianchi, "Analysis of two gradient-based algorithms for on-line regression," in Proc.12th Annu. Conf. Computational Learning Theory 1999, pp. 163-170, ACM Press.
7. N. Cesa-Bianchi, Y. Freund, D. Haussler, D. P. Helmbold, R. E. Schapire, and M. K. Warmuth, "How to use expert advice," J. Assoc. Comput. Mach., vol. 44, pp. 427-485, 1997.
8. N. Cesa-Bianchi, Y. Freund, D. P. Helmbold, and M. K. Warmuth, "On-line prediction and conversion strategies," Mach. Learn., vol. 25, pp. 71-110, 1996.
9. N. Cesa-Bianchi and G. Lugosi, "On the prediction of individual sequences," Ann. Statist., vol. 27, pp. 1865-1895, 2000.
10. N. Cesa-Bianchi and G. Lugosi, Prediction, Learning and Games. New York: Cambridge Univ. Press, 2006.
11. O. Catoni, "Statistical learning theory and stochastic optimization," in Ecole d'Eté de Probabilités de Saint-Flour 2001 (Lecture Notes in Mathematics). New York: Springer, 2001, vol. 1851.
12. D. P. Foster and R. Vohra, "Regret in the on-line decision problem," Games and Economic Behavior, vol. 29, pp. 1084-1090, 1999.
13. Y. Freund, "Predicting a binary sequence almost as well as the optimal biased coin," in Proc. 9th Annu. Conf. Computational Learning Theory New York, 1996, pp. 89-98, ACM Press.
14. A. Gelman, J. B. Carlin, H. S. Stern, and D. B. Rubin, Bayesian Data Analysis, 2nd ed. Boca Raton, FL: CRC, 2003.
15. L. Györfi and G. Lugosi, "Strategies for sequential prediction of stationary time series," in Modeling Uncertainty: An Examination of its Theory, Methods, and Applications, M. Dror, P. L. Ecuyer, and F. Szidarovszky, Eds. Boston, MA: Kluwer, 2001.
16. L. Györfi, G. Lugosi, and G. Morvai, "A simple randomized algorithm for consistent sequential prediction of ergodic time series," IEEE Trans. Inf. Theory, vol. 45, no. 7, pp. 2642-2650, De. 1999.
17. D. H. Haussler, J. Kivenen, and M. K. Warmuth, "Sequential prediction of individual sequences under general loss functions," IEEE Trans. Inf. Theory, vol. 44, no. 5, pp. 1906-1925, Sep. 1998.
18. A. Juditsky, A. Nazin, A. B. Tsybakov, and N. Vayatis, "Online aggregation with a mirror descent algorithm," Probl. Inf. Transm. vol. 41, pp. 368-384, 2005.
19. A. Juditsky and A. Nemirovski, "Functional aggregation for nonparametric regression," Ann. Statist., vol. 28, pp. 681-712, 2000.
20. J. Kivinen and M. K. Warmuth, "Averaging expert predictions," in Proc. 4th Europ. Conf. Computational Learning Theory (Euro-COLT99) (Lecture Notes in Computer Science), H. U. Simon and P. Fischer, Eds. Berlin, Germany: Springer-Verlag, 1999, pp. 153-167.
21. V. Koltchinskii, Local Rademacher Complexities and Oracle Inequalities in Risk Minimization, Preprint, 2004.
22. G. Leung and A. R. Barron, "Information theory and mixing least-squares regressions," IEEE Trans. Inf. Theory, vol. 52, no. 8, pp. 3396-3410, Aug. 2006.
23. N. Littlestone and M. K. Warmuth, "The weighted majority algorithm," Info. Comput., vol. 108, pp. 212-261, 1994.
24. N. Merhav and M. Feder, "Universal prediction," IEEE Trans. Info. Theory, vol. 44, no. 6, pp. 2124-2147, Oct. 1998.
25. A. Nemirovski, "Topics in nonparametric statistics," in Ecole d'Eté de Probabilités de Saint-Flour 1998 (Lecture Notes in Mathematics), P. Bernard, Ed. New York: Springer, 2000, XXVIII.
26. A. B. Nobel, "On optimal sequential decision schemes for general processes," IEEE Trans. Inf. Theory, vol. 49, no. 1, pp. 83-98, jan. 2003.
27. A. Singer and M. Feder, "Universal linear prediction by model order weighting," IEEE Trans. Signal Process., vol. 47, no. 10, pp. 2685-2699, Oct. 1999.
28. A. Singer and M. Feder, "Universal linear least squares prediction," in Proc. 2000 IEEE Int. Symp. Information Theory, 2000.
29. A. B. Tsybakov, "Optimal rates of aggregation," in Proc. 16th Annu. Conf. Learning Theory (COLT) and 7th Annu. Workshop on Kernel Machines Lecture Notes in Artificial Intelligence. Heidelberg, Germany: Springer-Verlag, 2003, vol. 2777, pp. 303-313.
30. V. Vovk, "Aggregating strategies," in Proc. 3rd Annual Workshop on Computational Learning Theory. San Mateo, CA: Morgan Kaufman, 1990, pp. 371-383.
31. V. Vovk, "Competitive on-line statistics," Int. Stat. Rev., vol. 69, pp. 213-248, 2001.
32. M. H. Wegkamp, "Model selection in nonparametric regression," Ann. Statist., vol. 31, pp. 252-273, 2003.
33. K. Yamanishi, "Minimax relative loss analysis for sequential prediction algorithms using parametric hypotheses," in Proc. COLT 98, 1998, pp. 32-43, ACM Press.
34. Y. Yang, "Adaptive estimation in pattern recognition by combining different procedures," Statistica Sinica, vol. 10, pp. 1069-1089, 2000.
35. Y. Yang, "Adaptive regression by mixing," J. Amer. Statist. Assoc. vol. 96, pp. 574-588, 2001.
36. Y. Yang, "Aggregating regression procedures for a better performance," Bernoulli, vol. 10, pp. 25-47, 2004.
37. Y. Yang, "Combining forecasting procedures: Some theoretical results," Econometric Theory, vol. 20, pp. 176-222, 2004.