Information-theoretic lower bounds on the oracle complexity of stochastic convex optimization

IEEE Transactions on Information Theory

Volumn 58, Issue 5, 2012, Pages 3235-3249

  Agarwal, Alekh ^a   Bartlett, Peter L ^a,b   Ravikumar, Pradeep ^c   Wainwright, Martin J ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b QUEENSLAND UNIVERSITY OF TECHNOLOGY   (Australia)

^c University of Texas at Austin   (United States)

Author keywords

Computational learning theory; convex optimization; Fano's inequality; information based complexity; minimax analysis; oracle complexity

Indexed keywords

COMPUTATIONAL LEARNING THEORY; FANO'S INEQUALITY; INFORMATION-BASED COMPLEXITY; MINIMAX; ORACLE COMPLEXITY;

CONVEX OPTIMIZATION; INFORMATION THEORY; PARAMETER ESTIMATION; STOCHASTIC SYSTEMS;

INFORMATION USE;

EID: 84860244324     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2011.2182178     Document Type: Conference Paper

References (25)

1. L. Bottou and O. Bousquet, "The tradeoffs of large scale learning," in Proc. Adv. Neural Inf. Process. Syst., 2008.
2. S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge, UK: Cambridge Univ. Press, 2004.
3. D. Bertsekas, Nonlinear Programming. Belmont, MA: Athena Scientific, 1995.
4. Y. Nesterov, Introductory Lectures on Convex Optimization: Basic Course. Norwell, MA: Kluwer, 2004.
5. A. S. Nemirovski and D. B. Yudin, Problem Complexity and Method Efficiency in Optimization. New York: Wiley, 1983.
6. A. S. Nemirovski, Efficient methods in convex programming 2010 [Online]. Available: http://www2.isye.gatech.edu/~nemirovs/OPTI-LectureNotes. pdf
7. A. Agarwal, P. Bartlett, P. Ravikumar, and M. J. Wainwright, "Information- theoretic lower bounds on the oracle complexity of convex optimization," in Proc. Adv. Neural Inf. Process. Syst., 2009, vol. 22, pp. 1-9.
8. M. Raginsky and A. Rakhlin, "Information-based complexity, feedback and dynamics in convex programming," IEEE Trans. Inf. Theory, vol. 57, no. 10, pp. 7036-7056, Oct. 2011, to be published.
9. R. Z. Has'minskii, "A lower bound on the risks of nonparametric estimates of densities in the uniform metric," Theory Prob. Appl., vol. 23, pp. 794-798, 1978.
10. L. Birgé, "Approximation dans les espaces metriques et theorie de l'estimation," Z. Wahrsch. verw. Gebiete, vol. 65, pp. 181-327, 1983.
11. Y. Yang and A. Barron, "Information-theoretic determination of minimax rates of convergence," Ann. Statist., vol. 27, no. 5, pp. 1564-1599, 1999.
12. B. Yu, "Assouad, Fano and Le Cam," in Festschrift for Lucien Le Cam. Berlin, Germany: Springer-Verlag, 1997, pp. 423-435.
13. V. N. Vapnik and A. Chervonenkis, Theory of Pattern Recognition (in Russian). Moscow, Russia: Nauka, 1974.
14. A. Ehrenfeucht, D. Haussler, M. Kearns, and L. Valiant, "A general lower bound on the number of examples needed for learning," in Proc. 1st Annu. Workshop Comput. Learn. Theory, 1988, pp. 139-154.
15. H. U. Simon, "Bounds on the number of examples needed for learning functions," SIAM J. Comput., vol. 26, pp. 751-763, Jun. 1997.
16. E. Hazan, A. Agarwal, and S. Kale, "Logarithmic regret algorithms for online convex optimization," Mach. Learning, vol. 69, no. 2-3, pp. 169-192, 2007.
17. E. Hazan and S. Kale, "Beyond the regret minimization barrier: An optimal algorithm for stochastic strongly-convex optimization," in Proc. Ann. Conf. Learning Theory, 2010.
18. A. Juditsky and Y. Nesterov, Primal-dual subgradient methods for minimizing uniformly convex functions Universite Joseph Fourier, Grenoble, France, 2010 [Online]. Available: http://hal.archives-ouvertes. fr/hal-00508933/
19. T. Cover and J. Thomas, Elements of Information Theory. New York: Wiley, 1991.
20. L. L. Cam, "Convergence of estimates under dimensionality restrictions," Ann. Statist., vol. 1, no. 1, pp. 38-53, 1973.
21. J. Matousek, Lectures on Discrete Geometry. New York: Springer- Verlag, 2002.
22. G. Raskutti, M. J.Wainwright, and B. Yu, "Minimax rates of estimation for high-dimensional linear regression over α-balls," IEEE Trans. Inf. Theory, vol. 57, no. 10, pp. 6976-6994, Oct. 2011.
23. A. Beck and M. Teboulle, "Mirror descent and nonlinear projected subgradient methods for convex optimization," Oper. Res. Lett., vol. 31, no. 3, pp. 167-175, 2003.
24. G. Rockafellar, Convex Analysis. Princeton, NJ: Princeton Univ. Press, 1970.
25. J. Hiriart-Urruty and C. Lemaréchal, Convex Analysis and Minimization Algorithms. New York: Springer-Verlag, 1993, vol. 1.