Information-based complexity, feedback and dynamics in convex programming

IEEE Transactions on Information Theory

Volumn 57, Issue 10, 2011, Pages 7036-7056

  Raginsky, Maxim ^a   Rakhlin, Alexander ^b  

^a Duke University   (United States)

^b UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

Convex optimization; Fano's inequality; feedback information theory; hypothesis testing with controlled observations; information based; information theoretic converse; minimax lower bounds; sequential optimization algorithms; statistical estimation complexity

Indexed keywords

FANO'S INEQUALITY; FEEDBACK INFORMATION THEORY; HYPOTHESIS TESTING; INFORMATION-BASED; INFORMATION-THEORETIC CONVERSE; MINIMAX; SEQUENTIAL OPTIMIZATION ALGORITHMS; STATISTICAL ESTIMATION;

ALGORITHMS; CONVEX OPTIMIZATION; ESTIMATION; INFORMATION THEORY; RISK PERCEPTION; SIGNAL TO NOISE RATIO;

OPTIMIZATION;

EID: 80053997013     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2011.2154375     Document Type: Article

References (34)

1. A. S. Nemirovski and D. B. Yudin, Problem Complexity and Method Efficiency in Optimization. Hoboken, NJ: Wiley, 1983.
2. Y. Nesterov, Introductory Lectures on Convex Optimization. Boston, MA: Kluwer, 2004.
3. A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro, "Robust stochastic approximation approach to stochastic programming" SIAM J. Optim., vol. 19, no. 4, pp. 1574-1609, 2009.
4. M. Raginsky and A. Rakhlin, "Information complexity of black-box convex optimization: A new look via feedback information theory" in Proc. 47th Annu. Allerton Conf. Communication, Control and Computing, Sep./Oct. 2009, pp. 803-810.
5. B. Yu, "Assouad, Fano and Le Cam" in Festschrift for Lucien Le Cam. New York: Springer, 1997, pp. 423-435.
6. Y. Yang and A. Barron, "Information-theoretic determination of minimax rates of convergence" Ann. Statist., vol. 27, no. 5, pp. 1564-1599, 1999.
7. A. B. Tsybakov, Introduction to Nonparametric Estimation. New York: Springer, 2009.
8. A. Shapiro and A. Nemirovski, "On complexity of stochastic programming" in Continuous Optimization: Trends and Applications, V. Jeyakumar and A. M. Rubinov, Eds. New York: Springer, 2005, pp. 111-144.
9. A. Agarwal, P. Bartlett, P. Ravikumar, and M. Wainwright, "Information- theoretic lower bounds on the oracle complexity of convex optimization" Advances in Neural Information Processing Systems 22. Y. Bengio, D. Schuurmans, J. Lafferty, C. K. I. Williams, and A. Culotta, Eds., pp. 1-9, 2009.
10. A. Agarwal, P. Bartlett, P. Ravikumar, and M. Wainwright, Information- Theoretic Lower Bounds on the Oracle Complexity of Convex Optimization, Sep. 2010 [Online].Available: http://arxiv.org/abs/1009. 0571
11. J.-B Hiriart-Urruty and C. Lemaréchal, Fundamentals of Convex Analysis. New York: Springer, 2001.
12. A. N. Shiryaev and V. G. Spokoiny, Statistical Experiments and Decisions: Asymptotic Theory. Singapore: World Scientific, 2000.
13. O. Kallenberg, Foundations of Modern Probability, 2nd ed. New York: Springer, 2002.
14. R. M. Gray, Entropy and Information Theory. New York: Springer, 1991.
15. J. F. Traub, G. W. Wasilkowski, and H. Wozniakowski, Information- Based Complexity. Boston, MA: Academic, 1988.
16. L. Plaskota, Noisy Information and Computational Complexity. Cambridge, U.K.: Cambridge Univ. Press, 1996.
17. B. T. Polyak and A. B. Juditsky," Acceleration of stochastic approximation by averaging" SIAM J. Control Optim., vol. 30, no. 4, pp. 838-855, Jul. 1992.
18. J. Pfanzagl, "On the measurability and consistency of minimum contrast estimates" Metrika, vol. 14, no. 1, pp. 249-272, Dec. 1969.
19. V. V. Fedorov, Theory of Optimal Experiments. New York: Academic, 1972.
20. D. J. C. MacKay, "Information-based objective functions for active data selection" Neur. Comput., vol. 4, pp. 590-604, 1992.
21. L. Paninski, "Asymptotic theory of information-theoretic experimental design" Neural Comput., vol. 17, pp. 1480-1507, 2005.
22. R. Castro and R. Nowak, "Minimax bounds for active learning" IEEE Trans. Inf. Theory, vol. 54, no. 5, pp. 2339-2353, May 2008.
23. M. V. Burnashev and K. S. Zigangirov, "An interval estimation problem for controlled observations" Problem. Inf. Transm., vol. 10, pp. 223-231, 1974.
24. M. V. Burnashev, "Sequential discrimination of hypotheses with control of observations" Math. USSR Izv, vol. 15, no. 3, pp. 419-440, 1980.
25. T. S. Han and S. Verdú "Generalizing the Fano inequality" IEEE Trans. Inf. Theory, vol. 40, no. 4, pp. 1247-1251, Jul. 1994.
26. T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed. Hoboken, NJ: Wiley, 2006.
27. M. L. Vil'k and S. V. Shil'man, "Convergence and optimality of implementable adaptation algorithms (informational approach)" Problems Inf. Transm., vol. 20, pp. 314-326, 1985.
28. Y. Yang," How powerful can any regression learning procedure be?" in Proc. 11th Int. Conf. Artif. Intell. Statist. (AISTATS), M. Meila and X. Shen, Eds., San Juan, PR, Mar. 2007.
29. R. D. Nowak, The Geometry of Generalized Binary Search, Oct. 2009 [Online]. Available: http://arxiv.org/abs/0910.4397
30. A. Guntuboyina," Lower bounds for the minimax risk using f-divergences and applications" IEEE Trans. Inf. Theory, vol. 57, no. 4, pp. 2386-2399, Apr. 2011.
31. L. Devroye, L. Györfi, and G. Lugosi, A Probabilistic Theory of Pattern Recognition. New York: Springer, 1996.
32. A. B. Tsybakov," Optimal aggregation of classifiers in statistical learning" Ann. Statist., vol. 32, no. 1, pp. 135-166, 2004.
33. M. Naghshvar and T. Javidi, "Active M-ary sequential hypothesis testing" in Proc. IEEE Int. Symp. Inf. Theory, Austin, TX, Jun. 2010, pp. 1623-1627.
34. J. Pearl, Causality: Models, Reasoning and Inference, 2nd ed. Cambridge, U.K.: Cambridge Univ. Press, 2009.