Stochastic zeroth-order optimization in high dimensions

International Conference on Artificial Intelligence and Statistics, AISTATS 2018

Volumn , Issue , 2018, Pages 1356-1365

  Wang, Yining ^a   Du, Simon S ^a   Balakrishnan, Sivaraman ^a   Singh, Aarti ^a  

^a CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; FUNCTIONS;

CONVERGENCE RATES; CONVEX FUNCTIONS; FUNCTION VALUES; GRADIENT ESTIMATES; HIGH DIMENSIONS; HIGH-DIMENSIONAL; ORDER OPTIMIZATIONS; SELECTION ALGORITHM;

STOCHASTIC SYSTEMS;

EID: 85064809780     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (44)

1. A. Agarwal, P. L. Bartlett, P. Ravikumar, and M. J. Wainwright. Information-theoretic lower bounds on the oracle complexity of stochastic convex optimization. IEEE Transactions on Information Theory, 58(5):3235–3249, 2012.
2. A. Agarwal, O. Dekel, and L. Xiao. Optimal algorithms for online convex optimization with multi-point bandit feedback. In Proceedings of the annual Conference on Learning Theory (COLT), 2010.
3. A. Agarwal, D. Foster, D. Hsu, S. Kakade, and A. Rakhlin. Stochastic convex optimization with bandit feedback. SIAM Journal on Optimization, 23(1):213–240, 2013.
4. A. S. Bandeira, K. Scheinberg, and L. N. Vicente. Computation of sparse low degree interpolating polynomials and their application to derivative-free optimization. Mathematical Programming, 134:223–257, 2012.
5. A. S. Bandeira, K. Scheinberg, and L. N. Vicente. Convergence of trust-region methods based on probabilistic models. SIAM Journal on Optimization, 24(3):1238–1264, 2014.
6. A. Beck and M. Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31(3):167–175, 2003.
7. P. J. Bickel, Y. Ritov, and A. B. Tsybakov. Simultaneous analysis of lasso and dantzig selector. The Annals of Statistics, 37(4):1705–1732, 2009.
8. S. Bubeck, R. Eldan, and Y. T. Lee. Kernel-based methods for bandit convex optimization. In Proceedings of the annual ACM SIGACT Symposium on Theory of Computing (STOC), 2017.
9. A. D. Bull. Convergence rates of efficient global optimization algorithms. Journal of Machine Learning Research, 12(Oct):2879–2904, 2011.
10. E. J. Candès, J. Romberg, and T. Tao. Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Transactions on Information Theory, 52(2):489–509, 2006.
11. D. L. Donoho. Compressed sensing. IEEE Transactions on Information Theory, 52(4):1289–1306, 2006.
12. J. C. Duchi, S. Shalev-Shwartz, Y. Singer, and A. Tewari. Composite objective mirror descent. In Proceedings of the annual Conference on Learning Theory (COLT), 2010.
13. A. D. Flaxman, A. T. Kalai, and H. B. McHanan. Online convex optimization in the bandit setting: gradient descent without a gradient. In Proceedings of the ACM-SIAM Symposium on Discrete Algorithms (SODA), 2005.
14. S. Ghadimi and G. Lan. Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization I: A generic algorithmic framework. SIAM Journal on Optimization, 22(4):1469–1492, 2012.
15. S. Ghadimi and G. Lan. Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization, II: shrinking procedures and optimal algorithms. SIAM Journal on Optimization, 23(4):2061–2089, 2013.
16. E. Hazan and K. Levy. Bandit convex optimization: Towards tight bounds. In Proceedings of Advances in Neural Information Processing Systems (NIPS), 2014.
17. W. Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58(301):13–30, 1963.
18. K. G. Jamieson, R. Nowak, and B. Recht. Query complexity of derivative-free optimization. In Proceedings of Advances in Neural Information Processing Systems (NIPS), 2012.
19. A. Javanmard and A. Montanari. Confidence intervals and hypothesis testing for high-dimensional regression. Journal of Machine Learning Research, 15(1):2869–2909, 2014.
20. R. Johnson and T. Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Proceedings of Advances in neural information processing systems (NIPS), 2013.
21. K. Knight and W. Fu. Asymptotics for lasso-type estimators. The Annals of statistics, 28(5):1356–1378, 2000.
22. G. Lan. An optimal method for stochastic composite optimization. Mathematical Programming, 133:365–397, 2012.
23. D. D. Leeds, J. A. Pyles, and M. J. Tarr. Exploration of complex visual feature spaces for object perception. Frontiers in computational neuroscience, 8, 2014.
24. Q. Lei, I. E.-H. Yen, C.-y. Wu, I. S. Dhillon, and P. Ravikumar. Doubly greedy primal-dual coordinate descent for sparse empirical risk minimization. In Proceedings of the International Conference on Machine Learning (ICML), 2017.
25. K. Lounici. Sup-norm convergence rate and sign concentration property of lasso and dantzig estimators. Electronic Journal of Statistics, 2:90–102, 2008.
26. N. Nakamura, J. Seepaul, J. B. Kadane, and B. Reeja-Jayan. Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry, 2017.
27. A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization, 19(4):1574–1609, 2009.
28. A. Nemirovski and D. Yudin. Problem complexity and method efficiency in optimization. A Wiley-Interscience Publication, 1983.
29. W. K. Newey, F. Hsieh, and J. M. Robins. Twicing kernels and a small bias property of semiparametric estimators. Econometrica, 72(3):947–962, 2004.
30. G. Raskutti, M. J. Wainwright, and B. Yu. Mini-max rates of estimation for high-dimensional linear regression over Lq-balls. IEEE Transactions on Information Theory, 57(10):6976–6994, 2011.
31. B. Reeja-Jayan, K. L. Harrison, K. Yang, C.-L. Wang, A. Yilmaz, and A. Manthiram. Microwave-assisted low-temperature growth of thin films in solution. Scientific reports, 2, 2012.
32. J. Scarlett, I. Bogunovic, and V. Cevher. Lower bounds on regret for noisy gaussian process bandit optimization. In Proceedings of the annual conference on Learning Theory (COLT), 2017.
33. S. Shalev-Shwartz, N. Srebro, and T. Zhang. Trading accuracy for sparsity in optimization problems with sparsity constraints. SIAM Journal on Optimization, 20(6):2807–2832, 2010.
34. S. Shalev-Shwartz and A. Tewari. Stochastic methods for l1-regularized loss minimization. Journal of Machine Learning Research, 12(Jun):1865–1892, 2011.
35. O. Shamir. On the complexity of bandit and derivative-free stochastic convex optimization. In Proceedings of the annual Conference on Learning Theory (COLT), 2013.
36. J. Snoek, H. Larochelle, and R. P. Adams. Practical bayesian optimization of machine learning algorithms. In Proceedings of Advances in Neural Information Processing Systems (NIPS), 2012.
37. N. Srebro, K. Sridharan, and A. Tewari. On the universality of online mirror descent. In Proceedings of Advanced in Neural Information Processing Systems (NIPS), 2011.
38. R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Statistical Methodology), 58(1):267–288, 1996.
39. S. Van de Geer, P. Bühlmann, Y. Ritov, R. Dezeure, et al. On asymptotically optimal confidence regions and tests for high-dimensional models. The Annals of Statistics, 42(3):1166–1202, 2014.
40. H. Victor. A general class of exponential inequalities for martingales and ratios. The Annals of Probability, 27(1):537–564, 1999.
41. M. J. Wainwright. Sharp thresholds for high-dimensional and noisy sparsity recovery using L1-constrained quadratic programming (Lasso). IEEE Transactions on Information T theory, 55(5):2183–2202, 2009.
42. L. Xiao and T. Zhang. A proximal-gradient homotopy method for the sparse least-squares problem. SIAM Journal on Optimization, 23(2):1062–1091, 2013.
43. C.-H. Zhang and S. S. Zhang. Confidence intervals for low dimensional parameters in high dimensional linear models. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 76(1):217–242, 2014.
44. P. Zhao and B. Yu. On model selection consistency of lasso. Journal of Machine learning research, 7(Nov):2541–2563, 2006.