Towards an optimal stochastic alternating direction method of multipliers

31st International Conference on Machine Learning, ICML 2014

Volumn 2, Issue , 2014, Pages 944-959

  Azadi, Samaneh ^a,b   Sra, Suvrit ^c,d  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b SHIRAZ UNIVERSITY   (Iran)

^c CARNEGIE MELLON UNIVERSITY   (United States)

^d MAX PLANCK INSTITUTE FOR INTELLIGENT SYSTEMS   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; CONVEX OPTIMIZATION; LEARNING SYSTEMS; OPTIMIZATION;

ALTERNATING DIRECTION METHOD OF MULTIPLIERS; CONVERGENCE RATES; LINEAR EQUALITY CONSTRAINTS; MINIMAX; OPTIMAL RATE; STOCHASTIC PROBLEMS;

STOCHASTIC SYSTEMS;

EID: 84919948575     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (27)

1. Baneijee, O., Ghaoui, L. E., and d'Aspremont, A. Model selection through sparse maximum likelihood estimation for multivarite Gaussian or binary data. JMLR, 9:485- 516, 2008.
2. Beck, A. and Teboulle, M. A Fast Iterative Shrinkage- Thresholding Algorithm for Linear Inverse Problems. SIAM J. Imaging Sciences, 2(1): 183-202, 2009.
3. Boyd, Stephen, Parikh, Neal, Chu, Eric, Peleato, Borja, and Eckstein, Jonathan. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends(§) in Machine Learning, 3(1): 1-122, 2011.
4. Chambolle, A. and Pock, T. A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging. J. Math. Imaging Vis., 40(1):120-145, 2011.
5. Chan, R. H., Tao, M., and Yuan, X. Constrained total variation models and fast algorithms based on alternating direction method of multipliers. SIAM J. Imaging Sci., 6 (1), 2013.
6. Chen, Xi, Lin, Qihang, and Pena, Javier. Optimal regularized dual averaging methods for stochastic optimization. In Advances in Neural Information Processing Systems (NIPS-I2), pp. 404-412, 2012.
7. Devolder, O., Glineur, F and Nesterov, Yu. First-order methods of smooth convex optimization with inexact oracle. Technical Report 2011/17, UCL, 2011.
8. Douglas, J. and Rachford, H. H. On the numerical solution of the heat conduction problem in 2 and 3 space variables. Tran. Amer. Math. Soc., 82:421-439, 1956.
9. Duchi, J. and Singer, Y. Online and Batch Learning using Forward-Backward Splitting. J. Mach. Learning Res. (JMLR), 2009.
10. Eckstein, J. and Bertsekas, D. P. On the Douglas- Rachoford splitting method and the proximal point algorithm for maximal monotone operators. Math. Prog., 55(3):292-318, 1992.
11. Ghadimi, S. and Lan, G. Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization, i: A generic algorithmic framework. SIAM J. Optimization, 22:1469-1492, 2012.
12. Goldfarb, D., Ma, S., and Scheinberg, K. Fast alternating linearization methods for minimizing the sum of two convex functions. Math. Prog. Ser. A, 2012.
13. He, B. and Yuan, X. On non-ergodic convergence rate of Douglas-Rachford alternating direction method of multipliers. Unpublished, 2012.
14. Kraska, T., Talwalkar, A., J.Duchi, Griffith, R., Franklin, M., and Jordan, M.I. ML base:A Distributed Machine Learning System. In Conf. Innovative Data Systems Research, 2013.
15. Lacoste-Julien, S., Schmidt, M., and Bach, F. A simpler approach to obtaining an 0(l/t) convergence rate for projected subgradient descent. arXiv:1212.2002, 2012.
16. Nemirovski, A., Juditsky, A., Lan, G., and Shapiro, A. Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization, 19(4): 1574- 1609, 2009.
17. Nesterov, Y. Introductory Lectures on Convex Optimization: A Basic Course. Springer, 2004.
18. Nesterov, Yu. Gradient methods for minimizing composite objective function. Technical Report 2007/76, Universite catholique de Louvain, Center for Operations Research and Econometrics (CORE), 2007.
19. Ouyang, Hua, He, Niao, Tran, Long, and Gray, Alexander G. Stochastic alternating direction method of multipliers. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pp. 80-88, 2013.
20. Parikh, N. and Boyd, S. Proximal Algorithms, volume 1. NOW, 2013.
21. Qin, Z. and Goldfarb, D. Structured sparsity via alternating direction methods. J. Machine Learning Research, 13: 1435-1468, 2012.
22. Rockafellar, R. T. Monotone Operators and the Proximal Point Algorithm. SIAM J. Control Optim., 14(5):877- 989, 1976.
23. Shamir, Ohad and Zhang, Tong. Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pp. 71-79, 2013.
24. Srebro, N. and Tewari, A. Stochastic Optimization for Machine Learning. ICML 2010 Tutorial, 2010.
25. Suzuki, Taiji. Dual averaging and proximal gradient descent for online alternating direction multiplier method. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pp. 392-400, 2013.
26. Tseng, P. On accelerated proximal gradient methods for convex-concave optimization. Unpublished, 2008.
27. Xiao, L. Dual averaging methods for regularized stochastic learning and online optimization. JMLR, 11:2543-2596, 2010.