Stochastic alternating direction method of multipliers

30th International Conference on Machine Learning, ICML 2013

Volumn , Issue PART 1, 2013, Pages 80-88

  Ouyang, Hua ^a   He, Niao ^b   Tran, Long Q ^a   Gray, Alexander ^a  

^a GEORGIA INSTITUTE OF TECHNOLOGY   (United States)

^b Georgia Institute of Technology   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

FUNCTIONS; LEARNING ALGORITHMS; LEARNING SYSTEMS; STOCHASTIC SYSTEMS;

ALTERNATING DIRECTION METHOD OF MULTIPLIERS; COMPOSITE OBJECTIVE FUNCTION; MACHINE LEARNING APPLICATIONS; OPTIMIZATION PROBLEMS; RATES OF CONVERGENCE; SEPARABLE LEAST SQUARES; STOCHASTIC FUNCTIONS; STRUCTURAL ASSUMPTION;

PROBLEM SOLVING;

EID: 84891310987     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (41)

1. Banerjee, O., Ghaoui, L. E., and d'Aspremont, A. Model selection through sparse maximum likelihood estimation for multivariate gaussian or binary data. Journal of Machine Learning Research, 9:485-516, 2008. (Pubitemid 351469014)
2. Beck, A. and Teboulle, M. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci., 2(1):193-202, 2009.
3. Boyd, S., Parikh, N., Chu, E., Peleato, B., and Eckstein, J. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning, 3(1), 2010.
4. Combettes, P. L. and Pesquet, J. Proximal splitting methods in signal processing. In Bauschke, H. H. et al. (ed.), Fixed-Point Algorithms for Inverse Problems in Science and Engineering, chapter 10, pp. 185-212. Springer, New York, 2011.
5. Combettes, P. L. and Wajs, V. R. Signal recovery by proximal forward-backward splitting. Multiscale Model. Simul., 4(4):11681200, 2005.
6. Daubechies, I., Defrise, M., and Mol, C. De. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Mathematics, 57(11):14131457, 2004.
7. Deng, W. and Yin, W. On the global and linear convergence of the generalized alternating direction method of multipliers. Technical Report TR12-14, Rice University CAAM Technical Report, 2012.
8. Duchi, J. and Singer, Y. Efficient online and batch learning using forward backward splitting. JMLR, 10:2899-2934, 2009.
9. Eckstein, J. and Bertsekas, D. P. On the douglas-rachford splitting method and the proximal point algorithm for maximal monotone operators. Mathematical Programming, 55(1-3):293-318, 1992.
10. Friedman, J. and Tibshirani, R. Sparse inverse covariance estimation with the graphical lasso. Biostatistics, 9(3): 432-441, 2007.
11. Gabay, D. Applications of the method of multipliers to variational inequalities. In Fortin, M. and Glowinski, R. (eds.), Augmented Lagrangian Methods: Applications to the Solution of Boundary-Value Problems. North-Holland: Amsterdam, 1983.
12. Gabay, D. and Mercier, B. A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Computers & Mathematics with Applications, 2(1), 1976.
13. Glowinski, R. and Marroco, A. Sur lapproximation, par elements nis dordre un, et la resolution, par penalisation-dualite, dune classe de problems de dirichlet non lincares. Revue Francaise dAutomatique, Informatique, et Recherche Operationelle, 9(2), 1975.
14. Glowinski, R. and Tallec, P. L. Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics. Studies in Applied and Numerical Mathematics. SIAM, 1989.
15. Goldfarb, D., Ma, S., and Scheinberg, K. Fast alternating linearization methods for minimizing the sum of two convex functions, 2010. URL http://arxiv.org/abs/0912.4571.
16. Goldstein, T. and Osher, S. The split bregman method for l1-regularized problems. SIAM J. Imaging Sci., 2(2):323-343, 2009.
17. He, B. and Yuan, X. On the o(1/n) convergence rate of the douglas-rachford alternating direction method. SIAM J. Numer. Anal., 50(2):700-709, 2012a.
18. He, B. and Yuan, X. On non-ergodic convergence rate of douglas-rachford alternating direction method of multipliers. 2012b.
19. Hong, Mingyi and Luo, Zhi-Quan. On the linear convergence of the alternating direction method of multipliers, http://arxiv.org/abs/1208.3922, 2012.
20. Hu, Chonghai, Kwok, James T., and Pan, Weike. Accelcrated gradient methods for stochastic optimization and online learning. In NIPS 22, 2009.
21. Kim, S., Sohn, K., and Xing, E. P. A multivariate regression approach to association analysis of a quantitative trait network. Bioinformatics, 25(12):204-212, 2009.
22. Kogan, S., Levin, D., Routledge, B. R., Sagi, J. S., and Smith, N. A. Predicting risk from financial reports with regression. In NAACL-HLT 2009, Boulder, CO, 2009.
23. Lan, G. An optimal method for stochastic composite optimization. Mathematical Programming, 2010. doi: DOI10.1007/s10107-010-0434-y.
24. Lan, G. and Ghadimi, S. Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization, i: a generic algorithmic framework. SIAM J. on Optimization, 2011.
25. Langford, J., Li, L., and Zhang, T. Sparse online learning via truncated gradient. Journal of Machine Learning Research, pp. 777-801, 2009.
26. Monteiro, Renato D. C. and Svaiter, B. F. Iteration-complexity of block-decomposition algorithms and the alternating minimization augmented lagrangian method. Technical report, Georgia Institute of Technology, 2010.
27. Nemirovski, A. and Yudin, D. Problem Complexity and Method Efficiency in Optimization. John Wiley and Sons, 1983.
28. Nemirovski, A., Juditsky, A., Lan, G., and Shapiro, A. Robust stochastic approximation approach to stochastic programming. SIAM J. on Optimization, 19(4):1574-1609, 2009.
29. Nesterov, Y. Introductory Lectures on Convex Optimization, A Basic Course. Kluwer Academic Publishers, 2004.
30. Nesterov, Y. Gradient methods for minimizing composite objective function. Technical Report CORE DISCUSSION PAPER 2007/76, 2007.
31. Suzuki, Taiji. Dual averaging and proximal gradient descent for online alternating direction multiplier method. In Proceedings of ICML, 2013.
32. Tibshirani, R., Saunders, M., Rosset, S., Zhu, J., and Knight, K. Sparsity and smoothness via the fused lasso. Journal of the Royal Statistical Society: Series B, 67(1):91-108, 2004. (Pubitemid 40167043)
33. Tibshirani, R. J. and Taylor, J. The solution path of the generalized lasso. Annals of Statistics, 39(3):1335-1371, 2011.
34. Tseng, P. On accelerated proximal gradient methods for convex-concave optimization. SIAM J. Optim., 2008.
35. Vapnik, V. N. The nature of statistical learning theory. Springer-Verlag New York Incorporated, 2000.
36. Wang, H. and Banerjee, A. Online alternating direction method. In Proceedings of ICML, 2012.
37. Wright, S. J., Nowak, R. D., and Figueiredo, M. A. T. Sparse reconstruction by separable approximation. IEEE Transactions on Signal Processing, 57(7):2479-2493, 2009.
38. Xiao, L. Dual averaging methods for regularized stochastic learning and online optimization. JMLR, 11:2543-2596, 2010.
39. Yang, J. and Yuan, X. Linearized augmented lagrangian and alternating direction methods for nuclear norm minimization. Mathematics of Computation, 2012. doi: http://dx.doi.org/10.1090/S0025-5718-2012-02598-1.
40. 1-problems in compressive sensing. SIAM J. on Scientific Computing, 33(1):250-278, 2011.
41. Zhang, X., Burger, M., and Osher, S. A unified primal-dual algorithm framework based on bregman iteration. J. of Scientific Computing, 46(1):20-46, 2011.