Parallel selective algorithms for nonconvex big data optimization

IEEE Transactions on Signal Processing

Volumn 63, Issue 7, 2015, Pages 1874-1889

  Facchinei, Francisco ^a   Scutari, Gesualdo ^b   Sagratella, Simone ^a  

^a SAPIENZA UNIVERSITY OF ROME   (Italy)

^b State University of New York   (United States)

Author keywords

distributed methods; Jacobi method; LASSO; Parallel optimization; sparse solution; variables selection

Indexed keywords

ALGORITHMS; CONVERGENCE OF NUMERICAL METHODS; ITERATIVE METHODS; NUMERICAL METHODS;

DISTRIBUTED METHODS; JACOBI METHODS; LASSO; PARALLEL OPTIMIZATION; SPARSE SOLUTIONS; VARIABLES SELECTIONS;

BIG DATA;

EID: 84924692154     PISSN: 1053587X     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSP.2015.2399858     Document Type: Article

References (41)

1. F. Facchinei, S. Sagratella, andG. Scutari, "Flexible parallel algorithms for big data optimization, " presented at the IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Florence, Italy, May 4-9, 2014.
2. R. Tibshirani, "Regression shrinkage and selection via the lasso, " J. Roy. Statist. Soc. Series B (Methodol.), pp. 267-288, 1996.
3. Z. Qin, K. Scheinberg, and D. Goldfarb, "Efficient block-coordinate descent algorithms for the group lasso, " Math. Programm. Comput., vol. 5, pp. 143-169, Jun. 2013.
4. A. Rakotomamonjy, "Surveying and comparing simultaneous sparse approximation (or group-lasso) algorithms, " Signal Process., vol. 91, no. 7, pp. 1505-1526, Jul. 2011.
5. G.-X.Yuan, K.-W.Chang, C.-J.Hsieh, andC.-J. Lin, "Acomparison of optimization methods and software for large-scale l1-regularized linear classification, " J. Mach. Learn. Res., vol. 11, pp. 3183-3234, 2010.
6. R. H. Byrd, J. Nocedal, and F. Oztoprak, "An inexact successive quadratic approximation method for convex L-1 regularized optimization, " 2013, arXiv preprint arXiv:1309.3529 [Online]. Available: http://arxiv.org/abs/1309.3529
7. K. Fountoulakis and J. Gondzio, "A second-order method for strongly convex L1-regularization problems, " 2013, arXiv preprint arXiv:1306. 5386 [Online]. Available: http://arxiv.org/abs/1306.5386
8. I. Necoara and D. Clipici, "Efficient parallel coordinate descent algorithm for convex optimization problems with separable constraints: Application to distributed MPC, " J. Process Control, vol. 23, no. 3, pp. 243-253, Mar. 2013.
9. Y. Nesterov, "Gradient methods for minimizing composite functions, " Math. Programm., vol. 140, pp. 125-161, Aug. 2013.
10. P. Tseng and S. Yun, "A coordinate gradient descent method for nonsmooth separable minimization, " Math. Programm., vol. 117, no. 1-2, pp. 387-423, Mar. 2009.
11. A. Beck and M. Teboulle, "A fast iterative shrinkage-thresholding algorithm for linear inverse problems, " SIAM J. Imag. Sci., vol. 2, no. 1, pp. 183-202, Jan. 2009.
12. S. J. Wright, R. D. Nowak, and M. A. Figueiredo, "Sparse reconstruction by separable approximation, " IEEE Trans. Signal Process., vol. 57, no. 7, pp. 2479-2493, Jul. 2009.
13. Z. Peng, M. Yan, and W. Yin, "Parallel and distributed sparse optimization, " in Proc. IEEE Asilomar Conf. Signals, Syst., Comput., 2013, pp. 659-646.
14. M. Razaviyayn, M. Hong, and Z.-Q. Luo, "A unified convergence analysis of block successive minimization methods for nonsmooth optimization, " SIAM J. Optim., vol. 23, no. 2, pp. 1126-1153, 2013.
15. P. L. Bühlmann, S. A. van de Geer, and S. Van de Geer, Statistics for High-Dimensional Data. New York, NY, USA: Springer, 2011.
16. Optimization for Machine Learning, ser. Neural Information Processing, S. Sra, S. Nowozin, and S. J. Wright, Eds. Cambridge, MA, USA: MIT Press, Sep. 2011.
17. F. Bach, R. Jenatton, J. Mairal, and G. Obozinski, "Optimization with sparsity-inducing penalties, " in Foundations and Trends® in Machine Learning. Norwell, MA, USA: Now Publishers, Dec. 2011.
18. Y. Li and S. Osher, "Coordinate descent optimization for minimization with application to compressed sensing; a greedy algorithm, " Inverse Probl. Imag., vol. 3, no. 3, pp. 487-503, 2009.
19. I. S. Dhillon, P. K. Ravikumar, and A. Tewari, , J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger, Eds., "Nearest neighbor based greedy coordinate descent, " in Advances in Neural Information Processing Systems 24. Red Hook, NY, USA: Curran Associates, 2011, pp. 2160-2168.
20. C. Scherrer, A. Tewari, M. Halappanavar, and D. Haglin, "Feature clustering for accelerating parallel coordinate descent, " in Proc. 26th Annu. Conf. Neural Inf. Process. Syst. (NIPS), Lake Tahoe, NV, USA, Dec. 3-6, 2012, pp. 28-36.
21. G. Scutari, F. Facchinei, P. Song, D. Palomar, and J.-S. Pang, "Decomposition by partial linearization: Parallel optimization of multi-agent systems, " IEEE Trans. Signal Process., vol. 62, pp. 641-656, Feb. 2014.
22. Y. Nesterov, "Efficiency of coordinate descent methods on huge-scale optimization problems, " SIAM J. Optim., vol. 22, no. 2, pp. 341-362, 2012.
23. J. K. Bradley, A. Kyrola, D. Bickson, and C. Guestrin, "Parallel coordinate descent for l1-regularized loss minimization, " presented at the 28th Int.Conf. Mach. Learn., Bellevue, WA, USA, Jun. 28-Jul. 2 2011.
24. I. Necoara, Y. Nesterov, and F. Glineur, "A random coordinate descent method on large optimization problems with linear constraints, " Tech. Rep., Jul. 2011, pp. 1-25 [Online]. Available: http://acse.pub.ro/person/ion-necoara/
25. I. Necoara and D. Clipici, "Distributed random coordinate descent method for composite minimization, " Tech. Rep., Nov. 2013, pp. 1-41 [Online]. Available: http://arxiv-web.arxiv.org/abs/1312.5302
26. Z. Lu and L. Xiao, "Randomized block coordinate non-monotone gradient method for a class of nonlinear programming, " 2013, arXiv preprint arXiv:1306.5918v1 [Online]. Available: http://arxiv.org/abs/1306.5918
27. P. Richtárik and M. Taká, "Parallel coordinate descent methods for big data optimization, " 2012, arXiv preprint arXiv:1212.0873 [Online]. Available: http://arxiv.org/abs/1212.0873
28. M.Yuan andY. Lin, "Model selection and estimation in regression with grouped variables, " J. Roy. Statist. Soc. Series B (Statist. Methodol.), vol. 68, no. 1, pp. 49-67, 2006.
29. S. K. Shevade and S. S. Keerthi, "A simple and efficient algorithm for gene selection using sparse logistic regression, " Bioinformatics, vol. 19, no. 17, pp. 2246-2253, 2003.
30. L. Meier, S. Van De Geer, and P. Bühlmann, "The group lasso for logistic regression, " J. Roy. Statist. Soc. Series B (Statist.Methodol.), vol. 70, no. 1, pp. 53-71, 2008.
31. J.Mairal, F. Bach, J. Ponce, and G. Sapiro, "Online dictionary learning for sparse coding, " presented at the 26th Int. Conf.Mach. Learn., Montreal, QC, Canada, Jun. 14-18, 2009.
32. D. Goldfarb, S. Ma, and K. Scheinberg, "Fast alternating linearization methods for minimizing the sum of two convex functions, " Math. Programm., vol. 141, pp. 349-382, Oct. 2013.
33. D. P. Bertsekas and J. N. Tsitsiklis, Parallel and Distributed Computation: Numerical Methods, 2nd ed. Belmont, MA, USA: Athena Scientific, 1989.
34. M. Patriksson, "Cost approximation: A unified framework of descent algorithms for nonlinear programs, " SIAM J. Optim., vol. 8, no. 2, pp. 561-582, 1998.
35. F. Facchinei and J.-S. Pang, Finite-Dimensional Variational Inequalities and Complementarity Problem. New York, NY, USA: Springer-Verlag, 2003.
36. G. Cohen, "Optimization by decomposition and coordination: A unified approach, " IEEE Trans. Autom. Control, vol. 23, no. 2, pp. 222-232, Apr. 1978.
37. G. Cohen, "Auxiliary problem principle and decomposition of optimization problems, " J. Optim. Theory Appl., vol. 32, no. 3, pp. 277-305, Nov. 1980.
38. M. Fukushima and H. Mine, "A generalized proximal point algorithm for certain non-convex minimization problems, " Int. J. Syst. Sci., vol. 12, no. 8, pp. 989-1000, 1981.
39. H. Mine and M. Fukushima, "A minimization method for the sum of a convex function and a continuously differentiable function, " J. Optim. Theory Appl., vol. 33, no. 1, pp. 9-23, Jan. 1981.
40. W. Deng, M.-J. Lai, Z. Peng, and W. Yin, "Parallel multi-block ADMM with convergence, " Mar. 2014, arXiv:1312.3040 [Online]. Available: http://arxiv.org/abs/1312.3040
41. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming. Cambridge, MA, USA: Athena Scientific, May 2011.