Hybrid Random/Deterministic Parallel Algorithms for Convex and Nonconvex Big Data Optimization

IEEE Transactions on Signal Processing

Volumn 63, Issue 15, 2015, Pages 3914-3929

  Daneshmand, Amir ^a   Facchinei, Francisco ^b   Kungurtsev, Vyacheslav ^c   Scutari, Gesualdo ^d  

^a the State University of New York   (United States)

^b SAPIENZA UNIVERSITY OF ROME   (Italy)

^c CZECH TECHNICAL UNIVERSITY IN PRAGUE   (Czech Republic)

^d SCHOOL OF INDUSTRIAL ENGINEERING   (United States)

Author keywords

Jacobi method; nonconvex problems; parallel and distributed methods; random selections; sparse solution

Indexed keywords

ALGORITHMS; ITERATIVE METHODS;

DISTRIBUTED METHODS; JACOBI METHODS; NONCONVEX PROBLEM; RANDOM SELECTION; SPARSE SOLUTIONS;

BIG DATA;

EID: 84934290103     PISSN: 1053587X     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSP.2015.2436357     Document Type: Article

References (58)

1. A. Daneshmand, F. Facchinei, V. Kungurtsev, and G. Scutari, "Flexible selective parallel algorithms for big data optimization," in Proc. 48th Asilomar Confe. Signals, Syst., Comput., Nov. 2-5, 2014, pp. 3-7.
2. R. Tibshirani, "Regression shrinkage and selection via the lasso," J. Roy. Statist. Soc. B (Methodol.), pp. 267-288, 1996.
3. Z. Qin, K. Scheinberg, and D. Goldfarb, "Efficient block-coordinate descent algorithms for the group lasso," Math. Programm. Comput., vol. 5, pp. 143-169, June 2013.
4. G.-X. Yuan, K.-W. Chang, C.-J. Hsieh, and C.-J. Lin, "A comparison of optimization methods and software for large-scale l1-regularized linear classification," J. Mach. Learn. Res., vol. 11, pp. 3183-3234, 2010.
5. K. Fountoulakis and J. Gondzio, "A second-order method for strongly convex L1-regularization problems," 2013, arXiv preprint arXiv:1306. 5386 [Online]. Available: Http://arxiv.org/abs/1306.5386
6. I. Necoara and D. Clipici, "Efficient parallel coordinate descent algorithm for convex optimization problems with separable constraints: Application to distributed MPC," J. Process Control, vol. 23, no. 3, pp. 243-253, March 2013.
7. Y. Nesterov, "Gradient methods for minimizing composite functions," Math. Programm., vol. 140, pp. 125-161, August 2013.
8. P. Tseng and S. Yun, "A coordinate gradient descent method for nonsmooth separable minimization," Math. Programm., vol. 117, no. 1-2, pp. 387-423, March 2009.
9. A. Beck and M. Teboulle, "A fast iterative shrinkage-thresholding algorithm for linear inverse problems," SIAM J. Imag. Sci., vol. 2, no. 1, pp. 183-202, Jan. 2009.
10. S. J. Wright, R. D. Nowak, and M. A. Figueiredo, "Sparse reconstruction by separable approximation," IEEE Trans. Signal Process., vol. 57, no. 7, pp. 2479-2493, Jul. 2009.
11. Z. Peng, M. Yan, and W. Yin, "Parallel and distributed sparse optimization," in Proc. Asilomar Conf. IEEE Signals, Syst., Comput., 2013, pp. 659-646.
12. K. Slavakis and G. B. Giannakis, "Online dictionary learning from big data using accelerated stochastic approximation algorithms," presented at the IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Florence, Italy, May 4-9, 2014.
13. K. Slavakis, G. B. Giannakis, and G. Mateos, "Modeling and optimization for big data analytics," IEEE Signal Process. Mag., vol. 31, no. 5, pp. 18-31, Sep. 2014.
14. M. De Santis, S. Lucidi, and F. Rinaldi, 'A fast active set block coordinate descent algorithm for-regularized least squares," Mar. 2014, eprint arXiv:1403.1738 [Online]. Available: Http://arxiv.org/abs/1403. 1738
15. , S. Sra, S. Nowozin, and S. J. Wright, Eds., Optimization for Machine Learning, ser. Neural Information Processing. Cambridge, MA, USA: MIT Press, Sep. 2011.
16. F. Bach, R. Jenatton, J. Mairal, and G. Obozinski, "Optimization with sparsity-inducing penalties," Found. Trends Mach. Learn., vol. 4, no. 1, pp. 1-106, 2011.
17. J. K. Bradley, A. Kyrola, D. Bickson, and C. Guestrin, "Parallel coordinate descent for l1-regularized loss minimization," presented at the 28th Int. Conf. Mach. Learn., Bellevue, WA, USA, Jun. 28-Jul. 2, 2011.
18. M. Patriksson, "Cost approximation: A unified framework of descent algorithms for nonlinear programs," SIAM J. Optim., vol. 8, no. 2, pp. 561-582, 1998.
19. F. Facchinei, S. Sagratella, and G. Scutari, "Flexible parallel algorithms for big data optimization," presented at the IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Florence, Italy, May 4-9, 2014.
20. F. Facchinei, S. Sagratella, and G. Scutari, "Flexible parallel algorithms for big data optimization," IEEE Trans. Signal Process. vol. 63, no. 7, pp. 1874-1889, Apr. 2015.
21. O. Fercoq, Z. Qu, P. Richtárik, and M. Taká?, "Fast distributed coordinate descent for non-strongly convex losses," 2014, arXiv preprint arXiv:1405.5300 [Online]. Available: Http://arxiv.org/abs/1405.5300
22. O. Fercoq and P. Richtárik, "Accelerated, parallel and proximal coordinate Descent," 2013, arXiv preprint arXiv:1312.5799 [Online]. Available: Http://arxiv.org/abs/1312.5799
23. Z. Lu and L. Xiao, "Randomized block coordinate non-monotone gradient method for a class of nonlinear programming," 2013, arXiv preprint arXiv:1306.5918v1 [Online]. Available: Http://arxiv.org/pdf/1306.5918.pdf
24. I. Necoara and D. Clipici, "Distributed random coordinate descent method for composite minimization," Tech. Rep., Nov. 2013, pp. 1-41
25. [Online]. Available: Http://arxiv-web.arxiv.org/abs/1312.5302
26. Y. Nesterov, "Efficiency of coordinate descent methods on huge-scale optimization problems," SIAM J. Optim., vol. 22, no. 2, pp. 341-362, 2012.
27. P. Richtárik and M. Taká?, "Parallel coordinate descent methods for big data optimization," 2012, arXiv preprint arXiv:1212.0873 [Online]. Available: Http://arxiv.org/abs/1212.0873
28. S. Shalev-Shwartz and A. Tewari, "Stochastic methods for-regularized loss minimization," J. Mach. Learn. Res., pp. 1865-1892, 2011.
29. Z. Lu and L. Xiao, "On the complexity analysis of randomized blockcoordinate descent methods," 2013, arXiv preprint arXiv:1305.4723
30. [Online]. Available: Http://arxiv.org/abs/1305.4723
31. I. Necoara and A. Patrascu, "A random coordinate descent algorithm for optimization problems with composite objective function and linear coupled constraints," Comput. Optim. Appl., vol. 57, no. 2, pp. 307-337, 2014.
32. A. Patrascu and I. Necoara, "Efficient random coordinate descent algorithms for large-scale structured nonconvex optimization," J. Global Optim., pp. 1-23, Feb. 2014.
33. P. Richtárik and M. Taká?, "Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function," Math. Programm., vol. 144, no. 1-2, pp. 1-38, 2014.
34. I. Dassios, K. Fountoulakis, and J. Gondzio, "A second-order method for compressed sensing problems with coherent and redundant dictionaries," 2014, arXiv preprint arXiv:1405.4146 [Online]. Available: Http://arxiv.org/abs/1405.4146
35. G.-X. Yuan, C.-H. Ho, and C.-J. Lin, "An improved glmnet for-regularized logistic regression," J. Mach. Learn. Res., vol. 13, no. 1, pp. 1999-2030, 2012.
36. G. Scutari, F. Facchinei, P. Song, D. Palomar, and J.-S. Pang, "Decomposition by Partial linearization: Parallel optimization of multi-agent systems," IEEE Trans. Signal Process., vol. 62, pp. 641-656, Feb. 2014.
37. C. Scherrer, A. Tewari, M. Halappanavar, and D. Haglin, Curran Associates, Inc., "Feature clustering for accelerating parallel coordinate descent," in Adv. Neural Inf. Process. Syst. (NIPS), 2012, pp. 28-36.
38. A. Auslender, Optimisation: Méthodes Numériques. Paris, France: Masson, 1976.
39. P. Tseng, "Convergence of a block coordinate descent method for nondifferentiable minimization," J. Optim. Theory Appl., vol. 109, no. 3, pp. 475-494, 2001.
40. M. Razaviyayn, M. Hong, and Z.-Q. Luo, "A unified convergence analysis of block successive minimization methods for nonsmooth optimization," SIAM J. Opt., vol. 23, no. 2, pp. 1126-1153, 2013.
41. M. Razaviyayn, M. Hong, Z.-Q. Luo, and J.-S. Pang, "Parallel successive convex approximation for nonsmooth nonconvex optimization," presented at the Annu. Conf. Neural Inf. Process. Syst. (NIPS), Montreal, QC, Canada, Dec. 2014, to appear.
42. M. Hong, M. Razaviyayn, and Z.-Q. Luo, "Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems," Oct. 2014, arXiv preprint arXiv:1410.1390 [Online]. Available: Http://arxiv.org/abs/1410.1390
43. J. T. Goodman, "Exponential priors for maximum entropy models," U.S. Patent 7 340 376, Mar. 4, 2008.
44. K.-W. Chang, C.-J. Hsieh, and C.-J. Lin, "Coordinate descent method for large-scale l2-loss linear support vector machines," J.Mach. Learn. Res., vol. 9, pp. 1369-1398, 2008.
45. R. Tappenden, P. Richtárik, and J. Gondzio, "Inexact coordinate descent: Complexity and preconditioning," 2013, arXiv preprint arXiv:1304.5530 [Online]. Available: Http://arxiv.org/abs/1304.5530
46. P. Richtárik and M. Taká?, "Distributed coordinate descent method for learning with big data," 2013, arXiv preprint arXiv:1310.2059 [Online]. Available: Http://arxiv.org/abs/1310.2059
47. M. Razaviyayn, H.-W. Tseng, and Z.-Q. Luo, "Dictionary learning for sparse representation: Complexity and algorithms," presented at the IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Florence, Italy, May 4-9, 2014.
48. T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning, ser. Springer Series in Statistics. New York, NY, USA: Springer, 2009.
49. H. A. Eiselt and V. Marianov, "Pioneering developments in location analysis," in Foundations of Location Analysis, International Series in Operations Research & Management Science, A. Eiselt and V. Marianov, Eds. New York, NY, USA: Springer, 2011, ch. 11, pp. 3-22.
50. A. Chambolle, "An algorithm for total variation minimization and applications," J. Math. Imag. Vis., vol. 20, no. 1-2, pp. 89-97, Jan. 2004.
51. J. Mairal, F. Bach, J. Ponce, and G. Sapiro, "Online Dictionary Learning for Sparse Coding," in Proc. 26th Int. Conf. Mach. Learn., Montreal, QC, Canada, Jun. 14-18, 2009.
52. D. Goldfarb, S. Ma, and K. Scheinberg, "Fast alternating linearization methods for minimizing the sum of two convex functions," Math. Programm., vol. 141, pp. 349-382, Oct. 2013.
53. D. P. Bertsekas and J. N. Tsitsiklis, Parallel and Distributed Computation: Numerical Methods, 2nd ed. Belmont, MA, USA: Athena Scientific, 1989.
54. J.-S. Pang, "Error bounds in mathematical programming," Math. Programm., vol. 79, no. 1-3, pp. 299-332, 1997.
55. P. Richtárik and M. Taká?, "On optimal probabilities in stochastic coordinate descent methods," 2013, arXiv preprint arXiv:1310.3438 [Online]. Available: Http://arxiv.org/abs/1310.3438
56. A. Daneshmand, "Numerical comparison of hybrid random/deterministic parallel algorithms for nonconvex big data optimization," Dept. of Elect. Eng., SUNY Buffalo, Buffalo, NY, USA, Tech. Rep., Aug. 2014 [Online]. Available: Http://www.eng.buffalo.edu/?amirdane/DaneshmandTechRepNumCompAug14.pdf
57. CBCL Face Database #1, MIT Center for Biological and Computation Learning [Online]. Available: Http://www.ai.mit.edu/projects/cbcl
58. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming. Cambridge, MA, USA: Athena Scientific, May 2011.