A Unified Algorithmic Framework for Block-Structured Optimization Involving Big Data: With applications in machine learning and signal processing

IEEE Signal Processing Magazine

Volumn 33, Issue 1, 2016, Pages 57-77

  Hong, Mingyi ^a   Razaviyayn, Meisam ^b   Luo, Zhi Quan ^b   Pang, Jong Shi ^c  

^a UNIVERSITY OF MINNESOTA   (United States)

^b University of Minnesota   (United States)

^c University of Illinois at Urbana Champaign   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTATIONAL EFFICIENCY; FACTORIZATION; LEARNING SYSTEMS; MATRIX ALGEBRA; MAXIMUM PRINCIPLE; SIGNAL PROCESSING;

ALGORITHMIC FRAMEWORK; BLOCK COORDINATE DESCENTS; COMMUNICATION OVERHEADS; DATA OPTIMIZATION; DESIGN FLEXIBILITY; EXPECTATION-MAXIMIZATION METHOD; MASSIVE DATA SETS; NONNEGATIVE MATRIX FACTORIZATION METHOD (NMF);

BIG DATA;

EID: 85032750870     PISSN: 10535888     EISSN: None     Source Type: Journal    
DOI: 10.1109/MSP.2015.2481563     Document Type: Article

References (107)

1. V. Krishnamurthy, O. Namvar, and M. Hamdi, "Signal processing of social networks: Interactive sensing and learning," Found. Trends Signal Process., vol. 7, nos. 1-2. pp. 1-19, 2014.
2. Special Issue on Signal Processing for Big Data, IEEE Signal Processing Mag., vol. 31, no. 5, Sept. 2014.
3. S. Sra, S. Nowozin, and S. J. Wright, Optimization for Machine Learning. Cambridge, MA: MIT Press, 2012.
4. A. Roberts and L. Pachter, "Streaming fragment assignment for real-time analysis of sequencing experiments," Nat. Methods, vol. 10, no. 1, pp. 71-73, 2013.
5. Y. Nesterov, "Efficiency of coordinate descent methods on huge-scale optimization problems," SIAM J. Optim., vol. 22, no. 2, pp. 341-362, 2012.
6. V. Cevher, S. Becker, and M. Schmidt, "Convex optimization for big data: Scalable, randomized, and parallel algorithms for big data analytics," IEEE Signal Process. Mag., vol. 31, no. 5, pp. 32-43, Sept. 2014.
7. N. S. Aybat, "First order methods for large-scale sparse optimization," Ph.D. dissertation, Columbia Univ., 2014.
8. P. Richtik and M. Tak?, "Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function," Math. Program., vol. 144, no. 1, pp. 1-38, 2014.
9. S. Cui, Z.-Q. Luo, A. Hero, and J. Moura, Big Data over Networks. Cambridge, U.K.: Cambridge Univ. Press, 2015.
10. Y.-L. You and M. Kaveh, "A regularization approach to joint blur identification and image restoration," IEEE Trans. Image Processing, vol. 5, no. 3, pp. 416-428, 1996.
11. J. Friedman, T. Hastie, and R. Tibshirani, "Regularization paths for generalized linear models via coordinate descent," J. Stat. Softw., vol. 33, no. 1, pp. 1-22, 2010.
12. Q. Shi, M. Razaviyayn, Z.-Q. Luo, and C. He, "An iteratively weighted MMSE approach to distributed sum-utility maximization for a MIMO interfering broadcast channel," IEEE Trans. Signal Processing, vol. 59, no. 9, pp. 4331-4340, 2011.
13. B. Chen, S. He, Z. Li, and S. Zhang, "Maximum block improvement and polynomial optimization," SIAM J. Optim., vol. 22, no. 1, pp. 87-107, 2012.
14. J. Nutini, M. Schmidt, I. H. Laradji, M. Friedlander, and H. Koepke, "Coordinate descent converges faster with the Gauss-Southwell rule than random selection," in Proc. 30th Int. Conf. Machine Learning (ICML), 2015, pp. 1632-1641.
15. P. Richtik and M. Tak?, "Parallel coordinate descent methods for big data optimization," Math. Program., to be published.
16. F. Facchinei, S. Sagratella, and G. Scutari, "Flexible parallel algorithms for big data optimization," in Proc. 2014 IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), 2014, pp. 7208-7212.
17. J. Liu, C. Re S. J. Wright, and V. Bittorf, "An asynchronous parallel stochastic coordinate descent algorithm," in Proc. Int. Conf. Machine Learning (ICML), 2014.
18. M. Razaviyayn, M. Hong, Z.-Q. Luo, and J. S. Pang, "Parallel successive convex approximation for nonsmooth nonconvex optimization," in Proc. Neural Information Processing (NIPS), 2014, pp. 1440-1448.
19. F. Facchinei, G. Scutari, and S. Sagratella, "Parallel selective algorithms for nonconvex big data optimization," IEEE Trans. Signal Processing, vol. 63, no. 7, pp. 1874-1889, 2015.
20. M. Razaviyayn, M. Sanjabi, and Z.-Q. Luo, "A stochastic successive minimization method for nonsmooth nonconvex optimization with applications to transceiver design in wireless communication networks," preprint, 2013.
21. M. Razaviyayn, M. Hong, and Z.-Q. Luo, "A unified convergence analysis of block successive minimization methods for nonsmooth optimization," SIAM J. Optim., vol. 23, no. 2, pp. 1126-1153, 2013.
22. J. Mairal, "Optimization with first-order surrogate functions," in Proc. Int. Conf. Machine Learning (ICML), 2013, pp. 783-791.
23. A. Beck and M. Teboulle, "A fast iterative shrinkage-thresholding algorithm for linear inverse problems," SIAM J. Imaging Sci., vol. 2, no. 1, pp. 183-202, 2009.
24. M. A. T. Figueiredo, R. D. Nowak, and S. J. Wright, "Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems," IEEE J. Sel. Top. Signal Processing, vol. 1, no. 4, pp. 586-597, 2007.
25. E. Cand? and B. Recht, "Exact matrix completion via convex optimization," Commun. ACM, vol. 55, no. 6, pp. 111-119, 2012.
26. R. Sun and Z.-Q. Luo, "Guaranteed matrix completion via nonconvex factorization," preprint, 2014.
27. J. Kim, Y. He, and H. Park, "Algorithms for nonnegative matrix and tensor factorizations: A unified view based on block coordinate descent framework," J. Glob Optim., vol. 58, no. 2, pp. 285-319, Feb. 2014.
28. T. G. Kolda and B. W. Bader, "Tensor decompositions and applications," SIAM Rev., vol. 51, no. 3, pp. 455-500, 2009.
29. S. J. Wright, "Coordinate descent algorithms," Math. Program., vol. 151, no. 1, pp. 3-34, 2015.
30. A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum likelihood from incomplete data via the EM algorithm," J. R. Stat. Soc. Series B, vol. 39, no. 1, pp. 1-38, 1977.
31. A. L. Yuille and A. Rangarajan, "The concave-convex procedure," Neural Comput., vol. 15, pp. 915-936, 2003.
32. D. D. Lee and H. S. Seung, "Algorithms for non-negative matrix factorization," in Proc. Neural Information Processing Systems (NIPS), 2000, pp. 556-562.
33. C. Shi, R. A. Berry, and M. L. Honig, "Monotonic convergence of distributed interference pricing in wireless networks," in Proc. IEEE Int. Conf. Symp. Information Theory, 2009, pp. 1619-1623.
34. M. Hong and Z.-Q. Luo, "Signal processing and optimal resource allocation for the interference channel," in Academic Press Library in Signal Processing. New York: Academic, 2013.
35. M. Hong, Q. Li, and Y.-F. Liu, "Decomposition by successive convex approximation: A unifying approach for linear transceiver design in interfering heterogeneous networks," IEEE Trans. Wireless Commun., to be published.
36. P. Tseng, "Convergence of a block coordinate descent method for nondifferentiable minimization," J. Optim. Theory Appl., vol. 103, no. 9, pp. 475-494, 2001.
37. M. J. D. Powell, "On search directions for minimization algorithms," Math. Program., vol. 4, no. 1, pp. 193-201, 1973.
38. M. Razaviyayn, "Successive convex approximation: Analysis and applications," Ph.D. dissertation, Univ. of Minnesota, 2014.
39. Z. Lu and L. Xiao, "On the complexity analysis of randomized block-coordinate descent methods," Math. Program., vol. 152, no. 1, pp. 615-642, Aug. 2015.
40. A. Beck and L. Tetruashvili, "On the convergence of block coordinate descent type methods," SIAM J. Optim., vol. 23, no. 4, pp. 2037-2060, 2013.
41. A. Beck, "On the convergence of alternating minimization with applications to iteratively reweighted least squares and decomposition schemes," SIAM J. Optim., vol. 25, no. 1, pp. 185-209, 2015.
42. M. Hong, X. Wang, M. Razaviyayn, and Z.-Q. Luo, "Iteration complexity analysis of block coordinate descent methods," preprint, 2013.
43. A. Saha and A. Tewari, "On the nonasymptotic convergence of cyclic coordinate descent method," SIAM J. Optim., vol. 23, no. 1, pp. 576-601, 2013.
44. M. Hong, T.-H. Chang, X. Wang, M. Razaviyayn, S. Ma, and Z.-Q. Luo, "A block successive upper bound minimization method of multipliers for linearly constrained convex optimization," preprint, 2013.
45. M. Kadkhodaei, M. Sanjabi, and Z.-Q. Luo, "On the linear convergence of approximate proximal splitting methods for non-smooth convex minimization," J. Oper. Res. Soc. China, vol. 2, no. 2, pp. 123-141, 2014.
46. Y. Xu and W. Yin, "A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion," SIAM J. Imaging Sci., vol. 6, no. 3, pp. 1758-1789, 2013.
47. Q. Lin, Z. Lu, and L. Xiao, "An accelerated proximal coordinate gradient method and its application to regularized empirical risk minimization," SIAM J. Optim., to be published.
48. O. Fercoq and P. Richtik, "Accelerated, parallel and proximal coordinate descent," SIAM J. Optim., vol. 25, no. 4, pp. 1927-2023, 2015.
49. D. R. Hunter and K. Lange, "A tutorial on MM algorithms," Am. Stat., vol. 58, no. 1, pp. 30-37, 2004.
50. D. P. Bertsekas and J. N. Tsitsiklis, Parallel and Distributed Computation: Numerical Methods, 2nd ed. Belmont, MA: Athena-Scientific, 1999.
51. L. Grippo and M. Sciandrone, "On the convergence of the block nonlinear Gauss-Seidel method under convex constraints," Oper. Res. Lett., vol. 26, no. 3, pp. 127-136, 2000.
52. P. Combettes and J.-C. Pesquet, "Proximal splitting methods in signal processing," in Fixed-Point Algorithms for Inverse Problems in Science and Engineering, Springer Optimization and Its Applications. New York: Springer, 2011, pp. 185-212.
53. Z. Lu and L. Xiao, "Randomized block coordinate non-monotone gradient method for a class of nonlinear programming," preprint, 2013.
54. J. Mairal, F. Bach, J. Ponce, and G. Sapiro, "Online dictionary learning for sparse coding," in Proc. 26th Annu. Int. Conf. Machine Learning (ICML), 2009, pp. 689-696.
55. Q. Li, "On the convergence of Lee-Seung's multiplicative update for NMF," 2012, Tech. Note.
56. C. F?otte and J. Idier, "Algorithms for nonnegative matrix factorization with the beta-divergence," Neural Comput., vol. 23, no. 3, pp. 1-24, 2011.
57. I. Daubechies, R. DeVore, M. Fornasier, and C. S. Gunturk, "Iteratively reweighted least squares minimization for sparse recovery," Commun. Pure Appl. Math., vol. 63, no. 1, pp. 1-38, 2010.
58. Z-.Q. Luo and S. Zhang, "Dynamic spectrum management: Complexity and duality," IEEE J. Sel. Top. Signal Processing, vol. 2, no. 1, pp. 57-73, 2008.
59. H. Baligh, M. Hong, W.-C. Liao, Z.-Q. Luo, M. Razaviyayn, M. Sanjabi, and R. Sun, "Cross-layer provision of future cellular networks: A WMMSE-based approach," IEEE Signal Processing Mag., vol. 31, no. 6, pp. 56-68, Nov. 2014.
60. M. Razaviyayn, H. Baligh, A. Callard, and Z.-Q. Luo, "Joint user grouping and transceiver design in a mimo interfering broadcast channel," IEEE Trans. Signal Processing, vol. 62, no. 1, pp. 85-94, Jan. 2014.
61. S. S. Christensen, R. Agarwal, E. D. Carvalho, and J. M. Cioffi, "Weighted sum-rate maximization using weighted MMSE for MIMO-BC beamforming design," IEEE Trans. Wireless Commun., vol. 7, no. 12, pp. 4792-4799, 2008.
62. M. Hong, R. Sun, H. Baligh, and Z.-Q. Luo, "Joint base station clustering and beamformer design for partial coordinated transmission in heterogenous networks," IEEE J. Sel. Areas Commun., vol. 31, no. 2, pp. 226-240, 2013.
63. C. Shi, R. A. Berry, and M. L. Honig, "Distributed interference pricing with MISO channels," in Proc. 46th Annual Allerton Conf. Communication, Control, and Computing 2008, Sept. 2008, pp. 539-546.
64. S.-J. Kim and G. B. Giannakis, "Optimal resource allocation for MIMO Ad Hoc Cognitive Radio Networks," IEEE Trans. Inform. Theory, vol. 57, no. 5, pp. 3117-3131, 2011.
65. C. T. K. Ng and H. Huang, "Linear precoding in cooperative MIMO cellular networks with limited coordination clusters," IEEE J. Sel. Areas Commun., vol. 28, no. 9, pp. 1446-1454, 2010.
66. B. Langmead, C. Trapnell, M. Pop, and S. L. Salzberg, "Ultrafast and memoryefficient alignment of short dna sequences to the human genome," Genome Biol., vol. 10, no. 3, pp. R25, 2009.
67. H. Li and R. Durbin, "Fast and accurate short read alignment with Burrows-Wheeler transform," Bioinformatics, vol. 25, no. 14, pp. 1754-1760, 2009.
68. N. Bray, H. Pimentel, P. Melsted, and L. Pachter, "Near-optimal RNA-Seq quantification," preprint, 2015.
69. A. Shashua and T. Hazan, "Non-negative tensor factorization with applications to statistics and computer vision," in Proc. 22nd Int. Conf. Machine Learning (ICML), 2005, pp. 792-799.
70. I. Ibraghimov, "Application of the three-way decomposition for matrix compression," Numer. Linear Algebra Appl., vol. 9, no. 6-7, pp. 551-565, 2002.
71. J. H?tad, "Tensor rank is NP-complete," J. Algorithms, vol. 11, no. 4, pp. 644-654, 1990.
72. J. D. Carroll and J. J. Chang, "Analysis of individual differences in multidimensional scaling via an N-way generalization of 'Eckart-Young' decomposition," Psychometrika, vol. 35, no. 3, pp. 283-319, 1970.
73. R. A. Harshman, "Foundations of the parafac procedure: Models and conditions for an "explanatory" multimodal factor analysis," UCLA Working Papers in Phonetics, 16, pp. 1-84, 1970.
74. C. Navasca, L. De Lathauwer, and S. Kindermann, "Swamp reducing technique for tensor decomposition," in Proc. 16th European Signal Processing Conf. (EUSIPCO), Aug. 2008, pp. 1-5.
75. J. A. Bazerque, G. Mateos, and G. B. Giannakis, "Rank regularization and Bayesian inference for tensor completion and extrapolation," IEEE Trans. Signal Processing, vol. 61, no. 21-24, pp. 5689-5703, 2013.
76. D. L. Donoho, "Compressed sensing," IEEE Trans. Inform. Theory, vol. 52, no. 4, pp. 1289-1306, 2006.
77. E. J. Cand?, J. Romberg, and T. Tao, "Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information," IEEE Trans. Inform. Theory, vol. 52, no. 2, pp. 489-509, 2006.
78. M. Razaviyayn, H.-W. Tseng, and Z.-Q. Luo, "Dictionary learning for sparse representation: Complexity and algorithms," in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), 2014, pp. 5247-5251.
79. H. Lee, A. Battle, R. Raina, and A. Y. Ng, "Efficient sparse coding algorithms," in Proc. Advances in Neural Information Processing Systems (NIPS), 2006, pp. 801-808.
80. M. Aharon, M. Elad, and A. Bruckstein, "K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation," IEEE Trans. Signal Processing, vol. 54, no. 11, pp. 4311-4322, 2006.
81. G. McLachlan, Discriminant Analysis and Statistical Pattern Recognition, vol. 544. Hoboken, NJ: Wiley, 2004.
82. D. M. Witten and R. Tibshirani, "Penalized classification using Fisher's linear discriminant," J. R. Stat. Soc. Series B (Stat. Methodol.), vol. 73, no. 5, pp. 753-772, 2011.
83. S. Kim, R. Pasupathy, and S. G. Henderson, "A guide to sample average approximation," in Handbook of Simulation Optimization, M. C. Fu, Ed. vol. 216: International Series in Operations Research and Management Science. New York: Springer, 2015, pp. 207-243.
84. J. Mairal, "Stochastic majorization-minimization algorithms for large-scale optimization," in Proc. Advances in Neural Information Processing Systems, 2013, pp. 2283-2291.
85. M. Razaviyayn, M. Sanjabi, and Z.-Q. Luo, "A stochastic weighted MMSE approach to sum rate maximization for a MIMO interference channel," in Proc. IEEE 14th Workshop on Signal Processing Advances in Wireless Communications (SPAWC), IEEE, 2013, pp. 325-329.
86. J. Mairal, F. Bach, J. Ponce, and G. Sapiro, "Online learning for matrix factorization and sparse coding," J. Mach. Learn. Res., vol. 11, pp. 19-60, Jan. 2010.
87. K. Slavakis, S.-J. Kim, G. Mateos, and G. B. Giannakis, "Stochastic approximation vis-a-vis online learning for big data analytics," IEEE Signal Processing Mag., vol. 31, no. 6, pp. 124-129, 2014.
88. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, "Distributed optimization and statistical learning via the alternating direction method of multipliers," Found. Trends Mach. Learn., vol. 3, no. 1, pp. 1-122, 2011.
89. R. Glowinski, Numerical Methods for Nonlinear Variational Problems. New York: Springer-Verlag, 1984.
90. M. Hong, Z.-Q. Luo, and M. Razaviyayn, "Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems," submitted for publication.
91. B. Ames and M. Hong, "Alternating directions method of multipliers for l1-penalized zero variance discriminant analysis and principal component analysis," submitted for publication.
92. I. Necoara, Y. Nestrov, and F. Glineur, "A random coordinate descent method on large-scale optimization problems with linear constraints," Univ. Politehnica Bucharest, Tech. Rep, 2011.
93. S. Reddi, A. Hefny, C. Downey, A. Dubey, and S. Sra, "Large-scale randomizedcoordinate descent methods with non-separable linear constraints," preprint, 2014.
94. N. S. Aybat, G. Iyengar, and Z. Wang, "An asynchronous distributed proximal method for composite convex optimization," in Proc. 32th Int. Conf. Machine Learning (ICML), 2015, pp. 2454-2462.
95. W.-C. Liao, M. Hong, H. Farmanbar, and Z.-Qu. Luo, "Semi-asynchronous routing for large-scale hierarchical networks," in IEEE Int. Conf. Acoustics, Speech and Signal Processing, 2015, pp. 2894-2898.
96. Gurobi Optimization, Inc. Gurobi Optimizer Reference Manual, 2013. [Online]. Available: http://www.gurobi.com/documentation/6.0/refman/
97. D. P. Bertsekas, P. Hosein, and P. Tseng, "Relaxation methods for network flow problems with convex arc costs," SIAM J. Control Optim., vol. 25, no. 5, pp. 1219-1243, Sept. 1987.
98. B. R. Marks and G. P. Wright, "A general inner approximation algorithm for nonconvex mathematical programs," Oper. Res., vol. 26, no. 4, pp. 681-683, 1978.
99. F. Facchinei, S. Sagratella, and G. Scutari, "Flexible parallel algorithms for big data optimization," preprint, 2013.
100. G. Scutari, F. Facchinei, P. Song, D. P. Palomar, and J.-S. Pang, "Decomposition by partial linearization: Parallel optimization of multi-agent systems," IEEE Trans. Signal Processing, vol. 63, no. 3, pp. 641-656, 2014.
101. J.-S. Pang and M. Razaviyayn, "A unified distributed algorithm for non-cooperative games with non-convex and non-differentiable objectives," preprint, 2014.
102. Yu. Nesterov, "Gradient methods for minimizing composite functions," Math. Program., vol. 140, no. 1, pp. 125-161, 2013.
103. M. Razaviyayn, M. Hong, Z.-Q. Luo, and J.-S. Pang, "Parallel successive convex approximation for nonsmooth nonconvex optimization," in Proc. Advances in Neural Information Processing Systems (NIPS), 2014, pp. 1440-1448.
104. P. Richtik and M. Tak?, "Distributed coordinate descent method for learning with big data," preprint, 2013.
105. Z. Peng, M. Yan, and W. Yin, "Parallel and distributed sparse optimization," in Proc. IEEE 2013 Asilomar Conf. Signals, Systems and Computers, 2013, pp. 659-646.
106. C. Scherrer, A. Tewari, M. Halapanavar, and D. J. Haglin, "Feature clustering for accelerating parallel coordinate descent," in Proc. Advanced in Neural Information Processing Systems (NIPS), 2012, pp. 28-36.
107. O. Fercoq, Z. Qu, P. Richtik, and M. Tak?, "Fast distributed coordinate descent for non-strongly convex losses," in Proc. IEEE Int. Workshop on Machine Learning for Signal Processing (MLSP), 2014, pp. 1-6.