Alternating Direction Method of Multipliers for Sparse Principal Component Analysis

Journal of the Operations Research Society of China

Volumn 1, Issue 2, 2013, Pages 253-274

  Ma, Shiqian ^a  

^a CHINESE UNIVERSITY OF HONG KONG   (Hong Kong)

Author keywords

Alternating direction method; Augmented Lagrangian method; Deflation; Projection onto the simplex; Semidefinite programming; Sparse PCA

Indexed keywords

CONSTRAINED OPTIMIZATION; LAGRANGE MULTIPLIERS; MATRIX ALGEBRA; NUMERICAL ANALYSIS; RELAXATION PROCESSES; TEXT PROCESSING;

ALTERNATING DIRECTION METHODS; AUGMENTED LAGRANGIAN METHODS; DEFLATION; PROJECTION ONTO THE SIMPLEX; SEMI-DEFINITE PROGRAMMING; SPARSE PCA;

NUMERICAL METHODS;

EID: 84899048296     PISSN: 2194668X     EISSN: 21946698     Source Type: Journal    
DOI: 10.1007/s40305-013-0016-9     Document Type: Article

References (42)

1. Alizadeh, F.: Interior point methods in semidefinite programming with applications to combinatorial optimization. SIAM J. Optim. 5, 13-51 (1993).
2. Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3, 1-122 (2011).
3. Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004).
4. Brucker, P.: An O(n) algorithm for quadratic knapsack problems. Oper. Res. Lett. 3, 163-166 (1984).
5. Candès, E. J., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52, 489-509 (2006).
6. Combettes, P. L., Pesquet, J.-C.: A Douglas-Rachford splitting approach to nonsmooth convex variational signal recovery. IEEE J. Sel. Top. Signal Process. 1, 564-574 (2007).
7. Combettes, P. L., Wajs, V. R.: Signal recovery by proximal forward-backward splitting. SIAM J. Multiscale Model. Simul. 4, 1168-1200 (2005).
8. d'Aspremont, A., Bach, F., El Ghaoui, L.: Optimal solutions for sparse principal component analysis. J. Mach. Learn. Res. 9, 1269-1294 (2008).
9. d'Aspremont, A., El Ghaoui, L., Jordan, M. I., Lanckriet, G. R. G.: A direct formulation for sparse pca using semidefinite programming. SIAM Rev. 49, 434-448 (2007).
10. Donoho, D.: Compressed sensing. IEEE Trans. Inf. Theory 52, 1289-1306 (2006).
11. Douglas, J., Rachford, H. H.: On the numerical solution of the heat conduction problem in 2 and 3 space variables. Trans. Am. Math. Soc. 82, 421-439 (1956).
12. Duchi, J., Shalev-Shwartz, S., Singer, Y., Chandra, T.: Efficient projections onto the l1-ball for learning in high dimensions. In: ICML (2008).
13. Eckstein, J.: Splitting methods for monotone operators with applications to parallel optimization. PhD thesis, Massachusetts Institute of Technology (1989).
14. Eckstein, J., Bertsekas, D. P.: On the Douglas-Rachford splitting method and the proximal point algorithm for maximal monotone operators. Math. Program. 55, 293-318 (1992).
15. Gabay, D.: Applications of the method of multipliers to variational inequalities. In: Fortin, M., Glowinski, R. (eds.) Augmented Lagrangian Methods: Applications to the Solution of Boundary Value Problems. North-Holland, Amsterdam (1983).
16. Glowinski, R., Le Tallec, P.: Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics. SIAM, Philadelphia (1989).
17. Goldfarb, D., Ma, S.: Fast multiple splitting algorithms for convex optimization. SIAM J. Optim. 22, 533-556 (2012).
18. Goldfarb, D., Ma, S., Scheinberg, K.: Fast alternating linearization methods for minimizing the sum of two convex functions. Math. Program. (2012). doi: 10. 1007/s10107-012-0530-2.
19. Goldstein, T., Osher, S.: The split Bregman method for L1-regularized problems. SIAM J. Imaging Sci. 2, 323-343 (2009).
20. He, B. S., Liao, L.-Z., Han, D., Yang, H.: A new inexact alternating direction method for monotone variational inequalities. Math. Program. 92, 103-118 (2002).
21. Journee, M., Nesterov, Yu., Richtarik, P., Sepulchre, R.: Generalized power method for sparse principal component analysis. J. Mach. Learn. Res. 11, 517-553 (2010).
22. Lions, P. L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16, 964-979 (1979).
23. Lu, Z., Zhang, Y.: An augmented Lagrangian approach for sparse principal component analysis. Math. Program. 135, 149-193 (2012).
24. Luss, R., Teboulle, M.: Conditional gradient algorithms for rank-one matrix approximations with a sparsity constraint. SIAM Rev.
25. Mackey, L.: Deflation methods for sparse PCA. In: Advances in Neural Information Processing Systems (NIPS) (2008).
26. Malick, J., Povh, J., Rendl, F., Wiegele, A.: Regularization methods for semidefinite programming. SIAM J. Optim. 20, 336-356 (2009).
27. 2). Dokl. Akad. Nauk SSSR 269, 543-547 (1983).
28. Nesterov, Y. E.: Smooth minimization for non-smooth functions. Math. Program., Ser. A 103, 127-152 (2005).
29. Pardalos, P. M., Kovoor, N.: An algorithm for a singly constrained class of quadratic programs subject to upper and lower bounds. Math. Program. 46, 321-328 (1990).
30. Peaceman, D. H., Rachford, H. H.: The numerical solution of parabolic elliptic differential equations. SIAM J. Appl. Math. 3, 28-41 (1955).
31. Shalev-Shwartz, S., Singer, Y.: Efficient learning of label ranking by soft projections onto polyhedra. J. Mach. Learn. Res. 7, 1567-1599 (2006).
32. Saad, Y.: Projection and deflation methods for partial pole assignment in linear state feedback. IEEE Trans. Autom. Control 33, 290-297 (1998).
33. Scheinberg, K., Ma, S., Goldfarb, D.: Sparse inverse covariance selection via alternating linearization methods. In: NIPS (2010).
34. Todd, M. J.: Semidefinite optimization. Acta Numer. 10, 515-560 (2001).
35. van den Berg, E., Friedlander, M. P.: Probing the Pareto frontier for basis pursuit solutions. SIAM J. Sci. Comput. 31, 890-912 (2008).
36. Wang, Y., Yang, J., Yin, W., Zhang, Y.: A new alternating minimization algorithm for total variation image reconstruction. SIAM J. Imaging Sci. 1, 248-272 (2008).
37. Wen, Z., Goldfarb, D., Yin, W.: Alternating direction augmented Lagrangian methods for semidefinite programming. Math. Program. Comput. 2, 203-230 (2010).
38. 1 problems in compressive sensing. SIAM J. Sci. Comput. 33, 250-278 (2011).
39. Yuan, X.: Alternating direction methods for sparse covariance selection. J. Sci. Comput. 51, 261-273 (2012).
40. Zhang, Y., d'Aspremont, A., El Ghaoui, L.: Sparse PCA: Convex relaxations, algorithms and applications. In: Anjos, M., Lasserre, J. B. (eds.) Handbook on Semidefinite, Cone and Polynomial Optimization (2011).
41. Zhang, Y., El Ghaoui, L.: Large-scale sparse principal component analysis with application to text data. In: NIPS (2011).
42. Zou, H., Hastie, T., Tibshirani, R.: Sparse principle component analysis. J. Comput. Graph. Stat. 15, 265-286 (2006).