A proximal point algorithm for log-determinant optimization with group Lasso regularization

SIAM Journal on Optimization

Volumn 23, Issue 2, 2013, Pages 857-893

  Yang, Junfeng ^a   Sun, Defeng ^b   Toh, Kim Chuan ^b  

^a NANJING UNIVERSITY   (China)

^b NATIONAL UNIVERSITY OF SINGAPORE   (Singapore)

Author keywords

Alternating direction method; Augmented Lagrangian; Covariance selection; Gaussian graphical model; Group Lasso regularization; Log determinant optimization; Newton's method; Proximal point algorithm

Indexed keywords

ALTERNATING DIRECTION METHODS; AUGMENTED LAGRANGIANS; COVARIANCE SELECTION; GAUSSIAN GRAPHICAL MODELS; GROUP LASSOS; NEWTON'S METHODS; PROXIMAL POINT ALGORITHM;

ALGORITHMS; COVARIANCE MATRIX; LAGRANGE MULTIPLIERS; MAXIMUM LIKELIHOOD ESTIMATION; NEWTON-RAPHSON METHOD; OPTIMIZATION;

INVERSE PROBLEMS;

EID: 84880719563     PISSN: 10526234     EISSN: None     Source Type: Journal    
DOI: 10.1137/120864192     Document Type: Article

References (66)

1. F. R. Bach, Consistency of the group lasso and multiple kernel learning, J. Mach. Learn. Res., 9 (2008), pp. 1179-1225.
2. O. Banerjee, L. El Ghaoui, and A. d'Aspremont, Model selection through sparse maximum likelihood estimation for multivariate Gaussian or binary data, J. Mach. Learn. Res., 9 (2008), pp. 485-516.
3. J. A. Bilmes, Factored sparse inverse covariance matrices, in Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, IEEE Computer Society, Washington, D.C., 2000, pp. 1009-1012.
4. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, Distributed optimization and statistical learning via the alternating direction method of multipliers, Found. Trends Machine Learing, 3 (2010), pp. 1-122.
5. E. J. Candès, J. Romberg, and T. Tao, Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information, IEEE Trans. Inform. Theory, 52 (2006), pp. 489-509.
6. Z. Chan and D. F. Sun, Constraint nondegeneracy, strong regularity, and nonsingularity in semidefinite programming, SIAM J. Optim., 19 (2008), pp. 370-396.
7. C. H. Chen, B. S. He, and X. M. Yuan, Matrix completion via an alternating direction method, IMA J. Numer. Anal., 32 (2012), pp. 227-245.
8. S. S. Chen, D. L. Donoho, and M. A. Saunders, Atomic decomposition by basis pursuit, SIAM J. Sci. Comput., 20 (1998), pp. 33-61.
9. F. H. Clarke, Optimization and Nonsmooth Analysis, Canadian Mathematical Society Series of Monographs and Advanced Texts, John Wiley & Sons, New York, 1983.
10. Dahl, Joachim, Vandenberghe, Lieven and Roychowdhury, Vwani, Covariance selection for nonchordal graphs via chordal embedding, Optim. Methods Softw., 23(4), (2008), pp. 501-520.
11. A. d'Aspremont, L. El Ghaoui, M. I. Jordan, and Gert R. G. Lanckriet, A direct formulation for sparse PCA using semidefinite programming, SIAM Rev., 49 (2007), pp. 434-448.
12. A. Dempster, Covariance selection, Biometrics, 28 (1972), pp. 157-175.
13. A. Dobra, C. Hans, B. Jones, J. R. Nevins, G. A. Yao, and M. West, Sparse graphical models for exploring gene expression data, J. Multivariate Anal., 90 (2004), pp. 196-212.
14. A. Dobra and M. West, Bayesian Covariance Selection, 2004, http://ftp.stat.duke.edu/WorkingPapers/04-23.pdf.
15. D. L. Donoho, Compressed sensing, IEEE Trans. Inform. Theory, 52 (2006), pp. 1289-1306.
16. J. Duchi, S. Gould, and D. Koller, Projected subgradient methods for learning sparse gaussians, in Proceedings of Conference on Uncertainty in Artificial Intelligence (UAI), 2008.
17. 1-ball for learning in high dimensions, in Proceedings of the 25th International Conference on Machine Learning, Andrew McCallum and Sam Roweis, eds., Helsinki, Finland, 2008, pp. 272-279.
18. J. Eckstein and D. P. Bertsekas, On the Douglas-Rachford splitting method and the proximal point algorithm for maximal monotone operators, Math. Program., 55 (1992), pp. 293-318.
19. E. Esser, Applications of Lagrangian-Based Alternating Direction Methods and Connections to Split Bregman, UCLA CAM Report 09-31, 2009.
20. J. Fan, Y. Feng, and Y. Wu, Network exploration via the adaptive lasso and SCAD penalties, Ann. Appl. Stat., 3 (2009), pp. 521-541.
21. M. Fornasier and H. Rauhut, Recovery algorithms for vector-valued data with joint sparsity constraints, SIAM J. Numer. Anal., 46 (2008), pp. 577-613.
22. J. Friedman, T. Hastie, and R. Tibshirani, Sparse inverse covariance estimation with the graphical lasso, Biostatistics, 9 (2008), pp. 432-441.
23. D. Gabay and B. Mercier, A dual algorithm for the solution of nonlinear variational problems via finite-element approximations, Comput. Math. Appl., 2 (1976), pp. 17-40.
24. Y. Gao and D. F. Sun, Calibrating least squares semidefinite programming with equality and inequality constraints, SIAM J. Matrix Anal. Appl., 31 (2009), pp. 1432-1457.
25. R. Glowinski and A. Marrocco, Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité, d'une classe de problèmes de Dirichlet non linéaires, Rev. Française Automat. Informat. Recherche Opérationnelle, 9 (1975), pp. 41-76.
26. M. R. Hestenes, Multiplier and gradient methods, J. Optim. Theory Appl., 4 (1969), pp. 303-320.
27. J.-B. Hiriart-Urruty and C. Lemaréchal, Convex Analysis and Minimization Algorithms, Springer-Verlag, Berlin, 1993.
28. J. Honorio and D. Samaras, Multi-task learning of gaussian graphical models, in Proceedings of the 27th International Conference on Machine Learning, Haifa, Israel, 2010, pp. 447-454.
29. J. Huang, N. Liu, M. Pourahmadi, and L. Liu, Covariance matrix selection and estimation via penalised normal likelihood, Biometrika, 93 (2006), pp. 85-98.
30. B. Kummer, Newton's method for nondifferentiable functions, in Advances in Mathematical Optimization, Math. Res. 15, Akademie-Verlag, Berlin, 1988, pp. 114-125.
31. S. L. Lauritzen, Graphical Models, Oxford Statist. Sci. Ser. 17, Clarendon Press, New York, 1996.
32. C. Lemaréchal and C. Sagastizábal, Practical aspects of the Moreau-Yosida regularization: Theoretical preliminaries, SIAM J. Optim., 7 (1997), pp. 367-385.
33. H. Li and J. Gui, Gradient directed regularization for sparse Gaussian concentration graphs, with applications to inference of genetic networks, Biostatistics, 7 (2006), pp. 302-317.
34. 1-regularized sparse covariance selection, Math. Program. Comput., 2 (2010), pp. 291-315.
35. Z. Lu, Smooth optimization approach for sparse covariance selection, SIAM J. Optim., 19 (2009), pp. 1807-1827.
36. Z. Lu, Adaptive first-order methods for general sparse inverse covariance selection, SIAM J. Matrix Anal. Appl., 31 (2010), pp. 2000-2016.
37. B. M. Marlin and K. P. Murphy, Sparse Gaussian graphical models with unknown block structures, in Proceedings of the 26th International Conference on Machine Learning, Montreal, Canada, 2009, pp. 705-712.
38. B. Martinet, Réegularisation d'inéquations variationnelles par approximations successives, Rev. Françcaise Informat. Recherche Opérationnelle, 4 (1970), pp. 154-158.
39. L. Meier, S. van de Geer, and P. Bühlmann, The group Lasso for logistic regression, J. Roy. Statist. Soc. Ser. B Stat. Methodol., 70 (2008), pp. 53-71.
40. N. Meinshausen and P. Buehlmann, High-dimensional graphs and variable selection with the Lasso, Ann. Statist., 34 (2006), pp. 1436-1462.
41. J.-J. Moreau, Proximit'eet dualit'e dans un espace hilbertien, Bull. Soc. Math. France, 93 (1965), pp. 273-299.
42. 2), Dokl. Akad. Nauk SSSR, 269 (1983), pp. 543-547.
43. Yu. Nesterov, Smooth minimization of non-smooth functions, Math. Program., 103 (2005), pp. 127-152.
44. M. J. D. Powell, A method for nonlinear constraints in minimization problems, in Optimization, Academic Press, London, 1969, pp. 283-298.
45. L. Qi and J. Sun, A nonsmooth version of Newton's method, Math. Program., 58 (1993), pp. 353-367.
46. R. T. Rockafellar, Convex analysis. Princeton Math. Ser. 28, Princeton University Press, Princeton, NJ, 1996.
47. R. T. Rockafellar, Augmented Lagrangians and applications of the proximal point algorithm in convex programming, Math. Oper. Res., 1 (1976), pp. 97-116.
48. R. T. Rockafellar, Monotone operators and the proximal point algorithm, SIAM J. Control Optim., 14 (1976), pp. 877-898.
49. N. Sagara and M. Fukushima, A trust region method for nonsmooth convex optimization, J. Ind. Manag. Optim., 1 (2005), pp. 171-180.
50. F. Santosa and W. W. Symes, Linear inversion of band-limited reflection seismograms, SIAM J. Sci. Statist. Comput., 7 (1986), pp. 1307-1330.
51. Y. Shen, Z. W. Wen, and Y. Zhang, Augmented Lagrangian alternating direction method for matrix separation based on low-rank factorization, Optim. Methods Softw., to appear.
52. D. F. Sun, J. Sun, and L. Zhang, The rate of convergence of the augmented Lagrangian method for nonlinear semidefinite programming, Math. Program., 114 (2008), pp. 349-391.
53. R. Tibshirani, Regression shrinkage and selection via the Lasso, J. Roy. Statist. Soc. Ser. B, 58 (1996), pp. 267-288.
54. C. Wang, D. F. Sun, and K.-C. Toh, Solving log-determinant optimization problems by a Newton-CG primal proximal point algorithm, SIAM J. Optim., 20 (2010), pp. 2994-3013.
55. Z. W. Wen, D. Goldfarb, and W. Yin, Alternating direciotn augmented Lagrangina methods for semidefinite porgramming, Math. Program. Comput., 2 (2010), pp. 203-230.
56. B. Wu, C. Ding, D. F. Sun, and K.-C. Toh, On the Moreau-Yosida regularization of the vector k-norm related functions, Optimization, 2011, http://www.optimizationonline. org/DB FILE/2011/03/2978.pdf.
57. J. F. Yang and X. M. Yuan, Linearized augmented Lagrangian and alternating direction methods for nuclear norm minimization, Math. Comput., 82 (2013), pp. 301-329.
58. 1-problems in compressive sensing, SIAM J. Sci. Comput., 33 (2011), pp. 250-278.
59. J. F. Yang, Y. Zhang, and W. Yin, A fast alternating direction method for TVL1-L2 signal reconstruction from partial Fourier data, IEEE J. Selected Topics in Signal Processing, 4 (2010), pp. 288-297.
60. K. Yosida, Functional Analysis, Grundlehren Math. Wiss. 123, Springer-Verlag, Berlin, 1980.
61. M. Yuan and Y. Lin, Model selection and estimation in regression with grouped variables, J. Roy. Statist. Soc. Ser. B, 68 (2006), pp. 49-67.
62. M. Yuan and Y. Lin, Model selection and estimation in the Gaussian graphical model, Biometrika, 94 (2007), pp. 19-35.
63. X. M. Yuan, Alternating direction method of multipliers for covariance selection models, J. Sci. Comput., 51 (2012), pp. 261-273.
64. S. Yun, P. Tseng, and K.-C. Toh, A block coordinate gradient descent method for regularized convex separable optimization and covariance selection, Math. Program., 129 (2011), pp. 331-355.
65. P. Zhao, G. Rocha, and B. Yu, The composite absolute penalties family for grouped and hierarchical variable selection, Ann. Statist., 37 (2009), pp. 3468-3497.
66. X.-Y. Zhao, D. F. Sun, and K.-C. Toh, A Newton-CG augmented Lagrangian method for semidefinite programming, SIAM J. Optim., 20 (2010), pp. 1737-1765.