Regularized M-estimators with nonconvexity: Statistical and algorithmic theory for local optima

Journal of Machine Learning Research

Volumn 16, Issue , 2015, Pages 559-616

  Loh, Po Ling ^a   Wainwright, Martin J ^b  

^a UNIVERSITY OF PENNSYLVANIA   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

High dimensional statistics; M estimation; Model selection; Nonconvex optimization; Nonconvex regularization

Indexed keywords

ALGORITHMS; IMAGE ENHANCEMENT; POPULATION STATISTICS; STATISTICS;

HIGH-DIMENSIONAL STATISTICS; M-ESTIMATION; MODEL SELECTION; NONCONVEX; NONCONVEX OPTIMIZATION;

OPTIMIZATION;

EID: 84930632658     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (39)

1. A. Agarwal, S. Negahban, and M. J. Wainwright. Fast global convergence of gradient methods for high-dimensional statistical recovery. Annals of Statistics, 40(5):2452-2482, 2012.
2. K. S. Alexander. Rates of growth and sample moduli for weighted empirical processes indexed by sets. Probability Theory and Related Fields, 75:379-423, 1987.
3. O. Banerjee, L. El Ghaoui, and A. d'Aspremont. Model selection through sparse maximum likelihood estimation for multivariate Gaussian or binary data. Journal of Machine Learning Research, 9:485-516, 2008.
4. D. P. Bertsekas. Nonlinear Programming. Athena Scientific, Belmont, MA, 1999.
5. P. J. Bickel, Y. Ritov, and A. Tsybakov. Simultaneous analysis of Lasso and Dantzig selector. Annals of Statistics, 37(4):1705-1732, 2009.
6. P. Breheny and J. Huang. Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection. Annals of Applied Statistics, 5(1):232-253, 2011.
7. R. J. Carroll, D. Ruppert, and L. A. Stefanski. Measurement Error in Nonlinear Models. Chapman and Hall, 1995.
8. L. Chen and Y. Gu. The convergence guarantees of a non-convex approach for sparse recovery. IEEE Transactions on Signal Processing, 62(15):3754-3767, 2014.
9. J. Fan and R. Li. Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association, 96:1348-1360, 2001.
10. J. Fan, Y. Feng, and Y. Wu. Network exploration via the adaptive LASSO and SCAD penalties. Annals of Applied Statistics, pages 521-541, 2009.
11. J. Fan, L. Xue, and H. Zou. Strong oracle optimality of folded concave penalized estimation. The Annals of Statistics, 42(3):819-849, 06 2014.
12. J. Friedman, T. Hastie, and R. Tibshirani. Sparse inverse covariance estimation with the graphical Lasso. Biostatistics, 9(3):432-441, July 2008.
13. R. A. Horn and C. R. Johnson. Matrix Analysis. Cambridge University Press, 1990.
14. D. R. Hunter and R. Li. Variable selection using MM algorithms. Annals of Statistics, 33(4):1617-1642, 2005.
15. M. Ledoux and M. Talagrand. Probability in Banach Spaces: Isoperimetry and Processes. Springer-Verlag, New York, NY, 1991.
16. E. L. Lehmann and G. Casella. Theory of Point Estimation. Springer Verlag, 1998.
17. P. Loh and M. J. Wainwright. High-dimensional regression with noisy and missing data: Provable guarantees with non-convexity. Annals of Statistics, 40(3):1637-1664, 2012.
18. P. Loh and M. J. Wainwright. Structure estimation for discrete graphical models: Generalized covariance matrices and their inverses. The Annals of Statistics, 41(6):3022-3049, 12 2013a.
19. P. Loh and M. J. Wainwright. Regularized M-estimators with nonconvexity: Statistical and algorithmic theory for local optima. arXiv e-prints, May 2013b. Available at http://arxiv.org/abs/1305.2436.
20. P. Loh and M. J. Wainwright. Regularized M-estimators with nonconvexity: Statistical and algorithmic theory for local optima. In NIPS, pages 476-484, 2013c.
21. R. Mazumder, J. H. Friedman, and T. Hastie. Sparsenet: Coordinate descent with nonconvex penalties. Journal of the American Statistical Association, 106(495):1125-1138, 2011.
22. P. McCullagh and J. A. Nelder. Generalized Linear Models (Second Edition). London: Chapman & Hall, 1989.
23. S. Negahban, P. Ravikumar, M. J. Wainwright, and B. Yu. A unified framework for highdimensional analysis of M-estimators with decomposable regularizers. Statistical Science, 27(4):538-557, December 2012. See arXiv version for lemma/propositions cited here.
24. Y. Nesterov. Gradient methods for minimizing composite objective function. CORE Discussion Papers 2007076, Universit Catholique de Louvain, Center for Operations Research and Econometrics (CORE), 2007. URL http://EconPapers.repec.org/RePEc:cor:louvco:2007076.
25. Y. Nesterov and A. Nemirovskii. Interior Point Polynomial Algorithms in Convex Programming. SIAM studies in applied and numerical mathematics. Society for Industrial and Applied Mathematics, 1987.
26. G. Raskutti, M. J. Wainwright, and B. Yu. Minimax rates of estimation for high-dimensional linear regression over ℓq -balls. IEEE Transactions on Information Theory, 57(10):6976-6994, 2011.
27. P. Ravikumar, M. J. Wainwright, G. Raskutti, and B. Yu. High-dimensional covariance estimation by minimizing ℓ1-penalized log-determinant divergence. Electronic Journal of Statistics, 4:935-980, 2011.
28. M. Rosenbaum and A. B. Tsybakov. Sparse recovery under matrix uncertainty. Annals of Statistics, 38:2620-2651, 2010.
29. A. J. Rothman, P. J. Bickel, E. Levina, and J. Zhu. Sparse permutation invariant covariance estimation. Electronic Journal of Statistics, 2:494-515, 2008.
30. S. van de Geer. Empirical Processes in M-Estimation. Cambridge University Press, 2000.
31. A. W. van der Vaart and J. A. Wellner. Weak Convergence and Empirical Processes. Springer Series in Statistics. Springer-Verlag, New York, 1996. With applications to statistics.
32. S. A. Vavasis. Complexity issues in global optimization: A survey. In Handbook of Global Optimization, pages 27-41. Kluwer, 1995.
33. J.-P. Vial. Strong convexity of sets and functions. Journal of Mathematical Economics, 9(1-2):187-205, January 1982.
34. M. J. Wainwright. Structured regularizers for high-dimensional problems: Statistical and computational issues. Annual Review of Statistics and Its Application, 1(1):233-253, 2014.
35. Z. Wang, H. Liu, and T. Zhang. Optimal computational and statistical rates of convergence for sparse nonconvex learning problems. The Annals of Statistics, 42(6):2164-2201, 12 2014.
36. M. Yuan and Y. Lin. Model selection and estimation in the Gaussian graphical model. Biometrika, 94(1):19-35, 2007.
37. C.-H. Zhang. Nearly unbiased variable selection under minimax concave penalty. Annals of Statistics, 38(2):894-942, 2010.
38. C.-H. Zhang and T. Zhang. A general theory of concave regularization for high-dimensional sparse estimation problems. Statistical Science, 27(4):576-593, 2012.
39. H. Zou and R. Li. One-step sparse estimates in nonconcave penalized likelihood models. Annals of Statistics, 36(4):1509-1533, 2008.