Sparsistency and agnostic inference in sparse PCA

Annals of Statistics

Volumn 43, Issue 1, 2015, Pages 299-322

  Lei, B Y Jing ^a   Vu, Vincent Q ^a,b  

^a CARNEGIE MELLON UNIVERSITY   (United States)

^b OHIO STATE UNIVERSITY   (United States)

Author keywords

Agnostic inference; Principal components analysis; Sparsity; Subspace estimation; Variable selection

Indexed keywords

EID: 84922565321     PISSN: 00905364     EISSN: 21688966     Source Type: Journal    
DOI: 10.1214/14-AOS1273     Document Type: Article

References (45)

1. AKAIKE, H. (1973). Information theory and an extension of the likelihood principle. In Proceedings of the Second International Symposium of Information Theory. Akadémiai Kiado, Budapest.
2. AMINI, A. A. and WAINWRIGHT, M. J. (2009). High-dimensional analysis of semidefinite relaxations for sparse principal components. Ann. Statist. 37 2877-2921. MR2541450
3. BERK, R. H. (1966). Limiting behavior of posterior distributions when the model is incorrect. Ann. Math. Statist. 37 51-58; Correction, Ibid 37 745-746. MR0189176
4. BERTHET, Q. and RIGOLLET, P. (2013a). Optimal detection of sparse principal components in high dimension. Ann. Statist. 41 1780-1815. MR3127849
5. BERTHET, Q. and RIGOLLET, P. (2013b). Computational lower bounds for sparse PCA. Preprint. Available at arXiv:1304.0828.
6. BIRNBAUM, A., JOHNSTONE, I.M.,NADLER, B. and PAUL, D. (2013). Minimax bounds for sparse PCA with noisy high-dimensional data. Ann. Statist. 41 1055-1084. MR3113803
7. BOYD, S., PARIKH, N., CHU, E., PELEATO, B. and ECKSTEIN, J. (2010). Distributed optimization and statistical learning via the alternating direction method of multipliers. Faund. Trends Mach. Learn. 3 1-122.
8. BÜHLMANN, P. and VAN DE GEER, S. (2011). Statistics for High-Dimensional Data: Methods, Theory and Applications. Springer, Heidelberg. MR2807761
9. BUJA, A., HASTIE, T. and TIBSHIRANI, R. (1989). Linear smoothers and additive models. Ann. Statist. 17 453-555. MR0994249
10. CAI, T. T., M A, Z. and W U, Y. (2013). Sparse PCA: Optimal rates and adaptive estimation. Ann. Statist. 41 3074-3110. MR3161458
11. D 'A SPREMONT, A., BACH, F. and E L GHAOUI, L. (2008). Optimal solutions for sparse principal component analysis. J. Mach. Learn. Res. 9 1269-1294. MR2426043
12. D 'A SPREMONT, A., E L GHAOUI, L., JORDAN, M. I. and LANCKRIET, G. R. G. (2007). A direct formulation for sparse PCA using semidefinite programming. SIAM Rev. 49 434-448 (electronic). MR2353806
13. DESHPANDE, Y. andM ONTANARI, A. (2013). Finding hidden cliques of size √N/e in nearly linear time. Preprint. Available at arXiv:1304.7047.
14. FAN, J. and L I, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. J. Amer. Statist. Assoc. 96 1348-1360. MR1946581
15. GREENSHTEIN, E. and RITOV, Y. (2004). Persistence in high-dimensional linear predictor selection and the virtue of overparametrization. Bernoulli 10 971-988. MR2108039
16. nd ed. Springer, New York. MR2722294
17. HOTELLING, H. (1933). Analysis of a complex of statistical variables into principal components. J. Educ. Psychol. 498-520.
18. HUBER, P. J. (1967). The behavior of maximum likelihood estimates under nonstandard conditions. In Proc. Fifth Berkeley Sympos. Math. Statist. and Probability (Berkeley, Calif., 1965/66), Vol. I: Statistics 221-233. Univ. California Press, Berkeley. MR0216620
19. JOHNSTONE, I. M. and L U, A. Y. (2009). On consistency and sparsity for principal components analysis in high dimensions. J. Amer. Statist. Assoc. 104 682-693. MR2751448
20. JOLLIFFE, I. T., TRENDAFILOV, N. T. and UDDIN, M. (2003). A modified principal component technique based on the LASSO. J. Comput. Graph. Statist. 12 531-547. MR2002634
21. JOURNÉE, M., NESTEROV, Y., R ICHTÁRIK, P. and SEPULCHRE, R. (2010). Generalized power method for sparse principal component analysis. J. Mach. Learn. Res. 11 517-553. MR2600619
22. KEARNS, M. J., SCHAPIRE, R. E. and SELLIE, L. M. (1994). Toward efficient agnostic learning. Mach. Learn. 17 115-141.
23. KRAUTHGAMER, R., NADLER, B. and VILENCHIK, D. (2013). Do semidefinite relaxations solve sparse PCA up to the information limit? Preprint. Available at ArXiv:1306.3690.
24. LAM, C. and FAN, J. (2009). Sparsistency and rates of convergence in large covariance matrix estimation. Ann. Statist. 37 4254-4278. MR2572459
25. LOUNICI, K. (2013). Sparse principal component analysis with missing observations. Progr. Probab. 66 327-356.
26. M A, Z. (2013). Sparse principal component analysis and iterative thresholding. Ann. Statist. 41 772-801. MR3099121
27. MACKEY, L. W. (2009). Deflation methods for sparse PCA. In Advances in Neural Information Processing Systems 21 (D. Koller, D. Schuurmans, Y. Bengio and L. Bottou, eds.) 1017-1024. Curran Associates, Red Hook, NY.
28. MEINSHAUSEN, N. and BÜHLMANN, P. (2006). High-dimensional graphs and variable selection with the lasso. Ann. Statist. 34 1436-1462. MR2278363
29. NEGAHBAN, S. N., RAVIKUMAR, P., WAINWRIGHT, M. J. and Y U, B. (2012). A unified framework for high-dimensional analysis of M -estimators with decomposable regularizers. Statist. Sci. 27 538-557. MR3025133
30. OVERTON, M. L. and WOMERSLEY, R. S. (1992). On the sum of the largest eigenvalues of a symmetric matrix. SIAM J. Matrix Anal. Appl. 13 41-45. MR1146651
31. PAUL, D. and JOHNSTONE, I. M. (2012). Augmented sparse principal component analysis for high dimensional data. Preprint. Available at arXiv:1202.1242.
32. PEARSON, K. (1901). On lines and planes of closest fit to systems of points in space. Philos. Mag. 2 559-572.
33. RAVIKUMAR, P.,WAINWRIGHT, M. J., RASKUTTI, G. and Y U, B. (2011). High-dimensional covariance estimation by minimizingℓ1 -penalized log-determinant divergence. Electron. J. Stat. 5 935-980. MR2836766
34. ROTHMAN, A. J., BICKEL, P. J., LEVINA, E. and ZHU, J. (2008). Sparse permutation invariant covariance estimation. Electron. J. Stat. 2 494-515. MR2417391
35. SHEN, H. and HUANG, J. Z. (2008). Sparse principal component analysis via regularized low rank matrix approximation. J. Multivariate Anal. 99 1015-1034. MR2419336
36. VAN DER VAART, A. W. andW ELLNER, J. A. (1996). Weak Convergence and Empirical Processes: With Applications to Statistics. Springer, New York. MR1385671
37. V U, V. Q. and LEI, J. (2012). Minimax rates of estimation for sparse PCA in high dimensions. In Proc. Fifteenth International Conference on Artificial Intelligence and Statistics JMLR W&CP 22 1278-1286.
38. V U, V. Q. and LEI, J. (2013). Minimax sparse principal subspace estimation in high dimensions. Ann. Statist. 41 2905-2947. MR3161452
39. V U, V. Q., CHO, J., LEI, J. and ROHE, K. (2013). Fantope projection and selection: A near-optimal convex relaxation of sparse PCA. In Advances in Neural Information Processing Systems (NIPS) 26 (C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani and K. Q. Weinberger, eds.) 2670-2678. Curran Associates, Red Hook, NY.
40. WAINWRIGHT, M. J. (2009). Sharp thresholds for high-dimensional and noisy sparsity recovery using ℓ1 -constrained quadratic programming (Lasso). IEEE Trans . Inform. Theory 55 2183-2202. MR2729873
41. WHITE, H. (1982). Maximum likelihood estimation of misspecified models. Econometrica 50 1-25. MR0640163
42. WITTEN, D.M., TIBSHIRANI, R. and HASTIE, T. (2009). A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis. Biostatistics 10 515-534.
43. YUAN, X.-T. and ZHANG, T. (2013). Truncated power method for sparse eigenvalue problems. J. Mach. Learn. Res. 14 899-925. MR3063614
44. ZHAO, P. and Y U, B. (2006). On model selection consistency of Lasso. J. Mach. Learn. Res. 7 2541-2563. MR2274449
45. ZOU, H., HASTIE, T. and TIBSHIRANI, R. (2006). Sparse principal component analysis. J. Comput. Graph. Statist. 15 265-286. MR2252527