Sparse PCA: Optimal rates and adaptive estimation

Annals of Statistics

Volumn 41, Issue 6, 2013, Pages 3074-3110

  Tony Cai, T ^a   Ma, Zongming ^a   Wu, Yihong ^b  

^a UNIVERSITY OF PENNSYLVANIA   (United States)

^b University of Illinois at Urbana Champaign   (United States)

Author keywords

Adaptive estimation; Aggregation; Covariance matrix; Eigenvector; Group sparsity; Low rank matrix; Minimax lower bound; Optimal rate of convergence; Principal component analysis; Thresholding

Indexed keywords

EID: 84891928504     PISSN: 00905364     EISSN: None     Source Type: Journal    
DOI: 10.1214/13-AOS1178     Document Type: Article

References (60)

1. ABRAMOVICH, F., BENJAMINI, Y., DONOHO, D. L. and JOHNSTONE, I. M. (2006). Adapting to unknown sparsity by controlling the false discovery rate. Ann. Statist. 34 584-653. MR2281879
2. AMINI, A. A. and WAINWRIGHT, M. J. (2009). High-dimensional analysis of semidefinite relaxations for sparse principal components. Ann. Statist. 37 2877-2921. MR2541450
3. ANDERSON, T. W. (2003). An Introduction to Multivariate Statistical Analysis, 3rd ed. Wiley, Hoboken, NJ. MR1990662
4. BAIK, J. and SILVERSTEIN, J. W. (2006). Eigenvalues of large sample covariance matrices of spiked population models. J. Multivariate Anal. 97 1382-1408. MR2279680
5. BERTHET, Q. and RIGOLLET, P. (2013). Optimal detection of sparse principal components in high dimension. Ann. Statist. 41 1780-1815. MR3127849
6. BICKEL, P. J. and LEVINA, E. (2008). Covariance regularization by thresholding. Ann. Statist. 36 2577-2604. MR2485008
7. BIRGÉ, L. (1983). Approximation dans les espaces métriques et théorie de l'estimation. Z. Wahrsch. Verw. Gebiete 65 181-237. MR0722129
8. BIRGÉ, L. and MASSART, P. (2001). Gaussian model selection. J. Eur. Math. Soc. (JEMS) 3 203-268. MR1848946
9. BIRNBAUM, A., JOHNSTONE, I.M., NADLER, B. and PAUL, D. (2013). Minimax bounds for sparse PCA with noisy high-dimensional data. Preprint. Available at arXiv:1203.0967.
10. CAI, T. T., LIU, W. and ZHOU, H. H. (2012). Optimal estimation of large sparse precision matrices. Technical Report, Univ. Pennsylvania, PA.
11. CAI, T. T., MA, Z. and WU, Y. (2013). Optimal estimation and rank detection for sparse spiked covariance matrices. Preprint. Available at arXiv:1305.3235.
12. CAI, T. T., MA, Z. and WU, Y. (2013). Supplement to "Sparse PCA: Optimal rates and adaptive estimation." DOI:10.1214/13-AOS1178SUPP.
13. CAI, T. T. and YUAN, M. (2012). Adaptive covariance matrix estimation through block thresholding. Ann. Statist. 40 2014-2042. MR3059075
14. CAI, T. T., ZHANG, C.-H. and ZHOU, H. H. (2010). Optimal rates of convergence for covariance matrix estimation. Ann. Statist. 38 2118-2144. MR2676885
15. CAI, T. T. and ZHOU, H. H. (2012). Optimal rates of convergence for sparse covariance matrix estimation. Ann. Statist. 40 2389-2420. MR3097607
16. CHAMBERLAIN, G. and ROTHSCHILD, M. (1983). Arbitrage, factor structure, and mean-variance analysis on large asset markets. Econometrica 51 1281-1304. MR0736050
17. COVER, T. M. and THOMAS, J. A. (2006). Elements of Information Theory, 2nd ed. Wiley, Hoboken, NJ. MR2239987
18. D'ASPREMONT, A., EL GHAOUI, L., JORDAN, M. I. and LANCKRIET, G. R. G. (2007). A direct formulation for sparse PCA using semidefinite programming. SIAM Rev. 49 434-448 (electronic). MR2353806
19. DAVIDSON, K. R. and SZAREK, S. J. (2001). Handbook of the Geometry of Banach Spaces 317-366. North-Holland, Amsterdam. MR1863696
20. DAVIS, C. and KAHAN, W. M. (1970). The rotation of eigenvectors by a perturbation. III. SIAM J. Numer. Anal. 7 1-46. MR0264450
21. EATON, M. L. (1970). Some problems in covariance estimation. Technical Report 49, Dept. Statistics, Univ. Stanford, Stanford, CA.
22. HORN, R. A. and JOHNSON, C. R. (1990). Matrix Analysis. Cambridge Univ. Press, Cambridge. MR1084815
23. HOYLE, D. C. and RATTRAY, M. (2004). Principal-component-analysis eigenvalue spectra from data with symmetry-breaking structure. Phys. Rev. E (3) 69 026124.
24. JOHNSTONE, I. M. (2001). On the distribution of the largest eigenvalue in principal components analysis. Ann. Statist. 29 295-327. MR1863961
25. JOHNSTONE, I. M. (2001). Thresholding for weighted chi-squared. Statist. Sinica 11 691-704.
26. JOHNSTONE, I. M. and LU, A. Y. (2009). On consistency and sparsity for principal components analysis in high dimensions. J. Amer. Statist. Assoc. 104 682-693. MR2751448
27. JOLLIFFE, I. T., TRENDAFILOV, N. T. and UDDIN, M. (2003). A modified principal component technique based on the LASSO. J. Comput. Graph. Statist. 12 531-547. MR2002634
28. JOURNÉE, M., NESTEROV, Y., RICHTÁRIK, P. and SEPULCHRE, R. (2010). Generalized power method for sparse principal component analysis. J. Mach. Learn. Res. 11 517-553. MR2600619
29. JUDITSKY, A. and NEMIROVSKI, A. (2000). Functional aggregation for nonparametric regression. Ann. Statist. 28 681-712. MR1792783
30. JUNG, S. and MARRON, J. S. (2009). PCA consistency in high dimension, low sample size context. Ann. Statist. 37 4104-4130. MR2572454
31. KOLMOGOROV, A. N. and TIHOMIROV, V. M. (1959). ε-entropy and ε-capacity of sets in function spaces. Uspehi Mat. Nauk 14 3-86. MR0112032
32. KOLTCHINSKII, V., LOUNICI, K. and TSYBAKOV, A. B. (2011). Nuclear-norm penalization and optimal rates for noisy low-rank matrix completion. Ann. Statist. 39 2302-2329. MR2906869
33. LAURENT, B. and MASSART, P. (2000). Adaptive estimation of a quadratic functional by model selection. Ann. Statist. 28 1302-1338. MR1805785
34. LECAM, L. (1973). Convergence of estimates under dimensionality restrictions. Ann. Statist. 1 38-53. MR0334381
35. LOUNICI, K. (2013). Sparse principal component analysis with missing observations. In High Dimensional Probability VI 327-356. Springer, Basel.
36. LOUNICI, K., PONTIL, M., VAN DE GEER, S. and TSYBAKOV, A. B. (2011). Oracle inequalities and optimal inference under group sparsity. Ann. Statist. 39 2164-2204. MR2893865
37. MA, Z. (2013). Sparse principal component analysis and iterative thresholding. Ann. Statist. 41 772-801. MR3099121
38. MENDELSON, S. (2010). Empirical processes with a bounded ψ1 diameter. Geom. Funct. Anal. 20 988-1027. MR2729283
39. NADLER, B. (2008). Finite sample approximation results for principal component analysis: A matrix perturbation approach. Ann. Statist. 36 2791-2817. MR2485013
40. NEGAHBAN, S. and WAINWRIGHT, M. J. (2011). Estimation of (near) low-rank matrices with noise and high-dimensional scaling. Ann. Statist. 39 1069-1097. MR2816348
41. NEMIROVSKI, A. (2000). Topics in non-parametric statistics. In Lectures on Probability Theory and Statistics (Saint-Flour, 1998). Lecture Notes in Math. 1738 85-277. Springer, Berlin. MR1775640
42. PAUL, D. (2005). Nonparametric estimation of principal components. Ph.D. thesis, Univ. Stanford, Stanford, CA. MR2707156
43. PAUL, D. (2007). Asymptotics of sample eigenstructure for a large dimensional spiked covariance model. Statist. Sinica 17 1617-1642. MR2399865
44. RECHT, B., FAZEL, M. and PARRILO, P. A. (2010). Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Rev. 52 471-501. MR2680543
45. RIGOLLET, P. and TSYBAKOV, A. (2011). Exponential screening and optimal rates of sparse estimation. Ann. Statist. 39 731-771. MR2816337
46. ROHDE, A. and TSYBAKOV, A. B. (2011). Estimation of high-dimensional low-rank matrices. Ann. Statist. 39 887-930. MR2816342
47. SHEN, D., SHEN, H. and MARRON, J. S. (2013). Consistency of sparse PCA in high dimension, low sample size contexts. J. Multivariate Anal. 115 317-333.
48. SHEN, H. and HUANG, J. Z. (2008). Sparse principal component analysis via regularized low rank matrix approximation. J. Multivariate Anal. 99 1015-1034. MR2419336
49. STEIN, C. (1956). Some problems in multivariate analysis, part i. Technical Report 6, Dept. Statistics, Univ. Stanford.
50. SZAREK, S. J. (1982). Nets of Grassmann manifold and orthogonal group. In Proceedings of Research Workshop on Banach Space Theory (Iowa City, Iowa, 1981) 169-185. Univ. Iowa, Iowa City, IA. MR0724113
51. TSYBAKOV, A. B. (2009). Introduction to Nonparametric Estimation. Springer, New York. MR2724359
52. ULFARSSON, M. O. and SOLO, V. (2008). Sparse variable PCA using geodesic steepest descent. IEEE Trans. Signal Process. 56 5823-5832. MR2518261
53. VARMUZA, K. and FILZMOSER, P. (2009). Introduction to Multivariate Statistical Analysis in Chemometrics. CRC Press, Boca Raton, FL.
54. VU, V. Q. and LEI, J. (2012). Minimax rates of estimation for sparse PCA in high dimensions. In The Fifteenth International Conference on Artificial Intelligence and Statistics (AISTATS'12), available at http://arxiv.org/abs/ 1202.0786.
55. WEDIN, P.-Å. (1972). Perturbation bounds in connection with singular value decomposition. Nordisk Tidskr. Informationsbehandling (BIT) 12 99-111. MR0309968
56. WITTEN, D. M., TIBSHIRANI, R. and HASTIE, T. (2009). A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis. Biostatistics 10 515-534.
57. YANG, Y. (2000). Combining different procedures for adaptive regression. J. Multivariate Anal. 74 135-161. MR1790617
58. YANG, Y. and BARRON, A. (1999). Information-theoretic determination of minimax rates of convergence. Ann. Statist. 27 1564-1599. MR1742500
59. YUAN, X.-T. and ZHANG, T. (2013). Truncated power method for sparse eigenvalue problems. J. Mach. Learn. Res. 14 899-925. MR3063614
60. ZOU, H., HASTIE, T. and TIBSHIRANI, R. (2006). Sparse principal component analysis. J. Comput. Graph. Statist. 15 265-286. MR2252527