OptShrink: An algorithm for improved low-rank signal matrix Denoising by optimal, data-driven singular value shrinkage

IEEE Transactions on Information Theory

Volumn 60, Issue 5, 2014, Pages 3002-3018

  Nadakuditi, Raj Rao ^a  

^a UNIVERSITY OF MICHIGAN   (United States)

Author keywords

free probability; Haar measure; informational limit; phase transition; principal components analysis; random eigenvalues; random eigenvectors; Random matrices; random perturbation; sample covariance matrices.

Indexed keywords

ALGORITHMS; COVARIANCE MATRIX; EIGENVALUES AND EIGENFUNCTIONS; GAUSSIAN NOISE (ELECTRONIC); OPTIMIZATION; PHASE TRANSITIONS; PRINCIPAL COMPONENT ANALYSIS; RANDOM PROCESSES; RANDOM VARIABLES; SHRINKAGE;

FREE PROBABILITIES; HAAR MEASURE; INFORMATIONAL LIMIT; PRINCIPAL COMPONENTS ANALYSIS; RANDOM EIGENVALUES; RANDOM MATRICES; RANDOM PERTURBATIONS; SAMPLE COVARIANCES;

SIGNAL DENOISING;

EID: 84899641400     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2014.2311661     Document Type: Article

References (89)

1. Z. Bai and J. W. Silverstein, Spectral Analysis of Large Dimensional Random Matrices. New York, NY, USA: Springer-Verlag, 2010.
2. J. Baik, G. Ben Arous, and S. Pch, Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices, Ann. Probab., vol. 33, no. 5, pp. 1643-1697, 2005. (Pubitemid 41560766)
3. J. Baik and J. W. Silverstein, Eigenvalues of large sample covariance matrices of spiked population models, J. Multivariate Anal., vol. 97, no. 6, pp. 1382-1408, 2006. (Pubitemid 43705671)
4. G. Ben Arous and P. Bourgade, Extreme gaps between eigenvalues of random matrices, Ann. Probab., vol. 41, no. 4, pp. 2648-2681, 2013.
5. F. Benaych-Georges and R. R. Nadakuditi, The singular values and vectors of low rank perturbations of large rectangular random matrices, J. Multivariate Anal., vol. 111, pp. 120-135, Oct. 2012.
6. F. Benaych-Georges, Rectangular random matrices, related convolution, Probab. Theory Rel. Fields, vol. 144, nos. 3-4, pp. 471-515, 2009.
7. A. Birnbaum, I. M. Johnstone, B. Nadler, and D. Paul, Minimax bounds for sparse PCA with noisy high-dimensional data, Ann. Statist., vol. 41, no. 3, pp. 1055-1084, 2013.
8. A. Bloemendal, L. Erd?os, A. Knowles, H.-T. Yau, and J. Yin, Isotropic local laws for sample covariance and generalized Wigner matrices, arXiv: 1308.5729.
9. S. Boucheron, G. Lugosi, and P. Massart, Concentration Inequalities: A Nonasymptotic Theory of Independence. London, U.K.: Oxford Univ. Press, 2013.
10. C. Boutsidis and E. Gallopoulos, SVD based initialization: A head start for nonnegative matrix factorization, Pattern Recognit., vol. 41, no. 4, pp. 1350-1362, 2008. (Pubitemid 350212975)
11. L. Breiman, Better subset regression using the nonnegative garrote, Technometrics, vol. 37, no. 4, pp. 373-384, 1995.
12. C. Cacciapuoti, A. Maltsev, and B. Schlein, Local Marchenko-Pastur law at the hard edge of sample covariance matrices, vol. 54, no. 4, p. 043302, 2012.
13. J. A. Cadzow and D. M. Wilkes, Enhanced rational signal modeling, Signal Process., vol. 25, no. 2, pp. 171-188, 1991.
14. J. F. Cai, E. J. Cands, and Z. Shen, A singular value thresholding algorithm for matrix completion, SIAM J. Optim., vol. 20, no. 4, pp. 1956-1982, 2010.
15. E. J. Cands, X. Li, Y. Ma, and J. Wright, Robust principal component analysis? J. ACM, vol. 58, no. 3, pp. 11:1-11:37, Jun. 2011.
16. E. J. Candes and Y. Plan, Matrix completion with noise, Proc. IEEE, vol. 98, no. 6, pp. 925-936, Jun. 2010.
17. E. J. Cands and B. Recht, Exact matrix completion via convex optimization, Found. Comput. Math., vol. 9, no. 6, pp. 717-772, 2009.
18. E. J. Cands and T. Tao, The power of convex relaxation: Nearoptimal matrix completion, IEEE Trans. Inf. Theory, vol. 56, no. 5, pp. 2053-2080, May 2010.
19. V. Chandrasekaran, S. Sanghavi, P. A. Parrilo, and A. S.Willsky, Sparse and low-rank matrix decompositions, in Proc. 47th Annu. Allerton Conf. Commun., Control, Comput., Oct. 2009, pp. 962-967.
20. V. Chandrasekaran, S. Sanghavi, P. A. Parrilo, and A. S. Willsky, Ranksparsity incoherence for matrix decomposition, SIAM J. Optim., vol. 21, no. 2, pp. 572-596, 2011.
21. S. Chatterjee, Matrix estimation by universal singular value thresholding, arXiv:1212.1247, 2012.
22. P. Chen and D. Suter, Recovering the missing components in a large noisy low-rank matrix: Application to SFM, IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, no. 8, pp. 1051-1063, Aug. 2004.
23. M. T. Chu, R. E. Funderlic, and R. J. Plemmons, Structured low rank approximation, Linear Algebra Appl., vol. 366, pp. 157-172, Jun. 2003.
24. P. L. Combettes and J. W. Silverstein, Signal detection via spectral theory of large dimensional random matrices, IEEE Trans. Signal Process., vol. 8, no. 40, pp. 2100-2105, Aug. 1992.
25. S. Dasgupta, Learning mixtures of Gaussians, in Proc. 40th Annu. Symp. Found. Comput. Sci., 1999, pp. 634-644. (Pubitemid 30539944)
26. A. dAspremont, F. Bach, and L. El Ghaoui, Optimal solutions for sparse principal component analysis, J. Mach. Learn. Res., vol. 9, pp. 1269-1294, Jun. 2008.
27. C.-A. Deledalle, S. Vaiter, G. Peyr, J. Fadili, and C. Dossal, Risk estimation for matrix recovery with spectral regularization, arXiv:1205.1482, 2012.
28. P. Drineas, R. Kannan, and M. W. Mahoney, Fast Monte Carlo algorithms for matrices II: Computing a low-rank approximation to a matrix, SIAM J. Comput., vol. 36, no. 1, pp. 158-183, 2006. (Pubitemid 46374022)
29. C. Eckart and G. Young, The approximation of one matrix by another of lower rank, Psychometrika, vol. 1, no. 3, pp. 211-218, 1936.
30. N. El Karoui, Tracy-Widom limit for the largest eigenvalue of a large class of complex sample covariance matrices, Ann. Probab., vol. 35, no. 2, pp. 663-714, 2007.
31. L. Erd?os and H.-T. Yau, Universality of local spectral statistics of random matrices, Bull. Amer. Math. Soc., vol. 49, no. 3, pp. 377-414, 2012.
32. M. Fazel, E. J. Candes, B. Recht, and P. Parrilo, Compressed sensing and robust recovery of low rank matrices, in Proc. 42nd Asilomar Conf. Signals, Syst. Comput., Oct. 2008, pp. 1043-1047.
33. D. Fral and S. Pch, The largest eigenvalues of sample covariance matrices for a spiked population: Diagonal case, J. Math. Phys., vol. 50, no. 7, p. 073302, 2009.
34. G. H. Golub, A. Hoffman, and G. W. Stewart, A generalization of the Eckart-Young-Mirsky matrix approximation theorem, Linear Algebra Appl., vol. 88, pp. 317-327, Apr. 1987.
35. W. Hachem, P. Loubaton, X. Mestre, J. Najim, and P. Vallet, A subspace estimator for fixed rank perturbations of large random matrices, J. Multivariate Anal., vol. 114, pp. 427-447, Feb. 2013.
36. R. A. Horn and C. R. Johnson, Topics in Matrix Analysis. Cambridge, U.K.: Cambridge Univ. Press, 1991.
37. D. Hsu and S. M. Kakade, Learning gaussian mixture models: Moment methods and spectral decompositions, in Proc. 4th Conf. ITCS, 2012, pp. 11-20.
38. R. Jennaton, G. Obozinksi, and F. Bach, Structured sparse principal component analysis, in Proc. Int. Conf. Artif. Intell. Statist., vol. 9, pp. 366-373, 2010.
39. I. M. Johnstone and A. Lu, On consistency and sparsity for principal components analysis in high dimensions, J. Amer. Statist. Assoc., vol. 104, no. 486, pp. 682-693, 2009.
40. I. M. Johnstone, On the distribution of the largest eigenvalue in principal components analysis, Ann. Statist., vol. 29, no. 2, pp. 295-327, 2001.
41. I. M. Johnstone, High dimensional statistical inference and random matrices, in Proc. Int. Congr. Math., Madrid, 2006, pp. 307-333.
42. I. T. Jolliffe, Principal Component Analysis, vol. 2.New York,NY, USA: Wiley, 2002.
43. S. M. Kakade, S. Shalev-Shwartz, and A. Tewari, Regularization techniques for learning with matrices, J. Mach. Learn. Res., vol. 13, pp. 1865-1890, Mar. 2012.
44. R. Kannan, H. Salmasian, and S. Vempala, The spectral method for general mixture models, in Proc. Learn. Theory, 2005, pp. 155-199.
45. R. Kannan and S. Vempala, Spectral Algorithms. Norwell, MA, USA: Now Publishers Inc., 2009.
46. D. R. Karger, S. Oh, and D. Shah, Budget-optimal crowdsourcing using low-rank matrix approximations, in Proc. 49th Annu. Allerton Conf. Commun., Control, Comput., Sep. 2011, pp. 284-291.
47. R. H. Keshavan, A. Montanari, and S. Oh, Matrix completion from a few entries, IEEE Trans. Inf. Theory, vol. 56, no. 6, pp. 2980-2998, Jun. 2010.
48. V. C. Klema and A. Laub, The singular value decomposition: Its computation and some applications, IEEE Trans. Autom. Control, vol. 25, no. 2, pp. 164-176, Apr. 1980. (Pubitemid 10475293)
49. V. Koltchinskii, K. Lounici, and A. B. Tsybakov, Nuclear-norm penalization and optimal rates for noisy low-rank matrix completion, Ann. Statist., vol. 39, no. 5, pp. 2302-2329, 2011.
50. S. Kritchman and B. Nadler, Determining the number of components in a factor model from limited noisy data, Chemom. Intell. Lab. Syst., vol. 94, no. 1, pp. 19-32, 2008.
51. S. Kritchman and B. Nadler, Non-parametric detection of the number of signals: Hypothesis testing and random matrix theory, IEEE Trans. Signal Process., vol. 57, no. 10, pp. 3930-3941, Oct. 2009.
52. A. N. Langville, C. D. Meyer, R. Albright, J. Cox, and D. Duling, Initializations for the nonnegative matrix factorization, in Proc. 12th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2006, pp. 23-26.
53. R. Lata?a, Some estimates of norms of random matrices, Proc. Amer. Math. Soc., vol. 133, no. 5, pp. 1273-1282, 2005.
54. L. M. Le Cam, Locally Asymptotically Normal Families of Distributions: Certain Approximations to Families of Distributions and Their Use in the Theory of Estimation and Testing Hypotheses, vol. 3. Berkeley, CA, USA: Univ. California Press, 1960.
55. Y. Li, K. J. R. Liu, and J. Razavilar, A parameter estimation scheme for damped sinusoidal signals based on low-rank Hankel approximation, IEEE Trans. Signal Process., vol. 45, no. 2, pp. 481-486, Feb. 1997. (Pubitemid 127765912)
56. V. A. Marchenko and L. A. Pastur, Distribution of eigenvalues for some sets of random matrices, Matematicheskii Sbornik, vol. 114, no. 4, pp. 507-536, 1967.
57. I. Markovsky, Structured low-rank approximation and its applications, Automatica, vol. 44, no. 4, pp. 891-909, 2008.
58. L. Mirsky, Symmetric gauge functions and unitarily invariant norms, Quart. J. Math., vol. 11, no. 1, pp. 50-59, 1960.
59. R. R. Nadakuditi, Exploiting random matrix theory to improve noisy low-rank matrix approximation, in Proc. Conf. Rec. 45th Asilomar Conf. Signals, Syst. Comput., Nov. 2011, pp. 769-773.
60. R. R. Nadakuditi, When are the most informative components for inference also the principal components? arXiv:1302.1232, 2013.
61. R. R. Nadakuditi and M. E. J. Newman, Graph spectra and the detectability of community structure in networks, Phys. Rev. Lett., vol. 108, no. 18, p. 188701, 2012.
62. R. R. Nadakuditi and M. E. J. Newman, Spectra of random graphs with arbitrary expected degrees, Phys. Rev. E, vol. 87, no. 1, p. 012803, 2013.
63. R. R. Nadakuditi and A. Edelman, Sample eigenvalue based detection of high-dimensional signals in white noise using relatively few samples, IEEE Trans. Signal Process., vol. 56, no. 7, pp. 2625-2638, Jul. 2008.
64. B. Nadler, Nonparametric detection of signals by information theoretic criteria: Performance analysis and an improved estimator, IEEE Trans. Signal Process., vol. 58, no. 5, pp. 2746-2756, May 2010.
65. N. Srebro and T. Jaakkola, Weighted low-rank approximations, in Proc. 20th Int. Conf. Mach. Learn., vol. 20. 2003, pp. 720-727.
66. A. Onatski, Testing hypotheses about the numbers of factors in large factor models, Econometrica, vol. 77, no. 5, pp. 1447-1479, 2009.
67. A. Onatski, Determining the number of factors from empirical distribution of eigenvalues, Rev. Econ. Statist., vol. 92, no. 4, pp. 1004-1016, 2010.
68. D. Passemier and J.-F. Yao, On determining the number of spikes in a high-dimensional spiked population model, Random Matrices, Theory Appl., vol. 1, no. 1, p. 1150002, 2012.
69. S. Oymak and B. Hassibi, Finding dense clusters via low rank + sparse decomposition, arXiv:1104.5186, 2011.
70. D. Paul, Asymptotics of sample eigenstructure for a large dimensional spiked covariance model, Statist. Sinica, vol. 17, no. 4, pp. 1617-1642, 2007.
71. S. Pch, Universality results for the largest eigenvalues of some sample covariance matrix ensembles, Probab. Theory Rel. Fields, vol. 143, nos. 3-4, pp. 481-516, 2009.
72. N. S. Pillai and J. Yin, Universality of covariance matrices, Oct. 2011.
73. A. Rohde and A. B. Tsybakov, Estimation of high-dimensional lowrank matrices, Ann. Statist., vol. 39, no. 2, pp. 887-930, 2011.
74. A. Sanjeev and R. Kannan, Learning mixtures of arbitrary Gaussians, in Proc. 33rd Annu. ACM Symp. Theory Comput., 2001, pp. 247-257. (Pubitemid 32865740)
75. J. Saunderson, V. Chandrasekaran, P. A. Parrilo, and A. S. Willsky, Diagonal and low-rank matrix decompositions, correlation matrices, and ellipsoid fitting, SIAM J. Matrix Anal. Appl., vol. 33, no. 4, pp. 1395-1416, 2012.
76. L. L. Scharf, The SVD and reduced rank signal processing, Signal Process., vol. 25, no. 2, p. 113, 1991.
77. A. A. Shabalin and A. B. Nobel, Reconstruction of a low-rank matrix in the presence of Gaussian noise, J. Multivariate Anal., vol. 118, pp. 67-76, Jul. 2013.
78. J. W. Silverstein and S.-I. Choi, Analysis of the limiting spectral distribution of large dimensional random matrices, J. Multivariate Anal., vol. 54, no. 2, pp. 295-309, 1995.
79. A. Soshnikov, A note on universality of the distribution of the largest eigenvalues in certain sample covariance matrices, J. Statist. Phys., vol. 108, nos. 5-6, pp. 1033-1056, 2002.
80. M. Tao and X. Yuan, Recovering low-rank and sparse components of matrices from incomplete and noisy observations, SIAM J. Optim., vol. 21, no. 1, pp. 57-81, 2011.
81. T. Tao, Topics in Random Matrix Theory, vol. 132. Providence, RI, USA: AMS, 2012.
82. M. E. Tipping and C. M. Bishop, Mixtures of probabilistic principal component analyzers, Neural Comput., vol. 11, no. 2, pp. 443-482, 1999.
83. D. W. Tufts and A. A. Shah, Estimation of a signal waveform from noisy data using low-rank approximation to a data matrix, IEEE Trans. Signal Process., vol. 41, no. 4, pp. 1716-1721, Apr. 1993.
84. M. O. Ulfarsson and V. Solo, Dimension estimation in noisy PCA with SURE and random matrix theory, IEEE Trans. Signal Process., vol. 56, no. 12, pp. 5804-5816, Dec. 2008.
85. S. Vempala and G. Wang, A spectral algorithm for learning mixtures of distributions, in Proc. 43rd Annu. IEEE Symp. Found. Comput. Sci., 2002, pp. 113-122.
86. D. M. Wikes and M. H. Hayes, Iterated toeplitz approximation of covariance matrices, in Proc. ICASSP, Apr. 1988, pp. 1663-1666. (Pubitemid 18666343)
87. Y. Zhang, A. dAspremont, and L. El Ghaoui, Sparse PCA: Convex relaxations, algorithms and applications, in Handbook on Semidefinite, Conic and Polynomial Optimization. New York, NY, USA: Springer- Verlag, 2012, pp. 915-940.
88. Z. Zhang, H. Zha, and H. Simon, Low-rank approximations with sparse factors I: Basic algorithms and error analysis, SIAM J. Matrix Anal. Appl., vol. 23, no. 3, pp. 706-727, 2002.
89. H. Zou, T. Hastie, and R. Tibshirani, Sparse principal component analysis, J. Comput. Graph. Statist., vol. 15, no. 2, pp. 265-286, 2006. (Pubitemid 43935978)