Matrix estimation by Universal Singular Value Thresholding

Annals of Statistics

Volumn 43, Issue 1, 2015, Pages 177-214

  Chatterjee, Sourav ^a  

^a STANFORD UNIVERSITY   (United States)

Author keywords

Covariance matrix; Distance matrix; Graphons; Latent space model; Low rank matrices; Matrix completion; Matrix estimation; Singular value decomposition; Sochastic blockmodel

Indexed keywords

EID: 84922519987     PISSN: 00905364     EISSN: 21688966     Source Type: Journal    
DOI: 10.1214/14-AOS1272     Document Type: Article

References (97)

1. ACHLIOPTAS, D. and M C S HERRY, F. (2001). Fast computation of low rank matrix approximations. In Proceedings of the Thirty-Third Annual ACM Symposium on Theory of Computing 611-618 (electronic). ACM, New York. MR2120364
2. ADAMS, E. (2005). Bayesian analysis of linear dominance hierarchies. Animal Behaviour 69 1191-1201.
3. AGRESTI, A. (1990). Categorical Data Analysis. Wiley, New York. MR1044993
4. AIROLDI, E. M., BLEI, D.M., FIENBERG, S. E. and XING, E. P. (2008). Mixed membership stochastic blockmodels. J. Mach. Learn. Res. 9 1981-2014.
5. ALDOUS, D. J. (1981). Representations for partially exchangeable arrays of random variables. J. Multivariate Anal. 11 581-598. MR0637937
6. ALFAKIH, A. Y., KHANDANI, A. and WOLKOWICZ, H. (1999). Solving Euclidean distance matrix completion problems via semidefinite programming. Comput. Optim. Appl. 12 13-30. MR1704098
7. AMINI, A. A., CHEN, A., BICKEL, P. J. and LEVINA, E. (2013). Pseudo-likelihood methods for community detection in large sparse networks. Ann. Statist. 41 2097-2122. MR3127859
8. ANDERSON, G. W., GUIONNET, A. and ZEITOUNI, O. (2010). An Introduction to Random Matrices. Cambridge Studies in Advanced Mathematics 118. Cambridge Univ. Press, Cambridge. MR2760897
9. AUSTIN, T. (2008). On exchangeable random variables and the statistics of large graphs and hypergraphs. Probab. Surv. 5 80-145. MR2426176
10. AZAR, Y., FLAT, A., KARLIN, A.,M C S HERRY, F. and SALA, J. (2001). Spectral analysis of data. In Proceedings of the Thirty-third Annual ACM Symposium on Theory of Computing 619-626. ACM, New York.
11. BAKONYI, M. and JOHNSON, C. R. (1995). The Euclidean distance matrix completion problem. SIAM J. Matrix Anal. Appl. 16 646-654. MR1321802
12. BENNETT, G. (1962). Probability inequalities for sums of independent random variables. J. Amer. Statist. Assoc. 57 33-45.
13. BERNSTEIN, S. (1924). Sur une modification de l'inéqualité de Tchebichef. Annals Science Institute Sav. Ukraine, Sect. Math. I (Russian, French summary.) 38-49.
14. BHATIA, R. (1997). Matrix Analysis. Graduate Texts in Mathematics 169. Springer, New York. MR1477662
15. BHATIA, R. (2007). Positive Definite Matrices. Princeton Univ. Press, Princeton, NJ. MR2284176
16. BICKEL, P. J. and CHEN, A. (2009). A nonparametric view of network models and Newman-Girvan and other modularities. Proc. Natl. Acad. Sci. USA 106 21068-21073.
17. BICKEL, P. J., CHEN, A. and LEVINA, E. (2011). The method of moments and degree distributions for network models. Ann. Statist. 39 2280-2301. MR2906868
18. BISWAS, P., LIAN, T.-C.,WANG, T.-C. and Y E, Y. (2006). Semidefinite programming based algorithms for sensor network localization. ACM Trans. Sen. Netw. 2 188-220.
19. nd ed. Springer, New York. MR2158691
20. BORGS, C., CHAYES, J., L OVÁSZ, L., S ÓS, V. T. and VESZTERGOMBI, K. (2006). Counting graph homomorphisms. In Topics in Discrete Mathematics. Algorithms Combin. 26 315-371. Springer, Berlin. MR2249277
21. BORGS, C., CHAYES, J. T., L OVÁSZ, L., S ÓS, V. T. and VESZTERGOMBI, K. (2008). Convergent sequences of dense graphs. I. Subgraph frequencies, metric properties and testing. Adv. Math. 219 1801-1851. MR2455626
22. BORGS, C., CHAYES, J. T., L OVÁSZ, L., S ÓS, V. T. and VESZTERGOMBI, K. (2012). Convergent sequences of dense graphs II. Multiway cuts and statistical physics. Ann. of Math. (2) 176 151-219. MR2925382
23. BRADLEY, R. A. and TERRY, M. E. (1952). Rank analysis of incomplete block designs. I. The method of paired comparisons. Biometrika 39 324-345. MR0070925
24. CAI, J.-F., C ANDÈS, E. J. and SHEN, Z. (2010). A singular value thresholding algorithm for matrix completion. SIAM J. Optim. 20 1956-1982. MR2600248
25. C ANDÈS, E. J. and PLAN, Y. (2010). Matrix completion with noise. Proceedings of the IEEE 98 925-936.
26. CANDÈS, E. J. and RECHT, B. (2009). Exact matrix completion via convex optimization. Found. Comput. Math. 9 717-772. MR2565240
27. C ANDÈS, E. J., ROMBERG, J. and TAO, T. (2006). Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inform. Theory 52 489-509. MR2236170
28. C ANDÈS, E. J. and TAO, T. (2010). The power of convex relaxation: Near-optimal matrix completion. IEEE Trans. Inform. Theory 56 2053-2080. MR2723472
29. CARON, F. and DOUCET, A. (2012). Efficient Bayesian inference for generalized Bradley-Terry models. J. Comput. Graph. Statist. 21 174-196. MR2913362
30. CHATTERJEE, S. andD IACONIS, P. (2013). Estimating and understanding exponential random graph models. Ann. Statist. 41 2428-2461. MR3127871
31. CHATTERJEE, S., DIACONIS, P. and SLY, A. (2011). Random graphs with a given degree sequence. Ann. Appl. Probab. 21 1400-1435. MR2857452
32. CHATTERJEE, S. and VARADHAN, S. R. S. (2011). The large deviation principle for the Erdo∗Prime;s-Rényi random graph. European J. Combin. 32 1000-1017. MR2825532
33. CHATTERJEE, S. and VARADHAN, S. R. S. (2012). Large deviations for random matrices. Commun. Stoch. Anal. 6 1-13. MR2890846
34. CHAUDHURI, K., CHUNG, F. and TSIATAS, A. (2012). Spectral clustering of graphs with general degrees in the extended planted partition model. J. Mach. Learn. Res. 35 1-23.
35. CHOI, D. andW OLFE, P. J. (2014). Co-clustering separately exchangeable network data. Ann. Statist. 42 29-63. MR3161460
36. CHOI, D. S.,WOLFE, P. J. and AIROLDI, E. M. (2012). Stochastic blockmodels with a growing number of classes. Biometrika 99 273-284. MR2931253
37. CONDON, A. and KARP, R. M. (2001). Algorithms for graph partitioning on the planted partition model. Random Structures Algorithms 18 116-140. MR1809718
38. DAVENPORT, M. A., PLAN, Y., VAN DEN BERG, E. andW OOTTERS, M. (2012). 1-bit matrix completion. Preprint. Available at arXiv:1209.3672.
39. nd ed. Griffin's Statistical Monographs & Courses 41. Oxford Univ. Press, London. MR0947340
40. DAVIDSON, R. R. and FARQUHAR, P. H. (1976). A bibliography on the method of paired comparisons. Biometrics 32 241-252. MR0408134
41. DIACONIS, P. (1988). Group Representations in Probability and Statistics. Institute of Mathematical Statistics Lecture Notes-Monograph Series 11. IMS, Hayward, CA. MR0964069
42. DIACONIS, P. and JANSON, S. (2008). Graph limits and exchangeable random graphs. Rend. Mat. Appl. (7) 28 33-61. MR2463439
43. DONOHO, D. L. (2006). Compressed sensing. IEEE Trans. Inform. Theory 52 1289-1306. MR2241189
44. DONOHO, D. L. and JOHNSTONE, I. M. (1995). Adapting to unknown smoothness via wavelet shrinkage. J. Amer. Statist. Assoc. 90 1200-1224. MR1379464
45. FAZEL, M. (2002). Matrix rank minimization with applications. Ph.D. thesis, Stanford Univ., Stanford, CA.
46. F ÜREDI, Z. and K OMLÓS, J. (1981). The eigenvalues of random symmetric matrices. Combinatorica 1 233-241. MR0637828
47. GAVISH, M. and DONOHO, D. L. (2014). The optimal hard threshold for singular values is ? 3. IEEE Trans. Inform. Theory 60 5040-5053. MR3245370 4/
48. GORMLEY, I. C. andM URPHY, T. B. (2008). Exploring voting blocs within the Irish electorate: A mixture modeling approach. J. Amer. Statist. Assoc. 103 1014-1027. MR2528824
49. GORMLEY, I. C. and MURPHY, T. B. (2009). A grade of membership model for rank data. Bayesian Anal. 4 265-295. MR2507364
50. rd Annual International Conference on Machine Learning 361-368. ACM, New York.
51. GRONE, R., JOHNSON, C. R., DE S Á, E. M. and WOLKOWICZ, H. (1984). Positive definite completions of partial Hermitian matrices. Linear Algebra Appl. 58 109-124. MR0739282
52. th Annual International Conference on Machine Learning 377-384. ACM, New York.
53. HANDCOCK, M. S., RAFTERY, A. E. and TANTRUM, J. M. (2007). Model-based clustering for social networks. J. Roy. Statist. Soc. Ser. A 170 301-354. MR2364300
54. HASTIE, T. and TIBSHIRANI, R. (1998). Classification by pairwise coupling. Ann. Statist. 26 451-471. MR1626055
55. HOFF, P. D., RAFTERY, A. E. and HANDCOCK, M. S. (2002). Latent space approaches to social network analysis. J. Amer. Statist. Assoc. 97 1090-1098. MR1951262
56. HOLLAND, P.W., LASKEY, K. B. and LEINHARDT, S. (1983). Stochastic blockmodels: First steps. Social Networks 5 109-137. MR0718088
57. HOOVER, D. N. (1982). Row-column exchangeability and a generalized model for probability. In Exchangeability in Probability and Statistics (Rome, 1981) 281-291. North-Holland, Amsterdam. MR0675982
58. HUANG, T.-K.,WENG, R. C. and LIN, C.-J. (2006). Generalized Bradley-Terry models and multi-class probability estimates. J. Mach. Learn. Res. 7 85-115. MR2274363
59. HUNTER, D. R. (2004). MM algorithms for generalized Bradley-Terry models. Ann. Statist. 32 384-406. MR2051012
60. JAVANMARD, A. and MONTANARI, A. (2011). Localization from incomplete noisy distance measurements. In 2011 IEEE International Symposium on Information Theory Proceedings (ISIT) 1584-1588. IEEE, New York.
61. JOHNSON, C. R. (1990). Matrix completion problems: A survey. In Matrix Theory and Applications (Phoenix, AZ, 1989). Proc. Sympos. Appl. Math. 40 171-198. Amer. Math. Soc., Providence, RI. MR1059486
62. KESHAVAN, R. H., MONTANARI, A. and O H, S. (2010). Matrix completion from noisy entries. J. Mach. Learn. Res. 11 2057-2078. MR2678022
63. KESHAVAN, R. H.,MONTANARI, A. andO H, S. (2010).Matrix completion from a few entries. IEEE Trans. Inform. Theory 56 2980-2998. MR2683452
64. KOLTCHINSKII, V. (2011). Von Neumann entropy penalization and low-rank matrix estimation. Ann. Statist. 39 2936-2973. MR3012397
65. KOLTCHINSKII, V., LOUNICI, K. and TSYBAKOV, A. B. (2011). Nuclear-norm penalization and optimal rates for noisy low-rank matrix completion. Ann. Statist. 39 2302-2329. MR2906869
66. LANGE, K., HUNTER, D. R. and YANG, I. (2000). Optimization transfer using surrogate objective functions. J. Comput. Graph. Statist. 9 1-59. MR1819865
67. th International Conference on World Wide Web 695-704. ACM, Beijing, China.
68. L OVÁSZ, L. (2012). Large Networks and Graph Limits. American Mathematical Society Colloquium Publications 60. Amer. Math. Soc., Providence, RI. MR3012035
69. L OVÁSZ, L. and SZEGEDY, B. (2006). Limits of dense graph sequences. J. Combin. Theory Ser. B 96 933-957. MR2274085
70. LUBETZKY, E. and ZHAO, Y. (2012). On replica symmetry of large deviations in random graphs. Preprint. Available at arXiv:1210.7013.
71. LUCE, R. D. (1959). Individual Choice Behavior: A Theoretical Analysis. Wiley, New York. MR0108411
72. LUCE, R. D. (1977). The choice axiom after twenty years. J. Math. Psych. 15 215-233. MR0462675
73. MAZUMDER, R., HASTIE, T. and TIBSHIRANI, R. (2010). Spectral regularization algorithms for learning large incomplete matrices. J. Mach. Learn. Res. 11 2287-2322. MR2719857
74. MOSSEL, E., NEEMAN, J. and SLY, A. (2012). Stochastic block models and reconstruction. Preprint. Available at arXiv:1202.1499.
75. NADAKUDITI, R. R. (2013). OptShrink: An algorithm for improved low-rank signal matrix denoising by optimal, data-driven singular value shrinkage. Preprint. Available at arXiv:1306.6042.
76. NEGAHBAN, S. andWAINWRIGHT, M. J. (2011). Estimation of (near) low-rank matrices with noise and high-dimensional scaling. Ann. Statist. 39 1069-1097. MR2816348
77. NOWICKI, K. and SNIJDERS, T. A. B. (2001). Estimation and prediction for stochastic blockstructures. J. Amer. Statist. Assoc. 96 1077-1087. MR1947255
78. O H, S., MONTANARI, A. and KARBASI, A. (2010). Sensor network localization from local connectivity: Performance analysis for the MDS-MAP algorithm. In Information Theory Workshop (ITW) 1-5. IEEE, New York.
79. OLIVEIRA, R. I. (2009). Concentration of the adjacency matrix and of the Laplacian in random graphs with independent edges. Preprint. Available at arXiv:0911.0600.
80. PLACKETT, R. L. (1975). The analysis of permutations. J. R. Stat. Soc. Ser. C. Appl. Stat. 24 193-202. MR0391338
81. RADIN, C. and YIN, M. (2013). Phase transitions in exponential random graphs. Ann. Appl. Probab. 23 2458-2471. MR3127941
82. RAO, P. V. and KUPPER, L. L. (1967). Ties in paired-comparison experiments: A generalization of the Bradley-Terry model. J. Amer. Statist. Assoc. 62 194-204. MR0217963
83. nd International Conference on Machine Learning 713-719. ACM, New York.
84. ROHDE, A. and TSYBAKOV, A. B. (2011). Estimation of high-dimensional low-rank matrices. Ann. Statist. 39 887-930. MR2816342
85. ROHE, K., CHATTERJEE, S. and Y U, B. (2011). Spectral clustering and the high-dimensional stochastic blockmodel. Ann. Statist. 39 1878-1915. MR2893856
86. ROHE, K., QIN, T. and FAN, H. (2012). The highest dimensional Stochastic Blockmodel with a regularized estimator. Preprint. Available at arXiv:1206.2380.
87. RUDELSON, M. and VERSHYNIN, R. (2007). Sampling from large matrices: An approach through geometric functional analysis. J. ACM 54 Art. 21, 19 pp. (electronic). MR2351844
88. SIMONS, G. and YAO, Y.-C. (1999). Asymptotics when the number of parameters tends to infinity in the Bradley-Terry model for paired comparisons. Ann. Statist. 27 1041-1060. MR1724040
89. SINGER, A. (2008). A remark on global positioning from local distances. Proc. Natl. Acad. Sci. USA 105 9507-9511. MR2430205
90. SINGER, A. and CUCURINGU, M. (2009/10). Uniqueness of low-rank matrix completion by rigidity theory. SIAM J. Matrix Anal. Appl. 31 1621-1641. MR2595541
91. SNIJDERS, T. A. B. and NOWICKI, K. (1997). Estimation and prediction for stochastic blockmodels for graphs with latent block structure. J. Classification 14 75-100. MR1449742
92. SPENCE, I. and DOMONEY, D. (1974). Single subject incomplete designs for nonmetric multidimensional scaling. Psychometrika 39 469-490.
93. TALAGRAND, M. (1996). A new look at independence. Ann. Probab. 24 1-34. MR1387624
94. V U, V. H. (2007). Spectral norm of random matrices. Combinatorica 27 721-736. MR2384414
95. WOLFE, P. J. and OLHEDE, S. C. (2013). Nonparametric graphon estimation. Preprint. Available at arXiv:1309.5936.
96. YANG, J. J., HAN, Q. and AIROLDI, E. M. (2014). Nonparametric estimation and testing of exchangeable graph models. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics. Journal of Machine Learning Research, Conference and Workshop Proceedings, Vol. 33 1060-1067.
97. ZERMELO, E. (1929). Die Berechnung der Turnier-Ergebnisse als ein Maximumproblem der Wahrscheinlichkeitsrechnung. Math. Z. 29 436-460. MR1545015