Sketching as a tool for numerical linear algebra

Foundations and Trends in Theoretical Computer Science

Volumn 10, Issue 1-2, 2014, Pages 1-157

  Woodruff, David P ^a  

^a IBM ALMADEN RESEARCH CENTER   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; LEAST SQUARES APPROXIMATIONS; SURVEYS;

GRAPH SPARSIFICATION; LEAST SQUARE; LINEAR SKETCHING; LOW RANK APPROXIMATIONS; NUMERICAL LINEAR ALGEBRA; RANDOM MATRIX; ROBUST REGRESSIONS;

MATRIX ALGEBRA;

EID: 84915779647     PISSN: 1551305X     EISSN: 15513068     Source Type: Journal    
DOI: 10.1561/0400000060     Document Type: Review

References (125)

1. Dimitris Achlioptas. Database-friendly random projections: Johnsonlindenstrauss with binary coins. J. Comput. Syst. Sci., 66(4): 671-687, 2003.
2. Nir Ailon and Bernard Chazelle. Approximate nearest neighbors and the fast johnson-lindenstrauss transform. In ACM Symposium on Theory of Computing (STOC), 2006.
3. Nir Ailon and Edo Liberty. Fast dimension reduction using rademacher series on dual bch codes. In ACM-SIAM Symposium on Discrete Algorithms (SODA), 2008.
4. Alexandr Andoni. High frequency moment via max stability. Available at http://web.mit.edu/andoni/www/papers/fkStable.pdf, 2012.
5. Alexandr Andoni, Robert Krauthgamer, and Krzysztof Onak. Streaming algorithms via precision sampling. In FOCS, pages 363-372, 2011.
6. Sanjeev Arora, Elad Hazan, and Satyen Kale. A fast random sampling algorithm for sparsifying matrices. In APPROX-RANDOM, pages 272-279, 2006.
7. H. Auerbach. On the Area of Convex Curves with Conjugate Diameters. PhD thesis, University of Lwów, Lwów, Poland, 1930. (in Polish).
8. Haim Avron, Petar Maymounkov, and Sivan Toledo. Blendenpik: Supercharging lapack's least-squares solver. SIAM J. Scientific Computing, 32(3): 1217-1236, 2010.
9. Haim Avron, Huy L. Nguyen, and David P. Woodruff. Subspace embeddings for the polynomial kernel. In NIPS, 2014.
10. Haim Avron, Vikas Sindhwani, and David P.Woodruff. Sketching structured matrices for faster nonlinear regression. In NIPS, pages 2994-3002, 2013.
11. Baruch Awerbuch and Robert Kleinberg. Online linear optimization and adaptive routing. J. Comput. Syst. Sci., 74(1): 97-114, 2008.
12. Z. Bai and Y.Q. Yin. Limit of the smallest eigenvalue of a large dimensional sample covariance matrix. The Annals of Probability 21, 3: 1275-1294, 1993.
13. Maria-Florina Balcan, Vandana Kanchanapally, Yingyu Liang, and David P. Woodruff. Fast and communication efficient algorithms for distributd pca. In NIPS, 2014.
14. J.D. Batson, D.A. Spielman, and N. Srivastava. Twice-ramanujan sparsifiers. In Proceedings of the 41st annual ACM symposium on Theory of computing, pages 255-262. ACM, 2009.
15. Michael W Berry, Shakhina A Pulatova, and GW Stewart. Algorithm 844: Computing sparse reduced-rank approximations to sparse matrices. ACM Transactions on Mathematical Software (TOMS), 31(2): 252-269, 2005.
16. David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. the Journal of machine Learning research, 2003.
17. J. Bourgain, J. Lindenstrauss, and V. Milman. Approximation of zonoids by zonotopes. Acta Math, 162: 73-141, 1989.
18. Jean Bourgain and Jelani Nelson. Toward a unified theory of sparse dimensionality reduction in euclidean space. CoRR, abs/1311.2542, 2013.
19. Christos Boutsidis, Petros Drineas, and Malik Magdon-Ismail. Near optimal column based matrix reconstruction. SIAM Journal on Computing (SICOMP), 2013.
20. Christos Boutsidis, Michael W. Mahoney, and Petros Drineas. An improved approximation algorithm for the column subset selection problem. In Proceedings of the Nineteenth Annual ACM -SIAM Symposium on Discrete Algorithms (SODA), pages 968-977, 2009.
21. Christos Boutsidis and David P. Woodruff. Optimal cur matrix decompositions. In STOC, pages 353-362, 2014.
22. J.P. Brooks and J.H. Dulá. The 1-norm best-fit hyperplane problem. Technical report, Optimization Online, 2009.
23. J.P. Brooks, J.H. Dulá, and E.L. Boone. A pure 1-norm principal component analysis. Technical report, Optimization Online, 2010.
24. Moses Charikar, Kevin Chen, and Martin Farach-Colton. Finding frequent items in data streams. Theor. Comput. Sci., 312(1): 3-15, 2004.
25. K. Clarkson. Subgradient and sampling algorithms for 1 regression. In Proceedings of the 16th Annual ACM-SIAM Symposium on Discrete Algorithms, 2005.
26. Kenneth L. Clarkson, Petros Drineas, Malik Magdon-Ismail, Michael W. Mahoney, Xiangrui Meng, and David P. Woodruff. The fast cauchy transform and faster robust linear regression. In SODA, pages 466-477, 2013.
27. Kenneth L Clarkson and David P Woodruff. Low rank approximation and regression in input sparsity time. In In STOC, 2013.
28. K.L. Clarkson and D.P. Woodruff. Numerical linear algebra in the streaming model. In Proceedings of the 41st annual ACM symposium on Theory of computing (STOC), 2009.
29. Michael Cohen, Sam Elder, Cameron Musco, Christopher Musco, and Madalina Persu. Dimensionality reduction for k-means clustering and low rank approximation. Arxiv preprint arXiv: 1410.6801, 2014.
30. Michael Cohen, Jelani Nelson, and David P. Woodruff. Optimal approximate matrix product in terms of stable rank. Manuscript, 2014.
31. Michael B. Cohen, Yin Tat Lee, Cameron Musco, Christopher Musco, Richard Peng, and Aaron Sidford. Uniform sampling for matrix approximation. Arxiv preprint arXiv: 1408.5099, 2014.
32. Anirban Dasgupta, Petros Drineas, Boulos Harb, Ravi Kumar, and Michael W. Mahoney. Sampling algorithms and coresets for p regression. SIAM J. Comput., 38(5): 2060-2078, 2009.
33. Anirban Dasgupta, Ravi Kumar, and Tamás Sarlós. A sparse johnson: Lindenstrauss transform. In STOC, pages 341-350, 2010.
34. A. Deshpande and K. Varadarajan. Sampling-based dimension reduction for subspace approximation. In Proceedings of the thirty-ninth annual ACM symposium on Theory of computing, pages 641-650. ACM, 2007.
35. Amit Deshpande, Luis Rademacher, Santosh Vempala, and Grant Wang. Matrix approximation and projective clustering via volume sampling. Theory of Computing, 2(12): 225-247, 2006.
36. D. Donoho and P. Stark. Uncertainty principles and signal recovery. SIAM Journal on Applied Mathematics, 1989.
37. P. Drineas and R. Kannan. Pass efficient algorithms for approximating large matrices. In Proceedings of the 14th Annual ACM-SIAM Symposium on Discrete Algorithms, pages 223-232, 2003.
38. P. Drineas, R. Kannan, and M.W. Mahoney. Fast Monte Carlo algorithms for matrices III: Computing a compressed approximate matrix decomposition. SIAM Journal of Computing, 36(1): 184-206, 2006.
39. P. Drineas, M. W. Mahoney, and S. Muthukrishnan. Subspace sampling and relative-error matrix approximation: Column-based methods. In APPROX-RANDOM, pages 316-326, 2006.
40. P. Drineas, M. W. Mahoney, and S. Muthukrishnan. Subspace sampling and relative-error matrix approximation: Column-row-based methods. In Algorithms - ESA 2006, 14th Annual European Symposium, Zurich, Switzerland, September 11-13, 2006, Proceedings, volume 4168 of Lecture Notes in Computer Science, pages 304-314. Springer, 2006.
41. P. Drineas, M. W. Mahoney, and S. Muthukrishnan. Relative-error cur matrix decompositions. SIAM Journal Matrix Analysis and Applications, 30(2): 844-881, 2008.
42. P. Drineas, M.W. Mahoney, and S. Muthukrishnan. Sampling algorithms for 2 regression and applications. In Proceedings of the 17th Annual ACM-SIAM Symposium on Discrete Algorithms, pages 1127-1136, 2006.
43. P. Drineas, M.W. Mahoney, S. Muthukrishnan, and T. Sarlos. Faster least squares approximation, Technical Report, arXiv: 0710.1435, 2007.
44. Petros Drineas, Malik Magdon-Ismail, Michael W. Mahoney, and David P. Woodruff. Fast approximation of matrix coherence and statistical leverage. Journal of Machine Learning Research, 13: 3475-3506, 2012.
45. Petros Drineas, Michael W. Mahoney, and S. Muthukrishnan. Subspace sampling and relative-error matrix approximation: Column-based methods. In APPROX-RANDOM, pages 316-326, 2006.
46. Petros Drineas, Michael W. Mahoney, and S. Muthukrishnan. Subspace sampling and relative-error matrix approximation: Column-row-based methods. In ESA, pages 304-314, 2006.
47. Uriel Feige and Eran Ofek. Spectral techniques applied to sparse random graphs. Random Struct. Algorithms, 27(2): 251-275, 2005.
48. D. Feldman and M. Langberg. A unified framework for approximating and clustering data. In Proc. 41th Annu. ACM Symp. on Theory of Computing (STOC), to appear, 2011.
49. Dan Feldman, Morteza Monemizadeh, Christian Sohler, and David P. Woodruff. Coresets and sketches for high dimensional subspace approximation problems. In SODA, pages 630-649, 2010.
50. Dan Feldman, Melanie Schmidt, and Christian Sohler. Turning big data into tiny data: Constant-size coresets for k-means, pca and projective clustering. In Proceedings of the Annual ACM-SIAM Symposium on Discrete Algorithms, 2013.
51. Anna C. Gilbert, Yi Li, Ely Porat, and Martin J. Strauss. Approximate sparse recovery: Optimizing time and measurements. In STOC, pages 475-484, 2010.
52. Alex Gittens and Michael W Mahoney. Revisiting the nystrom method for improved large-scale machine learning. arXiv preprint arXiv: 1303.1849, 2013.
53. Gene H. Golub and Charles F. van Loan. Matrix computations (3. ed.). Johns Hopkins University Press, 1996.
54. S.A. Goreinov, EE Tyrtyshnikov, and NL Zamarashkin. A theory of pseudoskeleton approximations. Linear Algebra and its Applications, 261(1-3): 1-21, 1997.
55. S.A. Goreinov, N.L. Zamarashkin, and E.E. Tyrtyshnikov. Pseudoskeleton approximations by matrices of maximal volume. Mathematical Notes, 62(4): 515-519, 1997.
56. M. Gu and L. Miranian. Strong rank revealing Cholesky factorization. Electronic Transactions on Numerical Analysis, 17: 76-92, 2004.
57. Venkatesan Guruswami and Ali Kemal Sinop. Optimal column-based low-rank matrix reconstruction. In Proceedings of the Twenty-Third Annual ACM-SIAM Symposium on Discrete Algorithms, pages 1207-1214. SIAM, 2012.
58. Uffe Haagerup. The best constants in the khintchine inequality. Studia Mathematica, 70(3): 231-283, 1981.
59. Nathan Halko, Per-Gunnar Martinsson, and Joel A. Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review, 53(2): 217-288, 2011.
60. Moritz Hardt and David P. Woodruff. How robust are linear sketches to adaptive inputs? In STOC, pages 121-130, 2013.
61. T.M. Hwang, W.W. Lin, and D. Pierce. Improved bound for rank revealing LU factorizations. Linear algebra and its applications, 261(1- 3): 173-186, 1997.
62. P. Indyk and R. Motwani. Approximate nearest neighbors: Towards removing the curse of dimensionality. In Proceedings of the thirtieth annual ACM symposium on Theory of computing, pages 604-613, 1998.
63. Piotr Indyk. Stable distributions, pseudorandom generators, embeddings, and data stream computation. J. ACM, 53(3): 307-323, 2006.
64. Piotr Indyk. Uncertainty principles, extractors, and explicit embeddings of l2 into l1. In STOC, pages 615-620, 2007.
65. Yuri Ingster and I. A. Suslina. Nonparametric Goodness-of-Fit Testing Under Gaussian Models. Springer, 1st edition, 2002.
66. William Johnson and Gideon Schechtman. Very tight embeddings of subspaces of lp, 1 = p Geometric and Functional Analysis, 13(4): 845-851, 2003.
67. Ravi Kannan, Santosh Vempala, and David P. Woodruff. Principal component analysis and higher correlations for distributed data. In COLT, pages 1040-1057, 2014.
68. Michael Kapralov, Yin Tat Lee, Cameron Musco, Christopher Musco, and Aaron Sidford. Single pass spectral sparsification in dynamic streams. CoRR, abs/1407.1289, 2014.
69. Q. Ke and T. Kanade. Robust subspace computation using 1 norm, 2003. Technical Report CMU-CS-03-172, Carnegie Mellon University, Pittsburgh, PA.
70. Qifa Ke and Takeo Kanade. Robust l1 norm factorization in the presence of outliers and missing data by alternative convex programming. In CVPR (1), pages 739-746, 2005.
71. E. Kushilevitz and N. Nisan. Communication Complexity. Cambridge University Press, 1997.
72. Rafal Latala. Estimates of moments and tails of Gaussian chaoses. Ann. Probab., 34(6): 2315-2331, 2006.
73. Béatrice Laurent and Pascal Massart. Adaptive estimation of a quadratic functional by model selection. Ann. Statist., 28(5): 1302-1338, 2000.
74. Daniel D. Lee and H. Sebastian Seung. Algorithms for non-negative matrix factorization. Advances in Neural Information Processing Systems, 2001.
75. Joseph Lehec. Moments of the Gaussian chaos. In Séminaire de Probabilités XLIII, volume 2006 of Lecture Notes in Math., pages 327-340. Springer, Berlin, 2011.
76. D. Lewis. Finite dimensional subspaces of lp. Studia Math, 63: 207-211, 1978.
77. Mu Li, Gary L. Miller, and Richard Peng. Iterative row sampling. In FOCS, pages 127-136, 2013.
78. Yi Li, Huy L. Nguyen, and David P. Woodruff. On sketching matrix norms and the top singular vector. In SODA, 2014.
79. D.G. Luenberger and Y. Ye. Linear and nonlinear programming, volume 116. Springer Verlag, 2008.
80. M. Magdon-Ismail. Row Sampling for Matrix Algorithms via a Non- Commutative Bernstein Bound. Arxiv preprint arXiv: 1008.0587, 2010.
81. Malik Magdon-Ismail. Using a non-commutative bernstein bound to approximate some matrix algorithms in the spectral norm. CoRR, abs/1103.5453, 2011.
82. A. Magen and A. Zouzias. Low Rank Matrix-valued Chernoff Bounds and Approximate Matrix Multiplication. Proceedings of the 22nd Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), 2011.
83. Avner Magen. Dimensionality reductions in l2 that preserve volumes and distance to affine spaces. Discrete & Computational Geometry, 38(1): 139-153, 2007.
84. Michael W. Mahoney. Randomized algorithms for matrices and data. Foundations and Trends in Machine Learning, 3(2): 123-224, 2011.
85. Michael W. Mahoney and Petros Drineas. Cur matrix decompositions for improved data analysis. Proceedings of the National Academy of Sciences (PNAS), 106: 697-702, 2009.
86. Michael W Mahoney and Petros Drineas. Cur matrix decompositions for improved data analysis. Proceedings of the National Academy of Sciences, 106(3): 697-702, 2009.
87. O. L. Mangasarian. Arbitrary-norm separating plane. Operations Research Letters, 24: 15-23, 1997.
88. Jiri Matousek. Lectures on Discrete Geometry. Springer, 2002.
89. A. Maurer. A bound on the deviation probability for sums of nonnegative random variables. Journal of Inequalities in Pure and Applied Mathematics, 4(1: 15), 2003.
90. X. Meng, M. A. Saunders, and M. W. Mahoney. LSRN: A Parallel Iterative Solver for Strongly Over- or Under-Determined Systems. ArXiv e-prints, September 2011.
91. Xiangrui Meng and Michael W Mahoney. Low-distortion subspace embeddings in input-sparsity time and applications to robust linear regression. In In STOC, pages 91-100. ACM, 2013.
92. Peter Bro Miltersen, Noam Nisan, Shmuel Safra, and Avi Wigderson. On data structures and asymmetric communication complexity. J. Comput. Syst. Sci., 57(1): 37-49, 1998.
93. Thomas P. Minka. A comparison of numerical optimizers for logistic regression. Technical report, Microsoft, 2003.
94. L. Miranian and M. Gu. Strong rank revealing LU factorizations. Linear Algebra and its Applications, 367(C): 1-16, July 2003.
95. S. Muthukrishnan. Data streams: Algorithms and applications. Foundations and Trends in Theoretical Computer Science, 2005.
96. Jelani Nelson and Huy L Nguyen. Osnap: Faster numerical linear algebra algorithms via sparser subspace embeddings. In In FOCS, 2013.
97. Jelani Nelson and Huy L. Nguyen. Lower bounds for oblivious subspace embeddings. In ICALP (1), pages 883-894, 2014.
98. Jelani Nelson and Huy L. Nguyen. Sparsity lower bounds for dimensionality reducing maps. In STOC, pages 101-110, 2013.
99. Huy Le Nguyen. Personal communication, 2013.
100. C.T. Pan. On the existence and computation of rank-revealing LU factorizations. Linear Algebra and its Applications, 316(1-3): 199-222, 2000.
101. Oded Regev. Personal communication, 2014.
102. V. Rokhlin and M. Tygert. A fast randomized algorithm for overdetermined linear least-squares regression. Proceedings of the National Academy of Sciences, 105(36): 13212, 2008.
103. Mark Rudelson and Roman Vershynin. Non-asymptotic theory of random matrices: Extreme singular values. CoRR, abs/1003.2990v2, 2010.
104. T. Sarlós. Improved approximation algorithms for large matrices via random projections. In IEEE Symposium on Foundations of Computer Science (FOCS), 2006.
105. Schechtman. More on embedding subspaces of lp into nr . Composition Math, 61: 159-170, 1987.
106. N.D. Shyamalkumar and K. Varadarajan. Efficient subspace approximation algorithms. In Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, pages 532-540, 2007.
107. Christian Sohler and David P. Woodruff. Subspace embeddings for the l1-norm with applications. In STOC, pages 755-764, 2011.
108. Daniel A. Spielman and Shang-Hua Teng. Nearly-linear time algorithms for graph partitioning, graph sparsification, and solving linear systems. In STOC, pages 81-90, 2004.
109. Daniel A. Spielman and Shang-Hua Teng. Spectral sparsification of graphs. SIAM J. Comput., 40(4): 981-1025, 2011.
110. N. Srivastava and D.A. Spielman. Graph sparsifications by effective resistances. In Proceedings of the 40th ACM Symposium on Theory of Computing (STOC), 2008.
111. G.W. Stewart. Four algorithms for the efficient computation of truncated QR approximations to a sparse matrix. Numerische Mathematik, 83: 313-323, 1999.
112. Michel Talagrand. Embedding subspaces of l1 into n1 . Proceedings of the American Mathematical Society, 108(2): 363-369, 1990.
113. Zhihui Tang. Fast Transforms Based on Structured Matrices With Applications to The Fast Multipole Method. PhD thesis, PhD Thesis, University of Maryland College Park, 2004.
114. Mikkel Thorup and Yin Zhang. Tabulation-based 5-independent hashing with applications to linear probing and second moment estimation. SIAM J. Comput., 41(2): 293-331, 2012.
115. Joel Tropp. Improved analysis of the subsampled randomized hadamard transform. Adv. Adapt. Data Anal., special issue, "Sparse Representation of Data and Images, 2011.
116. Alexandre B. Tsybakov. Introduction to Nonparametric Estimation. Springer, 1st edition, 2008.
117. E. Tyrtyshnikov. Mosaic-skeleton approximations. Calcolo, 33(1): 47-57, 1996.
118. E. Tyrtyshnikov. Incomplete cross approximation in the mosaic-skeleton method. Computing, 64(4): 367-380, 2000.
119. Kasturi Varadarajan and Xin Xiao. On the sensitivity of shape fitting problems. In FSTTCS, pages 486-497, 2012.
120. Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. In Y. C. Eldar and G. Kutyniok, editors, Compressed Sensing: Theory and Applications. Cambridge University Press, 2011.
121. S. Wang and Z. Zhang. Improving cur matrix decomposition and the nystrom approximation via adaptive sampling. Journal of Machine Learning Research, 2013.
122. David P. Woodruff. Low rank approximation lower bounds in rowupdate streams. In NIPS, 2014.
123. David P. Woodruff and Qin Zhang. Subspace embeddings and p- regression using exponential random variables. CoRR, 2013.
124. Dean Foster Yichao Lu, Paramveer Dhillon and Lyle Ungar. Faster ridge regression via the subsampled randomized hadamard transform. In Proceedings of the Neural Information Processing Systems (NIPS) Conference, 2013.
125. Anastasios Zouzias. A matrix hyperbolic cosine algorithm and applications. In ICALP (1), pages 846-858, 2012.