Randomized extended Kaczmarz for solving least squares

SIAM Journal on Matrix Analysis and Applications

Volumn 34, Issue 2, 2013, Pages 773-793

  Zouzias, Anastasios ^a   Freris, Nikolaos M ^b  

^a UNIVERSITY OF TORONTO   (Canada)

^b EPFL   (Switzerland)

Author keywords

Iterative method; LAPACK; Linear least squares; Minimum length solution; Overdetermined system; Random sampling; Randomized algorithms; Sparse matrix; Underdetermined system

Indexed keywords

LAPACK; LINEAR LEAST SQUARES; OVERDETERMINED SYSTEMS; RANDOM SAMPLING; RANDOMIZED ALGORITHMS; SPARSE MATRICES; UNDERDETERMINED SYSTEMS;

ALGORITHMS; LINEAR SYSTEMS; NUMBER THEORY;

ITERATIVE METHODS;

EID: 84880534082     PISSN: 08954798     EISSN: 10957162     Source Type: Journal    
DOI: 10.1137/120889897     Document Type: Article

References (51)

1. [ABD+90] E. Anderson, Z. Bai, J. Dongarra, A. Greenbaum, A. McKenney, J. Du Croz, S. Hammerling, J. Demmel, C. Bischof, and D. Sorensen, LAPACK: A portable linear algebra library for high-performance computers, in Proceedings of the 1990 ACM/IEEE Conference on Supercomputing, Supercomputing '90, IEEE Computer Society Press, Los Alamitos, CA, 1990, pp. 2-11.
2. [AMT10] H. Avron, P. Maymounkov, and S. Toledo, Blendenpik: Supercharging LAPACK's least-squares solver, SIAM J. Sci. Comput., 32 (2010), pp. 1217-1236.
3. [Ans84] R. Ansorge, Connections between the Cimmino-method and the Kaczmarz-method for the solution of singular and regular systems of equations, Computing, 33 (1984), pp. 367-375.
4. [BBC+87] R. Barrett, M. Berry, T. F. Chan, J. Demmel, J. Donato, J. Dongarra, V. Eijkhout, R. Pozo, C. Romine, and H. van der Vorst, Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods, Software, Environments, Tools, SIAM, Philadelphia, 1987.
5. [Bj96] A. Björck, Numerical Methods for Least Squares Problems, SIAM, Philadelphia, 1996.
6. [CEG83] Y. Censor, P. Eggermont, and D. Gordon, Strong underrelaxation in Kaczmarz's method for inconsistent systems, Numer. Math., 41 (1983), pp. 83-92.
7. [Cen81] Y. Censor, Row-action methods for huge and sparse systems and their applications, SIAM Rev., 23 (1981), pp. 444-466.
8. [CP12] X. Chen and A. M. Powell, Almost sure convergence of the Kaczmarz algorithm with random measurements, J. Fourier Anal. Appl., 18 (2012), pp. 1195-1214.
9. [CRT11] E. S. Coakley, V. Rokhlin, and M. Tygert, A fast randomized algorithm for orthogonal projection, SIAM J. Sci. Comput., 33 (2011), pp. 849-868.
10. [CW09] K. L. Clarkson and D. P. Woodruff, Numerical linear algebra in the streaming model, in Proceedings of the Symposium on Theory of Computing (STOC '09), ACM, New York, 2009, pp. 205-214.
11. [CW12] K. L. Clarkson and D. P. Woodruff, Low rank approximation and regression in input sparsity time, in Proceedings of the 45th Annual ACM Symposium on the Theory of Computing (STOC'13), Palo Alto, CA, ACM, New York, 2013, pp. 81-90.
12. [CZ97] Y. Censor and S. A. Zenios, Parallel Optimization: Theory, Algorithms, and Applications, Numer. Math. Sci. Comput. Ser., Oxford University Press, New York, 1997.
13. [Dem88] J. W. Demmel, The probability that a numerical analysis problem is difficult, Math. Comp., 50 (1988), pp. 449-480.
14. [DH11] T. A. Davis and Y. Hu, The university of Florida sparse matrix collection, ACM Trans. Math. Software, 38 (2011).
15. 2-regression and applications, in Proceedings of the 17th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), SIAM, Philadelphia, 2006, pp. 1127-1136.
16. [DMMS11] P. Drineas, M. W. Mahoney, S. Muthukrishnan, and T. Sarlòs, Faster least squares approximation, Numer. Math., 117 (Feb. 2011), pp. 219-249.
17. [FCM+92] H. G. Feichtinger, C. Cenker, M. Mayer, H. Steier, and T. Strohmer, New variants of the POCS method using affine subspaces of finite codimension with applications to irregular sampling, in Visual Communications and Image Processing, Proc. SPIE 1818, 1992, pp. 299-310.
18. [FZ12] N. M. Freris and A. Zouzias, Fast distributed smoothing of relative measurments, IEEE Conference on Decision and Control (CDC), Maui, HI, 2012.
19. [Gal03] A. Galántai, Projectors and Projection Methods, Adv. Math. 6, Kluwer, Boston, 2004.
20. [GBH70] R. Gordon, R. Bender, and G. T. Herman, Algebraic reconstruction techniques (ART) for three-dimensional electron microscopy and X-ray photography, J.Theoret. Biol., 29 (1970), pp. 471-481.
21. [GL96] G. H. Golub and C. F. Van Loan, Matrix Computations, 3rd ed., Johns Hopkins University Press, Baltimore, MD, 1996.
22. [GV61] G. H. Golub and R. S. Varga, Chebyshev semi-iterative methods, successive overrelaxation iterative methods, and second order Richardson iterative methods, Numer. Math., 3 (1961), pp. 157-168.
23. [Her80] G. T. Herman, Image Reconstruction from Projections: The Fundamentals of Computerized Tomography, Comp. Sci. Appl. Math., Academic Press, San Francisco, 1980.
24. [Hig89] N. J. Higham, A Collection of Test Matrices in MATLAB, Technical report, Cornell University, Ithaca, NY, 1989.
25. [Hig96] N. J. Higham, Accuracy and Stability of Numerical Algorithms, SIAM, Philadelphia, 1996.
26. [HM93] G. T. Herman and L. B. Meyer, Algebraic reconstruction techniques can be made computationally efficient, IEEE Trans. Med. Imaging, 12 (1993), pp. 600-609.
27. [HN90] M. Hanke and W. Niethammer, On the acceleration of Kaczmarz's method for inconsistent linear systems, Linear Algebra Appl., 130 (1990), pp. 83-98.
28. [Kac37] S. Kaczmarz, Angenäherte auflösung von systemen linearer gleichungen, Bull. Internat. Acad. Polonaise Sci. Lett., 35 (1937), pp. 355-357.
29. [Kni93] P. A. Knight, Error Analysis of Stationary Iteration and Associated Problems. Ph.D. in Mathematics, Manchester University, Manchester, UK, 1993.
30. [Kni96] P. A. Knight, A Rounding Error Analysis of Row-Action Methods, manuscript.
31. [LL10] D. Leventhal and A. S. Lewis, Randomized methods for linear constraints: Convergence rates and conditioning, Math. Oper. Res., 35 (2010), pp. 641-654.
32. [McC75] S. F. McCormick, An iterative procedure for the solution of constrained nonlinear equations with application to optimization problems, Numer. Math., 23 (1975), pp. 371-385.
33. [MSM11] X. Meng, M. A. Saunders, and M. W. Mahoney, LSRN: A Parallel Iterative Solver for Strongly Over- and Under-Determined Systems, preprint, http://arxiv.org/abs/1109.5981, 2011.
34. [MZ11] A. Magen and A. Zouzias, Low rank matrix-valued Chernoff bounds and approximate matrix multiplication, in Proceedings of the 22nd ACM-SIAM Symposium on Discrete Algorithms (SODA), SIAM, Philadelphia, 2011, pp. 1422-1436.
35. [Nat01] F. Natterer, The Mathematics of Computerized Tomography, Classics in Appl. Math. 32, SIAM, Philadelphia, 2001.
36. [NDT09] N. H. Nguyen, T. T. Do, and T. D. Tran, A fast and efficient algorithm for lowrank approximation of a matrix, in Proceedings of the Symposium on Theory of Computing (STOC '09), ACM, New York, 2009, pp. 215-224.
37. [Nee10] D. Needell, Randomized Kaczmarz solver for noisy linear systems, BIT, 50 (2010), pp. 395-403.
38. [NT12] D. Needell and J. A. Tropp, Paved with good intentions: Analysis of a randomized block Kaczmarz method, Linear Algebra Appl., to appear.
39. [Pop95] C. Popa, Least-squares solution of overdetermined inconsistent linear systems using Kaczmarz's relaxation, Int. J. Comput. Math., 55 (1995), pp. 79-89.
40. [Pop98] C. Popa, Extensions of block-projections methods with relaxation parameters to inconsistent and rank-deficient least-squares problems, BIT, 38 (19698), pp. 151-176.
41. [PS82] C. C. Paige and M. A. Saunders, LSQR: An algorithm for sparse linear equations and sparse least squares, ACM Trans. Math. Software, 8 (1982), pp. 43-71.
42. [RT08] V. Rokhlin and M. Tygert, A fast randomized algorithm for overdetermined linear least-squares regression, Proc. Natl. Acad. Sci. USA, 105 (2008), pp. 13212-13218.
43. [Saa03] Y. Saad, Iterative Methods for Sparse Linear Systems, 2nd edition, SIAM, Philadelphia, 2003.
44. [Sar06] T. Sarlòs, Improved approximation algorithms for large matrices via random projections, in Proceedings of the Symposium on Foundations of Computer Science (FOCS), IEEE Computer Society, Los Alamitos, CA, 2006, pp. 143-152.
45. [Smi02] W. D. Smith, How to Sample from a Probability Distribution, Technical report, NEC, Princeton, NJ, 2002.
46. [SV09] T. Strohmer and R. Vershynin, A Randomized Kaczmarz algorithm with exponential convergence, J. Fourier Anal. Appl., 15 (2009), pp. 262-278.
47. [Tan71] K. Tanabe, Projection method for solving a singular system of linear equations and its applications, Numer. Math., 17 (1971), pp. 203-214.
48. [Tom55] C. Tompkins, Projection methods in calculation, in Proceedings of the 2nd Symposium in Linear Programming, National Bureau of Standards, Washington, DC, 1955, pp. 425-448.
49. [Vos91] M. D. Vose, A linear algorithm for generating random numbers with a given distribution, IEEE Trans. Software Engrg., 17 (1991), pp. 972-975.
50. [Wal77] A. J. Walker, An efficient method for generating discrete random variables with general distributions, ACM Trans. Math. Software, 3 (1977), pp. 253-256.
51. [WM67] T. M. Whitney and R. K. Meany, Two algorithms related to the method of steepest descent, SIAM J. Numer. Anal., 4 (1967), pp. 109-118.