Minimizing communication in numerical linear algebra

SIAM Journal on Matrix Analysis and Applications

Volumn 32, Issue 3, 2011, Pages 866-901

  Ballard, Grey ^a   Demmel, James ^a   Holtz, Olga ^a,b   Schwartz, Oded ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b TECHNISCHE UNIVERSITÄT BERLIN   (Germany)

Author keywords

Bandwidth; Communication avoiding; Latency; Linear algebra algorithms; Lower bound

Indexed keywords

ARITHMETIC OPERATIONS; CHOLESKY FACTORIZATIONS; COMMUNICATION-AVOIDING; COMPUTING POWER; DIRECT METHOD; EIGENVALUES; FAST MEMORY; GRAM-SCHMIDT ALGORITHMS; GRAPH-THEORETIC PROBLEM; INPUT MATRICES; LATENCY; LATENCY COSTS; LINEAR ALGEBRA ALGORITHMS; LINEAR ALGEBRA OPERATIONS; LOCAL MEMORIES; LOWER BOUND TECHNIQUES; LOWER BOUNDS; LU FACTORIZATION; MATRIX; MATRIX MULTIPLICATION; NUMERICAL LINEAR ALGEBRA; OPTIMAL ALGORITHM; QR FACTORIZATIONS; SINGULAR VALUES; SPARSE MATRICES;

ALGEBRA; ALGORITHMS; BANDWIDTH; COMMUNICATION; EIGENVALUES AND EIGENFUNCTIONS; FACTORIZATION; GRAPH THEORY;

MATRIX ALGEBRA;

EID: 80054034521     PISSN: 08954798     EISSN: 10957162     Source Type: Journal    
DOI: 10.1137/090769156     Document Type: Article

References (64)

1. E. ANDERSON, Z. BAI, C. BISCHOF, J. DEMMEL, J. DONGARRA, J. DU CROZ, A. GREENBAUM, S. HAMMARLING, A. MCKENNEY, S. OSTROUCHOV, AND D. SORENSEN, LAPACK's User's Guide, SIAM, Philadelphia, 1992. Also available from http://www.netlib. org/lapack/.
2. R. C. AGARWAL, S. M. BALLE, F. G. GUSTAVSON, M. JOSHI, AND P. PALKAR, A three-dimensional approach to parallel matrix multiplication, IBM J. Res. Dev., 39 (1995), pp. 575-582.
3. B. S. ANDERSEN, F. GUSTAVSON, AND J. WASNIEWSKI, A recursive formulation of Cholesky factorization of a matrix in packed storage format, ACM Trans. Math. Software, 27 (2001), pp. 214-244. (Pubitemid 33602326)
4. N. AHMED AND K. PINGALI, Automatic generation of block-recursive codes, in Euro-Par '00: Proceedings of the 6th International Euro-Par Conference on Parallel Processing, London, UK, Springer-Verlag, Berlin, 2000, pp. 368-378.
5. C. ASHCRAFT, A Taxonomy of Distributed Dense LU Factorization Methods, Boeing Computer Services Technical Report ECA-TR-161, 1991.
6. C. ASHCRAFT, The fan-both family of column-based distributed Cholesky factorization algorithms, in Graph Theory and Sparse Matrix Computation, IMA Volumes in Mathematics and Its Applications 56, J. R. Gilbert, A. George, and J. W. H. Liu, eds., Springer-Verlag, Berlin, 1993, pp. 159-190
7. A. AGGARWAL AND J. S. VITTER, The input/output complexity of sorting and related problems, Comm. ACM, 31 (1988), pp. 1116-1127. (Pubitemid 18662481)
8. M. A. BENDER, G. S. BRODAL, R. FAGERBERG, R. JACOB, AND E. VICARI, Optimal sparse matrix dense vector multiplication in the I/O-model, in SPAA '07: Proceedings of the 19th Annual ACM Symposium on Parallel Algorithms and Architectures, ACM, New York, 2007, pp. 61-70. (Pubitemid 47568555)
9. K. BRAMAN, R. BYERS, AND R. MATHIAS, The multishift QR algorithm. Part I: Maintaining well-focused shifts and level 3 performance, SIAM J. Matrix Anal. Appl., 23 (2002), pp. 929-947.
10. K. BRAMAN, R. BYERS, AND R. MATHIAS, The multishift QR algorithm. Part II: Aggressive early deflation, SIAM J. Matrix Anal. Appl., 23 (2002), pp. 948-973.
11. L. S. BLACKFORD, J. CHOI, A. CLEARY, E. DAZEVEDO, J. DEMMEL, I. DHILLON, J. DONGARRA, S. HAMMARLING, G. HENRY, A. PETITET, K. STANLEY, D. WALKER, AND R. C. WHALEY, ScaLAPACK Users' Guide, SIAM, Philadelphia, 1997. Also available from http://www.netlib.org/scalapack/.
12. L. S. BLACKFORD, J. DEMMEL, J. DONGARRA, I. DUFF, S. HAMMARLING, G. HENRY, M. HEROUX, L. KAUFMAN, A. LUMSDAINE, A. PETITET, R. POZO, K. REMINGTON, R. C. WHALEY, Z. MAANY, F. KROUGH, G. CORLISS, C. HU, B. KEAFOTT, W. WALSTER, AND J. WOLFF V. GUDENBERG, Basic linear algebra subprograms technical (BLAST) forum standard, Int. J. Supercomput. Appl. High Perform. Comput., 15 (2001), pp. 1-315.
13. L. S. BLACKFORD, J. DEMMEL, J. DONGARRA, I. DUFF, S. HAMMARLING, G. HENRY, M. HEROUX, L. KAUFMAN, A. LUMSDAINE, A. PETITET, R. POZO, K. REMINGTON, AND R. C. WHALEY, An updated set of basic linear algebra subroutines (BLAS), ACM Trans. Math. Software, 28 (2002), pp. 135-151. (Pubitemid 135701673)
14. G. BALLARD, J. DEMMEL, AND I. DUMITRIU, Communication-Optimal Parallel and Sequential Eigenvalue and Singular Value Algorithms, EECS Technical Report EECS-2011-14, University of California-Berkeley, 2011.
15. G. BALLARD, J. DEMMEL, O. HOLTZ, AND O. SCHWARTZ, Communication-optimal parallel and sequential Cholesky decomposition, SIAM J. Sci. Comput., 32 (2010), pp. 3495-3523.
16. G. BALLARD, J. DEMMEL, O. HOLTZ, AND O. SCHWARTZ, Graph expansion and communication costs of fast matrix multiplication, in Proceedings of the 23rd ACM Symposium on Parallelism in Algorithms and Architectures (SPAA 2011), 2011, to appear.
17. G. BALLARD, J. DEMMEL, O. HOLTZ, AND O. SCHWARTZ, Minimizing Communication in Linear Algebra, EECS Technical Report EECS-2011-15, University of California-Berkeley, 2011.
18. J. BUNCH AND L. KAUFMAN, Some stable methods for calculating inertia and solving symmetric linear systems, Math. Comp., 31 (1977), pp. 163-179.
19. A. BUTTARI, J. LANGOU, J. KURZAK, AND J. J. DONGARRA, A Class of Parallel Tiled Linear Algebra Algorithms for Multicore Architectures, Technical Report 191, LAPACK Working Note, 2007.
20. C. H. BISCHOF, B. LANG, AND X. SUN, Algorithm 807: The SBR toolbox-software for successive band reduction, ACM Trans. Math. Software, 26 (2000), pp. 602-616.
21. C. H. BISCHOF, B. LANG, AND X. SUN, A framework for symmetric band reduction, ACM Trans. Math. Software, 26 (2000), pp. 581-601.
22. C. BISCHOF AND C. VAN LOAN, The WY representation for products of Householder matrices, SIAM J. Sci. Statist. Comput., 8 (1987), pp. 2-13.
23. Y. D. BURAGO AND V. A. ZALGALLER, Geometric Inequalities, Grundlehren Math. Wiss. 285, Springer, Berlin, 1988.
24. L. CANNON, A Cellular Computer to Implement the Kalman Filter Algorithm, Ph.D. thesis, Montana State University, Bozeman, MT, 1969.
25. T. CORMEN, C. LEISERSON, R. RIVEST, AND C. STEIN, Introduction to Algorithms, 2nd ed., MIT Press, Cambridge, MA, 2001.
26. R. A. CHOWDHURY AND V. RAMACHANDRAN, Cache-oblivious dynamic programming, in Proceedings of the 17th Annual ACM-SIAM Symposium on Discrete Algorithms, SIAM, Philadelphia, ACM, New York, 2006, pp. 591-600. (Pubitemid 43275280)
27. P.-Y. DAVID, J. DEMMEL, L. GRIGORI, AND S. PEYRONNET, Brief announcement: Lower bounds on communication for sparse Cholesky factorization of a model problem, in Proceedings of the 22nd ACM Symposium on Parallelism in Algorithms and Architectures (SPAA), 2010.
28. J. DEMMEL, I. DUMITRIU, AND O. HOLTZ, Fast linear algebra is sable, Numer. Math., 108 (2007), pp. 59-91.
29. J. DEMMEL, CS 267 Course Notes: Applications of Parallel Processing, Computer Science Division, University of California, 1996. http://www.cs. berkeley.edu/∼demmel/cs267.
30. J. DEMMEL, Applied Numerical Linear Algebra, SIAM, Philadelphia, 1997.
31. J. DEMMEL, L. GRIGORI, M. HOEMMEN, AND J. LANGOU, Communication-Optimal Parallel and Sequential QR and LU Factorizations, EECS Technical Report EECS-2008-89, University of California-Berkeley, 2008, to appear in SIAM. J. Sci. Comput.
32. J. DEMMEL, L. GRIGORI, M. HOEMMEN, AND J. LANGOU, Implementing communication-optimal parallel and sequential QR and LU factorizations, 2008, available from http://arxiv. org/abs/0809.2407.
33. J. DEMMEL, L. GRIGORI, AND H. XIANG, Communication-Avoiding Gaussian elimination, Supercomputing 08, ACM/IEEE, Austin, TX, 2008.
34. J. DEMMEL, L. GRIGORI, AND H. XIANG, CALU: A Communication Optimal LU Factorization Algorithm, EECS Technical Report EECS-2010-29, University of California-Berkeley, 2010, submitted to SIAM J. Matrix Anal. Appl.
35. E. DEKEL, D. NASSIMI, AND S. SAHNI, Parallel matrix and graph algorithms, SIAM J. Comput., 10 (1981), pp. 657-675.
36. E. ELMROTH AND F. GUSTAVSON, New serial and parallel recursive QR factorization algorithms for SMP systems, in Applied Parallel Computing. Large Scale Scientific and Industrial Problems, Lecture Notes in Comput. Sci. 1541, B. Kågström et al., eds., Springer, Berlin, 1998, pp. 120-128.
37. E. ELMROTH AND F. GUSTAVSONApplying recursion to serial and parallel QR factorization leads to better performance, IBM J. Res. Dev., 44 (2000), pp. 605-624.
38. E. ELMROTH, F. GUSTAVSON, I. JONSSON, AND B. KÅGSTRÖM, Recursive blocked algorithms and hybrid data structures for dense matrix library software, SIAM Rev., 46 (2004), pp. 3-45.
39. M. FRIGO, C. E. LEISERSON, H. PROKOP, AND S. RAMACHANDRAN, Cache-Oblivious algorithms, in Proceedings of the 40th Annual IEEE Symposium on Foundations of Computer Science, IEEE Computer Society, Washington, DC, 1999, pp. 285-297. (Pubitemid 30539973)
40. J. D. FRENS AND D. S. WISE, QR factorization with Morton-ordered quadtree matrices for memory re-use and parallelism, SIGPLAN Not., 38 (2003), pp. 144-154.
41. A. GEORGE, Nested dissection of a regular finite element mesh, SIAM J. Numer. Anal., 10 (1973), pp. 345-363.
42. B. C. GUNTER AND R. A. VAN DE GEIJN, Parallel out-of-core computation and updating of the QR factorization, ACM Trans. Math. Software, 31 (2005), pp. 60-78. (Pubitemid 40557862)
43. J. A. GUNNELS, F. G. GUSTAVSON, G. M. HENRY, AND R. A. VAN DE GEIJN, FLAME: Formal linear algebra methods environment, ACM Trans. Math. Software, 27 (2001), pp. 422-455. (Pubitemid 33602331)
44. G. H. GOLUB, R. J. PLEMMONS, AND A. SAMEH, Parallel block schemes for large-scale least-squares computations, in High-Speed Computing: Scientific Applications and Algorithm Design University of Illinois Press, Champaign, IL, 1988, pp. 171-179.
45. J. R. GILBERT AND R. E. TARJAN, The analysis of a nested dissection algorithm, Numer. Math., 50 (1987), pp. 377-404.
46. F. G. GUSTAVSON, Recursion leads to automatic variable blocking for dense linear-algebra algorithms, IBM J. Res. Dev., 41 (1997), pp. 737-756.
47. G. GOLUB AND C. VAN LOAN, Matrix Computations, 3rd ed., Johns Hopkins University Press, Baltimore, MD, 1996.
48. J. W. HONG AND H. T. KUNG, I/O complexity: The red-blue pebble game, in Proceedings of the 13th Annual ACM Symposium on Theory of Computing, ACM, New York, 1981, pp. 326-333.
49. A. J. HOFFMAN, M. S. MARTIN, AND D. J. ROSE, Complexity bounds for regular finite difference and finite element grids, SIAM J. Numer. Anal., 10 (1973), pp. 364-369.
50. D. IRONY AND S. TOLEDO, Trading replication for communication in parallel distributed-memory dense solvers, Parallel Process. Lett., 12 (2002), pp. 79-94. (Pubitemid 34668795)
51. D. IRONY, S. TOLEDO, AND A. TISKIN, Communication lower bounds for distributed-memory matrix multiplication, J. Parallel Distrib. Comput., 64 (2004), pp. 1017-1026. (Pubitemid 40000755)
52. L. H. LOOMIS AND H. WHITNEY, An inequality related to the isoperimetric inequality, Bull. Am. Math. Soc., 55 (1949), pp. 961-962.
53. J. P. MICHAEL, M. PENNER, AND V. K. PRASANNA, Optimizing graph algorithms for improved cache performance, in Proceedings of the International Parallel and Distributed Processing Symposium (IPDPS 2002), Fort Lauderdale, FL, 2002, pp. 769-782.
54. W. F. MCCOLL AND A. TISKIN, Memory-efficient matrix multiplication in the BSP model, Algorithmica, 24 (1999), pp. 287-297. (Pubitemid 129715337)
55. C. PUGLISI, Modification of the Householder method based on compact WY representation, SIAM J. Sci. Statist. Comput., 13 (1992), pp. 723-726.
56. Y. SAAD, Communication complexity of the Gaussian elimination algorithm on multiprocessors, Linear Algebra Appl., 77 (1986), pp. 315-340.
57. Y. SAAD, Iterative Methods for Sparse Linear Systems, PWS Publishing Co., Boston, 1996.
58. J. E. SAVAGE, Extending the Hong-Kung model to memory hierarchies, in Computing and Combinatorics, Lecture Notes in Comput. Sci. 959, Springer, Berlin, 1995, pp. 270-281.
59. E. SOLOMONIK AND J. DEMMEL, Communication-Optimal Parallel 2.5D Matrix Multiplication and LU Factorization Algorithms, EECS Technical Report EECS-2011-10, University of California-Berkeley, 2011, to appear in EURO-PAR 2011.
60. R. SCHREIBER AND C. VAN LOAN, A storage-efficient WY representation for products of Householder transformations, SIAM J. Sci. Statist. Comput., 10 (1989), pp. 53-57.
61. S. TOLEDO, Locality of reference in LU decomposition with partial pivoting, SIAM J. Matrix Anal. Appl., 18 (1997), pp. 1065-1081.
62. R. VAN DE GEIJN, PLAPACK: Parallel Linear Algebra Package, www.cs.utexas.edu/users/plapack.
63. R. VUDUC, J. DEMMEL, AND K. YELICK, OSKI: A library of automatically tuned sparse matrix kernels, in Proceedings of SciDAC 2005, J. of Physics: Conference Series, Institute of Physics Publishing, London, 2005.
64. R. YUSTER AND U. ZWICK, Fast sparse matrix multiplication, ACM Trans. Algorithms, 1 (2005), pp. 2-13.