Recursive blocked algorithms and hybrid data structures for dense matrix library software

SIAM Review

Volumn 46, Issue 1, 2004, Pages 3-45

  Elmroth, Erik ^a   Gustavson, Fred ^b   Jonsson, Isak ^a   Kågström, Bo ^a  

^a UMEÅ UNIVERSITY   (Sweden)

^b IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

Automatic variable blocking; Dense linear algebra; ESSL; Factorizations; GEMM based; Hybrid data structures; LAPACK; Level 3 BLAS; Library software; Matrix equations; RECSY; Recursion; SLICOT; SMP parallelization; Superscalar; Superscalar kernels

Indexed keywords

ALGOL (PROGRAMMING LANGUAGE); ALGORITHMS; CODES (SYMBOLS); COMPUTER SOFTWARE; DIGITAL COMPUTERS; DIGITAL LIBRARIES; LINEAR SYSTEMS; MATRIX ALGEBRA; PROBLEM SOLVING;

AUTOMATIC VARIABLE BLOCKING; DENSE MATRIX LIBRARY SOFTWARE; FACTORIZATIONS; RECURSIVE BLOCKED ALGORITHMS;

DATA STRUCTURES;

EID: 1842832833     PISSN: 00361445     EISSN: None     Source Type: Journal    
DOI: 10.1137/S0036144503428693     Document Type: Article

References (104)

1. R. C. Agarwal, F. G. Gustavson, and M. Zubair, Exploiting functional parallism of POWER2 to design high-performance numerical algorithms, IBM J. Res. Develop., 38 (1994), pp. 563-576.
2. N. Ahmed and K. Pingali, Automatic generation of block-recursive codes, in Euro-Par 2000 Parallel Processing, A. Bode et al., eds., Lecture Notes in Comput. Sci. 1900, Springer-Verlag, New York, 2000, pp. 368-378.
3. B. Andersen, F. Gustavson, and J. Waśniewski, A recursive formulation of Cholesky factorization of a matrix in packed storage, ACM Trans. Math. Software, 27 (2001), pp. 214-244.
4. E. Anderson, Z. Bai, C. Bischof, S. Blackford, J. Demmel, J. Dongarra, J. Du Croz, A. Greenbaum, S. Hammarling, A. McKenney and D. Sorensen, LAPACK Users' Guide, 3rd ed., SIAM, Philadelphia, 1999.
5. ATLAS, Automatically Tuned Linear Algebra Software, http://math-atlas.sourceforge.net/.
6. Z. Bai, J. Demmel, and A. McKenney, On computing condition numbers for the nonsymmetric eigenproblem, ACM Trans. Math. Software, 19 (1993), pp. 202-223.
7. J. Barnes and P. Hut, A hierarchical O(n logn) force calculation algorithm, Nature, 324 (1986), pp. 446-449.
8. R. H. Bartels and G. W. Stewart, Solution of the equation AX + XB = C, Comm. Assoc. Comput. Mach., 15 (1972), pp. 820-826.
9. J. Bilmes, K. Asanovic, C.-W. Chin, and J. Demmel, Optimizing matrix multiply using PHiPAC: A portable high-performance ANSI C methodology, in Proceedings of the International Conference on Supercomputing, Vienna, 1997, pp. 340-347.
10. C. Bischof and C. Van Loan, The WY representation for products of Householder matrices, SIAM J. Sci. Statist. Comput., 8 (1987), pp. s2-s13.
11. K. Braman, R. Byers, and R. Mathias, The multishift QR algorithm. Part I: Maintaining well-focused shifts and level 3 performance, SIAM J. Matrix Anal. Appl., 23 (2002), pp. 929-947.
12. K. Braman, R. Byers, and R. Mathias, The multishift QR algorithm. Part II: Aggressive early deflation, SIAM J. Matrix Anal. Appl., 23 (2002), pp. 948-973.
13. J. R. Bunch, J. J. Dongarra, C. B. Moler, and G. W. Stewart, LINPACK User's Guide, SIAM, Philadelphia, 1979.
14. S. Carr and R. B. Lehoucq, Compiler blockability of dense matrix factorizations, ACM Trans. Math. Software, 23 (1997), pp. 336-361.
15. S. Chatterjee, A. R. Lebeck, P. K. Patnala, and M. Thottethodi, Recursive array layouts and fast matrix multiplication, IEEE Trans. Parallel Distrib. Systems, 13 (2002), pp. 1105-1123.
16. K.-W. E. Chu, The solution of the matrix equations AXB - CXD = E and (Y A - DZ, Y C - BZ) = (E, F), Linear Algebra Appl., 93 (1987), pp. 93-105.
17. J. J. M. Cuppen, A divide and conquer method for the symmetric tridiagonal eigenproblem, Numer. Math., 36 (1981), pp. 177-195.
18. K. Dackland and B. Kågström, Blocked algorithms and software for reduction of a regular matrix pair to generalized Schur form, ACM Trans. Math. Software, 25 (1999), pp. 425-454.
19. J. Demmel and B. Kågström, Computing stable eigendecompositions of matrix pencils, Linear Algebra Appl., 88/89, (1987), pp. 139-186.
20. J. W. Demmel and N. J. Higham, Stability of block algorithms with fast level-3 BLAS, ACM Trans. Math. Software, 18 (1992), pp. 274-291.
21. J. Dongarra, J. Du Croz, I. Duff, and S. Hammarling, Algorithm 679: A set of level 3 basic linear algebra subprograms, ACM Trans. Math. Software, 16 (1990), pp. 18-28.
22. J. Dongarra, J. Du Croz, I. Duff, and S. Hammarling, A set of level 3 basic linear algebra subprograms, ACM Trans. Math. Software, 16 (1990), pp. 1-17.
23. J. Dongarra, J. Du Croz, S. Hammarling, and R. J. Hanson, An extended set of Fortran basic linear algebra subroutines, ACM Trans. Math. Software, 14 (1988), pp. 1-17.
24. J. Dongarra, V. Eijkhout, and P. Łszczek, Recursive approach in sparse matrix LU factorization, Sci. Programming, 9 (2001), pp. 51-60.
25. J. J. Dongarra and D. W. Walker, Software libraries for linear algebra computations on high performance computers, SIAM Rev., 37 (1995), pp. 151-180.
26. C. G. Douglas, M. Heroux, G. Slishman, and R. M. Smith, GEMMW: A portable 3 BLAS Winograd variant of Strassen's matrix multiply algorithm, J. Comput. Phys., 110 (1994), pp. 1.-10.
27. E. Elmroth and F. G. Gustavson, New serial and parallel recursive QR factorization algorithms for SMP systems, in Applied Parallel Computing: Large Scale Scientific and Industrial Problems, B. Kågström et al. eds., Lecture Notes in Comput. Sci. 1541, Springer-Verlag, New York, 1998, pp. 120-128.
28. E. Elmroth and F. G. Gustavson, Applying recursion to serial and parallel QR factorization leads to better performance, IBM J. Res. Develop., 44 (2000), pp. 605-624.
29. E. Elmroth and F. G. Gustavson, A faster and simpler recursive algorithm for the LAPACK routine DGELS, BIT, 41 (2001), pp. 936-949.
30. E. Elmroth and F. G. Gustavson, High-performance library software for QR factorization, in Applied Parallel Computing: New Paradigms for HPC in Industry and Academia, T. Sørvik et al., eds., Lecture Notes in Comput. Sci. 1947, Springer-Verlag, New York, 2001, pp. 53-63.
31. G. E. Forsythe and C. B. Moler, Computer Solution of Linear Algebraic Systems, Prentice-Hall, Englewood Cliffs, NJ, 1967.
32. J. D. Frens and D. S. Wise, QR factorization with Morton-ordered quadtree matrices for memory re-use and parallelism , in Proceedings of the 2003 ACM Symposium on Principles and Practice of Parallel Programming, ACM SIGPLAN Notices, 38 (10) (2003), pp. 144-154.
33. M. Frigo, A fast Fourier transform compiler, in Proceedings of the 1999 ACM SIGPLAN Conference on Programming Language Design and Implementation, ACM SIGPLAN Notices, 34 (3) (1999), pp. 169-180.
34. M. Frigo and S. G. Johnson, FFTW: An adaptive software architecture for the FFT. in Proceedings of the 1998 IEEE International Conference on Acoustics Speech and Signal Processing, Vol. 3, IEEE Press, 1998, pp. 1381-1384.
35. M. Frigo, C. E. Leiserson, H. Prokop, and S. Ramachandran, Cache-oblivious algorithms. in Proceedings of the 40th Annual IEEE Symposium on Foundations of Computer Science, New York, 1999, IEEE Computer Society, Los Alamitos, CA, 1999.
36. K. A. Gallivan, R. J. Plemmons, and A. H. Sameh, Parallel algorithms for dense linear algebra computations, SIAM Rev., 32 (1990), pp. 54-135.
37. T = E, ACM Trans. Math. Software, 18 (1992) pp. 223-231.
38. G. Golub, S. Nash, and C. Van Loan, A Hessenberg-Schur method for the matrix problem AX + X B = C, IEEE Trans. Automat. Control, AC-24 (1979), pp. 909-913.
39. G. Golub and C. Van Loan, Matrix Computations, 3rd ed., Johns Hopkins University Press, Baltimore, MD, 1996.
40. K. Goto and R. van de Geijn, On Reducing TLB Misses in Matrix Multiplication, Technical Report TR-2002-55, FLAME Working Note 9, Department of Computer Sciences, University of Texas at Austin, 2002.
41. J. Gunnels, F. G. Gustavson, G. Henry, and R. van de Geijn, Formal linear algebra methods environment (FLAME), ACM Trans. Math. Software, 27 (2001) pp. 422-455.
42. J. A. Gunnels, D. S. Katz, E. S. Quintana-Orti, and R. van de Geijn, Fault-Tolerant High-Performance Matrix-Matrix Multiplication. FLAME Technical Report TR-2000-34, Working Note 2. Department of Computing Sciences, University of Texas at Austin, 2000.
43. F. G. Gustavson, Recursion leads to automatic variable blocking for dense linear-algebra algorithms, IBM J. Res. Develop., 41 (1997), pp. 737-755.
44. F. G. Gustavson, New generalized data structures for matrices lead to a variety of high performance algorithms, in The Architectures for Scientific Software, R. F. Boisvert and P. T. P. Tang, eds., IFIP Conference Proceedings 188, Kluwer Academic, Dordrecht, The Netherlands, pp. 211-234.
45. F. G. Gustavson, High-performance linear algebra algorithms using new generalized data structures for matrices, IBM J. Res. Develop., 47 (2003), pp. 31-554.
46. F. G. Gustavson, A. Henriksson, I. Jonsson, B. Kågström, and P. Ling, Recursive blocked data formats and BLAS's for dense linear algebra algorithms, in Applied Parallel Computing: Large Scale Scientific and Industrial Problems, B. Kågström et al., eds., Lecture Notes in Comput. Sci. 1541, Springer-Verlag, New York, 1998, pp. 195-206.
47. F. G. Gustavson, A. Henriksson, I. Jonsson, B. Kågström, and P. Ling, Superscalar GEMM-based level 3 *BLAS - The on-going evolution of a portable and high-performance library, in Applied Parallel Computing: Large Scale Scientific and Industrial Problems, B. Kåström et al., eds., Lecture Notes in Comput. Sci. 1541, Springer-Verlag, New York, 1998, pp. 207-215.
48. F. G. Gustavson and I. Jonsson, Minimal-storage high-performance Cholesky factorization via blocking and recursion, IBM J. Res. Develop., 44 (2000), pp. 823-849.
49. W. W. Hager, Condition estimates, SIAM J. Sci. Statist. Comput., 5 (1984), pp. 311-316.
50. S. J. Hammarling, Numerical solution of the stable, non-negative definite Lyapunov equation, IMA J. Numer. Anal., 2 (1982), pp. 303-323.
51. A. Henriksson and I. Jonsson, High-Performance Matrix Multiplication on the IBM SP High Node, Master's thesis, UMNAD-98.235, Department of Computing Science, Umeå University, Umeå, Sweden, 1998.
52. N. J. Higham, Fortran codes for estimating the one-norm of a real or complex matrix with applications to condition estimation, ACM Trans. Math. Software, 14 (1988), pp. 381-396.
53. N. J. Higham, Perturbation theory and backward error for AX - XB = C, BIT, 33 (1993), pp. 124-136.
54. N. J. Higham, Accuracy and Stability of Numerical Algorithms, SIAM, Philadelphia, 1996.
55. IBM, Engineering and Scientific Subroutine Library, Guide and Reference, 1990.
56. IBM Engineering and Scientific Subroutine Library, Guide and Reference, Ver. 3, Rel. 1, 1998.
57. IBM, Engineering and Scientific Subroutine Library, Guide and Reference, Ver. 3, Rel. 3, 2001.
58. D. Irony, G. Shklarski, and S. Toledo, Parallel and fully recursive multifrontal supernodal sparse Cholesky, in Computational Science - ICCS 2002, P. Sloot et al., eds., Lecture Notes in Comput. Sci. 2330, Springer-Verlag, Berlin, 2002, pp. 335-344.
59. I. Jonsson, Analysis of Processor and Memory Utilization of Recursive Algorithms for Sylvester-Type Matrix Equations Using Performance Monitoring, Technical Report UNINF-03.16, Department of Computing Science, Umeå University, Umeå, Sweden, 2003.
60. I. Jonsson and B. Kågström, Parallel triangular Sylvester-type matrix equation solvers for SMP systems using recursive blocking, in Applied Parallel Computing: New Paradigms for HPC Industry and Academia, T. Sørvik et al., eds., Lecture Notes in Comput. Sci. 1947, Springer-Verlag, New York, 2001, pp. 64-73.
61. I. Jonsson and B. Kågström, Parallel two-sided Sylvester-type matrix equation solvers for SMP systems using recursive blocking, in Applied Parallel Computing: Advanced Scientific Computing, J. Fagerhom et al., eds., Lecture Notes in Comput. Sci. 2367, Springer-Verlag, New York, 2002, pp. 297-306.
62. I. Jonsson and B. Kågström, Recursive blocked algorithms for solving triangular systems - Part I: One-sided and coupled Sylvester-type matrix equations, ACM Trans. Math. Software, 28 (2002), pp. 392-415.
63. I. Jonsson and B. Kågström, Recursive blocked algorithms for solving triangular systems - Part II: Two sided and generalized Sylvester and Lyapunov equations, ACM Trans. Math. Software, 28, (2002), pp. 416-435.
64. I. Jonsson and B. Kågström, RECSY - A High Performance Library for Sylvester-Type Matrix Equations, http://www.cs.umu.se/research/parallel/recsy, 2003.
65. B. Kåström, A perturbation analysis of the generalized Sylvester equation (AR - LB, DR - LE) = (C, F), SIAM J. Matrix Anal. Appl., 15 (1994), pp. 1045-1060.
66. B. Kågström, P. Ling, and C. Van Loan, GEMM-based level 3 BLAS: High-performance model implementations and performance evaluation benchmark, ACM Trans. Math. Software, 24 (1998), pp. 268-302.
67. B. Kågström, P. Ling, and C. Van Loan, Algorithm 784: GEMM-based level 3 BLAS: Portability and optimization issues, ACM Trans. Math. Software, 24, (1998), pp. 303-316.
68. -1 estimators, SIAM J. Matrix Anal. Appl., 13 (1992), pp. 90-101.
69. B. Kågström, and P. Poromaa, Computing eigenspaces with specified eigenvalues of a regular matrix pair (A, B) and condition estimation: Theory, algorithms and software, Numer. Algorithms, 12 (1996), pp. 369-407.
70. B. Kågström, and P. Poromaa, LAPACK-style algorithms and software for solving the generalized Sylvester equation and estimating the separation between regular matrix pairs, ACM Trans, Math. Software, 22 (1996), pp. 78-103.
71. B. Kågström, and P. Van Dooren, A generalized state-space approach for the additive decomposition of a transfer matrix, Internat. J. Numer. Linear Algebra Appl., 1 (1992), pp. 165-181.
72. B. Kågström, and L. Westin, Generalized Schur methods with condition estimators for solving the generalized Sylvester equation, IEEE Trans. Automat. Control, 34 (1989), pp. 745-751.
73. M. S. Lam, E. E. Rothberg, and M. E. Wolf, The cache performance and optimizations of blocked algorithms, in Proceedings of the Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, 1991, pp. 63-74.
74. C. Lawson, R. Hanson, D. Kincaid, and F. Krogh, Basic linear algebra subprograms for Fortran usage, ACM Trans. Math. Software, 5 (1979), pp. 308-323.
75. A. Lindkvist, High-Performance Recursive BLAS Kernels Using New Data Formats for the QR Factorization, Master's thesis, UMNAD-235.00, Department of Computing Science, Umeå University, Umeå, Sweden, 2000.
76. MathWorks, Using MATLAB, The MathWorks Inc., Natick, MA, 2002.
77. N. Park, B. Hong, and V. K. Prasanna, Tiling, block data layout, and memory hierarchy performance, IEEE Trans. Parallel Distrib. Systems, 14 (2003), pp. 640-654.
78. B. N. Parlett and Y. Wang, The influence of the compiler on the cost of mathematical software - In particular on the cost of triangular factorization, ACM Trans. Math. Software, 1 (1975), pp. 35-46.
79. T. Penzl, Numerical solution of generalized Lyapunov equations, Adv. Comput. Math., 8 (1998), pp. 33-48.
80. P. Poromaa, Parallel algorithms for triangular sylvester equations: Design, scheduling and scalability issues, in Applied Parallel Computing: Large Scale Scientific and Industrial Problems, B. Kågström et al., eds., Lecture Notes in Comput. Sci. 1541, Springer-Verlag, New York, 1998, pp. 438-446.
81. P. Poromaa, High Performance Computing: Algorithms and Library Software for Sylvester Equations and Certain Eigenvalve Problems with Applications in Condition Estimation, Ph.D. Thesis, UMINF-97.16, Department of Computing Science, Umeå University, Umeå, Sweden, 1997.
82. E. Rabani and S. Toledo, Out-of-core SVD and QR decompositions, in Proceedings of the 10th SIAM Conference on Parallel Processing for Scientific Computing, Portsmouth, VA, CD-ROM, SIAM, Philadelphia, 2001.
83. E. Rabani and S. Toledo, Very large electronic structure calculations using an out-of-core filter-diagonalization method, J. Comput. Phys., 180 (2002), pp. 256-269.
84. H. Sagan, Space-Filling Curves, Springer-Verlag, Berlin, 1994.
85. J. K. Salmon and M. S. Warren, Skeletons from the treecode closet, J. Comput. Phys., 111 (1994), pp. 136-155.
86. J. K. Salmon, M. S. Warren, and G. S. Winckelmans, Fast parallel tree codes for gravitational and fluid dynamical n-body problems, Internat. J. Supercomput. Appl., 8 (1994), pp. 129-142.
87. H. Samet, The quadtree and related hierarchical data structures, Comput. Surveys, 16 (1984), pp. 188-260.
88. R. Schrieber and C. Van Loan, A storage-efficient WY representation for products of Householder transformations, SIAM J. Sci. Statist. Comput., 10 (1989), pp. 53-57.
89. SGI, Scientific Computing Software Library (SCSL), software and documentation available from http://www.sgi.com/software/scsl.html, 1993-2003.
90. SLICOT, The SLICOT Library and the Numerics in Control Network (NICONET) website, http://www.win.tue.nl/niconet/.
91. B. T. Smith, J. M. Boyle, J. J. Dongarra, B. S. Garbow, Y. Ikebe, V. C. Klema, and C. B. Moler, Matrix Eigensystem Routines-EISPACK Guide, Lecture Notes in Comput. Sci. 6, Springer-Verlag, Berlin, 1976.
92. G. W. Stewart, Matrix Algorithms, Volume I: Basic Decompositions, SIAM, Philadelphia, 1998.
93. G. W. Stewart and J.-G. Sun, Matrix Perturbation Theory, Academic Press, New York, 1990.
94. V. Strassen, Gaussian elimination is not optimal, Numer. Math., 13 (1969), pp. 354-356.
95. S. Toledo, Locality of reference in LU decomposition with partial pivoting, SIAM J. Matrix Anal. Appl., 18 (1997), pp. 1065-1081.
96. TOP500, The Top 500 Supercomputer Sites, http://www.top500.org/.
97. V. Valsalam and A. Skjellum, A framework for high-performance matrix multiplication based on hierarchical abstractions, algorithms and optimized low-level kernels, Concurrency Computat. Pract. Exper., 14, (2002), pp. 805-839.
98. R. Vuduc, J. W. Demmel, and J. A. Bilmes, Statistical models for automatic performance tuning, in Proceedings of the International Conference on Computational Science, Lecture Notes in Comput. Sci. 2073, Springer-Verlag, New York 2001, pp. 117-126.
99. T Ax, in Proceedings of the ICCS Workshop on Parallel Linear Algebra, Lecture Notes in Comput. Sci. 2660, Springer-Verlag, New York, 2003, pp. 705-714.
100. R. C. Whaley, A. Patitet, and J. Dongarra, Automated empirical optimization of software and the ATLAS project, LAPACK Working Note 147, 2000; see also http://sourcegforge.net/projects/math-atlas/.
101. J. H. Wilkinson and C. Reinsch, Handbook for Automatic Computation, Vol. II. Linear Algebra, Springer-Verlag, New York, 1971.
102. D. S. Wise, G. A. Alexander, J. D. Frens, and Y. H. Gu, Language support for Morton order matrices. ACM SIGPLAN Notices, 36 (7) (2001), pp. 24-33.
103. M. E. Wolf and M. S. Lam, A data locality optimizing algorithm, in Proceedings of the ACM SIGPLAN'91 Conference on Programming Language Design and Implementation, 1991, pp. 30-44.
104. Q. Yi, V. Adve, and K. Kennedy, Transforming loops to recursion for multi-level memory hierarchies, ACM SIGPLAN Notices, 35 (5) (2000), pp. 169-181.