Self-adapting Linear Algebra algorithms and software

Proceedings of the IEEE

Volumn 93, Issue 2, 2005, Pages 293-311

  Demmel, James ^a   Dongarra, Jack ^b,c,d   Eijkhout, Victor ^b   Fuentes, Erika ^b   Petitet, Antoine ^e   Vuduc, Richard ^f   Whaley, R Clint ^g   Yelick, Katherine ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b University of Tennessee Knoxville   (United States)

^c OAK RIDGE NATIONAL LABORATORY   (United States)

^d Rice University   (United States)

^e SUN MICROSYSTEMS   (United States)

^f LAWRENCE LIVERMORE NATIONAL LABORATORY   (United States)

^g Florida State University   (United States)

Author keywords

Adaptive methods; Basic Linear Algebra Subprograms (BLAS); Dense kernels; Iterative methods; Linear systems; Matrix matrix product; Matrix vector product; Performance optimization; Preconditioners; Sparse kernels

Indexed keywords

ALGORITHMS; COMPUTER ARCHITECTURE; COMPUTER SIMULATION; COMPUTER SOFTWARE; ITERATIVE METHODS; LINEAR SYSTEMS; MATHEMATICAL MODELS; MATRIX ALGEBRA; OPTIMIZATION; VECTORS;

ADAPTIVE METHODS; BASIC LINEAR ALGEBRA SUBPROGRAMS (BLAS); DENSE KERNELS; MATRIX-MATRIX PRODUCTS; MATRIX-VECTOR PRODUCTS; PERFORMANCE OPTIMIZATION; PRECONDITIONERS; SPARSE KERNELS;

LINEAR ALGEBRA;

EID: 20744452904     PISSN: 00189219     EISSN: None     Source Type: Journal    
DOI: 10.1109/JPROC.2004.840848     Document Type: Conference Paper

References (81)

1. J. Bilmes, K. Asanović, C. Chin, and J. Demmel, "Optimizing matrix multiply using PHiPAC: A portable, high-performance, ANSI C coding methodology," presented at the Int. Conf. Supercomputing, Vienna, Austria, 1997.
2. R. C. Whaley, A. Petitet, and J. J. Dongarra, "Automated empirical optimization of software and the ATLAS project," Parallel Comput., vol. 27, no. 1-2, pp. 3-35, 2001.
3. E. I. Foster and C. Kesselman, The Grid - Blueprint for a New Computing Infrastructure. San Francisco, CA: Morgan Kaufmann, 1998.
4. J. Dongarra and E. Grosse, "Distribution of mathematical software via electronic mail," Commun. ACM, vol. 30, no. 5, pp. 403-407, 1987.
5. J. Du Croz and M. Pont, "The development of a floating-point validation package," presented at the 8th Symp. Computer Arithmetic, Como, Italy, 1987.
6. W. Kahan. (1987) Paranoia. [Online]. Available: http://www.netlib.org/
7. J. Dongarra, R. Pozo, and D. Walker, "An object oriented design for high performance linear algebra on distributed memory architectures," in Proc. Object Oriented Numerics Conf, 1993, pp. 268-269.
8. C. Koebel, D. Loveman, R. Schreiber, G. Steele, and M. Zosel, The High Performance Fortran Handbook. Cambridge, MA: MIT Press, 1994.
9. L. Blackford, J. Dongarra, C. Papadopoulos, and R. C. Whaley, "Installation guide and design of the HPF 1.1 interface to ScaLAPACK, SLHPF (LAPACK Working Note 137)," Univ. Tennessee, Knoxville, TN, Tech. Rep. UT CS-98-396, 1998.
10. R. C. Whaley and A. Petitet, "Minimizing development and maintenance costs in supporting persistently optimized BLAS," Softw. Pract. Exper., to be published.
11. R. C. Whaley and J. Dongarra, "Automatically Tuned Linear Algebra Software," in 9th SIAM Conf. Parallel Processing for Scientific Computing, San Antonio, TX, 1999.
12. _, "Automatically Tuned Linear Algebra Software," presented at the 1998 IEEE Conf. Supercomputing: Conf. High Performance Networking and Computing, San Jose, CA.
13. _, "Automatically Tuned Linear Algebra Software," Univ. Tennessee, Knoxville, Tech. Rep. UT-CS-97-366, 1997.
14. R. C. Whaley and A. Petitet. ATLAS home page. [Online]. Available: http://math-atlas.sourceforge.net/
15. M. Frigo and S. Johnson, "FFTW: An adaptive software architecture for the FFT," in Proc. Int. Conf. Acoustics, Speech, and Signal Processing (ICASSP), vol. 3, 1998, p. 1381.
16. M. Frigo and S. G. Johnson, "The Fastest Fourier Transform in the West," Massachusetts Inst. Technol., Tech. Rep. MIT-LCS-TR-728, 1997.
17. R. C. Whaley and A. Petitet. ATLAS timing page. [Online]. Available: http://math-atlas.sourceforge.net/timing/
18. R. Hanson, F. Krogh, and C. Lawson, "A proposal for standard linear algebra subprograms," ACM SIGNUM Newslett., vol. 8, no. 16, p. 16, 1973.
19. C. Lawson, R. Hanson, D. Kincaid, and F. Krogh, "Basic linear algebra subprograms for Fortran usage," ACM Trans. Math. Softw., vol. 5, no. 3, pp. 308-323, 1979.
20. J. Dongarra, J. D. Croz, S. Hammarling, and R. Hanson, "Algorithm 656: An extended set of basic linear algebra subprograms: Model implementation and test programs," ACM Trans. Math. Softw., vol. 14, no. 1, pp. 18-32, 1988.
21. _, "An extended set of FORTRAN basic linear algebra subprograms," ACM Trans. Math. Softw., vol. 14, no. 1, pp. 1-17, 1988.
22. J. Dongarra, J. D. Croz, I. Duff, and S. Hammarling, "A set of level 3 basic linear algebra subprograms," ACM Trans. Math. Softw., vol. 16, no. 1, pp. 1-17, 1990.
23. E. Anderson, Z. Bai, C. Bischof, J. Demmel, J. Dongarra, J. D. Croz, A. Greenbaum, S. Hammarling, A. McKenney, S. Ostrouchov, and D. Sorensen, LAPACK Users' Guide, 3rd ed. Philadelphia, PA: SIAM, 1999.
24. R. C. Whaley. User contribution to ATLAS. [Online]. Available: http://math-atlas.sourceforge.net/devel/atlas_contrib/
25. B. Kågström, P. Ling, and C. van Loan, "GEMM-based level 3 BLAS: High-performance model implementations and performance evaluation benchmark," Dept. Comput. Sci., Umeå Univ., Umeå, Sweden, Tech. Rep. UMINF 95-18, 1995.
26. _, "GEMM-based level 3 BLAS: High performance model implementations and performance evaluation benchmark," ACM Trans. Math. Softw., vol. 24, no. 3, pp. 268-302, 1998.
27. M. Dayde, I. Duff, and A. Petitet, "A parallel block implementation of level 3 BLAS for MIMD vector processors," ACM Trans. Math. Softw., vol. 20, no. 2, pp. 178-193, 1994.
28. F. Gustavson, A. Henriksson, I. Jonsson, B. Kågström, and P. Ling, "Recursive blocked data formats and BLAS's for dense linear algebra algorithms," in Lecture Notes in Computer Science, Applied Parallel Computing, PARA'98, B. Kågström, J. Dongarra, E. Elmroth, and J. Waśniewski, Eds., 1998, vol. 1541, pp. 195-206.
29. _, "Superscalar GEMM-based level 3 BLAS-The on-going evolution of a portable and high-performance library," in Lecture Notes in Computer Science, Applied Parallel Computing, PARA'98, vol. 1541, B. Kågström, J. Dongarra, E. Elmroth, and J. Waśniewski, Eds., 1998, pp. 207-215.
30. E.-J. Im, K. Yelick, and R. Vuduc, "Sparsity: Optimization framework for sparse matrix kernels," Int. J. High Perform. Comput. Appl., vol. 18, no. 1, pp. 135-158, Feb. 2004.
31. R. Vuduc, "Automatic performance tuning of sparse matrix kernels." Ph.D. dissertation, Univ. California, Berkeley, 2003.
32. B. C. Lee, R. Vuduc, J. W. Demmel, K. A. Yelick, M. deLorimier, and L. Zhong, "Performance modeling and analysis of cache blocking in sparse matrix vector multiply," Univ. California, Berkeley, Tech. Rep. UCB/CSD-04-1335, 2004.
33. R. Nishtala, R. Vuduc, J. W. Demmel, and K. A. Yelick, "When cache blocking sparse matrix multiply works and why." presented at the PARA '04 Workshop State-of-the-Art in Scientific Computing, Copenhagen, Denmark, 2004.
34. H. J. Moon, R. Vuduc, J. W. Demmel, and K. A. Yelick, "Matrix splitting and reordering for sparse matrix-vector multiply," Univ. California, Berkeley, to be published.
35. R. Vuduc, J. W. Demmel, K. A. Yelick, S. Kamil, R. Nishtala, and B. Lee, "Performance optimizations and bounds for sparse matrix-vector multiply," presented at the Conf. Supercomputing, Baltimore, MD, 2002.
36. E.-J. Im, "Optimizing the performance of sparse matrix-vector multiplication," Ph.D. dissertation, Univ. California, Berkeley, 2000.
37. Y. Saad. (1994) SPARSKIT: A basic toolkit for sparse matrix computations. [Online]. Available: www.cs.umn.edu/Research/arpa/SPARSKIT/sparskit.html
38. S. Balay, K. Buschelman, W. D. Gropp, D. Kaushik, M. Knepley, L. C. McInnes, B. F. Smith, and H. Zhang, "PETSc user's manual," Argonne Nat. Lab., Tech. Rep. ANL-95/11, 2.1.5 ed., 2002.
39. J. R. Gilbert, C. Moler, and R. Schreiber, "Sparse matrices in MATLAB: Design implementation," Xerox PARC, Tech. Rep. CSL-91-04, 1991.
40. S. Chatterjee, E. Parker, P. J. Hanlon, and A. R. Lebeck, "Exact analysis of the cache behavior of nested loops," in Proc. ACM SIGPLAN 2001 Conf. Programming Language Design and Implementation, pp. 286-297.
41. S. Ghosh, M. Martonosi, and S. Malik, "Cache miss equations: A compiler framework for analyzing and tuning memory behavior," ACM Trans. Program. Lang. Syst., vol. 21, no. 4, pp. 703-746, 1999.
42. K. S. McKinley, S. Carr, and C.-W. Tseng, "Improving data locality with loop transformations," ACM Trans. Program. Lang. Syst., vol. 18, no. 4, pp. 424-453, Jul. 1996.
43. M. E. Wolf and M. S. Lam, "A data locality optimizing algorithm," in Proc. ACM SIGPLAN 1991 Conf. Programming Language Design and Implementation, pp. 30-44.
44. K. Goto. (2004) High-performance BLAS. [Online]. Available: www.cs.utexas.edu/users/flame/goto
45. T. Davis. UF sparse matrix collection. [Online]. Available: www.cise.ufl.edu/research/sparse/matrices
46. R. Vuduc, S. Kamil, J. Hsu, R. Nishtala, J. W. Demmel, and K. A. Yelick, "Automatic performance tuning and analysis of sparse triangular solve," presented at the ICS 2002: Workshop Performance Optimization via High-Level Languages and Libraries, New York.
47. TAx," presented at the ICCS Workshop Parallel Linear Algebra, Melbourne, Australia, 2003.
48. J. W. Demmel, Applied Numerical Linear Algebra. Philadelphia, PA: SIAM, 1997.
49. A. H. Baker, E. R. Jessup, and T. Manteuffel, "A technique for accelerating the convergence of restarted GMRES," Dept. Comput. Sci., Univ. Colorado, Tech. Rep. CU-CS-045-03, 2003.
50. M. W. Berry, S. T. Dumais, and G. W. O'Brien, "Using linear algebra for intelligent information retrieval," SIAM Rev., vol. 37, no. 4, pp. 573-595, 1995.
51. J. B. White and P. Sadayappan, "On improving the performance of sparse matrix-vector multiplication," in Proc. Int. Conf. High-Performance Computing, 1997, pp. 66-71.
52. H. Okuda, K. Nakajima, M. Iizuka, L. Chen, and H. Nakamura, "Parallel finite element analysis platform for the Earth simulator: GeoFEM," in Proc. 3rd Int. Conf. Computational Science Workshop Computational Earthquake Physics and Solid Earth System Simulation, vol. 3, P. M. A. Sloot, D. Abramson, A. V. Bogdanov, J. J. Dongarra, A. Y. Zomaya, and Y. E. Gorbachev, Eds., 2003, pp. 773-780.
53. A. Pinar and M. Heath, "Improving performance of sparse matrix-vector multiplication," presented at the ACM/IEEE Conf. Supercomputing, Portland, OR, 1999.
54. W. J. Stewart. (1995) MARCA models home page. [Online]. Available: www.csc.ncsu.edu/faculty/WStewart/MARCA_Models/MARCA_Models.html
55. W. Wang and D. P. O'Leary, "Adaptive use of iterative methods in interior point methods for linear programming," Univ. Maryland, College Park, Tech. Rep. UMIACS-95-111, 1995.
56. J. M. Kleinberg, "Authoritative sources in a hyperlinked environment," J. ACM, vol. 46, no. 5, pp. 604-632, 1999.
57. J. Mellor-Crummey and J. Garvin, "Optimizing sparse matrix vector multiply using unroll-and-jam," presented at the Los Alamos Computer Science Institute 3rd Annu. Symp., Santa Fe, NM, 2002.
58. R. Geus and S. Röllin, "Toward a fast parallel sparse matrix-vector multiplication," in Proc. Int. Conf. Parallel Computing (ParCo), E. H. D'Hollander, J. R. Joubert, F. J. Peters, and H. Sips, Eds., 1999, pp. 308-315.
59. S. Toledo, "Improving memory-system performance of sparse matrix-vector multiplication," presented at the 8th SIAM Conf. Parallel Processing for Scientific Computing, Minneapolis, MN, 1997.
60. A. J. C. Bik and H. A. G. Wijshoff, "Advanced compiler optimizations for sparse computations," J. Parallel Distrib. Comput., vol. 31, no. 1, pp. 14-24, 1995.
61. N. Ahmed, N. Mateev, K. Pingali, and P. Stodghill, "A framework for sparse matrix code synthesis from high-level specifications." presented at the Conf. Supercomputing 2000, Dallas, TX.
62. W. Pugh and T. Shpeisman, "Generation of efficient code for sparse matrix computations," presented at the 11th Workshop Languages and Compilers for Parallel Computing, Chapel Hill, NC, 1998.
63. C. C. Douglas, J. Hu, M. Kowarschik, U. Rude, and C. Weiss, "Cache optimization for structured and unstructured grid multigrid," Electronic Trans. Numerical Analysis, vol. 10, pp. 21-40, 2000.
64. M. F. Adams, "Multigrid equation solvers for large scale nonlinear finite element simulations," Ph.D. dissertation, Univ. California, Berkeley, 1998.
65. G. Jin and J. Mellor-Crummey, "Experiences tuning SMG98 - A semicoarsening multigrid benchmark based on the hypre library," in Proc. Int. Conf. Supercomputing, 2002, pp. 305-314.
66. S. Sellappa and S. Chatterjee, "Cache-efficient multigrid algorithms," in Lecture Notes in Computer Science, Computational Science. Heidelberg, Germany: Springer-Verlag, 2001, vol. 2073, pp. 107-116.
67. F. G. Gustavson, "Two fast algorithms for sparse matrices: Multiplication and permuted transposition," ACM Trans. Math. Softw., vol. 4, no. 3, pp. 250-269, 1978.
68. P. Briggs, "Sparse matrix multiplication," SIGPLAN Notices, vol. 31, no. 11, pp. 33-37, Nov. 1996.
69. E. Cohen, "Structure prediction and computation of sparse matrix products," J. Combinatorial Optimization, vol. 2, no. 4, pp. 307-332, 1999.
70. P. D. Sulatycke and K. Ghosh, "Caching-efficient multithreaded fast multiplication of sparse matrices," presented at the 1st Merged International Parallel Processing Symp. and Symp. Parallel and Distributed Processing, Orlando, FL, 1998.
71. M. Challacombe, "A general parallel sparse-blocked matrix multiply for linear scaling scf theory," Comput. Phys. Commun., vol. 128, p. 93, 2000.
72. A. Chapman and Y. Saad, "Deflated and augmented Krylov subspace techniques," Numer. Linear Algebra Appl., vol. 4, no. 1, pp. 43-66, 1997.
73. Y. Saad and M. H. Schultz, "GMRes: A generalized minimal residual algorithm for solving nonsymmetric linear systems," SIAM J. Sci. Stat. Comput., vol. 7, pp. 856-869, 1986.
74. V. Eijkhout and E. Fuentes, "A proposed standard for numerical metadata," Innovative Comput. Lab., Univ. Tennessee, Tech. Rep. ICL-UT-03-02, 2003.
75. C. Lanczos, "An iteration method for the solution of the eigenvalue problem of linear differential and integral operators," J. Res. Nat. Bureau Stand., vol. 45, pp. 255-282, 1950.
76. C. Paige, "Computational variants of the Lanczos method for the eigenproblem," J. Inst. Math. Appl., vol. 10, pp. 373-381, 1972.
77. H. van der Vorst, "Bi-CGSTAB: A fast and smoothly converging variant of Bi-CG for the solution of nonsymmetric linear systems," SIAM J. Sci. Stat. Comput., vol. 13, pp. 631-644, 1992.
78. MatrixMarket [Online]. Available: http://math.nist.gov/Matrix-Market
79. K. Schloegel, G. Karypis, and V. Kumar, "Parallel multilevel algorithms for multi-constraint graph partitioning," in Proc. EuroPar 2000, pp. 296-310.
80. ParMETIS: Parallel graph partitioning [Online]. Available: http://www-users.cs.umn.edu/~karypis/metis/parmetis/index.html
81. V. Eijkhout, "Automatic determination of matrix blocks," Dept. Comput. Sci., Univ. Tennessee, Tech. Rep. ut-cs-01-458, 2001.