Recursive array layouts and fast matrix multiplication

IEEE Transactions on Parallel and Distributed Systems

Volumn 13, Issue 11, 2002, Pages 1105-1123

  Chatterjee, Siddhartha ^a   Lebeck, Alvin R ^b   Patnala, Praveen K ^c   Thottethodi, Mithuna ^b  

^a IBM T J WATSON RESEARCH CENTER   (United States)

^b Duke University   (United States)

^c Nexsi Corporation   (United States)

Author keywords

Data layout; Matrix multiplication

Indexed keywords

ALGORITHMS; COMPUTATIONAL COMPLEXITY; COMPUTATIONAL METHODS; DATA STRUCTURES; LINEAR ALGEBRA; PROGRAM COMPILERS; STORAGE ALLOCATION (COMPUTER);

FAST MATRIX MULTIPLICATION; RECURSIVE ARRAY LAYOUTS;

PARALLEL PROCESSING SYSTEMS;

EID: 0036870763     PISSN: 10459219     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPDS.2002.1058095     Document Type: Article

References (58)

1. M. Thottethodi, S. Chatterjee, and A.R. Lebeck, "Tuning Strassen's Matrix Multiplication for Memory Efficiency," Proc. SC98 (CD-ROM), Nov. 1998, available from http://www.supercomp.org/sc98.
2. S. Chatterjee, V.V. Jain, A.R. Lebeck, Mundhra, S. and M. Thottethodi, "Nonlinear Array Layouts for Hierarchical Memory Systems," Proc. 1999 ACM Int'l Conf. Supercomputing, pp. 444-453, June 1999.
3. S. Chatterjee, A.R. Lebeck, P.K. Patnala, and M. Thottethodi, "Recursive Array Layouts and Fast Parallel Matrix Multiplication," Proc. 11th Ann. ACM Symp. Parallel Algorithms and Architectures, pp. 222-231 June 1999.
4. J.J. Dongara, J. Du Croz, I.S. Duff, and S. Hammarling, "A Set of Level 3 Basic Linear Algebra Subprograms," ACM Trans. Math. Software, vol. 16, no. 1, pp. 1-17, Jan. 1990.
5. D. Hilbert, "Eber Stetige Abbildung Einer Linie Auf Ein Flächenstück," Mathematische Annalen, vol. 38, pp. 459-460, 1891.
6. G. Peano, "Sur une Courbe Qui Remplit Toute une Aire Plaine," Mathematische Annalen, vol. 36, pp. 157-160, 1890.
7. M.S. Warren and J.K. Salmon, "A Parallel Hashed Oct-Tree N-Body Algorithm," Proc. Supercomputing '93, pp. 12-21, Nov. 1993.
8. I. Banicescu and S.F. Hummel, "Balancing Processor Loads and Exploiting Data Locality in N-Body Simulations," Proc. Supercomputing '95 (CD-ROM), Dec. 1995, available from http://www.supercomp.org/sc95proceedings/594_BHUM/SC95.HTM.
9. S.F. Hummel, I. Banicescu, C.-T. Wang, and J. Wein, "Load Balancing and Data Locality via Fractiling: An Experimental Study," Language, Compilers, and Run-Time Systems for Scalable Computers, 1995.
10. Y.C. Hu, S.L. Johnsson, and S.-H. Teng, "High Performance Fortran for Highly Irregular Problems," Proc. the Sixth ACM SIGPLAN Symp. Principles and Practice of Parallel Programming, pp. 13-24, June 1997.
11. J.R. Pilkington and S.B. Baden, "Dynamic Partitioning of Non-Uniform Structured Workloads with Spacefilling Curves," IEEE Trans. Parallel and Distributed Systems, vol. 7, no. 3, pp. 288-300, Mar. 1996.
12. J.P. Singh, T. Joe, J.L. Hennessy, and A. Gupta, "An Empirical Comparison of the Kendall Square Research KSR-1 and the Stanford DASH Multiprocessors," Proc. Supercomputing '93, pp. 214-225, Nov. 1993.
13. T. Bially, "Space-Filling Curves: Their Generation and Their Application to Bandwidth Reduction," IEEE Trans. Information Theory, vol. 15, no. 6, pp. 658-664, Nov. 1969.
14. M.F. Goodchild and A.W. Grandfield, "Optimizing Raster Storage: An Examination of Four Alternatives," Proc. Auto-Carto 6, vol. 1, pp. 400-407, Oct. 1983.
15. R. Laurini, "Graphical Data Bases Built on Peano Space-Filling Curves," Proc. EUROGRAPHICS '85 Conf., C.E. Vandoni, ed., pp. 327-338, 1985.
16. H.V. Jagadish, "Linear Clustering of Objects with Multiple Attributes," Proc. 1990 ACM SIGMOD Int'l Conf. Management of Data, H. Garcia-Molina and H.V. Jagadish, eds., pp. 332-342, May 1990.
17. J.D. Frens and D.S. Wise, "Auto-Blocking Matrix-Multiplication or Tracking BLAS3 Performance with Source Code," Proc. Sixth ACM SIGPLAN Symp. Principles and Practice of Parallel Programming, pp. 206-216, June 1997.
18. D.S. Wise, "Ahnentafel Indexing into Morton-Ordered Arrays, or Matrix Locality for Free," Euro-Par 2000 Parallel Processing, A. Bode, T. Ludwig, W. Karl, and R. Wismüller, eds., 2000.
19. D.S. Wise, J.D. Frens, Y. Gu, and G.A. Alexander, "Language Support for Morton-Order Matrices," Proc. Eighth ACM SIGPLAN Symp. Principles and Practices of Parallel Programming, pp. 24-33, June 2001.
20. B. Moon, H.V. Jagadish, C. Faloutsos, and J.H. Saltz, "Analysis of the Clustering Properting of Hilbert Space-Filling Curve," Technical Report CS-TR-3611, Computer Science Dept., Univ. of Maryland, College Park, 1996.
21. M.S. Lam, E.E. Rothberg, and M.E. Wolf, "The Cache Performance and Optimizations of Blocked Algorithms," Proc. Fourth Int'l Conf. Architectural Support for Programming Languages and Operating Systems, pp. 63-74, Apr. 1991.
22. Basic Linear Algebra Subroutine Technical (BLAST) Forum, "Basic Linear Algebra Subroutine Technical (BLAST) Forum Standard," http://www.netlib.org/blas/blast-forum/, Aug. 2001.
23. C.E. Leiserson, "Personal Communication," Aug. 1998.
24. V. Strassen, "Gaussian Elimination is Not Optimal," Numerical Mathematics, vol. 13, pp. 354-356, 1969.
25. R.D. Blumofe, C.F. Joerg, B.C. Kuszmaul, C.E. Leiserson, K.H. Randall, and Y. Zhou, "Cilk: An Efficient Multithreaded Runtime System," Proc. Fifth ACM SIGPLAN Symp. Principles and Practice of Parallel Programming, pp. 207-216, July 1995, also see http://theory.lcs.mit.edu/~cilk.
26. P.C. Fischer and R.L. Probert, "Efficient Procedures for Using Matrix Algorithms," Automata, Languages, and Programming, 1974.
27. M.D. Hill and A.J. Smith, "Evaluating Associativity in CPU Caches," IEEE Trans. Computers, vol. 38, no. 12, pp. 1612-1630, Dec. 1989.
28. N.J. Higham, Accuracy and Stability of Numerical Algorithms. Philadelphia, Penn: SIAM, 1996.
29. M. Cierniak, and W. Li, "Unifying Data and Control Transformations for Distributed Shared-Memory Machines," Proc. ACM SIGPLAN '95 Conf. Programming Language Design and Implementation, pp. 205-517, June 1995.
30. D. Culler and J.P. Singh, Parallel Computer Architecture: A Hardware/Software Approach. Morgan Kaufmann, 1998.
31. C.H. Koelbel, D.B. Loveman, R.S. Schreiber, G.L. Steele Jr., and M.E. Zosel, The High Performance Fortran Handbook. Scientific and Eng. Computation, Cambridge, Mass.: MIT Press, 1994.
32. H. Sagan, Space-Filling Curves. Springer-Verlag, 1994.
33. M. Mano, Digital Design. Prentice-Hall, 1984.
34. S. Huss-Lederman, E.M. Jacobson, J.R. Johnson, A. Tsao, and T. Turnbull, "Implementation of Strassen's Algorithm for Matrix Multiplication," Proc. Supercomputing '96, 1996.
35. C. Douglas, M. Heroux, G. Slishman, and R.M. Smith, "GEMMW: A Portable Level 3 BLAS Winograd Variant of Strassen's Matrix-Matrix Multiply Algorithm," J. Computational Physics, vol. 110, pp. 1-10, 1994.
36. J. Bilmes, K. Asanovic, C.-W. Chin, and J. Demmel, "Optimizing Matrix Multiply Using PHiPAC: A Portable, High-Performance, ANSI C Coding Methodology," Proc. Int'l Conf. Supercomputing, pp. 340-347, July 1997.
37. R. C. Whaley, "Automatically Tuned Linear Algebra Software," Proc. Conf. Supercomputing, 1998.
38. M. Frigo and S.G. Johnson, "FFTW: An Adaptive Software Architecure for the FFT," Proc. Int'l Conf. Acoustics, Speech, and Signal Processing, vol. 3, p. 1381, 1998.
39. S. Toledo, "Locality of Reference in LU Decomposition with Partial Pivoting," SIAM J. Matrix Analysis and Applications, vol. 18, no. 4, pp. 1065-1081, Oct. 1997.
40. F.G. Gustavson, "Recursion Leads to Automatic Variable Blocking for Dense Linear-Algebra Algorithms," IBM J. Research and Development, vol. 41, no. 6, pp. 737-755, Nov. 1997.
41. E. Elmroth and F. Gustavson, "Applying Recursion to Serial and Parallel QR Factorization Leads to Better Performance," IBM J. Research and Development, vol. 44, no. 4, pp. 605-624, July 2000.
42. J. Wasniewski, B.S. Anderson, and F. Gustavson, "Recursive Formulation of Colesky Algorithm in Fortran 90," Proc. Fourth Int'l Workshop, Applied Parallel Computing, Large Scale Scientific and Industrial Problems, PARA '98, 'B. Kågström, J. Dongarra, E. Elmroth, and J. Wasniewski, eds., June 1998.
43. B.S. Andersen, F. Gustavson, J. Wasniewski, and P.Y. Yalamov, "Recursive Formulation of Some Dense Linear Algebra Algorithms," Proc. Ninth SIAM Conf. Parallel Processing for Scientific Computing (PPSC '99), B. Hendrickson, K.A. Yelick, C.H. Bischof, I.S. Duff, A.S. Edelman, G.A. Geist, M.T. Heath, M.A. Heroux, C. Koelbel, R.S. Schreiber, R.F. Sincovec, and M.F. Wheeler, eds., Mar. 1999.
44. IBM, Engineering and Scientific Subroutine Library Version 2 Release 2, Guide and Reference. vols. 1-3, 1994.
45. F.G. Gustavson, "New Generalized Data Structures for Matrices Lead to a Variety of High-Performance Algorithms," Simulation and Visualization on the Grid, B. Engquist, L. Johnson, M. Hammill, and F. Short, eds., 2000.
46. L. Stals and U. Rüde, "Techniques for Improving the Data Locality of Iterative Methods," Technical Report MRR97-038, Institut für Mathematik, Universität Augsburg, Germany, Oct. 1997.
47. G. Gibson, J.S. Vitter, and J. Wilkes, "Report of the Working Group on Storage I/O for Large-Scale Computing," ACM Computing Surveys, Dec. 1996.
48. C.E. Leiserson, S. Rao, and S. Toledo, "Efficient Out-of-Core Algorithms for Linear Relaxation Using Blocking Covers," J. Computer and System Sciences, vol. 54, no. 2, pp. 332-344, 1997.
49. S. Sen and S. Chatterjee, "Towards a Theory of Cache-Efficient Algorithms," Proc. 11th Annual ACM-SIAM Symp. Discrete Algorithms, pp. 829-833, Jan. 2000.
50. M.J. Wolfe, "More Iteration Space Tiling," Proc. Supercomputing '89, pp. 655-664, Nov. 1989.
51. M.E. Wolf and M.S. Lam, "A Data Locality Optimizing Algorithm," Proc. ACM SIGPLAN'91 Conf. Programming Language Design and Implementation, pp. 30-44, June 1991.
52. S. Carr, K.S. McKinley, and C.-W. Tseng, "Compiler Optimizations for Improving Data Locality," Proc. Sixth Int'l Conf. Architectural Support for Programming Languages and Operating Systems, pp. 252-262, Oct. 1994.
53. L. Carter, J. Ferrante, and S. Flynn Hummel, "Hierarchical Tiling for Improved Superscalar Performance," Proc. Int'l Parallel Processing Symp., Apr. 1995.
54. M.E. Mace, Memory Storage Patterns in Parallel Processing. Norwell, Mass.: Kluwer Academic Press, 1987.
55. K. Knobe, J.D. Lukas, and G.L. Steele, Jr., "Data Optimization: Allocation of Arrays to Reduce Communication on SIMD Machines," J. Parallel and Distributed Computing, vol. 8, no. 2, pp. 102-118, Feb. 1990.
56. M. Gupta, "Automatic Data Partitioning on Distributed Memory Multicomputers," PhD Thesis, University of Illinois at Urbana-Champaign, Urbana, Sept. 1992.
57. S. Chatterjee, J.R. Gilbert, R. Schreiber, and S.-H. Teng, "Optimal Evaluation of Array Expressions on Massively Parallel Machines," ACM Trans. Programming Languages and Systems, vol. 17, no. 1, pp. 123-156, Jan. 1995.
58. K. Kennedy and U. Kremer, "Automatic Data Layout for Distributed Memory Machines," ACM Trans. Programming Languages and Systems, 1998.