Communication lower bounds for distributed-memory matrix multiplication

Journal of Parallel and Distributed Computing

Volumn 64, Issue 9, 2004, Pages 1017-1026

  Irony, Dror ^a   Toledo, Sivan ^a   Tiskin, Alexander ^b  

^a TEL AVIV UNIVERSITY   (Israel)

^b UNIVERSITY OF WARWICK   (United Kingdom)

Author keywords

Communication; Distributed memory; Lower bounds; Matrix multiplication

Indexed keywords

ALGORITHMS; CACHE MEMORY; COMPUTER WORKSTATIONS; CONSTRAINT THEORY; DISTRIBUTED COMPUTER SYSTEMS; THEOREM PROVING;

DISTRIBUTED MEMORY; LOWER BOUNDS; MATRIX MULTIPLICATION;

PARALLEL PROCESSING SYSTEMS;

EID: 10844258198     PISSN: 07437315     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jpdc.2004.03.021     Document Type: Article

References (39)

1. R.C. Agarwal, S.M. Balle, F.G. Gustavson, M. Joshi, P. Palkar, A three-dimensional approach to parallel matrix multiplication, IBM J. Res. Devel. 39 (5) (1995) 575-582, available online at http://www.research.ibm.com/journal/rd39-5.html.
2. R.C. Agarwal, F.G. Gustavson, M. Zubair, Exploiting functional parallelism of POWER2 to design high-performance numerical algorithms, IBM J. Res. Dev. 38 (5) (1994) 563-576.
3. R.C. Agarwal, F.G. Gustavson, M. Zubair, A high-performance matrix-multiplication algorithm on a distributed-memory parallel computer using overlapped communication, IBM J. Res. Devel. 38 (6) (1994) 673-681, available online at http://www.research.ibm.com/ journal/rd38-6.html.
4. R.C. Agarwal, F.G. Gustavson, M. Zubair, Improving performance of linear algebra algorithms for dense matrices using algorithmic prefetch, IBM J. Res. Devel. 38 (3) (1994) 265-275.
5. A. Aggarwal, A. Chandra, M. Snir, Communication complexity of PRAMs, Theoret. Comput. Sci. 71 (1990) 3-28.
6. J. Bernsten, Communication efficient matrix multiplication on hypercubes, Parallel Comput. 12 (1989) 335-342.
7. J. Bilmes, K. Asanovic, C.W. Chin, J. Demmel, Optimizing matrix multiply using PHIPAC: a portable, high-performance, ANSI C coding methodology, in: Proceedings of the International Conference on Supercomputing, Vienna, Austria, 1997.
8. Yu.D. Burago, V.A. Zalgaller, Geometric Inequalities, Grundlehren der mathematischen Wissenschaften, vol. 285, Springer, Berlin, 1988.
9. L.E. Cannon, A cellular computer to implement the Kalman filter algorithm, Ph.D. Thesis, Montana State University, 1969.
10. T.H. Cormen, C.E. Leiserson, R.L. Rivest, C. Stein, Introduction to Algorithms, second ed., MIT Press, McGraw-Hill, Cambridge, MA, New York, 2001.
11. M.J. Dayde, I.S. Duff, A blocked implementation of level 3 BLAS for RISC processors, Technical Report RT/APO/96/1, ENSEEIHT-IRIT, France, 1996.
12. E. Dekel, D. Nassimi, S. Sahni, Parallel matrix and graph algorithms, SIAM J. Comput. 10 (1981) 657-675.
13. G.C. Fox, S.W. Otto, A.J.G. Hey, Matrix algorithms on a hypercube i: Matrix Multiplication, Parallel Comput. 4 (1987) 17-31.
14. A. Gupta, V. Kumar, The scalability of matrix multiplication algorithms on parallel computers, Technical Report TR 91-54, Department of Computer Science, University of Minnesota, 1991, available online from ftp://ftp.cs.umn.edu/users/kumar/matrix.ps, A short version appeared in Proceedings of 1993 International Conference on Parallel Processing, 1993, pp. III-115-III-119.
15. A. Gupta, V. Kumar, The scalability of matrix multiplication algorithms on parallel computers, Technical Report TR 91-54, Department of Computer Science, University of Minnesota, 1991, available online from ftp://ftp.cs.umn.edu/users/kumar/matrix.ps, A short version appeared in Proceedings of 1993 International Conference on Parallel Processing, 1993, pp. III-115-III-119.
16. H. Hadwiger, Vorlesungen über Inhalt, Oberfläche und Isoperimetrie, Grundlehren der mathematischen Wissenschaften, vol. 93, Springer, Berlin, 1957.
17. G. Henry, P. Fay, B. Cole, T.G. Mattson, The performance of the Intel TFLOPS supercomputer, Intel Tech. J. 98 (1) (1998) available online at http://developer.intel.com/technology/it j/.
18. J.-W. Hong, H.T. Kung, I/O complexity: the red-blue pebble game, in: Proceedings of the 13th Annual ACM Symposium on Theory of Computing, 1981, pp. 326-333.
19. D. Irony, S. Toledo, Trading replication for communication in parallel distributed-memory dense solvers, Parallel Process. Lett. 12 (2002) 79-94.
20. S.L. Johnsson, Minimizing the communication time for matrix multiplication on multiprocessors, Parallel Comput. 19 (1993) 1235-1257.
21. S.L. Johnsson, L.F. Ortiz, Local basic linear algebra subroutines LBLAS for the Connection Machine system CM-200, Internat. J. Supercomput. Appl. 7 (1993) 322-350.
22. B. Kågström, P. Ling, C. Van Loan, High performance GEMM-based level-3 BLAS: sample routines for double precision real data, in: M. Durand, F. El Dabaghi (Eds.), High Performance Computing H, North-Holland, Amsterdam, 1991, pp. 269-281.
23. B. Kågström, P. Ling, C. Van Loan, Portable high performance GEMM-based level 3 BLAS, in: Proceedings of the Sixth SIAM Conference on Parallel Processing for Scientific Computing, Philadelphia, 1993, pp. 339-346.
24. B. Kågström, C. Van Loan, GEMM-based level-3 BLAS, Technical Report CTC-91-TR47, Department of Computer Science, Cornell University, 1989.
25. C. Kamath, R. Ho, D.P. Manley, DXML: a high-performance scientific subroutine library, Digital Tech. J. 6 (3) (1994) 44-56.
26. D. Kramer, S.L. Johnsson, Yu Hu, Local basic linear algebra subroutines (LBLAS) for the CM-5/5E, Internat. J. Supercomput. Appl. 10 (1996) 300-335.
27. L.H. Loomis, H. Whitney, An inequality related to the isoperimetric inequality, Bull. AMS 55 (1949) 961-962.
28. W.F. McColl, A. Tiskin, Memory-efficient matrix multiplication in the BSP model, Algorithmica 24 (3/4) (1999) 287-297.
29. A.C. McKeller, E.G. Coffman Jr., Organizing matrices and matrix operations for paged memory systems, Commun. ACM 12 (3) (1969) 153-165.
30. M.S. Paterson, Private communication, 1993
31. J. Rutledge, H. Rubinstein, Matrix algebra programs for the UNIVAC, Presented at the Wayne Conference on Automatic Computing Machinery and Applications, Copies available from Sivan Toledo, March 1951.
32. J. Rutledge, H. Rubinstein, High order matrix computation on the UNIVAC, Presented at the meeting of the Association for Computing Machinery, Copies available from Sivan Toledo, May 1952.
33. V. Strassen, Gaussian elimination is not optimal, Numer. Math. 13 (1969) 354-355.
34. A. Tiskin, Bulk-synchronous parallel multiplication of Boolean matrices, in: K.G. Larsen, S. Skyum, W. Winskel (Eds.), Proceedings of ICALP, Lecture Notes in Computer Science, vol. 1443, Springer, Berlin, 1998, pp. 494-506.
35. A. Tiskin, The bulk-synchronous parallel random access machine, Theoret. Comput. Sci. 196 (1-2) (April 1998) 109-130.
36. A. Tiskin, Erratum: Bulk-synchronous parallel multiplication of Boolean matrices, in: J. Wiedermann, P. van Emde Boas, M. Nielsen (Eds.), Proceedings of ICALP, Lecture Notes in Computer Science, vol. 1644, Springer, Berlin, 1999, p. 717.
37. S. Toledo, A survey of out-of-core algorithms in numerical linear algebra, in: James M. Abello, Jeffrey Scott Vitter (Eds.), External Memory Algorithms, DIMACS Series in Discrete Mathematics and Theoretical Computer Science, American Mathematical Society, Providence, RI, 1999, pp. 161-179.
38. R. van de Geijn, J. Watts, SUMMA: scalable universal matrix multiplication algorithm, Concurrency: Pract. Exp. 9 (1997) 255-274.
39. R.C. Whaley, J.J. Dongarr, Automatically tuned linear algebra software, Technical Report, Computer Science Department, University of Tennessee, 1998, available online at www.netlib.org/atlas.