MadLINQ: Large-scale distributed matrix computation for the cloud

EuroSys'12 - Proceedings of the EuroSys 2012 Conference

Volumn , Issue , 2012, Pages 197-210

  Qian, Zhengping ^a   Chen, Xiuwei ^a   Kang, Nanxi ^b   Chen, Mingcheng ^b   Yu, Yuan ^c   Moscibroda, Thomas ^a   Zhang, Zheng ^a  

^a MICROSOFT RESEARCH ASIA   (China)

^b SHANGHAI JIAO TONG UNIVERSITY   (China)

^c MICROSOFT RESEARCH   (United States)

Author keywords

Cluster computing; Dataflow; Distributed systems; Fault tolerance; Matrix computation; Pipelining

Indexed keywords

APPLICATION DEVELOPERS; COMPUTING SYSTEM; DATA PARALLEL; DATA-INTENSIVE APPLICATION; DATAFLOW; DISTRIBUTED EXECUTION; DISTRIBUTED MATRIX; DISTRIBUTED SYSTEMS; DRYADLINQ; EXECUTION ENGINE; FAILURE RECOVERY; FAULT-TOLERANT; GENERAL PURPOSE; KEY TECHNIQUES; LARGE CLUSTERS; MATRIX ALGORITHMS; MATRIX COMPUTATION; PARALLELIZATIONS; PROGRAMMING MODELS; REAL-WORLD APPLICATION; RELATIONAL ALGEBRA; RESEARCH PROBLEMS; SEAMLESS INTEGRATION; SEQUENTIAL PROGRAMS; STATE-OF-THE-ART SYSTEM; SUB-MATRICES;

ALGORITHMS; CLUSTER COMPUTING; COMPUTER PROGRAMMING; FAULT TOLERANCE; FAULT TOLERANT COMPUTER SYSTEMS; PARALLEL ARCHITECTURES; PIPE LININGS;

MATRIX ALGEBRA;

EID: 84860577617     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2168836.2168857     Document Type: Conference Paper

References (36)

1. Intel Math Kernel Library. http://software.intel.com/en-us/articles/ intel-mkl/
2. Hadoop project. http://hadoop.apache.org/
3. HIVE project. http://hadoop.apache.org/hive/
4. Mahout project. http://mahout.apache.org/
5. TLC - The TLA+ Model Checker. http://research.microsoft.com/en-us/um/ people/lamport/tla/tlc.html
6. AGULLO, E., DEMMEL, J., DONGARRA, J., ET AL. Numerical linear algebra on emerging architectures: The PLASMA and MAGMA projects. Journal of Physics: Conference Series 180, 2009.
7. ANDERSON, E., BAI, Z., DONGARRA, J., ET AL. LAPACK: A portable linear algebra library for high-performance computers. In Supercomputing, 2002.
8. BENNETT, J., LANNING, S. The Netflix Prize. In KDD, 2007.
9. BOSILCA, G., BOUTEILLER, A., DANALIS, A., ET AL. Dague: A generic distributed dag engine for high performance computing. Tech. Rep. ICL-UT-10-01, EInnovative Computing Laboratory, University of Tennessee, 2010.
10. BRANDES., U. A faster algorithm for betweenness centrality. Journal of Mathematical Sociology 25, 2001.
11. BREESE, J.S., HECKERMAN, D., KADIE, C., ET AL. Empirical analysis of predictive algorithms for collaborative filtering. In Conf. on Uncertainty in Artificial Intelligence, 1998.
12. BULUÇ, A. Linear algebraic primitives for parallel computing on large graphs. PhD thesis, University of California at Santa Barbara, 2010.
13. CHAIKEN, R., JENKINS, B., LARSON, P.Å., ET AL. Scope: easy and efficient parallel processing of massive data sets. In VLDB, 2008.
14. CHAKRABARTI, D., ZHAN, Y., FALOUTSOS, C. R-MAT: A recursive model for graph mining. SIAM Data Mining 6, 2004.
15. CHAMBERS, C., RANIWALA, A., PERRY, F., ET AL. Flume-java: easy, efficient data-parallel pipelines. In PLDI, 2010.
16. CHANDY, K., LAMPORT, L. Distributed snapshots: Determining global states of distributed systems. ACM Transactions on Computer Systems (TOCS) 3, 1985.
17. CHOI, J., DONGARRA, J., POZO, R., WALKER, D. Scala-pack: A scalable linear algebra library for distributed memory concurrent computers. In Symposium on the Frontiers of Massively Parallel Computation, 1992.
18. CONDIE, T., CONWAY, N., ALVARO, P., ET AL. MapReduce online. In NSDI, 2010.
19. DEAN, J., GHEMAWAT, S. MapReduce: Simplified data processing on large clusters. In OSDI, 2004.
20. FREEMAN, L. A set of measures of centrality based on betweenness. Sociometry 40, 1977.
21. GOLUB, G.H., VAN LOAN, C.F. Matrix computations. Johns Hopkins Univ Pr., 1996.
22. GUNNELS, J., GUSTAVSON, F., HENRY, G., VAN DE GEIJN, R. Flame: Formal linear algebra methods environment. ACM Trans. on Math. Software (TOMS) 27, 2001.
23. ISARD, M., BUDIU, M., YU, Y., ET AL. Dryad: Distributed data-parallel programs from sequential building blocks. In EuroSys, 2007.
24. LAMPORT, L. Specifying systems: The TLA+ language and tools for hardware and software engineers, 2002.
25. LAWSON, C., HANSON, R., KINCAID, D., KROGH, F. Basic linear algebra subprograms for Fortran usage. ACM Trans. on Math. Software (TOMS) 5, 1979.
26. LIU, C., YANG, H., FAN, J., ET AL. Distributed nonnegative matrix factorization for web-scale dyadic data analysis on mapreduce. In WWW, 2010.
27. MALEWICZ, G., AUSTERN, M.H., BIK, A.J.C., ET AL. Pregel: a system for large-scale graph processing. In SIGMOD, 2010.
28. LOW, Y., GONZALEZ, J., KYROLA, A., ET AL. Graphlab: A new framework for parallel machine learning. In Conf. on Uncertainty in Artificial Intelligence, 2010.
29. MURRAY, D.G., SCHWARZKOPF, M., SMOWTON, C., ET AL. CIEL: A Universal Execution Engine for Distributed Data-Flow Computing. In NSDI, 2011.
30. OLSTON, C., REED, B., SRIVASTAVA, U., ET AL. Pig Latin: A not-so-foreign language for data processing. In SIGMOD, 2008.
31. SCHOENIG, S., DUCASSé, M. A backward slicing algorithm for prolog. Static Analysis, 1996.
32. SEO, S., YOON, E., KIM, J., ET AL. HAMA: An Efficient Matrix Computation with the MapReduce Framework. In CloudCom, 2010.
33. KANG, U., TSOURAKAKIS, C.E., FALOUTSOS, C. PEGASUS: mining peta-scale graphs. Knowledge and Information Systems 27, 2011.
34. VAN DONGEN, S. Graph clustering via a discrete uncoupling process. SIAM J. Matrix Anal. Appl 30, 2008.
35. WANG, Q., XU, J., LI, H., CRASWELL, N. Regularized latent semantic indexing. In SIGIR, 2011.
36. YU, Y., ISARD, M., FETTERLY, D., ET AL. DryadLINQ: A system for general-purpose distributed data-parallel computing using a high-level language. In OSDI, 2008.