GraphX: Graph processing in a distributed dataflow framework

Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014

Volumn , Issue , 2014, Pages 599-613

  Gonzalez, Joseph E ^a   Xin, Reynold S ^a,b   Dave, Ankur ^a   Crankshaw, Daniel ^a   Franklin, Michael J ^a   Stoica, Ion ^a,b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b Databricks Inc   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ABSTRACTING; FAULT TOLERANCE; FAULT TOLERANT COMPUTER SYSTEMS; ITERATIVE METHODS; SYSTEMS ANALYSIS;

DISTRIBUTED DATA FLOW; EMBEDDED GRAPHS; GRAPH ALGORITHMS; GRAPH PROCESSING; MATERIALIZED VIEW MAINTENANCE; PERFORMANCE GAIN; REAL WORKLOADS; TAILORED PROGRAMMING;

DATA FLOW ANALYSIS;

EID: 85072954017     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (39)

1. ABADI, D. J., MARCUS, A., MADDEN, S. R., AND HOLLENBACH, K. SW-Store: A vertically partitioned DBMS for semantic web data management. PVLDB 18, 2 (2009), 385-406.
2. AFRATI, F. N., AND ULLMAN, J. D. Optimizing joins in a map-reduce environment. In EDBT (2010), pp. 99-110.
3. BLANAS, S., PATEL, J. M., ERCEGOVAC, V., RAO, J., SHEKITA, E. J., AND TIAN, Y. A comparison of join algorithms for log processing in MapReduce. In Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data (New York, NY, USA, 2010), SIGMOD'10, ACM, pp. 975-986.
4. BOLDI, P., ROSA, M., SANTINI, M., AND VIGNA, S. Layered label propagation: A multiresolution coordinate-free ordering for compressing social networks. In WWW (2011), pp. 587-596.
5. BOLDI, P., AND VIGNA, S. The WebGraph framework I: Compression techniques. In WWW'04.
6. BROEKSTRA, J., KAMPMAN, A., AND HARMELEN, F. V. Sesame: A generic architecture for storing and querying rdf and rdf schema. In Proceedings of the First International Semantic Web Conference on The Semantic Web (2002), ISWC'02, pp. 54-68.
7. BULUÇ, A., AND GILBERT, J. R. The combinatorial BLAS: design, implementation, and applications. IJHPCA 25, 4 (2011), 496-509.
8. ÇATALYÜREK, U. V., AYKANAT, C., AND UÇAR, B. On two-dimensional sparse matrix partitioning: Models, methods, and a recipe. SIAM J. Sci. Comput. 32, 2 (2010), 656-683.
9. CHENG, R., HONG, J., KYROLA, A., MIAO, Y., WENG, X., WU, M., YANG, F., ZHOU, L., ZHAO, F., AND CHEN, E. Kineograph: taking the pulse of a fast-changing and connected world. In EuroSys (2012), pp. 85-98.
10. DEAN, J., AND GHEMAWAT, S. Mapreduce: simplified data processing on large clusters. In OSDI (2004).
11. EWEN, S., TZOUMAS, K., KAUFMANN, M., AND MARKL, V. Spinning fast iterative data flows. Proc. VLDB 5, 11 (July 2012), 1268-1279.
12. FEIGE, U., HAJIAGHAYI, M., AND LEE, J. R. Improved approximation algorithms for minimum-weight vertex separators. In Proceedings of the Thirty-seventh Annual ACM Symposium on Theory of Computing (New York, NY, USA, 2005), STOC'05, ACM, pp. 563-572.
13. GONZALEZ, J. E., LOW, Y., GU, H., BICKSON, D., AND GUESTRIN, C. Powergraph: Distributed graph-parallel computation on natural graphs. OSDI'12, USENIX Association, pp. 17-30.
14. HUANG, A., AND WU, W. Mining ecommerce graph data with spark at alibaba taobao. http://databricks.com/blog/2014/08/14/mining-graph-datawith-spark-at-alibaba-taobao.html, 2014.
15. ISARD, M., BUDIU, M., YU, Y., BIRRELL, A., AND FETTERLY, D. Dryad: distributed data-parallel programs from sequential building blocks. In EuroSys (2007), pp. 59-72.
16. KARYPIS, G., AND KUMAR, V. Multilevel k-way partitioning scheme for irregular graphs. J. Parallel Distrib. Comput. 48, 1 (1998), 96-129.
17. KYROLA, A., BLELLOCH, G., AND GUESTRIN, C. GraphChi: Large-scale graph computation on just a PC. In OSDI (2012).
18. LESKOVEC, J., LANG, K. J., DASGUPTA, A.,, AND MAHONEY, M. W. Community structure in large networks: Natural cluster sizes and the absence of large well-defined clusters. Internet Mathematics 6, 1 (2008), 29-123.
19. LOW, Y., ET AL. GraphLab: A new parallel framework for machine learning. In UAI (2010), pp. 340-349.
20. LOW, Y., GONZALEZ, J., KYROLA, A., BICKSON, D., GUESTRIN, C., AND HELLERSTEIN, J. M. Distributed GraphLab: A Framework for Machine Learning and Data Mining in the Cloud. PVLDB (2012).
21. MACKERT, L. F., AND LOHMAN, G. M. R* optimizer validation and performance evaluation for distributed queries. In VLDB'86 (1986), pp. 149-159.
22. MALEWICZ, G., AUSTERN, M. H., BIK, A. J., DEHNERT, J., HORN, I., LEISER, N., AND CZAJKOWSKI, G. Pregel: a system for large-scale graph processing. In SIGMOD (2010), pp. 135-146.
23. MANOLA, F., AND MILLER, E. RDF primer. W3C Recommendation 10 (2004), 1-107.
24. MONDAL, J., AND DESHPANDE, A. Managing large dynamic graphs efficiently. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data (New York, NY, USA, 2012), SIGMOD'12, ACM, pp. 145-156.
25. MURRAY, D. Building new frameworks on Naiad. blog post: http://bigdataatsvc.wordpress.com/2014/04/29/ building-new-frameworks-for-naiad/, April 2014.
26. MURRAY, D. G., MCSHERRY, F., ISAACS, R., ISARD, M., BARHAM, P., AND ABADI, M. Naiad: A timely dataflow system. In SOSP'13.
27. NAJORK, M., FETTERLY, D., HALVERSON, A., KENTHAPADI, K., AND GOLLAPUDI, S. Of hammers and nails: An empirical comparison of three paradigms for processing large graphs. In Proceedings of the Fifth ACM International Conference on Web Search and Data Mining (2012), WSDM'12, ACM, pp. 103-112.
28. NEUMANN, T., AND WEIKUM, G. RDF-3X: A RISC-style engine for RDF. VLDB'08.
29. OLSTON, C., REED, B., SRIVASTAVA, U., KUMAR, R., AND TOMKINS, A. Pig Latin: A not-so-foreign language for data processing. SIGMOD (2008).
30. PAGE, L., BRIN, S., MOTWANI, R., AND WINOGRAD, T. The pagerank citation ranking: Bringing order to the web. Technical Report 1999-66, Stanford InfoLab, 1999.
31. PRUD'HOMMEAUX, E., AND SEABORNE, A. SPARQL query language for RDF. Latest version available as http://www.w3.org/TR/rdf-sparql-query/, January 2008.
32. PUJOL, J. M., ERRAMILLI, V., SIGANOS, G., YANG, X., LAOUTARIS, N., CHHABRA, P., AND RODRIGUEZ, P. The little engine(s) that could: scaling online social networks. In SIGCOMM (2010), pp. 375-386.
33. ROBINSON, I., WEBBER, J., AND EIFREM, E. Graph Databases. O'Reilly Media, Incorporated, 2013.
34. ROY, A., MIHAILOVIC, I., AND ZWAENEPOEL, W. X-stream: Edge-centric graph processing using streaming partitions. SOSP'13, ACM, pp. 472-488.
35. SAAD, Y. Iterative Methods for Sparse Linear Systems, 2nd ed. Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 2003.
36. STANTON, I., AND KLIOT, G. Streaming graph partitioning for large distributed graphs. Tech. Rep. MSR-TR-2011-121, Microsoft Research, November 2011.
37. STUTZ, P., BERNSTEIN, A., AND COHEN, W. Signal/collect: graph algorithms for the (semantic) web. In ISWC (2010).
38. UGANDER, J., AND BACKSTROM, L. Balanced label propagation for partitioning massive graphs. In Proceedings of the Sixth ACM International Conference on Web Search and Data Mining (New York, NY, USA, 2013), WSDM'13, ACM, pp. 507-516.
39. ZAHARIA, M., CHOWDHURY, M., DAS, T., DAVE, A., MA, J., MCCAULEY, M., FRANKLIN, M. J., SHENKER, S., AND STO-ICA, I. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. NSDI'12.