Pregelix: Big(ger) graph analytics on a dataflow engine

Proceedings of the VLDB Endowment

Volumn 8, Issue 2 2, 2014, Pages 161-172

  Bu, Yingyi ^a   Borkar, Vinayak ^a,b   Jia, Jianfeng ^a   Carey, Michael J ^a   Condie, Tyson ^c  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b Harris and Associates   (United States)

^c UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DATA FLOW ANALYSIS; OPEN SYSTEMS;

DATA-FLOW DESIGN; DATAFLOW; GRAPH PROCESSING; GRAPH-ANALYTIC; LARGE GRAPHS; MEMORY PRESSURE; MESSAGE-PASSING; OPEN-SOURCE; PROCESSING SYSTEMS; SCALINGS;

MESSAGE PASSING;

EID: 84938089087     PISSN: None     EISSN: 21508097     Source Type: Journal    
DOI: 10.14778/2735471.2735477     Document Type: Conference Paper

References (46)

1. AsterixDB. http://asterixdb.ics.uci.edu.
2. BTC. http://km.aifb.kit.edu/projects/btc-2009/.
3. Genomix. https://github.com/uci-cbcl/genomix.
4. Giraph. http://giraph.apache.org/.
5. Hadoop/HDFS. http://hadoop.apache.org/.
6. Hama. http://hama.apache.org/.
7. Pivotal. http://www.gopivotal.com/products/pivotal-greenplum-database.
8. Teradata. http://www.teradata.com.
9. Vertica. http://www.vertica.com.
10. F. Bancilhon and R. Ramakrishnan. An amateur's introduction to recursive query processing strategies. In SIGMOD, pages 16-52, 1986.
11. D. Battré, S. Ewen, F. Hueske, O. Kao, V. Markl, and D. Warneke. Nephele/pacts: a programming model and execution framework for web-scale analytical processing. In SoCC, pages 119-130, 2010.
12. V. R. Borkar, M. J. Carey, R. Grover, N. Onose, and R. Vernica. Hyracks: A flexible and extensible foundation for data-intensive computing. In ICDE, pages 1151-1162, 2011.
13. Y. Bu, V. R. Borkar, M. J. Carey, J. Rosen, N. Polyzotis, T. Condie, M. Weimer, and R. Ramakrishnan. Scaling datalog for machine learning on big data. CoRR, abs/1203.0160, 2012.
14. Y. Bu, V. R. Borkar, G. H. Xu, and M. J. Carey. A bloat-aware design for big data applications. In ISMM, pages 119-130, 2013.
15. Y. Bu, B. Howe, M. Balazinska, and M. D. Ernst. HaLoop: Efficient iterative data processing on large clusters. PVLDB, 3(1):285-296, 2010.
16. S. Chaudhuri. An overview of query optimization in relational systems. In PODS, pages 34-43, 1998.
17. R. Chen, M. Yang, X. Weng, B. Choi, B. He, and X. Li. Improving large graph processing on partitioned graphs in the cloud. In SoCC, page 3, 2012.
18. J. Cheng, Y. Ke, S. Chu, and C. Cheng. Efficient processing of distance queries in large graphs: a vertex cover approach. In SIGMOD Conference, pages 457-468, 2012.
19. D. Comer. The ubiquitous b-tree. ACM Comput. Surv., 11(2):121-137, 1979.
20. J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. In OSDI, pages 137-150, 2004.
21. D. J. DeWitt, S. Ghandeharizadeh, D. A. Schneider, A. Bricker, H.-I. Hsiao, and R. Rasmussen. The Gamma database machine project. IEEE Trans. Knowl. Data Eng., 2(1):44-62, 1990.
22. D. J. DeWitt and J. Gray. Parallel database systems: The future of high performance database systems. Commun. ACM, 35(6):85-98, 1992.
23. S. Even. Graph Algorithms. Cambridge University Press, New York, NY, USA, 2nd edition, 2011.
24. S. Ewen, K. Tzoumas, M. Kaufmann, and V. Markl. Spinning fast iterative data flows. PVLDB, 5(11):1268-1279, 2012.
25. S. Fushimi, M. Kitsuregawa, and H. Tanaka. An overview of the system software of a parallel relational database machine grace. In VLDB, pages 209-219, 1986.
26. A. V. Gerbessiotis and L. G. Valiant. Direct bulk-synchronous parallel algorithms. J. Parallel Distrib. Comput., 22(2):251-267, 1994.
27. G. Graefe. Query evaluation techniques for large databases. ACM Comput. Surv., 25(2):73-170, 1993.
28. H. Herodotou and S. Babu. Profiling, what-if analysis, and cost-based optimization of mapreduce programs. PVLDB, 4(11):1111-1122, 2011.
29. I. Hoque and I. Gupta. LFGraph: Simple and fast distributed graph analytics. In TRIOS, 2013.
30. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: distributed data-parallel programs from sequential building blocks. In Euro Sys, pages 59-72, 2007.
31. Y. Low, J. Gonzalez, A. Kyrola, D. Bickson, C. Guestrin, and J. M. Hellerstein. Distributed GraphLab: A framework for machine learning in the cloud. PVLDB, 5(8):716-727, 2012.
32. G. Malewicz, M. H. Austern, A. J. C. Bik, J. C. Dehnert, I. Horn, N. Leiser, and G. Czajkowski. Pregel: a system for large-scale graph processing. In SIGMOD Conference, pages 135-146, 2010.
33. S. R. Mihaylov, Z. G. Ives, and S. Guha. REX: Recursive, delta-based data-centric computation. PVLDB, 5(11):1280-1291, 2012.
34. P. E. O'Neil, E. Cheng, D. Gawlick, and E. J. O'Neil. The log-structured merge-tree (LSM-tree). Acta Inf., 33(4):351-385, 1996.
35. L. Page, S. Brin, R. Motwani, and T. Winograd. The pagerank citation ranking: Bringing order to the web. Technical Report 1999-66, Stanford InfoLab, November 1999.
36. S. Salihoglu and J. Widom. GPS: a graph processing system. In SSDBM, page 22, 2013.
37. B. Shao, H. Wang, and Y. Li. Trinity: a distributed graph engine on a memory cloud. In SIGMOD Conference, pages 505-516, 2013.
38. Y. Tian, A. Balmin, S. A. Corsten, S. Tatikonda, and J. McPherson. From "think like a vertex" to "think like a graph". PVLDB, 7(3):193-204, 2013.
39. V. K. Vavilapalli, A. C. Murthy, C. Douglas, S. Agarwal, M. Konar, R. Evans, T. Graves, J. Lowe, H. Shah, S. Seth, B. Saha, C. Curino, O. O'Malley, S. Radia, B. Reed, and E. Baldeschwieler. Apache Hadoop YARN: yet another resource negotiator. In SoCC, page 5, 2013.
40. R. S. Xin, J. E. Gonzalez, M. J. Franklin, and I. Stoica. GraphX: a resilient distributed graph system on spark. In GRADES, page 2, 2013.
41. Yahoo! Webscope Program. http://webscope.sandbox.yahoo.com/.
42. D. Yan, J. Cheng, K. Xing, W. Ng, and Y. Bu. Practical pregel algorithms for massive graphs. In Technique Report, CUHK, 2013.
43. S. Yang, X. Yan, B. Zong, and A. Khan. Towards effective partition management for large graphs. In SIGMOD Conference, pages 517-528, 2012.
44. M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J. Franklin, S. Shenker, and I. Stoica. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computingt. In NSDI, 2012.
45. D. R. Zerbino and E. Birney. Velvet: Algorithms for de novo short read assembly using de bruijn graphs. Genome Research In Genome Research, 18(5):821-829, 2008.
46. Y. Zhang, Q. Gao, L. Gao, and C. Wang. PrIter: a distributed framework for prioritized iterative computations. In SoCC, page 13, 2011.