Improving large graph processing on partitioned graphs in the cloud

Proceedings of the 3rd ACM Symposium on Cloud Computing, SoCC 2012

Volumn , Issue , 2012, Pages

  Chen, Rishan ^a   Weng, Xuetian ^a   He, Bingsheng ^b   Yang, Mao ^c   Choi, Byron ^d   Li, Xiaoming ^a  

^a PEKING UNIVERSITY   (China)

^b NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

^c MICROSOFT RESEARCH ASIA   (China)

^d HONG KONG BAPTIST UNIVERSITY   (Hong Kong)

Author keywords

Cloud computing; Data center network; Graph partitioning; Large graph processing

Indexed keywords

AMAZON EC2; DATA CENTERS; DATA-INTENSIVE APPLICATION; EXECUTION MODEL; EXISTING SYSTEMS; GRAPH PARTITIONING; GRAPH PROCESSING; LARGE GRAPHS; LOCAL CLUSTER; NETWORK ENVIRONMENTS; NETWORK TRAFFIC; RANDOM ACCESS; SOCIAL NETWORKS; SYNTHETIC GRAPHS; SYSTEM PROTOTYPE;

CLOUD COMPUTING; DIGITAL STORAGE; EXPERIMENTS; INTERNET; NETWORK PERFORMANCE; OPTIMIZATION;

GRAPH THEORY;

EID: 84870544283     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2391229.2391232     Document Type: Conference Paper

References (43)

1. A. Abouzeid, K. Bajda-Pawlikowski, D. Abadi, A. Silberschatz, and A. Rasin. Hadoopdb: an architectural hybrid of mapreduce and dbms technologies for analytical workloads. Proc. VLDB Endow., 2009.
2. G. Adomavicius and A. Tuzhilin. Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE TKDE, 17(6):734-749, 2005.
3. Apache Giraph. http://giraph.apache.org/.
4. Apache Hama. http://hama.apache.org/.
5. T. Benson, A. Akella, and D. A. Maltz. Network traffic characteristics of data centers in the wild. In IMC, 2010.
6. D. Chakrabarti, Y. Zhan, and C. Faloutsos. R-mat: A recursive model for graph mining. In Fourth SIAM International Conference on Data Mining, April 2004.
7. F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. E. Gruber. Bigtable: a distributed storage system for structured data. In OSDI, 2006.
8. R. Chen, X. Weng, B. He, and M. Yang. Large graph processing in the cloud. In Proceedings of the 2010 ACM SIGMOD International Conference on Management of data, SIGMOD '10, pages 1123-1126, New York, NY, USA, 2010. ACM.
9. S. Das, D. Agrawal, and A. El Abbadi. G-store: a scalable data store for transactional multi key access in the cloud. In SoCC: ACM symposium on Cloud computing, 2010.
10. J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. In OSDI, 2004.
11. B. Derbel, M. Mosbah, and A. Zemmari. Fast distributed graph partition and application. In IPDPS, 2006.
12. S. Ghemawat, H. Gobioff, and S.-T. Leung. The google file system. SIGOPS Oper. Syst. Rev., 37(5):29-43, 2003.
13. D. Gregor and A. Lumsdaine. The parallel bgl: A generic library for distributed graph computations. In Parallel Object-Oriented Scientific Computing (POOSC), 2005.
14. C. Guo, H. Wu, K. Tan, L. Shi, Y. Zhang, and S. Lu. Dcell: a scalable and fault-tolerant network structure for data centers. SIGCOMM, 38(4), 2008.
15. Hadoop. http://hadoop.apache.org/.
16. B. He, M. Yang, Z. Guo, R. Chen, B. Su, W. Lin, and L. Zhou. Comet: batched stream processing for data intensive distributed computing. In Proceedings of the 1st ACM symposium on Cloud computing, SoCC '10, pages 63-74, New York, NY, USA, 2010. ACM.
17. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: distributed data-parallel programs from sequential building blocks. SIGOPS Oper. Syst. Rev., 41(3):59-72, 2007.
18. M. Isard, V. Prabhakaran, J. Currey, U. Wieder, K. Talwar, and A. Goldberg. Quincy: fair scheduling for distributed computing clusters. In SOSP, 2009.
19. S. Kandula, S. Sengupta, A. Greenberg, P. Patel, and R. Chaiken. The nature of data center traffic: measurements & analysis. In IMC, 2009.
20. U. Kang, C. Tsourakakis, A. P. Appel, C. Faloutsos, and J. Leskovec. HADI: Fast diameter estimation and mining in massive graphs with hadoop. Technical Report CMU-ML-08-117, CMU, 2008.
21. U. Kang, C. E. Tsourakakis, and C. Faloutsos. PEGASUS: A peta-scale graph mining system - implementation and observations. In ICDM, 2009.
22. G. Karypis and V. Kumar. Parallel multilevel k-way partitioning scheme for irregular graphs. In Supercomputing, 1996.
23. G. Karypis and V. Kumar. A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM J. Sci. Comput., 20(1):359-392, 1998.
24. G. Karypis and V. Kumar. A parallel algorithm for multilevel graph partitioning and sparse matrix ordering. J. Parallel Distrib. Comput., 48(1):71-95, 1998.
25. B. W. Kernighan and S. Lin. An efficient heuristic procedure for partitioning graphs. The Bell system technical journal, 49(1):291-307, 1970.
26. S. Koranne. A distributed algorithm for k-way graph partitioning. In EUROMICRO, 1999.
27. Y. Low, D. Bickson, J. Gonzalez, C. Guestrin, A. Kyrola, and J. M. Hellerstein. Distributed graphlab: a framework for machine learning and data mining in the cloud. Proc. VLDB Endow., 5(8):716-727, Apr. 2012.
28. Y. Low, J. Gonzalez, A. Kyrola, D. Bickson, C. Guestrin, and J. Hellerstein. Graphlab: A new framework for parallel machine learning. In UAI, 2010.
29. G. Malewicz, M. Austern, A. Bik, J. Dehnert, I. Horn, N. Leiser, and G. Czajkowski. Pregel: A system for large-scale graph processing. In SIGMOD, 2010.
30. Metis. http://glaros.dtc.umn.edu/gkhome/views/metis/.
31. L. Page, S. Brin, R. Motwani, and T. Winograd. The pagerank citation ranking: Bringing order to the web. Technical Report 1999-66, 1999.
32. J. Pješivac-Grbović, T. Angskun, G. Bosilca, G. E. Fagg, E. Gabriel, and J. J. Dongarra. Performance analysis of mpi collective operations. Cluster Computing, 10(2):127-143, June 2007.
33. R. Power and J. Li. Piccolo: Building fast, distributed programs with partitioned tables. In OSDI, 2010.
34. J. M. Pujol, V. Erramilli, G. Siganos, X. Yang, N. Laoutaris, P. Chhabra, and P. Rodriguez. The little engine(s) that could: Scaling online social networks. In SIGCOMM, 2010.
35. B. Shao, H. Wang, and Y. Li. The trinity graph engine. Technical Report 161291, Microsoft Research, 2012.
36. A. Trifunović and W. J. Knottenbelt. Parallel multilevel algorithms for hypergraph partitioning. J. Parallel Distrib. Comput., 68, May 2008.
37. G. Wang and T. S. E. Ng. The impact of virtualization on network performance of amazon ec2 data center. In INFOCOM, 2010.
38. H. Wang, Q. Jing, R. Chen, B. He, Z. Qian, and L. Zhou. Distributed systems meet economics: pricing in the cloud. In Proceedings of the 2nd USENIX conference on Hot topics in cloud computing, HotCloud'10, pages 6-6, Berkeley, CA, USA, 2010. USENIX Association.
39. J. Wang, S. Wu, H. Gao, J. Li, and B. C. Ooi. Indexing multi-dimensional data in a cloud system. In SIGMOD, 2010.
40. H. T. Welser, E. Gleave, D. Fisher, and M. Smith. Visualizing the signatures of social roles in online discussion groups. The Journal of Social Structure, 2(8), 2007.
41. W. Xue, J. Shi, and B. Yang. X-rime: Hadoop based large scale social network analysis. In SCC, 2010.
42. M. Zaharia, A. Konwinski, A. D. Joseph, R. Katz, and I. Stoica. Improving mapreduce performance in heterogeneous environments. In OSDI, 2008.
43. A. Zhou, W. Qian, D. Tao, and Q. Ma. DisG: A distributed graph repository for web infrastructure (invited paper). International Symposium on Universal Communication, 0:141-145, 2008.