Memory or time: Performance evaluation for iterative operation on hadoop and spark

Proceedings - 2013 IEEE International Conference on High Performance Computing and Communications, HPCC 2013 and 2013 IEEE International Conference on Embedded and Ubiquitous Computing, EUC 2013

Volumn , Issue , 2014, Pages 721-727

  Gu, Lei ^a   Li, Huan ^a  

^a BEIHANG UNIVERSITY   (China)

Author keywords

Hadoop; Iterative Operation; Spark; System Performance

Indexed keywords

UBIQUITOUS COMPUTING;

BETTER PERFORMANCE; DATA-INTENSIVE APPLICATION; GENERAL PURPOSE FRAMEWORK; HADOOP; INTERMEDIATE RESULTS; ITERATIVE OPERATION; PERFORMANCE CRITERION; SYSTEM PERFORMANCE;

ELECTRIC SPARKS;

EID: 84903975456     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/HPCC.and.EUC.2013.106     Document Type: Conference Paper

References (22)

1. M. Zaharia, M. Chowdhury, S. S. Michael J. Franklin, and I. Stoica, "Spark: Cluster computing with working sets", In HotCloud, June 2010.
2. J. Dean and S. Ghemawat, "MapReduce: Simplified data processing on large clusters", In OSDI, 2004.
3. Apache Hadoop. [Online]. Available: http://hadoop. apache.org
4. Spark. [Online]. Available: http://spark-project.org
5. M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J. Franklin, S. Shenker, and I. Stoica, "Resilient Distributed Datasets: A fault-tolerant abstraction for in-memory cluster computing", In NSDI, April 2012.
6. L. Page, S. Brin, R. Motwani, and T. Winograd, "The PageRank citation ranking: Bringing order to the Web", Technical Report, Stanford InfoLab, 1999.
7. J. Ekanayake, H. Li, B. Zhang, T. Gunarathne, S.-H. Bae, J. Qiu, and G. Fox, "Twister: A runtime for iterative MapReduce", in Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, 2010, pp. 810-818.
8. Y. Bu, B. Howe, M. Balazinska, and M. D. Ernst, "HaLoop: Efficient iterative data processing on large clusters", in Proceedings of the VLDB Endowment, September 2010, pp. 285-296.
9. Y. Zhang, Q. Gao, L. Gao, and C. Wang, "iMapReduce: A distributed computing framework for iterative computation", in Proceedings of the 2011 IEEE International Symposium on Parallel and Distributed Processing Workshops and PhD Forum, 2011, pp. 1112-1121.
10. B. Hindman, A. Konwinski, M. Zaharia, A. Ghodsi, A. D. Joseph, R. Katz, S. Shenker, and I. Stoica, "Mesos: A platform for fine-grained resource sharing in the data center", in Proceedings of the 8th USENIX conference on Networked systems design and implementation. USENIX Association, 2011, pp. 22-22.
11. Apache Mesos. [Online]. Available: http://incubator.apache.org/mesos
12. Stanford SNAP. [Online]. Available: http://snap. stanford.edu
13. Twitter graph dataset. [Online]. Available: http://socialcomputing.asu. edu/datasets/Twitter
14. J. Leskovec, D. Chakrabarti, J. Kleinberg, and C. Faloutsos, "Realistic, mathematically tractable graph generation and evolution, using Kronecker multiplication", in Proceedings of the 9th European conference on Principles and Practice of Knowledge Discovery in Databases. Springer, 2005, pp. 133-145.
15. M. Bianchini, M. Gori, and F. Scarselli, "Inside PageRank", in ACM Transactions on Intenet Technology, vol. 5, 2005, pp. 92-128.
16. J. Lin and C. Dyer, Data-Intensive Text Processing with MapReduce. Morgan & Claypool, 2010.
17. D. Jiang, B. C. Ooi, L. Shi, and S. Wu, "The performance of Map-Reduce: An in-depth study", Proceedings of the VLDB Endowment, vol. 3, no. 1-2, pp. 472-483, 2010.
18. K.-H. Lee, Y.-J. Lee, H. Choi, Y. D. Chung, and B. Moon, "Parallel data processing with MapReduce: a survey", ACM SIGMOD Record, vol. 40, no. 4, pp. 11-20, 2012.
19. "Pegasus: A peta-scale graph mining system implementation and observations", in Data Mining, 2009. ICDM'09. Ninth IEEE International Conference on. IEEE, 2009, pp. 229-238.
20. "Pregel: A system for large-scale graph processing", in Proceedings of the 2010 ACM SIGMOD International Conference on Management of data. ACM, 2010, pp. 135-146.
21. "Distributed GraphLab: A framework for machine learning and data mining in the cloud", Proceedings of the VLDB Endowment, vol. 5, no. 8, pp. 716-727, 2012.
22. R. Zafarani and H. Liu, "Social computing data repository at ASU", 2009. [Online]. Available: http://socialcomputing.asu.edu