An Adaptive and Memory Efficient Sampling Mechanism for Partitioning in MapReduce

International Journal of Parallel Programming

Volumn 43, Issue 3, 2015, Pages 489-507

  Slagter, Kenn ^a   Hsu, Ching Hsien ^b   Chung, Yeh Ching ^a  

^a NATIONAL TSING HUA UNIVERSITY   (Taiwan)

^b CHUNG HUA UNIVERSITY   (Taiwan)

Author keywords

Cloud computing; Hadoop; Load balance; MapReduce; Partitioning; Sampling

Indexed keywords

BIG DATA; CLOUD COMPUTING; DISTRIBUTED DATABASE SYSTEMS; SAMPLING;

DISTRIBUTED ENVIRONMENTS; DISTRIBUTED PROGRAM; DISTRIBUTED SYSTEMS; HADOOP; LOAD BALANCE; MAP-REDUCE; PARTITIONING; PARTITIONING SYSTEMS;

DISTRIBUTED COMPUTER SYSTEMS;

EID: 84924221125     PISSN: 08857458     EISSN: 15737640     Source Type: Journal    
DOI: 10.1007/s10766-013-0288-z     Document Type: Article

References (28)

1. Apache Software Foundation, “Hadoop”, http://hadoop.apache.org/core
2. Candan, K., Kim, J.W., Nagarkar, P., Nagendra, M., Yu, R.: RanKloud: scalable multimedia data processing in server clusters. IEEE MultiMed. 18, 64–77 (2011)
3. Chang, F., Dean, J., Ghemawat, S., Hsieh, W.C., Wallach, D.A., Burrows, M., Chandra, T., Fikes, A., Gruber, R.E.: Bigtable: a distributed storage system for structured data. In: 7th USENIX Symposium on Operating Systems Design and Implementation, pp. 205–218 (2006)
4. Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51, 107–113 (2008)
5. Fadika, Z., Govindaraju, M.: Delma: dynamically elastic mapreduce framework for cpu-intensive applications. In: IEEE/ACM International Symposium on Cluster, Cloud and Grid, Computing, pp.454–463 (2011)
6. Ghemawat, S., Gobioff, H., Leung, S.-T.: The google file system. In: ACM SIGOPS Operating Systems Review, ACM, pp. 29–43 (2003)
7. Groot, S., Kitsuregawa, M.: Jumbo: Beyond MapReduce for Workload Balancing. VLDB, Phd Workshop (2010)
8. HBase, http://hadoop.apache.org/hbase/
9. Heinz, S., Zobel, J., Williams, H.: Burst tries: a fast, efficient data structure for string keys. Trans. Inf. Syst. (TOIS) 20(12), 192–223 (2002)
10. Jiang, W., Agrawal, G.: Ex-mate: data intensive computing with large reduction objects and its application to graph mining. In: Cluster, Cloud and Grid Computing (CCGrid): 11th IEEE/ACM International Symposium on, IEEE 2011, pp. 475–484 (2011)
11. Jin, C., Vecchiola, C., Buyya, R.: MRPGA: an extension of MapReduce for parallelizing genetic algorithms. In: IEEE Fourth International Conference on eScience pp. 214–221 (2008)
12. Kavulya, S., Tan, J., Gandhi, R., Narasimhan, P.: An analysis of traces from a production mapreduce cluster. In: Cluster, Cloud and Grid Computing (CCGrid), 2010 10th IEEE/ACM International Conference, pp. 94–103 (2010)
13. Krishnan, A.: GridBLAST: a globus-based high-throughput implementation of BLAST in a Grid computing framework. Concurr. Comput. Pract. Exp. 17(13), 1607–1623 (2005)
14. Liu, H., Orban, D.: Cloud mapreduce: a mapreduce implementation on top of a cloud operating system. In: 11th IEEE/ACM International Symposium, pp. 464–474 (2011)
15. Lynden, S., Tanimura, Y., Kojima, I., Matono, A.: Dynamic data redistribution for MapReduce joins. In: IEEE International Conference on Coud Computing Technology and Science, pp. 713–717 (2011)
16. Matsunaga, A., Tsugawa, M., Fortes, J.: “Programming abstractions for data intensive computing on clouds and grids. In: IEEE Fourth International Conference on eScience, pp. 489–493 (2008)
17. Miceli, C., Miceli, M., Jha, S., Kaiser, H., Merzky, A.: Programming abstractions for data intensive computing on clouds and grids. In: IEEE/ACM International Symposium on Cluster Computing and the Grid, pp. 480–483 (2009)
18. O’Malley, O.: TeraByte Sort on Apache Hadoop (2008)
19. Panda, B., Riedewald, M., Fink, D.: The model-summary problem and a solution for trees. In: Data Engineering, International Conference on Data, Engineering, pp. 452–455 (2010)
20. Papadimitriou, S., Sun, J.: Disco: distributed co-clustering with map-reduce: a case study towards petabyte-scale end-to-end mining. In: IEEE International Conference on Data Mining, p. 519 (2008)
21. Shafer, J., Rixner, S., Cox, A.L.: The Hadoop distributed filesystem: balancing portability and performance. In: IEEE International Symposium on Performance Analysis of System and Software(ISPASS), p. 123 (2010)
22. Slagter, K., Hsu, C.-H., Chung, Y.-C., Zhang, D.: An improved partitioning mechanism for optimizing massive data analysis using MapReduce. J. Supercomput. 66(1), 539–555 (2013)
23. Stockinger, H., Pagni, M., Cerutti, L., Falquet, L.: Grid approach to embarrassingly parallel CPU-intensive bioinformatics problems. In: IEEE International Conference on e-Science and Grid Computing (2006)
24. Tan, J., Pan, X., Kavulya, S., Gandhi, R., Narasimhan, P.: Mochi: visual log-analysis based tools for debugging Hadoop. In: USENIX Workshop on Hot Topics in Cloud Computing (HotCloud) (2009)
25. Vashishtha, H., Smit, M., Stroulia, E.: Moving text analysis tools to the cloud. In: IEEE World Congress on Services, pp. 110–112 (2010)
26. Verma, A., Llora, X., Goldberg, D.E., Campbell, R.H.: Scaling genetic algorithms using mapreduce. In: Intelligent Systems Design and Applications (2009)
27. White, T.: “Hadoop the definitive guide 2nd edition”, Published Oreilly (2010)
28. Xu, W., Huang, L., Fox, A., Patterson, D., Jordan, M.I.: Detecting large-scale system problems by mining console logs. In: Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles (2009)