G-Hadoop: MapReduce across distributed data centers for data-intensive computing

Future Generation Computer Systems

Volumn 29, Issue 3, 2013, Pages 739-750

  Wang, Lizhe ^a,b   Tao, Jie ^c   Ranjan, Rajiv ^d   Marten, Holger ^c   Streit, Achim ^c   Chen, Jingying ^e   Chen, Dan ^a  

^a CHINA UNIVERSITY OF GEOSCIENCES   (China)

^b CENTER FOR EARTH OBSERVATION AND DIGITAL EARTH   (China)

^c KARLSRUHE INSTITUTE OF TECHNOLOGY   (Germany)

^d CSIRO   (Australia)

^e CENTRAL CHINA NORMAL UNIVERSITY   (China)

Author keywords

Cloud computing; Data intensive computing; Hadoop; MapReduce; Massive data processing

Indexed keywords

CLUSTER ENVIRONMENTS; COMPUTATIONAL REQUIREMENTS; COMPUTING CENTER; DATA CENTERS; DATA-INTENSIVE COMPUTING; DISTRIBUTED DATA; DISTRIBUTED DATA PROCESSING; HADOOP; LARGE HADRON COLLIDER LHC; MAP-REDUCE; MULTIPLE CLUSTERS; PETABYTES; PROGRAMMING MODELS; SCIENTIFIC DATA; WORK-FLOW SYSTEMS;

CLOUD COMPUTING; HIGH ENERGY PHYSICS;

DATA PROCESSING;

EID: 84870312247     PISSN: 0167739X     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.future.2012.09.001     Document Type: Article

References (46)

1. F. Berman Got data?: a guide to data preservation in the information age Communications of the ACM 51 2008 50 56
2. Data intensive computing, Website. http://en.wikipedia.org/wiki/ DataIntensiveComputing.
3. L. Wang, M. Kunze, J. Tao, and G. von Laszewski Towards building a cloud for scientific applications Advances in Engineering Software 42 9 2011 714 722
4. G. Brumfiel High-energy physics: down the petabyte highway Nature 7330 2011 282 283
5. L. Wang, and C. Fu Research advances in modern cyberinfrastructure New Generation Computing 28 2 2010 111 112
6. Condor dagman, Website. http://www.cs.wisc.edu/condor/dagman/.
7. E. Deelman, G. Singh, M.-H. Su, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, G.B. Berriman, J. Good, A. Laity, J.C. Jacob, and D.S. Katz Pegasus: a framework for mapping complex scientific workflows onto distributed systems Science Programming 13 2005 219 237
8. Y. Zhao, M. Hategan, B. Clifford, I.T. Foster, G. von Laszewski, V. Nefedova, I. Raicu, T. Stef-Praun, and M. Wilde Swift: fast, reliable, loosely coupled parallel computation IEEE SCW 2007 IEEE Computer Society Salt Lake City, Utah, USA 199 206
9. I. Altintas, B. Ludaescher, S. Klasky, and M.A. Vouk Introduction to scientific workflow management and the kepler system Proceedings of the 2006 ACM/IEEE Conference on Supercomputing ser. SC'06 2006 ACM New York, NY, USA
10. L. Wang, D. Chen, and F. Huang Virtual workflow system for distributed collaborative scientific applications on grids Computers & Electrical Engineering 37 3 2011 300 310
11. L. Wang, M. Kunze, and J. Tao Performance evaluation of virtual machine-based grid workflow system Concurrency and Computation: Practice and Experience 20 15 2008 1759 1771
12. L. Wang, G. von Laszewski, J. Tao, and M. Kunze Virtual data system on distributed virtual machines in computational grids International Journal of Ad Hoc and Ubiquitous Computing 6 4 2010 194 204
13. P. Missier, S. Soiland-Reyes, S. Owen, W. Tan, A. Nenadic, I. Dunlop, A. Williams, T. Oinn, and C. Goble Taverna, reloaded Proceedings of the 22nd International Conference on Scientific and Statistical Database Management SSDBM'10 2010 Springer-Verlag Berlin, Heidelberg 471 481
14. L. Wang, G. von Laszewski, A.J. Younge, X. He, M. Kunze, J. Tao, and C. Fu Cloud computing: a perspective study New Generation Computing 28 2 2010 137 146
15. R. Ranjan, L. Zhao, X. Wu, A. Liu, A. Quiroz, and M. Parashar Peer-to-peer cloud provisioning: service discovery and load-balancing Cloud Computing 2010 195 217
16. R. Buyya, and R. Ranjan Special section: federated resource management in grid and cloud computing systems Future Generation Computer Systems 26 8 2010 1189 1191
17. D. Chen, L. Wang, G. Ouyang, and X. Li Massively parallel neural signal processing on a many-core platform Computing in Science and Engineering 2011
18. X. Yang, L. Wang, and G. von Laszewski Recent research advances in e-science Cluster Computing 12 4 2009 353 356
19. H. Zhu, T.K.Y. Chan, L. Wang, and R.C. Jegathese A distributed 3d rendering application for massive data sets IEICE Transactions 87-D 7 2004 1805 1812
20. C. Moretti, J. Bulosan, D. Thain, P.J. Flynn, All-pairs: an abstraction for data-intensive cloud computing, in: IPDPS, April 2008, pp. 1-11.
21. R. Grossman, and Y. Gu Data mining using high performance data clouds: experimental studies using sector and sphere Proceeding of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining KDD'08 2008 ACM New York, NY, USA 920 927
22. Y. Yu, M. Isard, D. Fetterly, M. Budiu, U. Erlingsson, P.K. Gunda, and J. Currey Dryadlinq: a system for general-purpose distributed data-parallel computing using a high-level language Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation OSDI'08 2008 USENIX Association Berkeley, CA, USA 1 14
23. D. Logothetis, and K. Yocum Wide-scale data stream management USENIX 2008 Annual Technical Conference on Annual Technical Conference 2008 USENIX Association Berkeley, CA, USA 405 418
24. J. Dean, and S. Ghemawat Mapreduce: simplified data processing on large clusters Communications of the ACM 51 2008 107 113
25. Apache Hadoop project, Web Page. http://hadoop.apache.org/.
26. B. He, W. Fang, Q. Luo, N.K. Govindaraju, and T. Wang Mars: a mapreduce framework on graphics processors Proceedings of the 17th International Conference on Parallel Architectures and Compilation Techniques PACT'08 2008 ACM New York, NY, USA 260 269
27. R. Chen, H. Chen, and B. Zang Tiled-mapreduce: optimizing resource usages of data-parallel applications on multicore with tiling Proceedings of the 19th International Conference on Parallel Architectures and Compilation Techniques PACT'10 2010 ACM New York, NY, USA 523 534
28. Y. Shan, B. Wang, J. Yan, Y. Wang, N. Xu, and H. Yang Fpmr: mapreduce framework on fpga Proceedings of the 18th Annual ACM/SIGDA International Symposium on Field Programmable Gate Arrays FPGA'10 2010 ACM New York, NY, USA 93 102
29. C. Ranger, R. Raghuraman, A. Penmetsa, G.R. Bradski, C. Kozyrakis, Evaluating mapreduce for multi-core and multiprocessor systems, in: 13st International Conference on High-Performance Computer Architecture, 2007, pp. 13-24.
30. R.M. Yoo, A. Romano, and C. Kozyrakis Phoenix rebirth: scalable mapreduce on a large-scale shared-memory system Proceedings of the 2009 IEEE International Symposium on Workload Characterization 2009 IEEE Austin, TX, USA 198 207
31. M.M. Rafique, B. Rose, A.R. Butt, and D.S. Nikolopoulos Supporting mapreduce on large-scale asymmetric multi-core clusters SIGOPS Operating Systems Review 43 2009 25 34
32. S. Ibrahim, H. Jin, B. Cheng, H. Cao, S. Wu, and L. Qi Cloudlet: towards mapreduce implementation on virtual machines Proceedings of the 18th ACM International Symposium on High Performance Distributed Computing HPDC'09 2009 ACM New York, NY, USA 65 66
33. C. Jin, and R. Buyya Mapreduce programming model for.net-based cloud computing Euro-Par Lecture Notes in Computer Science vol. 5704 2009 Springer 417 428
34. S. Pallickara, J. Ekanayake, and G. Fox Granules: a lightweight, streaming runtime for cloud computing with support for map-reduce CLUSTER 2009 IEEE New Orleans, Louisiana, USA 1 10
35. C. Miceli, M. Miceli, S. Jha, H. Kaiser, and A. Merzky Programming abstractions for data intensive computing on clouds and grids Proceedings of the 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid CCGRID'09 2009 IEEE Computer Society Washington, DC, USA 478 483
36. H. Lin, X. Ma, J. Archuleta, W.-C. Feng, M. Gardner, and Z. Zhang Moon: mapreduce on opportunistic environments Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing HPDC'10 2010 ACM New York, NY, USA 95 106
37. A. Dou, V. Kalogeraki, D. Gunopulos, T. Mielikainen, and V.H. Tuulos Misco: a mapreduce framework for mobile systems Proceedings of the 3rd International Conference on PErvasive Technologies Related to Assistive Environments PETRA'10 2010 ACM New York, NY, USA 32:1 32:8
38. Apache Hadoop on Demand (HOD), Website. http://hadoop.apache.org/common/ docs/r0.21.0/hodscheduler.html.
39. S. Krishnan, M. Tatineni, C. Baru, myhadoop - hadoop-on-demand on traditional hpc resources, University of California, San Diego, Technical Report, 2011.
40. O. Tatebe, K. Hiraga, and N. Soda Gfarm grid file system New Generation Computing 28 3 2010 257 275
41. Torque resource manager, Website. http://www.clusterresources.com/ products/torque-resource-manager.php.
42. Distributed Resource Management Application API (DRMAA), Website. http://drmaa.org/.
43. W. Sun, J. Tao, R. Ranjan, A. Streit, A security framework in G-Hadoop for data-intensive computing applications across distributed clusters, Journal of Computer and System Sciences.
44. D.H. Bailey, The bbp algorithm for pi, Sep. 2006. http://crd.lbl.gov/ dhbailey/dhbpapers/bbp-alg.pdf.
45. Hadoop sort benchmark, Website. http://wiki.apache.org/hadoop/Sort.
46. R. Ranjan, A. Harwood, and R. Buyya Coordinated load management in peer-to-peer coupled federated grid systems Journal of Supercomputing 61 2 2012 292 316 10.1007/s11227-010-0426-y [Online], Available