HOG: Distributed hadoop MapReduce on the grid

Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012

Volumn , Issue , 2012, Pages 1276-1283

  He, Chen ^a   Weitzel, Derek ^a   Swanson, David ^a   Lu, Ying ^a  

^a UNIVERSITY OF NEBRASKA LINCOLN   (United States)

Author keywords

Grid computing; MapReduce; Middleware

Indexed keywords

FAILURE DOMAINS; HADOOP FRAMEWORKS; HADOOP MAPREDUCE; MAP-REDUCE; MAPPING DATA; OPEN SCIENCE GRID; PROCESSING PLATFORM; WIDE AREA;

DATA PROCESSING; FAULT TOLERANCE; MIDDLEWARE;

GRID COMPUTING;

EID: 84876542016     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/SC.Companion.2012.154     Document Type: Conference Paper

References (25)

1. J. Dean and S. Ghemawat, "Mapreduce: simplified data processing on large clusters," in Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation-Volume 6, ser. OSDI'04. Berkeley, CA, USA: USENIX Association, 2004, pp. 10-10.
2. A. Bialecki, M. Cafarella, D. Cutting, and O. OMALLEY, "Hadoop: a framework for running applications on large clusters built of commodity hardware," Wiki at http://lucene.apache.org/hadoop, vol. 11, 2005.
3. M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmeleegy, S. Shenker, and I. Stoica, "Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling," in Proceedings of the 5th European conference on Computer systems. ACM, 2010, pp. 265-278.
4. D. GOTTFRID, "Self-service, prorated supercomputing fun!" http://open.blogs.nytimes.com/2007/11/01/self-service-prorated-supercomputing- fun/.
5. CERN, "Large hadron collider," http://lhc.web.cern.ch/lhc/.
6. R. Pordes, D. Petravick, B. Kramer, D. Olson, M. Livny, A. Roy, P. Avery, K. Blackburn, T. Wenaus, F. Würthwein et al., "The open science grid," in Journal of Physics: Conference Series, vol. 78. IOP Publishing, 2007, p. 012057.
7. M. Schatz, "Blastreduce: high performance short read mapping with mapreduce," University of Maryland, http://cgis.cs.umd.edu/Grad/ scholarlypapers/papers/MichaelSchatz.pdf.
8. D. Ford, F. Labelle, F. Popovici, M. Stokely, V. Truong, L. Barroso, C. Grimes, and S. Quinlan, "Availability in globally distributed storage systems," in Proceedings of the 9th USENIX Symposium on Operating Systems Design and Implementation, 2010.
9. Apache, "Hdfs," http://apache.hadoop.org/hdfs/.
10. Hadoop, "HDFS Architecture," September 2012, https://hadoop.apache.org/docs/r0.20.2/hdfs design.html.
11. A. Foundation, "Yarn," https://hadoop.apache.org/docs/r0.23.0/ hadoopyarn/hadoop-yarn-site/YARN.html.
12. I. Foster, C. Kesselman, and S. Tuecke, "The anatomy of the grid: Enabling scalable virtual organizations," International journal of high performance computing applications, vol. 15, no. 3, pp. 200-222, 2001.
13. V. Welch, I. Foster, C. Kesselman, O. Mulmo, L. Pearlman, S. Tuecke, J. Gawor, S. Meder, and F. Siebenlist, "X. 509 proxy certificates for dynamic delegation," in 3rd annual PKI R&D workshop, vol. 14, 2004.
14. I. Foster and C. Kesselman, "Globus: A metacomputing infrastructure toolkit," International Journal of High Performance Computing Applications, vol. 11, no. 2, pp. 115-128, 1997.
15. G. Alliance, "Globus toolkit," Online at: http://www.globus. org/toolkit/about.html, 2006.
16. K. Czajkowski, I. Foster, N. Karonis, C. Kesselman, S. Martin,W. Smith, and S. Tuecke, "A resource management architecture for metacomputing systems," in Job Scheduling Strategies for Parallel Processing. Springer, 1998, pp. 62-82.
17. J. Frey, T. Tannenbaum, M. Livny, I. Foster, and S. Tuecke, "Condor-g: A computation management agent for multi-institutional grids," Cluster Computing, vol. 5, no. 3, pp. 237-246, 2002.
18. A. Shoshani, A. Sim, and J. Gu, "Storage resource managers: Middleware components for grid storage," in NASA Conference Publication. NASA; 1998, 2002, pp. 209-224.
19. W. Allcock, J. Bresnahan, R. Kettimuthu, M. Link, C. Dumitrescu, I. Raicu, and I. Foster, "The globus striped gridftp framework and server," in Proceedings of the 2005 ACM/IEEE conference on Supercomputing. IEEE Computer Society, 2005, p. 54.
20. C. He, Y. Lu, and D. Swanson, "Matchmaking: A new mapreduce scheduling technique," in Cloud Computing Technology and Science (CloudCom), 2011 IEEE Third International Conference on. IEEE, 2011, pp. 40-47.
21. Hadoop. (2011, Mar) Hod scheduler. [Online]. Available: https://hadoop.apache.org/common/docs/r0.20.203.0/hod scheduler.html
22. H. Lin, X. Ma, J. Archuleta,W. Feng, M. Gardner, and Z. Zhang, "Moon: Mapreduce on opportunistic environments," in Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing. ACM, 2010, pp. 95-106.
23. D. Solo, R. Housley, and W. Ford, "Internet x. 509 public key infrastructure certificate and crl profile," 1999.
24. J. Steiner, C. Neuman, and J. Schiller, "Kerberos: An authentication service for open network systems," in USENIX conference proceedings, vol. 191, 1988, p. 202.
25. I. Raicu, I. Foster, and Y. Zhao, "Many-task computing for grids and supercomputers," in Many-Task Computing on Grids and Supercomputers, 2008. MTAGS 2008. Workshop on. IEEE, 2008, pp. 1-11.