Investigation of data locality in MapReduce

Proceedings - 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2012

Volumn , Issue , 2012, Pages 419-426

  Guo, Zhenhua ^a   Fox, Geoffrey ^a   Zhou, Mo ^a  

^a INDIANA UNIVERSITY   (United States)

Author keywords

data locality; Hadoop; MapReduce; scheduling

Indexed keywords

COMMODITY HARDWARE; DATA ACCESS REQUIREMENTS; DATA LOCALITY; DATA PARALLEL; DATA-INTENSIVE COMPUTING; GOOD DATA; HADOOP; HIGH-SPEED LINKS; IMPACT DATA; JOB EXECUTION; MAP-REDUCE; MULTIPLE TASKS; MULTIUSER ENVIRONMENTS; NETWORK TRAFFIC; OPTIMAL DATA; PERFORMANCE IMPROVEMENTS; STORAGE NODES;

ALGORITHMS; GRID COMPUTING; MATHEMATICAL MODELS;

SCHEDULING;

EID: 84863651849     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CCGrid.2012.42     Document Type: Conference Paper

References (19)

1. C.S. Chang. "Needs for Extreme Scale DM, Analysis and Visualization in Fusion Particle Code XGC," CPES at Exascale data management, Analysis, and Vis., Feb. 22-23, 2011, Houston, TX.
2. Scientific Grand Challenges: Fusion Energy Sciences and the Role of Computing at the Extreme Scale. U. S. Department of Energy, March 18-20, 2009. Washington DC
3. J. Chen and J. Bell, "Combustion Exascale Co-Design Center," 6th IESP Workshop, San Francisco, CA, April 6-7, 2011
4. S. Ghemawat, H. Gobioff, and S. Leung, "The Google file system,"In Proc. of SOSP 2003. New York, NY, USA, 29-43.
5. K. Shvachko, H. Kuang, S. Radia, and R. Chansler, "The Hadoop Distributed File System," IEEE MSST 2010
6. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, "Dryad: distributed data-parallel programs from sequential building blocks,"In Proc. of EuroSys 2007. ACM, New York, NY, USA, 59-72.
7. J. Ekanayake, T. Gunarathne, J. Qiu, G. Fox, et. al., "Applicability of DryadLINQ to Scientific Applications," January 30, 2010, Tech report in Community Grids Laboratory, Indiana University.
8. Y. Gu and R. Grossman, "Sector and Sphere: The Design and Implementation of a High Performance Data Cloud," Crossing boundaries: computational science, e-Science and global e-Infrastructure, I. Trans. R. Soc. A, 2009. vol. 367, p. 2429-2445.
9. K. Ranganathan and I. Foster, "Decoupling computation and data scheduling in distributed data-intensive applications," In Proc. of HPDC 2002, p. 352- 358, 2002
10. F. Chung, R. Graham, R. Bhagwan, S. Savage, and G. M. Voelker, "Maximizing data locality in distributed systems," J. Comput. Syst. Sci. 72, 8 (December 2006), 1309-1316.
11. H. H. Mohamed and D. H. J. Epema, "An evaluation of the close-tofiles processor and data co-allocation policy in multiclusters," In Proc. of Cluster 2004, p. 287- 298, 20-23 Sept. 2004
12. M. Zaharia, D. Borthakur, J. Sarma, K. Elmeleegy, S. Shenker, and I. Stoica, "Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling," In Proc. of EuroSys 2010, New York, NY, USA, 265-278.
13. B. Palanisamy, A. Singh, L. Liu and B. Jain, "Purlieus: Localityaware Resource Allocation for MapReduce in a Cloud," In Proc. of SC 2011
14. Z. Guo, M. Pierce, G. Fox and M. Zhou, "Automatic Task Reorganization in MapReduce" CLUSTER2011, Sep. 2011, Austin, TX.
15. Y. Luo, Z. Guo, Y. Sun, et. al. "A Hierarchical Framework for Cross-Domain MapReduce Execution," HPDC workshop ECMLS 2011
16. M. Zaharia, A. Konwinski, A. D. Joseph, R. Katz, and I. Stoica, "Improving MapReduce performance in heterogeneous environments," In Proc. of OSDI 2008, Berkeley, CA, USA
17. https://issues.apache.org/jira/browse/HADOOP-2141
18. http://www.assignmentproblems.com/doc/LSAPIntroduction.pdf
19. M. Tsugawa and J. Fortes, "A Virtual Network (ViNe) Architecture for Grid Computing," IPDPS 2006, p. 10, April 2006