Improving metadata management for small files in HDFS

Proceedings - IEEE International Conference on Cluster Computing, ICCC

Volumn , Issue , 2009, Pages

  MacKey, Grant ^a   Sehrish, Saba ^a   Wang, Jun ^a  

^a UNIVERSITY OF CENTRAL FLORIDA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ANALYSIS RESULTS; COMPRESSION METHODS; FILE SYSTEMS; LARGE SCALE DATA; MAIN MEMORY; METADATA MANAGEMENT; SCIENTIFIC APPLICATIONS; SINGLE SERVER; SMALL FILES; SPACE UTILIZATION;

CLUSTER COMPUTING; COMPUTER SCIENCE; TECHNICAL PRESENTATIONS;

METADATA;

EID: 72049093234     PISSN: 15525244     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CLUSTR.2009.5289133     Document Type: Conference Paper

References (17)

1. http://hadoop.apache.org/core/.
2. http://hadoop.apache.org/core/docs/r0.20.0/capacityscheduler.html.
3. http://hadoop.apache.org/core/docs/r0.20.0/fairscheduler.html.
4. http://hadoop.apache.org/core/docs/r0.20.0/hadooparchives.html.
5. https://issues.apache.org/jira/browse/hadoop-1687.
6. https://issues.apache.org/jira/browse/hadoop-3307.
7. http://www.cloudera.com/blog/2009/02/02/the-small-files-problem.
8. Apache. Introduction to Hadoop, 2008.
9. James K. Bonfield and Rodger Staden. ZTR: A New Format for DNA sequence trace data. Bioinformatics, 18(1):3-10, 2002.
10. Philip Carns, Sam Lang, Robert Ross, Murali Vilayannur, Julian Kunkel, and Thomas Ludwig. Small-File Access in Parallel File Systems. In Proceedings of the 23rd IEEE International Parallel and Distributed Processing Symposium, April 2009.
11. Ann Chervenak, Jennifer M. Schopf, Laura Pearlman, Mei-Hui Su, Shishir Bharathi, Luca Cinquini, Mike D'Arcy, Neill Miller, and David Bernholdt. Monitoring the Earth System Grid with MDS4. International Conference on e-Science and Grid Computing, 0:69, 2006.
12. Devaraj Das. Meet Hadoop! Open Source Grid Computing.
13. Jeffrey Dean and Sanjay Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. In Proceedings of the 6th Symposium on Operating System Design and Implementation (OSDI 04), Usenix Assoc., pages 137-150, 2004.
14. Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung. The Google File System. In SOSP '03: Proceedings of the nineteenth ACM symposium on Operating systems principles, pages 29-43, New York, NY, USA, 2003. ACM.
15. Eric H. Neilsen Jr. The Sloan Digital Sky Survey Data Archive Server. Computing in Science and Engineering, 10(1):13-17, 2008.
16. Grant Mackey, Saba Sehrish, Julio Lopez, John Bent, Salman Habib, and Jun Wang. Introducing Map-Reduce to High End Computing. In Petascale Data Storage Workshop at SC08, Austin, Texas, November 2008.
17. Michael C. Schatz. CloudBurst: Highly Sensitive Read Mapping with MapReduce. Bioinformatics, April 2009.