An optimized approach for storing and accessing small files on cloud storage

Journal of Network and Computer Applications

Volumn 35, Issue 6, 2012, Pages 1847-1862

  Dong, Bo ^a   Zheng, Qinghua ^a   Tian, Feng ^a   Chao, Kuo Ming ^b   Ma, Rui ^a   Anane, Rachid ^b  

^a XI'AN JIAOTONG UNIVERSITY   (China)

^b Institute for Clean Growth and Future Mobility   (United Kingdom)

Author keywords

Access efficiency; Cloud storage; Prefetching; Small file storage; Storage efficiency

Indexed keywords

CORRELATION FEATURES; DATA PLACEMENT; DISTRIBUTED FILE SYSTEMS; I/O PERFORMANCE; INTERNET SERVICES; PRE-FETCHING SCHEME; PREFETCHING; SMALL FILES; STORAGE AND ACCESS; STORAGE EFFICIENCY;

COMPUTER APPLICATIONS; COMPUTER NETWORKS;

OPTIMIZATION;

EID: 84867570478     PISSN: 10848045     EISSN: 10958592     Source Type: Journal    
DOI: 10.1016/j.jnca.2012.07.009     Document Type: Article

References (31)

1. S. Altschul, W. Gish, W. Miller, E. Myers, and D. Lipman Basic local alignment search tool Journal of Molecular Biology 215 3 1990 403 410
2. J. Bonfield, and R. Staden Ztr: a new format for dna sequence trace data Bioinformatics 18 1 2002 3
3. R. Buyya, C. Yeo, S. Venugopal, J. Broberg, and I. Brandic Cloud computing and emerging it platforms: vision, hype, and reality for delivering computing as the 5th utility Future Generation Computer Systems 25 6 2009 599 616
4. Chervenak A, Schopf J, Pearlman L, Su M, Bharathi S, Cinquini L, et al. Monitoring the earth system grid with mds4. In: second IEEE international conference on e-science and grid computing, 2006. e-Science'06. IEEE; 2006. p. 69.
5. J. Douceur, and W. Bolosky A large-scale study of file-system contents ACM SIGMETRICS Performance Evaluation Review 27 1 1999 59 70
6. D. Essary, and A. Amer Predictive data grouping: defining the bounds of energy and latency reduction through predictive data grouping and replication ACM Transactions on Storage (TOS) 4 1 2008 2
7. M. Gumińska, and J. Madejski Web based e-learning platform as a source of the personalized teaching materials Journal of Achievements in Materials and Manufacturing Engineering 24 2 2007 227 230
8. Hadoop. Hadoop official site; December 2011. 〈 http://hadoop.apache. org/ 〉.
9. Hadoop Archives. Hadoop archives guide; 2011. 〈 http://hadoop. apache.org/common/docs/current/hadoop-archives.html 〉.
10. Hendricks J, 2006. Improving small file performance in object-based storage. Technical report, DTIC Document.
11. Li M, Varki E, Bhatia S, Merchant A. Tap: Table-based prefetching for storage caches. In: Proceedings of the 6th USENIX conference on file and storage technologies. USENIX Association; 2008. p. 6.
12. Liu X, Han J, Zhong Y, Han C, He X, 2009. Implementing webgis on hadoop: a case study of improving small file i/o performance on hdfs. In: IEEE international conference on cluster computing and workshops, 2009. CLUSTER'09. IEEE; 2009. p. 1-8.
13. J. Lou, Q. Fu, Y. Wang, and J. Li Mining dependency in distributed systems through unstructured logs analysis ACM SIGOPS Operating Systems Review 44 1 2010 91 96
14. Mackey G, Sehrish S, Wang J. Improving metadata management for small files in hdfs. In: IEEE international conference on cluster computing and workshops, 2009. CLUSTER09. IEEE; 2009. p. 1-4.
15. MapFile. Mapfile api; 2011. 〈 http://hadoop.apache.org/common/docs/ current/api/org/apache/hadoop/io/MapFile.html 〉.
16. Min L, Yokota H. Comparing hadoop and fat-btree based access method for small file i/o applications. Web-age information management. Lecture notes in computer science, vol. 6184. Springer; 2010. p. 182-93.
17. E. Neilsen The sloan digital sky survey data archive server Computing in Science & Engineering 10 1 2008 13 17
18. NERSC. Petascale and data and storage institute nersc file system statistics; 2007. 〈 http://pdsi.nersc.gov/filesystem.htm 〉.
19. SequenceFile. Sequence file wiki; 2011. 〈 http://wiki.apache.org/ hadoop/SequenceFile 〉.
20. Shafer J, Rixner S, Cox A. The hadoop distributed filesystem: balancing portability and performance. In: IEEE international symposium on performance analysis of systems & software (ISPASS). IEEE; 2010. p. 122-33.
21. Shen C, Lu W, Wu J, Wei B. A digital library architecture supporting massive small files and efficient replica maintenance. In: Proceedings of the 10th annual joint conference on digital libraries. ACM; 2010. p. 391-2.
22. Shvachko K. Name-node memory size estimates and optimization proposal; 2007. 〈 https://issues.apache.org/jira/browse/HADOOP-1687 〉.
23. Shvachko K, Kuang H, Radia S, Chansler R. The hadoop distributed file system. In: IEEE 26th symposium on mass storage systems and technologies (MSST). IEEE; 2010. p. 1-10.
24. Soundararajan G, Mihailescu M, Amza C. Context-aware prefetching at the storage server. In: USENIX 2008 annual technical conference on annual technical conference. USENIX Association; 2008. p. 377-90.
25. Tantisiriroj W, Patil S, Gibson G. Data-intensive file systems for internet services: a rose by any other name. Technical report. UCB/EECS-2008-99. Parallel Data Laboratory; 2008.
26. J. Venner Pro Hadoop 2009 Springer
27. M. Vrable, S. Savage, and G. Voelker Cumulus: Filesystem backup to the cloud ACM Transactions on Storage (TOS) 5 4 2009 14
28. White T. The small files problem; 2009 〈 http://www.cloudera.com/ blog/2009/02/02/the-small-files-problem/ 〉.
29. T. White Hadoop: the definitive guide 2010 Yahoo Press
30. Xia P, Feng D, Jiang H, Tian L, Wang F. Farmer: a novel approach to file access correlation mining and evaluation reference model for optimizing peta-scale file system performance. In: Proceedings of the 17th international symposium on high performance distributed computing. ACM; 2008. p. 185-96.
31. Zhang Z, Kulkarni A, Ma X, Zhou Y. Memory resource allocation for file system prefetching: from a supply chain management perspective. In: Proceedings of the 4th ACM European conference on computer systems. ACM; 2009. p. 75-88.