A distributed cache for Hadoop Distributed File System in real-time cloud services

Proceedings - IEEE/ACM International Workshop on Grid Computing

Volumn , Issue , 2012, Pages 12-21

  Zhang, Jing ^a   Wu, Gongqing ^a   Hu, Xuegang ^a   Wu, Xindong ^a,b  

^a HEFEI UNIVERSITY OF TECHNOLOGY   (China)

^b University of Vermont   (United States)

Author keywords

Cloud storage; Distributed Cache System; HDFS; In memory cloud; Real time file access

Indexed keywords

CACHE SYSTEMS; CLIENT LIBRARIES; CLOUD SERVICES; CONCURRENT ENVIRONMENT; DISTRIBUTED CACHE; DISTRIBUTED FILE SYSTEMS; DISTRIBUTED HASH TABLE; FILE ACCESS; HDFS; MULTIPLE APPLICATIONS; NETWORK ENVIRONMENTS; SERVICE NODES; SHARED MEMORIES;

DISTRIBUTED DATABASE SYSTEMS; GRID COMPUTING; LOADING; WEB SERVICES;

CACHE MEMORY;

EID: 84869099437     PISSN: 15505510     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/Grid.2012.17     Document Type: Conference Paper

References (37)

1. Apache Hadoop. Available at http://hadoop.apache.org.
2. Apache Hadoop Distributed File System. Available at http://hadoop.apache. org/hdfs.
3. S. Ghemawat, H. Gobioff and S. T. Leung, "Google File System", In Proc. of the Nineteenth ACM Symposium on Operating Systems Principles (SOSP'03), Lake George New York, 2003, pp. 29-43.
4. J. Dean and S. Ghemawat, "Mapreduce: Simplified Data Processing on Large Clusters". In Proceedings of the 6th Conference on Symposium on Opearting Systems Design & Implementation (OSDI'04), Berkeley, CA, USA, 2004, pp.137-150.
5. J. Shafer and S Rixner, "The Hadoop distributed filesystem: balancing portability and performance", In 2010 IEEE International Symposium on Performance Analysis of Systems & Software (ISPASS2010), White Plains, NY, March 2010. pp.122-133.
6. J. Han, Y. Zhong, C. Han, X. He. "Implementing WebGIS on Hadoop: A case study of improving small file I/O performance on HDFS", In IEEE International Conference on Cluster Computing and Workshops (CLUSTER '09), New Orleans, LA, 2009. pp.1-8.
7. L. Jiang, B. Li, M. Song, "THE optimization of HDFS based on small files", In 3rd IEEE International Conference on Broadband Network and Multimedia Technology (IC-BNMT2010), Beijing, 2010. pp. 912-915.
8. G. Mackey, S. Sehrish, J. Wang, "Improving metadata management for small files in HDFS", In 2009 IEEE International Conference on Cluster Computing and Workshops (CLUSTER'09), New Orleans, Sept, 2009, pp.1-4.
9. J. Xie, S. Yin, et al. "Improving MapReduce performance through data placement in heterogeneous Hadoop clusters", In 2010 IEEE International Symposium on Parallel & Distributed Processing (IPDPSW), Atlanta, April, 2010, pp.1-9.
10. B. Dong, J. Qiu, et al. "A novel approach to improving the efficiency of storing and accessing small files on Hadoop: a case study by PowerPoint files", In 2010 IEEE International Conference on Service Computing (SCC), Miami, July 2010, pp.65-72.
11. H. Zhang, Y. Han, F. Chen and J. Wen, "A Novel Approach in Improving I/O Performance of Small Meteorological Files on HDFS", Applied Mechanics and Materials, vol.117-199, Oct. 2011, pp.1759-1765.
12. Apache HBase. Available at http://hbase.apache.org.
13. F. Chang, J, Dean, et al. "Bigtable: A Distributed Storage System for Structured Data", ACM Transactions on Computer Systems, Vol. 26(2), 2008, pp.205-218.
14. K. Dana, Hadoop HBase Performance Evaluation, unpublished, Available at http://www.cs.duke.edu/~kcd/hadoop/kcdhadoopreport.pdf.
15. J. Ousterhout, P. Agrawal, et al, "The Case for RAMClouds: Scalable High-Performance Storage Entirely in DRAM", ACM SIGOPS Operating Systems Review, Vol.43(4), Jan 2010, pp.92-105.
16. J. Ousterhout, P. Agrawal, et al,"The case for RAMCloud", Communications of the ACM, Vol.54(7), July 2011, pp.121-130.
17. Memcached: a distributed memory object caching system. Available at http://www.danga.com/memcached.
18. J. Petrovic, "Using Memcached for Data Distribution in Industrial Environment", In Third International Conference on Systems (ICONS08), Cancun, 2008, pp.358-372.
19. S. Zhang, J. Han, Z. Liu, K. Wang,"Accelerating MapReduce with Distributed Memory Cache ", In 15th International Conference on Parallel and Distributed Systems (ICPADS09), Shenzhen, 2009, pp.472-478.
20. C. Xu, X. Huang, N. Wu, P. Xu, G. Yang, "Using Memcached to Promote Read Throughput in Massive Small-File Storage System", In 9th Internatinal Conference on Grid and Cooperative Computing (GCC), Nanjing, 2010, pp.24-29.
21. Redis. Avialable at http://code.google.com/p/redis
22. D. Borthakur et al. "Apache Hadoop goes realtime at Facebook", In Proceedings of the 2011 International Conference on Management of Data (SIGMOD'11), New York, 2011.
23. S. Ekanayake, J. Mitchell, Y. Sun and J. Qiu, "Memcached Integration with Twister", unpublished, avialable at http://salsahpc.org/CloudCom2010/ EPoster/cloudcom2010-submission-264.pdf.
24. G. DeCandia, D.Hastorun, and M. Jampani, et al,"Dynamo: amazon's highly available key-value store", In Proceedings of twenty-first ACM SIGOPS Symposium on Operating Systems Principles, Stevenson, Washington, USA, October, 2007, pp.205-220. (Pubitemid 351429377)
25. G. Fowler, L. Noll, K Vo and D. Eastlake. "The FNV Non-Cryptographic Hash Algorithm", ietf-draft, 2011, Available at http://tools.ietf.org/ html/draft-eastlake-fnv-03.
26. W. Ehrhardt, "CRC Hash", Available at http://www. wolfgangehrhardt.de/crchash-en.html
27. Ketama Hash. Available at http://www.audioscrobbler.net/development/ ketama.
28. P. Hunt, M. Konar, F. P. Junqueira, and B. Reed, "ZooKeeper: waitfree coordination for internet-scale systems", In Proceedings of the 2010 USENIX Conference on USENIX Annual Technical Conference, Boston, MA, June 23-25, 2010.
29. Hadoop ZooKeeper. Available at http://zookeeper.apache.org/.
30. M. Burrows, "The Chubby lock service for loosely-coupled distributed systems", In Proceedings of the 7th Symposium on Operating Systems Design and Implementation (OSDI'06), 2006, pp.335-350.
31. B. Reed and F. P. Junqueira. "A simple totally ordered broadcast protocol". In LADIS'08: Proceedings of the 2nd Workshop on Large-Scale Distributed Systems and Middleware, pages 1-6, New York, NY, USA, 2008. ACM.
32. Hadoop Distributed File System C/C++ APIs Documnet, Available at http://hadoop.apache.org/common/docs/current/libhdfs.html.
33. The Apache Cassandra Project. Available at http://http://cassandra. apache.org/.
34. The Mongo Database Project. Available at http://www.mongodb.org/.
35. B. F. Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan and Russell Sears. "Benchmarking cloud serving systems with YCSB". In Proceedings of the 1st ACM Symposium on Cloud Computing (SoCC'10), Indianapolis, IN, 2010. pp.143-154.
36. Robin Hecht and Stefan Jablonski. "A Use Case Oriented Survey". In 2011 International Conference on Cloud and Service Computing (CSC2011), Hong Kong, pp.336 - 341.
37. Luiz A. Barroso, Jeffrey Dean, and Urs Holzle. Web search for a planet: The Google cluster architecture. IEEE Micro, April 2003, 23(2), pp.22-28.