BlobSeer: Bringing high throughput under heavy concurrency to hadoop map-reduce applications

Proceedings of the 2010 IEEE International Symposium on Parallel and Distributed Processing, IPDPS 2010

Volumn , Issue , 2010, Pages

  Nicolae, Bogdan ^a   Moise, Diana ^b   Antoniu, Gabriel ^b   Bougé, Luc ^b   Dorier, Matthieu ^b  

^a UNIVERSITÉ DE RENNES 1   (France)

^b CAMPUS DE BEAULIEU   (France)

Author keywords

BlobSeer; Data intensive; Distributed file systems; Hadoop; Heavy access concurrency; High Throughput; Large scale distributed computing; Map Reduce based application

Indexed keywords

DATA ACCESS; DATA MANAGEMENT SERVICES; DATA STORAGE; DISTRIBUTED COMPUTING; DISTRIBUTED FILE SYSTEMS; HADOOP; HIGH THROUGHPUT; PROGRAMMING MODELS; SOFTWARE FRAMEWORKS; STORAGE SYSTEMS; VERSIONING;

DISTRIBUTED PARAMETER NETWORKS;

THROUGHPUT;

EID: 77953991459     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/IPDPS.2010.5470433     Document Type: Conference Paper

References (17)

1. J. Dean and S. Ghemawat, "MapReduce: simplified data processing on large clusters," Communications of the ACM, vol.51, no.1, pp. 107-113, 2008.
2. "The Apache Hadoop Project," http://www.hadoop.org.
3. "HDFS. The Hadoop Distributed File System," http://hadoop.apache.org/common/docs/r0.20.1/hdfs-design.html.
4. S. Ghemawat, H. Gobioff, and S.-T. Leung, "The google file system," SIGOPS - Operating Systems Review, vol.37, no.5, pp. 29-43, 2003.
5. "Kosmix CloudStore Distributed File System," http://kosmosfs.sourceforge.net/index.html.
6. "Amazon Elastic Map Reduce," http://aws.amazon.com/ elasticmapreduce/.
7. "Amazon Elastic Compute Cloud (EC2)," http://aws.amazon.com/ ec2/.
8. "Amazon Simple Storage Service (S3)," http://aws.amazon.com/s3/ .
9. F. B. Schmuck and R. L. Haskin, "GPFS: A shared-disk file system for large computing clusters," in FAST '02: Proceedings of the Conference on File and Storage Technologies. USENIX Association, 2002, pp. 231-244. [Online]. Available: http://portal.acm.org/citation.cfm?id=651312
10. PVFS, "Parallel virtual file system, version 2," http://pvfs2.org/.
11. G. G. Wittawat Tantisiriroj, Swapnil Patil, "Data-intensive file systems for internet services: A rose by any other name. " Parallel Data Laboratory, Tech. Rep. UCB/EECS-2008-2099, October 2008.
12. B. Nicolae, G. Antoniu, and L. Bouge, "BlobSeer: How to enable efficient versioning for large object storage under heavy access concurrency," in Proc. 2nd Workshop on Data Management in Peer-to-Peer Systems (DAMAP'2009), Saint Petersburg, Russia, Mar. 2009, held in conjunction with EDBT'2009. [Online]. Available: http://hal.archives-ouvertes.fr/inria- 00382354/en/
13. -, "Enabling high data throughput in desktop grids through decentralized data and metadata management: The blobseer approach," in Proc. 15th International Euro-Par Conference on Parallel Processing (Euro-Par '09), ser. Lect. Notes in Comp. Science, vol.5704. Delft, The Netherlands: Springer-Verlag, 2009, pp. 404-416. [Online]. Available: http://hal.archives- ouvertes.fr/inria-00410956/en/
14. C. Zheng, G. Shen, S. Li, and S. Shenker, "Distributed Segment Tree: Support range query and cover query over DHT," in Proceedings of the Fifth International Workshop on Peer-to-Peer Systems (IPTPS), Santa Barbara, California, 2006.
15. M. P. Herlihy and J. M. Wing, "Linearizability: a correctness condition for concurrent objects," ACM Trans. Program. Lang. Syst., vol.12, no.3, pp. 463-492, 1990.
16. Y. Jégou, S. Lantéri, J. Leduc, M. Noredine, G. Mornet, R. Namyst, P. Primet, B. Quetier, O. Richard, E.-G. Talbi, and T. Iréa, "Grid'5000: a large scale and highly reconfigurable experimental grid testbed." International Journal of High Performance Computing Applications, vol.20, no.4, pp. 481-494, November 2006.
17. M. Zaharia, A. Konwinski, A. D. Joseph, R. H. Katz, and I. Stoica, "Improving mapreduce performance in heterogeneous environments," EECS Department, University of California, Berkeley, Tech. Rep. UCB/EECS-2008-2099, Aug 2008.