BitDew: A data management and distribution service with multi-protocol file transfer and metadata abstraction

Journal of Network and Computer Applications

Volumn 32, Issue 5, 2009, Pages 961-975

  Fedak, Gilles ^a,b   He, Haiwu ^a,b   Cappello, Franck ^a,b  

^a INRIA SACLAY   (France)

^b UFR 919 Laboratoire d'Informatique Pour la Mécanique et les Sciences de l'Ingénieur   (France)

Author keywords

Cloud computing; Content network; Desktop Grid; P2P

Indexed keywords

APPLICATION PROGRAMMERS; BIOINFORMATICS APPLICATIONS; BITTORRENT; CLIENT/SERVER; CLOUD COMPUTING; CONTENT NETWORK; DATA DISTRIBUTION; DATA MANAGEMENT; DATA MANAGEMENT SERVICES; DESKTOP GRID; DISTRIBUTED APPLICATIONS; DISTRIBUTED DATA; DISTRIBUTED HASH TABLE; DISTRIBUTED SERVICE; DISTRIBUTION SERVICES; FILE TRANSFER PROTOCOLS; FILE TRANSFERS; GENERAL APPROACH; GRID CONTEXT; HIGH LEVEL OF ABSTRACTION; ITS ARCHITECTURE; MULTIPROTOCOL; NEW CLASS; P2P; PERFORMANCE EVALUATION; PLUG-IN; PROGRAMMING INTERFACE; RUNTIME; RUNTIME ENVIRONMENTS; STORAGE RESOURCES; STORAGE SERVICES; TRANSPORT PROTOCOLS; USAGE SCENARIOS; WIDE AREA STORAGE;

ABSTRACTING; BENCHMARKING; BIOINFORMATICS; DATA COMMUNICATION SYSTEMS; FAULT TOLERANT COMPUTER SYSTEMS; GRID COMPUTING; INTERNET; INTERNET PROTOCOLS; MANAGEMENT; METADATA; NETWORK ARCHITECTURE; QUALITY ASSURANCE;

FAULT TOLERANCE;

EID: 67649831348     PISSN: 10848045     EISSN: 10958592     Source Type: Journal    
DOI: 10.1016/j.jnca.2009.04.002     Document Type: Article

References (47)

1. Adya A., et al. Farsite: federated, available, and reliable storage for an incompletely trusted environment. SIGOPS Operating Systems Review 36 SI (2002) 1-14
2. Allcock W, Bresnahan J, Kettimuthu R, Link M, Dumitrescu C, Raicu I, et al. The globus striped GridFTP framework and server. In: Proceedings of the super computing (SC05), Seattle, USA, 2005.
3. Amazon s3: Developer guide; March 2006 〈http://developer.amazonwebservices.com/〉.
4. Amazon web services 〈http://s3.amazonaws.com〉.
5. Anderson R. The eternity service. In: Proceedings of Pragocrypt '96, Prague, Czech, 1996.
6. Anderson D. BOINC: a system for public-resource computing and storage. In: Proceedings of the 5th IEEE/ACM international GRID workshop, Pittsburgh, USA, 2004.
7. Antonioletti M., et al. The design and implementation of grid database services in OGSA-DAI. Concurrency and Computation: Practice and Experience 17 (2005) 357-376
8. Antoniu G., Boug L., and Jan M. JuxMem: an adaptive supportive platform for data sharing on the grid. Scalable Computing: Practice and Experience 6 3 (2005) 45-55
9. Bassi A, Beck M, Fagg G, Moore T, Plank JS, Swany M, et al. The internet BackPlane protocol: A study in resource sharing. In: Second IEEE/ACM international symposium on cluster computing and the grid, Berlin, Germany, 2002.
10. Bolze R, et al. Grid5000: a large scale highly reconfigurable experimental grid testbed. International Journal on High Performance Computing and Applications 2006;20(4):481-94.
11. Butt AR, Johnson TA, Zheng Y, Charlie Hu Y. Kosha: A peer-to-peer enhancement for the network file system. In: Proceeding of international symposium on supercomputing SC'04, Pittsburgh, PA, 2004.
12. Cirne W., Brasileiro F., Andrade N., Costa L., Andrade A., Novaes R., and Mowbray M. Labs of the World, unite!!!. Journal of Grid Computing 4 3 (2006) 225-246
13. Cohen B. Incentives build robustness in BitTorrent. In: Workshop on economics of peer-to-peer systems, Berkeley, CA, 2003.
14. Deelman E, Singh G, Atkinson MP, Chervenak A, Chue Hong NP, Kesselman C, et al. Grid-based metadata services. In: SSDBM04, Santorini, Greece, June 2004.
15. Enabling Grids for E-Science in Europe 〈http://www.eu-egee.org〉.
16. Fedak G, Germain C, Néri V, Cappello F. Xtremweb: a generic global computing platform. In: Proceedings of CCGRID'2001 special session global computing on personal devices, Brisbane, Australia, May 2001. IEEE/ACM, IEEE Press; 2001.
17. Fedak G, He H, Cappello F. BitDew: a programmable environment for large-scale data management and distribution. In: Proceedings of the ACM/IEEE supercomputing conference (SC'08), Austin, USA, November 2008a.
18. Fedak G., He H., and Cappello F. A file transfer service with client/server, P2P and wide area storage protocols. Proceedings of the first international conference on data management in Grid and P2P systems, Globe'08. Lecture notes in computer science, Turin (2008)
19. Fernandess Y, Malkhi D. On collaborative content distribution using multi-message gossip. In: Proceeding of the IEEE IPDPS, Rhodes Island, 2006.
20. Garfinkel SL. An evaluation of Amazon's grid computing services: EC2, S3 and SQS. Technical Report TR-08-07, Harvard University Cambridge, MA; 2007.
21. Gelernter D. Generative communications in Linda. ACM transactions on programming languages and systems, 1985.
22. Gkantsidis C, Rodriguez P. Network coding for large scale content distribution. In: Proceedings of the IEEE/INFOCOM 2005, Miami, USA, March 2005.
23. Gkantsidis C, Miller J, Rodriguez P. Comprehensive view of a live network coding P2P system. In: ACM SIGCOMM/USENIX IMC'06, Brazil, 2006a.
24. Gkantsidis C, Miller J, Rodriguez P. Anatomy of a p2p content distribution system with network coding. In: IPTPS'06, CA, USA, 2006b.
25. Iamnitchi A., Doraimani S., and Garzoglio G. Filecules in high-energy physics: characteristics and impact on resource management. Proceeding of 15th IEEE international symposium on high performance distributed computing HPDC 15, Paris (2006)
26. Jin H, Xiong M, Wu S, Zou D. Replica based distributed metadata management in grid environment. In: Computational science-lecture notes in computer science, vol. 3994. Berlin: Springer; 2006. p. 1055-62.
27. Keahey K, Doering K, Foster I. From Sandbox to Playground: dynamic virtual environments in the grid. In: 5th international workshop in grid computing (Grid 2004), Pittsburgh, 2004.
28. Kondo D, Chien A, Casanova H. Resource management for rapid application turnaround on enterprise desktop grids. In: ACM conference on high performance computing and networking (SC'04), Pittsburgh, 2004.
29. Kondo D., Araujo F., Malecot P., Domingues P., Silva L.M., Fedak G., et al. Characterizing result errors in internet desktop grids. Proceedings of the EuroPar'07, Rennes, France (August 2007)
30. Kosar T, Livny M. Stork: making data placement a first class citizen in the grid. In: Proceedings of the 24th IEEE international conference on distributed computing systems (ICDCS 2004), Tokyo, Japan, March 2004.
31. Kubiatowicz J., et al. OceanStore: an architecture for global-scale persistent storage. Proceedings of the ACM ASPLOS (November 2000), ACM
32. Litzkow M.J., Livny M., and Mutka M.W. Condor-a hunter of idle workstations. Proceedings of the 8th international conference on distributed computing systems (1988), IEEE Computer Society, Washington, DC 104-111
33. Luna J., Flouris M., Marazakis M., and Bilas A. Providing security to the desktop data grid. 2nd workshop on desktop grids and volunteer computing systems (PCGrid'08) (2008)
34. Maymounkov P., and Mazires D. Kademlia: a peer-to-peer information system based on the XOR metric. Proceedings of the 1st international workshop on peer-to-peer systems (IPTPS'02) (2002), MIT, Cambridge, MA, USA
35. Onana Alima L, El-Ansary S, Brand P, Haridi S. DKS(N, k, f) a family of low-communication, scalable and fault-tolerant infrastructures for P2P applications. In: The 3rd international CGP2P workshop, Tokyo, Japan, 2003.
36. Otoo E., Rotem D., and Romosan A. Optimal file-bundle caching algorithms for data-grids. SC '04: Proceedings of the 2004 ACM/IEEE conference on supercomputing (2004), IEEE Computer Society, Washington, DC, USA 6
37. Palankar M., Onibokun A., Iamnitchi A., and Ripeanu M. Amazon S3 for science grids: a viable solution ?. 4th USENIX symposium on networked systems design and implementation (NSDI'07) (2007)
38. PPDG. From fabric to physics. Technical Report, The Particle Physics Data Grid; 2006 〈http://www.ppdg.net/〉.
39. Rowstron A, Druschel P. Pastry: scalable, distributed object location and routing for large-scale peer-to-peer systems. In: IFIP/ACM international conference on distributed systems platforms (middleware), Heidelberg, Germany, 2001.
40. Reinefeld A, Schintke F, Schatt T. Scalable and self-optimizing data grids. Annual Review of Scalable Computing, Singapore University Press 2004;6:30-60.
41. Santos-Neto E., Cirne W., Brasileiro F., and Lima A. Exploiting replication and data reuse to efficiently schedule data-intensive applications on grids. Proceedings of the 10th workshop on job scheduling strategies for parallel processing (2004)
42. Sarmenta L.F.G. Sabotage-tolerance mechanisms for volunteer computing systems. Future Generation Computer Systems 18 4 (2002) 561-572
43. Singh G, Bharathi S, Chervenak A, Deelman E, Kesselman C, Manohar M, et al. A metadata catalog service for data intensive applications. In: Proceedings of the SuperComputing'03, Phoenix, Arizona, USA, November 2003.
44. Stoica I, Morris R, Karger D, Frans Kaashoek M, Balakrishnan H. Chord: a scalable peer-to-peer lookup service for internet applications. In: Proceedings of the ACM SIGCOMM '01 conference, San Diego, CA, August 2001.
45. Tatebe O, Morita Y, Matsuoka S, Soda N, Sekiguchi S. Grid datafarm architecture for petascale data intensive computing. In: Proceedings of the 2nd IEEE/ACM symposium on cluster computing and the grid (CCGrid'02), Berlin, Germany, 2002.
46. Vazhkudai S, Freeh V, Ma X, Strickland J, Tammineedi N, Scott SL. FreeLoader: scavenging desktop storage resources for scientific data. In: Proceedings of the Supercomputing 2005 (SC'05), Seattle, 2005.
47. Wei B, Fedak G, Cappello F. Scheduling independent tasks sharing large data distributed with BitTorrent. In: 6th IEEE/ACM international workshop on grid computing, Seattle, USA, November 2005. p. 219-26.