Managing very large distributed data sets on a data grid

Concurrency and Computation: Practice and Experience

Volumn 22, Issue 11, 2010, Pages 1338-1364

  Branco, Miguel ^a,b   Zaluska, Ed ^b   De RourE , David ^b   Lassnig, Mario ^a,c   GaronnE , Vincent ^a  

^a CERN   (Switzerland)

^b UNIVERSITY OF SOUTHAMPTON   (United Kingdom)

^c UNIVERSITY OF INNSBRUCK   (Austria)

Author keywords

Data management; Distributed systems; Grid computing; Modelling; Simulation

Indexed keywords

DISTRIBUTED DATABASE SYSTEMS; INFORMATION MANAGEMENT; MIDDLEWARE; MODELS; UNCERTAINTY ANALYSIS;

DATA DISTRIBUTION ALGORITHM; DEGREE OF UNCERTAINTY; DISTRIBUTED SYSTEMS; FUTURE RESEARCH DIRECTIONS; LARGE HADRON COLLIDER LHC; MODELLING AND SIMULATIONS; OPERATIONAL EXPERIENCE; SIMULATION;

GRID COMPUTING;

EID: 77954492785     PISSN: 15320626     EISSN: 15320634     Source Type: Journal    
DOI: 10.1002/cpe.1489     Document Type: Article

References (43)

1. Gray J, Liu D, Nieto-Santisteban M, Szalay A, DeWitt D, Heber G. Scientific data management in the coming decade. SIGMOD Record 2005; 34(4):34-41.
2. Hsieh W, Madhavan J, Pike R. Data management projects at Google. SIGMOD Record 2008; 37(1):34-38.
3. Chervenak A, Foster I, Kesselman C, Salisbury C, Tuecke S. The data grid: Towards an architecture for the distributed management and analysis of large scientific data sets. Journal of Network and Computer Applications 2000; 23(3): 187-200.
4. The ATLAS Collaboration. Available at: http://atlasexperiment.org/ [Retrieved on 4 August 2009].
5. Branco M, Zaluska E, Roure D, Salgado P, Garonne V, Lassnig M, Rocha R. Managing Very-large Distributed Data sets (Lecture Notes in Computer Science, vol.5331/2008). Springer: New York, 2008; 775-792.
6. Sandberg R, Golgberg D, Kleiman S, Walsh D, Lyon B. Design and implementation of the sun network filesystem. Proceedings of the Summer 1985 USENIX Conference, Phoenix, AZ, U.S.A., 1985; 119-130.
7. Howard J, Kazar M, Menees S, Nichols D, Satyanarayanan M, Sidebotham R, West M. Scale and performance in a distributed file system. ACM Transactions on Computer Systems 1988; 6(1):51-81. (Pubitemid 18574134)
8. Satyanarayanan M, Kistler J, Kumar P, Okasaki M, Siegel E, Steere D. Coda: A highly available file system for a distributed workstation environment. IEEE Transactions on Computers 1990; 39(4):447-459. (Pubitemid 20702255)
9. Schmuck F, Haskin R. GPFS: A shared-disk file system for large computing clusters. Proceedings of the 2002 Conference on File and Storage Technologies, Monterey, CA, U.S.A., 2002; 231-244.
10. Andrews P, Kovatch P, Jordan C. Massive high-performance global file systems for grid computing. Proceedings of the ACM/IEEE SC 2005 Conference, Seattle, WA, U.S.A., 2005.
11. Schwan P. Lustre: Building a file system for 1000-node clusters. Proceedings of the 2003 Linux Symposium, Ottawa, Canada, 2003.
12. Weil S, Brandt S, Miller E, Long D, Maltzahn C. Ceph: A scalable, high-performance distributed file system. Proceedings of the 7th Conference on Operating Systems Design and Implementation, Seattle, WA, U.S.A., 2006; 307-320.
13. Ghemawat S, Gobioff H, Leung ST. The google file system. SIGOPS Operating Systems Review 2003; 37(5):29-43.
14. Apache Hadoop Project. Available at: http://hadoop.apache.org/core/ [Retrieved on 4 August 2009].
15. White B, Walker M, Humphrey M, Grimshaw A. LegionFS: A secure and scalable file system supporting cross-domain high-performance applications. Proceedings of the ACM/IEEE SC 2001 Conference, Denver, CO, U.S.A., 2001.
16. Cohen B. Incentives build robustness in BitTorrent. Workshop on Economics of Peer-to-Peer Systems, Berkeley, CA, U.S.A., 2003.
17. Lv Q, Cao P, Cohen E, Li K, Shenker S. Search and replication in unstructured peer-to-peer networks. Proceedings of the 16th International Conference on Supercomputing, New York, NY, U.S.A., 2002; 84-95. (Pubitemid 35039987)
18. Cohen E, Shenker S. Replication strategies in unstructured peer-to-peer networks. Proceedings of the 2002 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications, Pittsburgh, PA, U.S.A., 2002; 177-190. (Pubitemid 43843783)
19. Muthitacharoen A, Morris R, Gil TM, Chen B. Ivy: Read/write peer-to-peer file system. SIGOPS Operating Systems Review 2002; 36(SI):31-44.
20. Chun BG, Dabek F, Haeberlen A, Sit E, Weatherspoon H, Kaashoek M, Kubiatowicz J, Morris R. Efficient replica maintenance for distributed storage systems. Proceedings of the 3rd Conference on Networked Systems Design and Implementation, San Jose, CA, U.S.A., vol.3, 2006; 45-58.
21. Bhagwan R, Tati K, Cheng Y, Savage S, Voelker G. Total recall: System support for automated availability management. Proceedings of the 1st Conference on Symposium on Networked Systems Design and Implementation, San Francisco, CA, U.S.A., 2004.
22. Risson J, Moors T. Survey of research towards robust peer-to-peer networks: Search methods. Computer Networks 2006; 50(17):3485-3521.
23. Bester J, Foster I, Kesselman C, Tedesco J, Tuecke S. GASS: A data movement and access service for wide-area computing systems. Proceedings of the 6th Workshop on I/O in Parallel and Distributed Systems, Atlanta, GA, U.S.A., 1999; 78-88.
24. Samar A, Stockinger H. Grid data management pilot (GDMP): A tool for wide area replication. IASTED International Conference on Applied Informatics, Innsbruck, Austria, 2001.
25. Allcock W, Bester J, Bresnahan J, Chervenak A, Foster I, Kesselman C, Meder S, Nefedova V, Quesnel D, Tuecke S. Data management and transfer in high-performance computational grid environments. Parallel Computing 2002; 28(5):749-771.
26. Chervenak A, Deelman E, Foster I, Guy L, Hoschek W, Iamnitchi A, Kesselman C, Kunszt P, Ripeanu M, Schwartzkopf R, Stockinger H, Stockinger K, Tierney B. Giggle: A framework for constructing scalable replica location services. Proceedings of the ACM/IEEE SC 2002 Conference, Baltimore, MD, U.S.A., 2002; 1-17.
27. Lamehamedi H, Szymanski B, Shentu Z, Deelman E. Data replication strategies in grid environments. Proceedings of the 5th International Conference on Algorithms and Architectures for Parallel Processing, Beijing, China, 2002; 378-383.
28. Baru C, Moore R, Rajasekar A, Wan M. The SDSC storage resource broker. Proceedings of the 1998 Conference of the Centre for Advanced Studies on Collaborative Research, Toronto, Ontario, Canada, 1998.
29. Tatebe O, Sekiguchi S, Morita Y. Gfarm v2: A grid file system that supports high-performance distributed and parallel data computing. Available at: http://datafarm.apgrid.org/pdf/CHEP04-gfarmv2.pdf [24 June 2009].
30. Chervenak A, Schuler R, Kesselman C, Koranda S, Moe B. Wide area data replication for scientific collaborations. International Journal of High Performance Computing and Networking 2008; 5(3):124-134.
31. Bell W, Cameron D, Capozza L, Millar A, Stockinger K, Zini F. Simulation of Dynamic Grid Replication Strategies in OptorSim (Lecture Notes in Computer Science, vol.2536/2002). Springer: New York, 2002; 46-57.
32. Casanova H, Legrand A, Quinson M. SimGrid: A generic framework for large-scale distributed experiments. Computer Modeling and Simulation 2008; 1(3):126-131.
33. Sulistio A, Cibej U, Venugopal S, Robic B, Buyya R. A toolkit for modelling and simulating data grids: An extension to GridSim. Concurrency and Computation: Practice and Experience 2008; 20(13):1591-1609.
34. Shoshani A, Sim A, Gu J. Storage resource managers: Middleware components for grid storage. Proceedings of the 19th IEEE Symposium on Mass Storage Systems, College Park, MD, U.S.A., 2002.
35. Fielding R. Architectural styles and the design of network-based software architectures. PhD Thesis, University of California, 2000.
36. Vogels W. Eventually consistent. Communications of the ACM 2009; 52(1):40-44.
37. Kunszt P, Badino P. Frohner A, McCance G, Nienartowicz K, Rocha R, Rodrigues D. Data storage, access and catalogs in gLite. Local to Global Data Interoperability-Challenges and Technologies, Sardinia, Italy, 2005; 166-170. Available at: http://ieeexplore.ieee.org/xpls/abs all.jsp?tp=&arnumber= 1612487&isnumber=33856.
38. Bird I, Jones B, Kee K. The organization and management of grid infrastructures. IEEE Computer 2009; 42(1):36-46.
39. Adams D et al. The ATLAS Computing Model. Available at: http://cdsweb.cern.ch/record/811058 [2004].
40. Jain R. The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling. Wiley: New York, 1991.
41. Givens G, Hoeting J. Computational Statistics. Wiley: New York, 2005.
42. Dempster A, Laird N, Rubin D. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society 1977; 39(1):1-38.
43. Schwarz G. Estimating the dimension of a model. The Annals of Statistics 1978; 6(2):461-464.