Data pipelines: Enabling large scale multi-protocol data transfers

ACM International Conference Proceeding Series

Volumn 76, Issue , 2004, Pages 63-68

  Kosar, Tevfik ^a   Kola, George ^a   Livny, Miron ^a  

^a University of Wisconsin Madison   (United States)

Author keywords

Bulk data transfers; Data pipelines; Distributed systems; Fault tolerance; Grid; Mass storage systems; Replication

Indexed keywords

AUTOMATED SYSTEMS; BULK DATA TRANSFER; DATA PIPELINES; DISTRIBUTED SYSTEMS; HARDWARE FAILURES; HUMAN INTERVENTION; MASS STORAGE; MASS STORAGE SYSTEM; MULTIPROTOCOLS; SAN DIEGO; STORAGE SYSTEMS; SUPERCOMPUTING CENTERS; WHOLE PROCESS;

AUTOMATION; DATA STORAGE EQUIPMENT; DATA TRANSFER; FAULT TOLERANCE; MIDDLEWARE; NETWORK PROTOCOLS; PIPELINES; QUALITY ASSURANCE;

GRID COMPUTING;

EID: 77953001239     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1028493.1028504     Document Type: Conference Paper

References (22)

1. B. Allcock, J. Bester, J. Bresnahan, A. Chervenak, I. Foster, C. Kesselman, S. Meder, V. Nefedova, D. Quesnel, and S. Tuecke. Secure, efficient data transport and replica management for high-performance data-intensive computing. In IEEE Mass Storage Conference, San Diego, CA, April 2001.
2. C. Baru, R. Moore, A. Rajasekar, and M. Wan. The SDSC Storage Resource Broker. In Proceedings of CASCON, Toronto, Canada, 1998.
3. M. Butler, R. Pennington, and J. A. Terstriep. Mass Storage at NCSA: SGI DMF and HP UniTree. In Proceedings of 40th Cray User Group Conference, 1998.
4. Condor. The Directed Acyclic Graph Manager. http://www.cs.wisc.edu/ condor/dagman, 2003.
5. S. G. Djorgovski, R. R. Gal, S. C. Odewahn, R. R. de Carvalho, R. Brunner, G. Longo, and R. Scaramella. The Palomar Digital Sky Survey (DPOSS). Wide Field Surveys in Cosmology, 1988.
6. W. Feng. High Performance Transport Protocols. Los Alamos National Laboratory, 2003.
7. FNAL. Enstore mass storage. http://hppc.fnal.gov/enstore/, 2003.
8. I. Foster, C. Kesselman, and S. Tuecke. The Anatomy of the Grid: Enabling Scalable Virtual Organizations. International Journal of Supercomputing Applications, 2001.
9. I. Foster and C. Kesselmann. Globus: A Toolkit-Based Grid Architecture. In The Grid: Blueprints for a New Computing Infrastructure, pages 259-278, Morgan Kaufmann, 1999.
10. J. Frey, T. Tannenbaum, I. Foster, and S. Tuecke. Condor-G: A Computation Management Agent for Multi-Institutional Grids. In Tenth IEEE Symp. on High Performance Distributed Computing, San Francisco, CA, August 2001.
11. G. Kola and M. Livny. Diskrouter: A Flexible Infrastructure for High Performance Large Scale Data Transfers. Technical Report CS-TR-2003-11484, University of Wisconsin, 2003.
12. S. Koranda and B. Moe. Lightweight Data Replicator. http://www.lsc-group. phys.uwm.edu/lscdatagrid/LDR.
13. T. Kosar and M. Livny. Stork: Making Data Placement a First Class Citizen in the Grid. In Proceedings of the 24th International Conference on Distributed Computing Systems (ICDCS 2004), Tokyo, Japan, March 2004.
14. LBNL. High Performance Storage System. http://www.nersc.gov/nusers/ resources/HPSS/, 2003.
15. S.-Y. R. Li, R. W. Yeung, and N. Cai. Linear network coding. IEEE transactions on information theory,, 49(2):371-380, February 2003.
16. LIGO. Laser Interferometer Gravitational Wave Observatory. http://www.ligo.caltech.edu/, 2003.
17. M. J. Litzkow, M. Livny, and M. W. Mutka. Condor - A Hunter of Idle Workstations. In Proceedings of the 8th International Conference of Distributed Computing Systems, pages 104-111, 1988.
18. R. Madduri and B. Allcock. Reliable File Transfer Service. http://www-unix.mcs.anl.gov/ madduri/main.html, 2003.
19. V. Paxson. End-to-End Internet Packet Dynamics. IEEE/ACM Transactions on Networking, 7(3):277-292, 1999.
20. J. H. Saltzer, D. P. Reed, and D. D. Clark. End-to-end arguments in system design. ACM Transactions on Computer Systems, 2(4):277-288, nov 1984.
21. D. Thain, J. Basney, S. Son, and M. Livny. The kangaroo approach to data movement on the grid. In Proceedings of the Tenth IEEE Symposium on High Performance Distributed Computing, San Francisco, California, August 2001.
22. D. Thain, T. Tannenbaum, and M. Livny. Condor and the Grid. In Grid Computing: Making the Global Infrastructure a Reality., Fran Berman and Geoffrey Fox and Tony Hey, editors. John Wiley and Sons Inc., 2002.