Building reliable and efficient data transfer and processing pipelines

Concurrency and Computation: Practice and Experience

Volumn 18, Issue 6, 2006, Pages 609-620

  Kosar, T ^a   Kola, G ^a   Livny, M ^a  

^a University of Wisconsin Madison   (United States)

Author keywords

Data intensive computing; Data pipelines; Data transfer; Distributed systems; Fault tolerance; Grid computing; Workflows

Indexed keywords

COMPUTATIONAL METHODS; DATA ACQUISITION; DATA PROCESSING; DATA STORAGE EQUIPMENT; DISTRIBUTED COMPUTER SYSTEMS; FAULT TOLERANT COMPUTER SYSTEMS; INTERFACES (COMPUTER); WIDE AREA NETWORKS;

DATA INTENSIVE COMPUTING; DATA PIPELINES; GRID COMPUTING; WORKFLOWS;

DATA TRANSFER;

EID: 33646191487     PISSN: 15320626     EISSN: 15320634     Source Type: Journal    
DOI: 10.1002/cpe.969     Document Type: Conference Paper

References (24)

1. Foster I, Kesselman C, Tuecke S. The anatomy of the Grid: Enabling scalable virtual organizations. International Journal of Supercomputer Applications 2001; 15(3):200-222.
2. Butler M. Pennington R, Terstriep JA. Mass storage at NCSA: SGI DMF and HP UniTree. Proceedings of the 40th Cray-User Group Conference, 1998.
3. Baru C, Moore R, Rajasekar A, Wan M. The SDSC Storage Resource Broker. Proceedings of the Centers for Advanced Studies Conference (CASCON'98), Toronto, Canada, 1998.
4. LBNL. High performance storage system, 2003. http://www.nersc.gov/nusers/ resources/HPSS/.
5. FNAL. Enstore mass storage, 2003. http://hppc.fnal.gov/enstore/.
6. Djorgovski SG, Gal RR, Odewahn SC, de Carvalho RR, Brunner R, Longo G, Scaramella R. The Palomar Digital Sky Survey (DPOSS). Wide Field Surveys in Cosmology, 1988.
7. Thorn C. Creating new histories of learning for Math and science instruction: Using NVivo and Transana to manage and study large multimedia datasets. Proceedings of the Conference on Strategies in Qualitative Research: Methodological Issues and Practices in Using QSR NVivo and NUD, London, February 2004.
8. Paxson V. End-to-end internet packet dynamics. IEEE/ACM Transactions on Networking 1999; 7(3):277-292.
9. Saltzer JH, Reed DP, Clark DD. End-to-end arguments in system design. ACM Transactions on Computer Systems 1984; 2(4):277-288.
10. Kosar T. Livny M. Stork: Making data placement a first class citizen in the Grid. Proceedings of the 24th International Conference on Distributed Computing Systems (ICDCS 2004), Tokyo, Japan, March 2004.
11. Li S-YR, Yeng RW, Cai N. Linear network coding. IEEE Transactions on Information Theory 2003; 49(2):371-380.
12. Litzkow MJ, Livny M, Mutka MW. Condor - a hunter of idle workstations. Proceedings of the 8th International Conference of Distributed Computing Systems, 1988; 104-111.
13. Frey J, Tannenbaum T, Foster I, Tuecke S. Condor-G: A computation management agent for multi-institutional Grids. Proceedings of the 10th IEEE Symposium on High Performance Distributed Computing, San Francisco, CA, August 2001.
14. Foster I, Kesselmann C. Globus: A toolkit-based Grid architecture. The Grid: Blueprints for a New Computing Infrastructure. Morgan Kaufmann: San Francisco, CA, 1999; 259-278.
15. Condor. The Directed Acyclic Graph Manager, 2003. http://www.cs.wisc.edu/ condor/dagman.
16. Thain D, Tannenbaum T. Livny M. Condor and the Grid. Grid Computing: Making the Global Infrastructure a Reality, Berman F, Fox G, Hey T (eds.). Wiley: New York, 2002.
17. Kola G, Livny M. Diskrouter: A flexible infrastructure for high performance large scale data transfers. Technical Report CS-TR-2003-1484, University of Wisconsin, 2003.
18. Allcock B. Bester J. Bresnahan J. Chervenak A, Foster I. Kesselman C, Meder S, Nefedova V. Quesnel D. Tuecke S. Secure, efficient data transport and replica management for high-performance data-intensive computing. Proceedings of the IEEE Mass Storage Conference, San Diego, CA, April 2001.
19. Madduri R. Allcock B. Reliable file transfer service, 2003. http://www-unix.mcs.anl.gov/~madduri/main.html.
20. Thain D, Basney J. Son S, Livny M. The kangaroo approach to data movement on the Grid. Proceedings of the 10th IEEE Symposium on High Performance Distributed Computing, San Francisco, CA, August 2001.
21. Feng W. High performance transport protocols. Los Alamos National Laboratory, 2003.
22. Koranda S, Moe B. Lightweight data replicator. http://www.lsc-group.phys. uwm.edu/LDR [2004].
23. LIGO. Laser Interferometer Gravitational Wave Observatory, 2003. http://www.ligo.caltech.edu/.
24. Bent J. Thain D, Arpaci-Dusseau A, Arpaci-Dusseau R. Explicit control in a batch-aware distributed file system. Proceedings of the First USENIX/ACM Conference on Networked Systems Design and Implementation, March 2004.