A data transfer framework for large-scale science experiments

HPDC 2010 - Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing

Volumn , Issue , 2010, Pages 717-724

  Liu, Wantao ^a,e   Tieman, Brian ^c   Kettimuthu, Rajkumar ^d,e   Foster, Ian ^b,d,e  

^a BEIHANG UNIVERSITY   (China)

^b THE UNIVERSITY OF CHICAGO   (United States)

^c ARGONNE NATIONAL LABORATORY   (United States)

^d ARGONNE NATIONAL LABORATORY   (United States)

^e UNIVERSITY OF CHICAGO   (United States)

Author keywords

Concurrent; Data scheduling; Data transfer; Error recovery; GridFTP

Indexed keywords

AUTOMATIC TUNING; CONCURRENT; CONCURRENT THREADS; DATA ANALYSIS; DATA SCHEDULING; ERROR RECOVERY; GRIDFTP; PERFORMANCE DATA; PETABYTES; RESTART MECHANISM; RUNTIMES; SCIENCE EXPERIMENTS; SMALL FILES; TRANSFER PERFORMANCE; TRANSFER SYSTEMS;

DATA TRANSFER; GRAPHICAL USER INTERFACES; RECOVERY; SCHEDULING;

DATA HANDLING;

EID: 78650006114     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1851476.1851582     Document Type: Conference Paper

References (27)

1. CERN: http://www.cern.ch/.
2. LIGO: http://www.ligo.caltech.edu/
3. APS: http://www.aps.anl.gov/
4. SNS: http://neutrons.ornl.gov/
5. W. Allcock, "GridFTP: Protocol Extension to FTP for the Grid," Global Grid Forum GFD-R-P.020, 2003.
6. W. Allcock, J. Bresnahan, R. Kettimuthu, M. Link, C. Dumitrescu, I. Raicu, and I. Foster, "The Globus Striped GridFTP Framework and Server," SC'05, ACM Press, 2005
7. J. Postel and J. Reynolds, "File Transfer Protocol," IETF, RFC 959, 1985
8. PhEDEx: http://cmsweb.cern.ch/phedex/
9. J. Rehn, T. Barrass, D. Bonacorsi, J. Hernandez, I. Semoniouk, L. Tuura, and Y. Wu, "PhEDEx High-Throughput Data Transfer Management System," CHEP06, Mumbai, India, 2006
10. R. O. Sinnott, C. Bayliss, C. Davenhall, B. Harbulot, M. Jones, C. Millar, G. Roy, S. Roy, G. Stewart, J. Watt, and A. Aseno, "Secure, Performance-Oriented Data Management for nanoCMOS Electronics," eScience 2008
11. caGrid: http://cagrid.org/display/cagridhome/Home
12. caGrid Transfer: http://cagrid.org/display/transfer/Home
13. caGrid Bulk Data Transfer: http://cagrid.org/display/bdt/Home
14. T. Kosar and M. Livny, "Stork: Making Data Placement a First Class Citizen in the Grid,"24th International Conference on Distributed Computing Systems (ICDCS 2004), Tokyo, Japan, March 2004.
15. T. Kosar and M. Balman, "A New Paradigm: Data-Aware Scheduling in Grid Computing," Future Generation Computer Systems, 25, no. 4, April 2009, pp. 406-413
16. N. Ali and M. Lauria. "Improving the Performance of Remote I/O Using asynchronous Primitives," 15th IEEE International Symposium on High Performance Distributed Computing, 2006, pp. 218-228
17. CoG jglobus:http://dev.globus.org/wiki/CoG-jglobus
18. Java web start technology: http://java.sun.com/javase/technologies/ desktop/javawebstart/index.jsp
19. RFT:http://globus.org/toolkit/docs/latest-stable/data/rft/#rft
20. IGTF, http://www.igtf.net/
21. S. Vazhkudai, "Bulk Data Transfer Forecasts and Implications to Grid Scheduling," Ph.D. Dissertation, The University of Mississippi, May 2003
22. T. J. Hacker, B. D. Noble, and B. D. Atley. "The end-to-end performance effects of parallel TCP Sockets on a Lossy Wide Area Network," IPDPS '02, p. 314. IEEE, April 2002.
23. D. Lu, Y. Qiao, P. A. Dinda, and F. E. Bustamante. "Modeling and Taming Parallel TCP on the Wide Area Network," IPDPS '05, p. 68.2. IEEE, April 2005.
24. Esma Yildirim, Mehmet Balman, and Tevfik Kosar, "Dynamically Tuning Level of Parallelism in Wide Area Data Transfers," International Workshop on Data-aware Distributed Computing, pp. 39-48, June 2008.
25. Wantao Liu, Rajkumar Kettimuthu, Brian Tieman, Ravi Madduri, Bo Li and Ian Foster, "GridFTP GUI: An Easy and Efficient Way to Transfer Data in Grid," GridNets2009, September 2009.
26. Branco, M., Cameron, D., Gaidioz, B., Garonne, V., Koblitz, B., Lassnig, M., Rocha, R., Salgado, P. and Wenaus, T. "Managing ATLAS data on a petabyte-scale with DQ2," Journal of Physics: Conference Series, Volume 119, Issue 6, pp. 062017 (2008).
27. gLite File Transfer Service: www.gridpp.ac.uk/wiki/GLite-File-Transfer- Service