How GridFTP pipelining, parallelism and concurrency work: A guide for optimizing large dataset transfers

Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012

Volumn , Issue , 2012, Pages 506-515

  Yildirim, Esma ^a   Kim, Jangyoung ^b   Kosar, Tevfik ^b  

^a FATIH UNIVERSITY   (Turkey)

^b State University of New York   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

HIGH-BANDWIDTH NETWORKS; LARGE DATASET; LARGE DATASETS; NETWORK SPEED; PARALLELISM AND CONCURRENCIES; PRACTICAL USE; ROUND-TRIP TIME; TRANSFER PROTOCOL;

COMPLEX NETWORKS;

OPTIMIZATION;

EID: 84876551765     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/SC.Companion.2012.73     Document Type: Conference Paper

References (25)

1. J. Bresnahan, M. Link, R. Kettimuthu, D. Fraser, and I. Foster, "Gridftp pipelining," in TeraGrid 2007, 2007.
2. B. Allen, J. Bresnahan, L. Childers, I. Foster, G. Kandaswamy, R. Kettimuthu, J. Kordas, M. Link, S. Martin, K. Pickett, and S. Tuecke, "Software as a service for data scientists," Communications of the ACM, vol. 55:2, pp. 81-88, 2012.
3. R. S. Prasad, M. Jain, and C. Davrolis, "Socket buffer auto-sizing for high-performance data transfers," Journal of Grid Computing, vol. 1(4), pp. 361-376, Aug. 2004.
4. G. Hasegawa, T. Terai, T. Okamoto, and M. M, "Scalable socket buffer tuning for high-performance web servers," in International Conference on Network Protocols(ICNP01), 2001, p. 281.
5. A. Morajko, "Dynamic tuning of parallel/distributed applications," Ph.D. dissertation, Universitat Autonoma de Barcelona, 2004.
6. T. Ito, H. Ohsaki, and M. Imase, "On parameter tuning of data transfer protocol gridftp for wide-Area networks," International Journal of Computer Science and Engineering, vol. 2(4), pp. 177-183, Sept. 2008.
7. K. M. Choi, E. Huh, and H. Choo, "Efficient resource management scheme of tcp buffer tuned parallel stream to optimize system performance," in Proc. Embedded and ubiquitous computing, Nagasaki, Japan, Dec. 2005.
8. E. Yildirim, M. Balman, and T. Kosar, Data-intensive Distributed Computing: Challenges and Solutions for Large-scale Information Management. IGI-Global, 2012, ch. Data-Aware Distributed Computing.
9. T. J. Hacker, B. D. Noble, and B. D. Atley, "The end-to-end performance effects of parallel tcp sockets on a lossy wide area network," in Proc. IEEE International Symposium on Parallel and Distributed Processing(IPDPS'02), 2002, pp. 434-443.
10. D. Lu, Y. Qiao, P. A. Dinda, and F. E. Bustamante, "Modeling and taming parallel tcp on the wide area network," in Proc. IEEE International Symposium on Parallel and Distributed Processing (IPDPS'05), Apr. 2005, p. 68b.
11. E. Yildirim, D. Yin, and T. Kosar, "Prediction of optimal parallelism level in wide area data transfers," IEEE Transactions on Parallel and Distributed Systems, vol. 22(12), 2011.
12. D. Yin, E. Yildirim, and T. Kosar, "Prediction of optimal parallelism level in wide area data transfers," In IEEE Transactions on Parallel and Distributed Systems, 2012.
13. E. Altman, D. Barman, B. Tuffin, and M. Vojnovic, "Parallel tcp sockets: Simple model, throughput and validation," in Proc. IEEE Conference on Computer Communications (INFOCOM'06), Apr. 2006, pp. 1-12.
14. J. Crowcroft and P. Oechslin, "Differentiated end-to-end internet services using a weighted proportional fair sharing tcp," ACM SIGCOMM Computer Communication Review, vol. 28(3), pp. 53-69, July 1998.
15. L. Eggert, J. Heideman, and J. Touch, "Effects of ensemble tcp," ACM Computer Communication Review, vol. 30(1), pp. 15-29, 2000.
16. H. Sivakumar, S. Bailey, and R. L. Grossman, "Psockets: The case for application-level network striping for data intensive applications using high speed wide area networks," in Proc. IEEE Super Computing Conference (SC00), Texas, USA, Nov. 2000, pp. 63-63.
17. T. Kosar, M. Balman, E. Yildirim, S. Kulasekaran, and B. Ross, "Stork data scheduler: Mitigating the data bottleneck in e-science," The Philosophical Transactions of the Royal Society A, vol. 369(3254-3267), 2011.
18. T. Kosar and M. Livny, "Stork: Making data placement a first class citizen in the grid," in Proceedings of ICDCS'04, March 2004, pp. 342-349.
19. T. Kosar and M. Balman, "A new paradigm: Data-Aware scheduling in grid computing," Future Generation Computing Systems, vol. 25, no. 4, pp. 406-413, 2009.
20. W. Liu, B. Tieman, R. Kettimuthu, and I. Foster, "A data transfer framework for large-scale science experiments," in Proc. 3rd International Workshop on Data Intensive Distributed Computing (DIDC '10) in conjunction with 19th International Symposium on High Performance Distributed Computing (HPDC '10), June 2010.
21. "Louisiana optical network initiative (LONI)," http://www.loni.org/.
22. "Futuregrid testbed," http://www.futuregrid.org.
23. Emulab-network emulation testbed. [Online]. Available: http://www.emulab.net/
24. E. Yildirim, J. Kim, and T. Kosar, "Optimizing the sample size for a cloud-hosted data scheduling service," in Proc. 2nd International Workshop on Cloud Computing and Scientific Applications (CCSA in conjunction with CCGRID12), 2012.
25. E. Yildirim, , and T. Kosar, "Network-Aware end-to-end data throughput optimization," in Proceedings of Network-Aware Data Management Workshop (NDM 2011), November 2012.