A data throughput prediction and optimization service for widely distributed many-task computing

IEEE Transactions on Parallel and Distributed Systems

Volumn 22, Issue 6, 2011, Pages 899-909

  Yin, Dengpan ^a   Yildirim, Esma ^a   Kulasekaran, Sivakumar ^a   Ross, Brandon ^a   Kosar, Tevfik ^a  

^a Louisiana State University   (United States)

Author keywords

Many task computing; modeling; optimization; parallel TCP streams; prediction; scheduling; stork.

Indexed keywords

MODELING; PARALLEL TCP; PREDICTION; STORK.; TASK COMPUTING;

DATA TRANSFER; MATHEMATICAL MODELS; NETWORK PERFORMANCE; OPTIMIZATION; SCHEDULING; THROUGHPUT;

FORECASTING;

EID: 79955524221     PISSN: 10459219     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPDS.2010.187     Document Type: Article

References (33)

1. I. Raicu, I. Foster, and Y. Zhao, "Many-Task Computing for Grids and Supercomputers," Proc. IEEE Workshop Many-Task Computing on Grids and Supercomputers (MTAGS), 2008.
2. Louisiana Optical Network Initiative (LONI), http://www.loni.org/, 2010.
3. Energy Sciences Network (ESNet), http://www.es.net/, 2010.
4. TeraGrid, http://www.teragrid.org/, 2010.
5. S. Floyd, "Rfc 3649: Highspeed TCP for Large Congestion Windows," 2003.
6. R. Kelly, "Scalable TCP: Improving Performance in Highspeed Wide Area Networks," Computer Comm. Rev., vol. 32, no. 2, pp. 83- 91, 2003.
7. C. Jin, D.X. Wei, S.H. Low, G. Buhrmaster, J. Bunn, D.H. Choe, R.L.A. Cottrell, J.C. Doyle, W. Feng, O. Martin, H. Newman, F. Paganini, S. Ravot, and S. Singh, "Fast TCP: From Theory to Experiments," IEEE Network, vol. 19, no. 1, pp. 4-11, Feb. 2005. (Pubitemid 40195955)
8. H. Sivakumar, S. Bailey, and R.L. Grossman, "Psockets: The Case for Application-Level Network Striping for Data Intensive Applications Using High Speed Wide Area Networks," Proc. IEEE Super Computing Conf. (SC '00), p. 63, Nov. 2000.
9. J. Lee, D. Gunter, B. Tierney, B. Allcock, J. Bester, J. Bresnahan, and S. Tuecke, "Applied Techniques for High Bandwidth Data Transfers across Wide Area Networks," Proc. Int'l Conf. Computing in High Energy and Nuclear Physics (CHEP '01), Sept. 2001.
10. H. Balakrishman, V.N. Padmanabhan, S. Seshan, and R.H.K.M. Stemm, "TCP Behavior of a Busy Internet Server: Analysis and Improvements," Proc. IEEE INFOCOM '98, pp. 252-262, Mar. 1998.
11. T.J. Hacker, B.D. Noble, and B.D. Atley, "Adaptive Data Block Scheduling for Parallel Streams," Proc. IEEE Int'l Symp. High Performance Distributed Computing (HPDC '05), pp. 265-275, July 2005. (Pubitemid 41543494)
12. L. Eggert, J. Heideman, and J. Touch, "Effects of Ensemble TCP," ACM Computer Comm. Rev., vol. 30, no. 1, pp. 15-29, 2000.
13. R.P. Karrer, J. Park, and J. Kim, "TCP-ROME: Performance and Fairness in Parallel Downloads for Web and Real Time Multimedia Streaming Applications," technical report, Deutsche Telekom Labs, 2006.
14. D. Lu, Y. Qiao, and P.A. Dinda, "Characterizing and Predicting TCP Throughput on the Wide Area Network," Proc. IEEE Int'l Conf. Distributed Computing Systems (ICDCS '05), pp. 414-424, 2005. (Pubitemid 41675619)
15. E. Yildirim, D. Yin, and T. Kosar, "Prediction of Optimal Parallelism Level in Wide Area Data Transfers," to be published in IEEE Trans. Parallel and Distributed Systems, 2010.
16. A. Tirumala, L. Cottrell, and T. Dunnigan, "Measuring End-to- End Bandwidth with Iperf Using Web100," Proc. Passive and Active Measurement Workshop (PAM '03), 2003.
17. W. Allcock, J. Bresnahan, R. Kettimuthu, M. Link, C. Dumitrescu, I. Raicu, and I. Foster, "The Globus Striped Gridftp Framework and Server," Proc. 2005 ACM/IEEE Conf. Supercomputing (SC '05), p. 54, 2005.
18. T. Kosar and M. Livny, "Stork: Making Data Placement a First Class Citizen in the Grid," Proc. IEEE Int'l Conf. Distributed Computing Systems (ICDCS '04), pp. 342-349, 2004.
19. T.J. Hacker, B.D. Noble, and B.D. Atley, "The End-to-End Performance Effects of Parallel TCP Sockets on a Lossy Wide Area Network," Proc. IEEE Int'l Symp. Parallel and Distributed Processing (IPDPS '02), pp. 434-443, 2002.
20. D. Lu, Y. Qiao, P.A. Dinda, and F.E. Bustamante, "Modeling and Taming Parallel TCP on the Wide Area Network," Proc. IEEE Int'l Symp. Parallel and Distributed Processing (IPDPS '05), p. 68b, Apr. 2005.
21. E. Altman, D. Barman, B. Tuffin, and M. Vojnovic, "Parallel TCP Sockets: Simple Model, Throughput and Validation," Proc. IEEE INFOCOM '06, pp. 1-12, Apr. 2006.
22. J. Crowcroft and P. Oechslin, "Differentiated End-to-End Internet Services Using a Weighted Proportional Fair Sharing TCP," ACM SIGCOMM Computer Comm. Rev., vol. 28, no. 3, pp. 53-69, July 1998.
23. G. Kola and M.K. Vernon, "Target Bandwidth Sharing Using Endhost Measures," Performance Evaluation, vol. 64, nos. 9-12, pp. 948-964, Oct. 2007. (Pubitemid 47302142)
24. M. Jain and C. Dovrolis, "Pathload: A Measurement Tool for Endto- End Available Bandwidth," Proc. Passive and Active Measurements Workshop (PAM), pp. 14-25, 2002.
25. V.J. Ribeiro, R.H. Riedi, R.G. Baraniuk, J. Navratil, and L. Cottrell, "Pathchirp: Efficient Available Bandwidth Estimation for Network," Proc. Passive and Active Measurement Workshop, 2003.
26. P. Primet, R. Harakaly, and F. Bonnassieux, "Experiments of Network Throughput Measurement and Forecasting Using the Network Weather," Proc. Second IEEE/ACM Int'l Symp. Cluster Computing and the Grid (CCGRID '02), p. 413, 2002.
27. E. Yildirim, I.H. Suslu, and T. Kosar, "Which Network Measurement Tool is Right for You? A Multidimensional Comparison Study," Proc. Ninth IEEE/ACM Int'l Conf. Grid Computing (GRID '08), pp. 266-275, 2008.
28. J. Strauss, D. Katabi, and M.F. Kaashoek, "A Measurement Study of Available Bandwidth Estimation Tools," Proc. Internet Measurement Conf., pp. 39-44, 2003. (Pubitemid 40730637)
29. T. Dunigan, M. Mathis, and B. Tierney, "A TCP Tuning Daemon," Proc. IEEE Int'l Conf. Supercomputing (SC '02), pp. 1-16, Nov. 2002.
30. G. Jin, G. Yang, B.R. Crowley, and D.A. Agarwal, "Network Characterization Service (NCS)," Proc. IEEE Int'l Symp. High Performance Distributed Computing (HPDC '01), pp. 289-299, 2001.
31. LBNL "Netest," http://www-didc.lbl.gov/NCS/netest/, 2010.
32. D. Thain, T. Tannenbaum, and M. Livny, "Distributed Computing in Practice: The Condor Experience: Research Articles," Concurrency and Computation: Practice and Experience, vol. 17, nos. 2-4, pp. 323-356, 2005. (Pubitemid 40285781)
33. D. Yin, E. Yildirim, and T. Kosar, "A Data Throughput Prediction and Optimization Service for Widely Distributed Many-Task Computing," Proc. Second Workshop Many-Task Computing on Grids and Supercomputers (MTAGS '09), Nov. 2009.