Using regression techniques to predict large data transfers

International Journal of High Performance Computing Applications

Volumn 17, Issue 3, 2003, Pages 249-268

  Vazhkudai, Sudharshan ^a   Schopf, Jennifer M ^a  

^a Dept of Comp and Info Sci ^*

Author keywords

Data transfer prediction; Grids; Replica selection

Indexed keywords

COMPUTER ARCHITECTURE; FILE ORGANIZATION; PERFORMANCE; REGRESSION ANALYSIS; RESOURCE ALLOCATION;

DATA GRIDS; REPLICA SELECTION;

DATA TRANSFER;

EID: 0042388124     PISSN: 10943420     EISSN: None     Source Type: Journal    
DOI: 10.1177/1094342003173004     Document Type: Article

References (54)

1. Allcock, W., Chervenak, A., Foster, I., Kesselman, C., Salisbury, C., and Tuecke, S. 2002. The Data Grid: Towards an Architecture for the Distributed Management and Analysis of Large Scientific Datasets. Network and Computer Applications.
2. Adve, V.S. 1993. Analyzing the Behavior and Performance of Parallel Programs, Department of Computer Science, University of Wisconsin.
3. Allcock, W., Foster, I., Nefedova, V., Chevrenak, A., Deelman, E., Kesselman, C., Sim, A., Shoshani, A., Drach, B., and Williams, D. 2001. High-Performance Remote Access to Climate Simulation Data: A Challenge Problem for Data Grid Technologies. In Supercomputing.
4. Basu, S., Mukherjee, A., and Kilvansky, S. 1996. Time Series Models for Internet Traffic, Georgia Institute of Technology.
5. Baru, C., Moore, R., Rajasekar, A., and Wan, M. 1998. The SDSC Storage Resource Broker. In CASCON'98.
6. Cole, M. 1989. Algorithmic Skeletons: Structured Management of Parallel Computation. Pitman/MIT Press.
7. Clement, M.J., and Quinn, M.J. 1993. Analytical Performance Prediction on Multicomputers. In Supercomputing'93.
8. Crovella, M.E. 1999. Performance Prediction and Tuning of Parallel Programs, Department of Computer Science, University of Rochester.
9. Cardwell, N., Savage, S., and Anderson, T. 1998. Modeling the Performance of Short TCP Connections, Computer Science Department, Washington University.
10. Data Grid Project. 2002. http://www.eu-datagrid.org.
11. Dinda, P. and O'Hallaron, D. 2000. Host Load Prediction Using Linear Models. Cluster Computing, 3(4).
12. Downey, A. 1997. Queue Times on Space-Sharing Parallel Computers. In 11th International Parallel Processing Symposium.
13. Edwards, A.L. 1984. An Introduction to Linear Regression and Correlation, W.H. Freeman.
14. Foster, I., and Kesselman, C. 1998. The Globus Project: A Status Report. In IPPS/SDP '98 Heterogeneous Computing Workshop.
15. Faerman, M., Su, A., Wolski, R., and Berman, F. 1999. Adaptive Performance Prediction for Distributed Data-Intensive Applications. In ACM/IEEE SC99 Conference on High Performance Networking and Computing, Portland, Oregon.
16. Globus Project. 2002. http://www.globus.org.
17. Guo, L., and Matta, I. 2001. The War between Mice and Elephants, Computer Science Department, Boston University.
18. Groschwitz, N., and Polyzos, G. 1994. A Time Series Model of Long-Term Traffic on the NSFnet Backbone. In IEEE Conference on Communications (ICC'94).
19. GriPhyN Project. 2002. http://www.griphyn.org.
20. Gray, J., and Shenoy, P. 2002. Rules of Thumb in Data Engineering. In International Conference on Data Engineering ICDE2000, IEEE Press, San Diego.
21. Geisler, J., and Taylor, V. 1999. Performance Coupling: Case Studies for Measuring the Interactions of Kernels in Modern Applications. In SPEC Workshop on Performance Evaluation with Realistic Applications.
22. Harchol-Balter, M., and Downey, A. 1996. Exploiting Process Lifetime Distributions for Dynamic Load Balancing. In 1996 Sigmetrics Conference on Measurement and Modeling of Computer Systems.
23. Hoschek, W., Jaen-Martinez, J., Samar, A., and Stockinger, H. 2000. Data Management in an International Grid Project. In 2000 International Workshop on Grid Computing (GRID 2000), Bangalore, India.
24. Holtman, K. 2000. Object Level Replication for Physics. In 4th Annual Globus Retreat, Pittsburgh.
25. Haddad, R., and Parsons, T. 1991. Digital Signal Processing: Theory, Applications and Hardware, Computer Science Press.
26. Hafeez, M., Samar, A., and Stockinger, H. 2000. Prototype for Distributed Data Production in CMS. In 7th International Workshop on Advanced Computing and Analysis Techniques in Physics Research (ACAT2000).
27. Jones, R. 2002. The Public Netperf Homepage, http://www.netperf.org/netperf/NeperfPage.html.
28. Lamehamedi, H., Szymanski, B., Zujun, S., and Deelman, E. 2002. Data Replication Strategies in Grid Environments, In 5th International Conference on Algorithms and Architecture for Parallel Processing, ICA3PP'2002, Bejing, China, October, IEEE Computer Science Press, Los Alamitos, CA, 2002, pp. 378-383
29. Lamehamedi, H., Szymanski, N., Zujun, S., and Deelman, E. 2003. Simulation of Dynamic Replication Strategies in Data Grids. In 12th Heterogeneous Computing Workshop (HCW2003), Nice, France, April.
30. LIGO Experiment. 2002. http://www.ligo.caltech.edu/.
31. Mak, V.W., and Lundstrom, S.F. 1990. Predicting Performance of Parallel Computations. IEEE Transactions on Parallel and Distributed Systems, 1(3):257-270.
32. Malon, D., May, E., Resconi, S., Shank, J., Vaniachine, A., Wenaus, T., and Youssef, S. 2001. Grid-enabled Data Access in the ATLAS Athena Framework. In Computing and High Energy Physics 2001 (CHEP'01) Conference.
33. NetLogger. 2002. NetLogger: A Methodology for Monitoring and Analysis of Distributed Systems.
34. Newman, H., and Mount, R. 2002. The Particle Physics Data Grid, http://www.cacr.caltech.edu/ppdg.
35. Ostle, B., and Malone, L.C. 1988. Statistics in Research, Iowa State University Press.
36. Pankratz, A. 1991. Forecasting with Dynamic Regression Models, Wiley, New York.
37. Rangahathan, K., and Foster, I. 2001. Design and Evaluation of Replication Strategies for a High Performance Data Grid. In Computing and High Energy and Nuclear Physics 2001 (CHEP'01) Conference.
38. SARA. 2002. SARA: The Synthetic Aperture Radar Atlas, http://sara.unile.it/sara/.
39. Schopf, J.M., and Berman, F. 1998. Performance Predictions in Production Environments. In IPPS/SPDP'98.
40. Shen, X., and Choudhary, A. 2000. A Multi-Storage Resource Architecture and I/O, Performance Prediction for Scientific Computing. In 9th IEEE Symposium on High Performance Distributed Computing, IEEE Press.
41. Schopf, J.M. 1997. Structural Prediction Models for High Performance Distributed Applications. In Cluster Computing (CCC'97).
42. Smith, W., Foster, I., and Taylor, V. 1998 Predicting Application Run Times Using Historical Information. In IPPS/SPDP '98 Workshop on Job Scheduling Strategies for Parallel Processing.
43. Swany, M., and Wolski, R. 2002, Multivariate Resource Performance Forecasting in the Network Weather Service, University of California Santa Barbara Computer Science Technical Report 2002-12.
44. SYSSTAT Utilities Homepage., 2002. http://perso.wanadoo.fr/sebastien.godard/.
45. Thomasian, A., and Bay, P.F. 1986, Analytic Queuing Network Models for Parallel Processing of Task Systems. IEEE Transactions on Computers, 35(12):1045-1054
46. Tirumala, A., and Ferguson, J. 2001, Iperf 1.2 - The TCP/UDP Bandwidth Measurement Tool, http://dast.nlanr.net/Projects/Iperf.
47. Vazhkudai, S., Schopf, J., and Foster, I. 2002 Predicting the Performance Wide-Area Data Transfers. In 16th International Parallel and Distributed Processing Symposium (IPDPS), Fort Lauderdale, Florida, IEEE Press.
48. Vazhkudai, S., Tuecke, S., and Foster, I. 2001. Replica Selection in the Globus Data Grid. In First IEEE/ACM International Conference on Cluster Computing and the Grid (CCGRID 2001), Brisbane, Australia, IEEE Press.
49. Web100 Project. 2002. http://www.web100.org.
50. Wolski, R. 1998. Dynamically Forecasting Network Performance Using the Network Weather Service. Journal of Cluster Computing, 1:119-132.
51. Yilmaz, S., and Matta, I. 2001. On Class-based Isolation of UDP, Short-lived and Long-lived TCP Flows, Computer Science Department, Boston University.
52. Zaki, M.J., Li, W., and Parthasarathy, S. 1996, Customized Dynamic Lad Balancing for Network of Workstations, In High Performance Distributed Computing (HPDC '96).
53. Zhang, Y., Qiu, L., and Keshav, S. 1999, Optimizing {TCP} Start-up Performance, Department of Computer Science, Cornell University.
54. Zhang, Y., Qiu, L., and Keshav, S. 2000. Speeding Up Short Data Transfers: Theory, Architecture Support and Simulation Results. In NOSSDAV 2000, Chapel Hill. NC.