A framework for reliable and efficient data placement in distributed computing systems

Journal of Parallel and Distributed Computing

Volumn 65, Issue 10, 2005, Pages 1146-1157

  Kosar, Tevfik ^a   Livny, Miron ^a  

^a University of Wisconsin Madison   (United States)

Author keywords

Data intensive applications; Distributed computing; I O subsystem; Protocol auto tuning; Reliable and efficient data placement; Scheduling run time adaptation

Indexed keywords

COMPUTER OPERATING SYSTEMS; DATA HANDLING; OPTIMIZATION; SCHEDULING;

DATA-INTENSIVE APPLICATIONS; DISTRIBUTED COMPUTING; I/O SUBSYSTEMS; PROTOCOL AUTO-TUNING; RELIABLE AND EFFICIENT DATA PLACEMENT; SCHEDULING RUN-TIME ADAPTION;

PARALLEL PROCESSING SYSTEMS;

EID: 24144445675     PISSN: 07437315     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jpdc.2005.04.019     Document Type: Article

References (46)

1. B. Allcock, J. Bester, J. Bresnahan, A. Chervenak, I. Foster, C. Kesselman, S. Meder, V. Nefedova, D. Quesnel, S.T. ke, Secure, efficient data transport and replica management for high-performance data-intensive computing, in: IEEE Mass Storage Conference, San Diego, CA, April 2001.
2. W. Allcock, I. Foster, R. Madduri, Reliable data transport: a critical service for the grid, in: Building service based grids workshop, global grid forum 11, June 2004.
3. C. Baru, R. Moore, A. Rajasekar, M. Wan, The SDSC storage resource broker, in: Proceedings of CASCON, Toronto, Canada, 1998.
4. M. Beck, T. Moore, J. Plank, and M. Swany Logistical networking S. Hariri C. Lee C. Raghavendra Active Middleware Services 2000 Kluwer Academic Publishers Dordrecht
5. J. Bent, D. Thain, A. Arpaci-Dusseau, R. Arpaci-Dusseau, Explicit control in a batch-aware distributed file system, in: Proceedings of the First USENIX/ACM Conference on Networked Systems Design and Implementation, March 2004.
6. F. Berman, R. Wolski, S. Figueira, J. Schopf, G. Shao, Application level scheduling on distributed heterogeneous networks, in: Proceedings of Supercomputing '96, Pittsburgh, Pennsylvenia, November 1996.
7. L. Bic, and A.C. Shaw The Logical Design of Operating Systems 1974 Prentice-Hall Englewood cliffs, NJ
8. I. Bird, B. Hess, A. Kowalski, Building the mass storage system at Jefferson Lab, in: Proceedings of 18th IEEE Symposium on Mass Storage Systems, San Diego, CA, April 2001.
9. M. Butler, R. Pennington, J.A. Terstriep, Mass Storage at NCSA: SGI DMF and HP UniTree, in: Proceedings of 40th Cray User Group Conference, 1998.
10. R.L. Carter, M.E. Crovella, Dynamic server selection using bandwidth probing in wide-area networks, Technical Report TR-96-007, Computer Science Department, Boston University, 1996.
11. L. Chervenak, N. Palavalli, S. Bharathi, C. Kesselman, R. Schwartzkopf, Performance and scalability of a replica location service, in: Proceedings of the International Symposium on High Performance Distributed Computing Conference (HPDC-13), Honolulu, Hawaii, June 2004.
12. Condor, NeST: Network Storage, http://www.cs.wisc.edu/condor/nest/, 2003.
13. E. Deelman, J. Blythe, Y. Gil, C. Kesselman, Pegasus: Planning for execution in grids, in: GriPhyN Technical Report, 2002.
14. H.M. Deitel I/O control system An Introduction to Operating Systems 1990 Addison-Wesley, Longman Reading, MA, New York
15. S.G. Djorgovski, R.R. Gal, S.C. Odewahn, R.R. de Carvalho, R. Brunner, G. Longo, R. Scaramella, The Palomar Digital Sky Survey (DPOSS), Wide Field Surveys in Cosmology, 1988.
16. C. Dovrolis, P. Ramanathan, D. Moore, What do packet dispersion techniques measure? in: INFOCOMM, 2001.
17. M. Faerman, A. Su, R. Wolski, F. Berman, Adaptive performance prediction for distributed data-intensive applications, in: Proceedings of the IEE/ACM Conference on High Performance Networking and Computing, Portland, Oregon, November 1999.
18. W. Feng, High performance transport protocols, Los Alamos National Laboratory, 2003.
19. M. Fisk, W. Weng, Dynamic right-sizing in TCP, in: ICCCN, 2001.
20. FNAL, Enstore mass storage system, http://www.fnal.gov/docs/products/ enstore/
21. I. Foster, and C. Kesselmann Globus a toolkit-based grid architecture The Grid Blueprints for a New Computing Infrastructure 1999 Morgan Kaufmann Los Altos, CA 259 278
22. I. Foster, D. Kohr, R. Krishnaiyer, and J. Mogill Remote I/O fast access to distant storage Proceedings of the IOPADS'97 1997 ACM Press New York 14 25
23. I. Foster, J. Vockler, M. Wilde, Y. Zhao, Chimera: a virtual data system for representing, querying, and automating data derivation, in: 14th International Conference on Scientific and Statistical Database Management (SSDBM 2002), Edinburgh, Scotland, July 2002.
24. J. Frey, T. Tannenbaum, I. Foster, S. Tuecke, Condor-G: a computation management agent for multi-institutional grids, in: Proceedings of the Tenth IEEE Symposium on High Performance Distributed Computing, San Francisco, CA, August 2001.
25. G. Kola, M. Livny, Diskrouter: a flexible infrastructure for high performance large scale data transfers, Technical Report CS-TR-2003-1484, University of Wisconsin, 2003.
26. S. Koranda, B. Moe, Lightweight data replicator, http://www.lsc-group. phys.uwm.edu/lscdatagrid/LDR/overview.html, 2003.
27. J. Kubiatowicz, D. Bindel, Y. Chen, S. Czerwinski, P. Eaton, D. Geels, R. Gummadi, S. Rhea, H. Weatherspoon, W. Weimer, C. Wells, B. Zhao, Oceanstore: an architecture for global-scale persistent storage, in: Proceedings of the Ninth International Conference on Architectural Support for Programming Languages and Operating Systems (ASP- LOS 2000), November 2000.
28. LIGO, Laser Interferometer Gravitational Wave Observatory, http://www.ligo.caltech.edu/, 2003.
29. M.J. Litzkow, M. Livny, M.W. Mutka, Condor - a hunter of idle workstations, in: Proceedings of the Eighth International Conference of Distributed Computing Systems, 1988, pp. 104-111.
30. R. Maddurri, B. Allcock, Reliable file transfer service, http://www-unix.mcs.anl.gov/madduri/main.html, 2003.
31. S.E. Madnick, and J.J. Donovan Operating Systems 1974 McGraw-Hill New York
32. Y. Morita, H. Sato, Y. Watase, O. Tatebe, S. Sekiguchi, S. Matsuoka, N. Soda, A. Dell'Acqua, Building a high performance parallel file system using grid datafarm and root i/o, in: Proceedings of the 2003 Computing in High Energy and Nuclear Physics (CHEP03), La Jolla, CA, March 2003.
33. J. Nieplocha, I. Foster, H. Dachsel, Distant I/O: one-sided access to secondary storage on remote processors, in: Proceedings of the Seventh IEEE International Symposium on High Performance Distributed Computing, July 1998, pp. 148-154.
34. NLANR/DAST, Iperf: The TCP/UDP bandwidth measurement tool, http://dast.nlanr.net/Projects/Iperf/, 2003.
35. S. Ogura, H. Nakada, S. Matsuoka, Evaluation of the inter-cluster data transfer on Grid environment, in: Proceedings of the Third IEEE/ACM Symposium on Cluster Computing and the Grid (CCGrid2003), Tokyo, Japan, May 2003.
36. J. Postel, FTP: file transfer protocol specification. RFC-765, 1980.
37. PPDG, PPDG deliverables to CMS, http://www.ppdg.net/archives/ppdg/2001/ doc00017.doc.
38. R. Raman, M. Livny, M. Solomon, Matchmaking: distributed resource management for high throughput computing, in: Proceedings of the Seventh IEEE International Symposium on High Performance Distributed Computing (HPDC7), Chicago, IL, July 1998.
39. SDSC, High Performance Storage System (HPSS), http://www.sdsc.edu/hpss/
40. J. Semke, J. Mahdavi, M. Mathis, Automatic TCP buffer tuning, in: SIGCOMM, 1998, pp. 315-323.
41. A. Shishani, A. Sim, J. Gu, Storage resource managers: middleware components for grid storage, in: Nineteenth IEEE Symposium on Mass Storage Systems, 2002.
42. D. Thain, J. Basney, S. Son, M. Livny, The kangaroo approach to data movement on the grid, in: Proceedings of the Tenth IEEE Symposium on High Performance Distributed Computing, San Francisco, CA, August 2001.
43. D. Thain, M. Livny, The ethernet approach to grid computing, in: Proceedings of the 12th IEEE Symposium on High Performance Distributed Computing, Seattle, Washington, June 2003.
44. D. Thain, T. Tannenbaum, and M. Livny Condor and the Grid F. Berman G. Fox T. Hey Grid Computing Making the Global Infrastructure a Reality 2002 Wiley New York
45. S. Vazhkudai, J. Schopf, I. Foster, Predicting the performance of wide area data transfers, in: Proceedings of the 16th International Parallel and Distributed Processing Symposium (IPDPS 2002), 2002.
46. R. Wolski, Dynamically forecasting network performance to support dynamic scheduling using the Network Weather Service, in: Proceedings of the Sixth IEEE Symposium on High Performance Distributed Computing (HPDC6), Portland, Oregon, August 1997.