A new paradigm in data intensive computing: Stork and the data-aware schedulers

Proceedings - Challenges of Large Applications in Distributed Environments, CLADE 2006

Volumn 2006, Issue , 2006, Pages 5-12

  Kosar, Tevfik ^a  

^a Louisiana State University   (United States)

Author keywords

Data placement; Data aware; Data intensive; Grid; Scheduling; Staging; Storage management; Stork

Indexed keywords

DATA STORAGE EQUIPMENT; DATABASE SYSTEMS; DISTRIBUTED COMPUTER SYSTEMS; EMBEDDED SYSTEMS; INFORMATION RETRIEVAL; SCHEDULING; SECURITY OF DATA;

DATA AWARE SCHEDULERS; DATA INTENSIVE; DATA PLACEMENT; GRID; STORAGE MANAGEMENT; STORK;

COMPUTATION THEORY;

EID: 33845932640     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (34)

1. G. E. Moore, "Cramming more components onto integrated circuits," Electronics, vol. 38, no. 8, 1965.
2. J. Gray and P. Shenoy, "Rules of thumb in data engineering," in Proceedings of the IEEE International Conference on Data Engineering, San Diego, CA, February 2000.
3. "NCBI: Growth of genbank," http://www.ncbi.nlm.nih.gov/Genbank/ genbankstats.html.
4. "The Visible and Infrared Survey Telescope for Astronomy," http://www.vista.ac.uk/.
5. "Laser Interferometer Gravitational Wave Observatory," http://www.ligo.caltech.edu/.
6. G. Kola, T. Kosar, and M. Livny, "A fully automated fault-tolerant system for distributed video processing and off-site replication," in Proceedings of the 14th ACM International Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV 2004), Kinsale, Ireland, June 2004.
7. "The Large Synoptic Survey Telescope (LSST)," http://www.lsst.org/.
8. S. F. Altschul, W. Gish, W. Miller, E. W. Myers, and D. J. Lipman, "Basic Local Alignment Search Tool," Journal of Molecular Biology, vol. 3, no. 215, pp. 403-410, October 1990.
9. "A Toroidal LHC Apparatus Project (ATLAS)," http://atlas.web.cern.ch/.
10. "The Compact Muon Solenoid Project (CMS)," http://cmsinfo.cern. ch/.
11. L. Lamport and N. Lynch, "Distributed computing: Models and methods," Handbook of Theoretical Computer Science, pp. 1158-1199, Elsevier Science Publishers, 1990.
12. T. L. Casavant and J. G. Kuhl, "A taxonomy of scheduling in general-purpose distributed computing systems," IEEE Transactions on Software Engineering, vol. 14, no. 2, pp. 141-154, February 1998.
13. J. Waldo, G. Wyant, A. Wollrath, and S. Kendall, "A note on distributed computing," Sun Microsystems Laboratories, Tech. Rep. TR-94-29, November 1994.
14. E. Gabriel, M. Resch, T. Beisel, and R. Keller, "Distributed computing in a heterogeneous computing environment," Lecture Notes in Computer Science, vol. 1497, p. 180, January 1998.
15. R. Henderson and D. Tweten, "Portable Batch System: External reference specification," 1996.
16. "Using and administering IBM LoadLeveler," IBM Corporation SC23-3989, 1996.
17. M. J. Litzkow, M. Livny, and M. W. Mutka, "Condor - A hunter of idle workstations," in Proceedings of the 8th International Conference of Distributed Computing Systems, 1988, pp. 104-111.
18. S. Zhou, "LSF: Load sharing in large-scale heterogeneous distributed systems," in Proc. of Workshop on Cluster Computing, 1992.
19. I. Foster, C. Kesselman, and S. Tuecke, "The anatomy of the Grid: Enabling scalable virtual organizations," International Journal of Supercomputing Applications, 2001.
20. W. Hoschek, J. Jaen-Martinez, A. Samar, H. Stockinger, and K. Stockinger, "Data management in an international DataGrid project," in First IEEE/ACM Int'l Workshop on Grid Computing, Bangalore, India, December 2000.
21. "The Grid2003 production grid," http://www.ivdgl.org/grid2003/.
22. B. Sagal, "Grid Computing: The European DataGrid Project," in IEEE Nuclear Science Symposium and Medical Imaging Conference, Lyon, France, October 2000.
23. L. Bic and A. C. Shaw, "The Organization of Computer Systems," in The Logical Design of Operating Systems., Prentice Hall., 1974.
24. H. M. Deitel, "I/O Control System," in An Introduction to Operating Systems., Addison-Wesley Longman Publishing Co., Inc., 1990.
25. S. E. Madnick and J. J. Donovan, "I/O Scheduler," in Operating Systems., McGraw-Hill, Inc., 1974.
26. W. Feng, "High performance transport protocols," Los Alamos National Laboratory, 2003.
27. T. Kosar and M. Livny, "Stork: Making data placement a first class citizen in the Grid," in Proceedings of the 24th Int. Conference on Distributed Computing Systems (ICDCS 2004), Tokyo, Japan, March 2004.
28. J. Bent, D. Thain, A. Arpaci-Dusseau, and R. Arpaci-Dusseau, "Explicit control in a batch-aware distributed file system," in Proceedings of the First USENIX/ACM Conference on Networked Systems Design and Implementation, March 2004.
29. J. Bent, "Data-driven batch scheduling," Ph.D. dissertation, University of Wisconsin-Madison, 2005.
30. D. Thain, J. Bent, A. Arpaci-Dusseau, and R. Arpaci-Dusseau, "Pipeline and batch sharing in grid workloads," in Proceedings of the Twelfth IEEE Symposium on High Performance Distributed Computing, June 2003.
31. T. Kosar and M. Livny, "A framework for reliable and efficient data placement in distributed computing systems," Journal of Parallel and Distributed Computing, 2005.
32. C. Baru, R. Moore, A. Rajasekar, and M. Wan, "The SDSC Storage Resource Broker," in Proceedings of CASCON, Toronto, Canada, 1998.
33. L. Chervenak, N. Palavalli, S. Bharathi, C. Kesselman, and R. Schwartzkopf, "Performance and scalability of a Replica Location Service," in Proceedings of the International Symposium on High Performance Distributed Computing Conference (HPDC-13), Honolulu, Hawaii, June 2004.
34. "NeST: Network Storage Technology," http://www.cs.wisc.edu/ condor/nest/.