Stork data scheduler: Mitigating the data bottleneck in e-Science

Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences

Volumn 369, Issue 1949, 2011, Pages 3254-3267

  Kosar, Tevfik ^a,c   Balman, Mehmet ^b   Yildirim, Esma ^a   Kulasekaran, Sivakumar ^c   Ross, Brandon ^c  

^a the State University of New York   (United States)

^b LAWRENCE BERKELEY NATIONAL LABORATORY   (United States)

^c LOUISIANA STATE UNIVERSITY   (United States)

Author keywords

Data intensive computing; E Science; Input output scheduling; Stork; Throughput optimization

Indexed keywords

DATA TRANSFER; OPTIMIZATION; PLANNING;

DATA-INTENSIVE COMPUTING; E-SCIENCE; INPUT/OUTPUT SCHEDULING; STORK; THROUGHPUT OPTIMIZATION;

SCHEDULING;

EID: 80052102508     PISSN: 1364503X     EISSN: None     Source Type: Journal    
DOI: 10.1098/rsta.2011.0148     Document Type: Article

References (42)

1. Hey, T. & Trefethen, A. 2003 The data deluge: An e-science perspective. In Grid computing- making the global infrastructure a reality, ch. 36, pp. 809-824. New York, NY: Wiley and Sons.
2. CCSP. 2003 Strategic plan for the US Climate Change Science Program. A Report by the Climate Change Science Program and the Subcommittee on Global Change Research. Washington, DC: CCSP.
3. Kosar, T., Balman, M., Suslu, I., Yildirim, E. & Yin, D. 2009 Data-aware distributed computing with stork data scheduler. In Proc. SEE-GRID-SCI'09, Istanbul, Turkey, 9-10 December 2009.
4. Kosar, T. & Livny, M. 2004 Stork: Making data placement a first class citizen in the grid. In Proc. 24th Int. Conf. on Distributed Computing Systems (ICDCS'04), Tokyo, Japan, 23-26, pp. 342-349. Washington, DC: IEEE Computer Society.
5. SCOOP. SURA Coastal Ocean Observing and Prediction. See http://scoop.sura.org/.
6. UCoMS. Ubiquitous Computing and Monitoring System for discovery and management of energy resources. See http://www.ucoms.org/overview.html.
7. DPOSS. The Palomar digital sky survey. See http://www.astro.caltech.edu/ ~george/dposs/.
8. WCER. Wisconsin Center for Education Research digital video processing project. See http://www.wcer.wisc.edu/.
9. NumRel. Numerical Relativity at LSU. See http://www.cct.lsu.edu/ numerical/.
10. MSCFD. Multiscale Computational Fluid Dynamics at LSU. See http://www.cct.lsu.edu/ IGERT/.
11. DOE. 2004 Report from the DOE Office of Science Data-Management Workshops, San Jose, CA, and Chicago, IL, March-May 2004. Washington, DC: The Office of Science Data-Management Challenge.
12. Tierney, B. L., Lee, J., Crowley, B., Holding, M., Hylton, J. & Drake, F. L. 1999 A networkaware distributed storage cache for data-intensive environments. In Proc. 8th Int. Symp. on High Performance Distributed Computing (HPDC'99), Redondo Beach, CA, 3-6 August 1999, pp. 185-189. Washington, DC: IEEE Computer Society.
13. Johnston, W. E., Gannon, D., Nitzberg, B., Tanner, L. A., Thigpen, B. & Woo, A. 2000 Computing and data grids for science and engineering. Dallas, TX: Supercomputing.
14. Allcock, B. et al. 2001 Secure, efficient data transport and replica management for highperformance data-intensive computing. In Proc. 18th Conf. IEEE Mass Storage Systems, San Diego, CA, 17-20 April 2001. Washington, DC: IEEE Computer Society.
15. Allcock, B. et al. 2002 Data management and transfer in high performance computational grid environments. Parallel Comput. 28, 749-771. (Pubitemid 34801987)
16. Allcock, B. et al. 2001 High-performance remote access to climate simulation data: A challenge problem for data grid technologies. In Proc. 2001 ACM/IEEE Conf. on Supercomputing, Denver, CO, 10-16 November 2001. Washington, DC: IEEE Computer Society.
17. Ranganathan, R. & Foster, I. 2002 Decoupling computation and data scheduling in distributed data-intensive applications. In Proc. 11th IEEE Int. Symp. on High Performance Distributed Computing (HPDC-11), Edinburgh, UK, 23-26 July 2002, pp. 352. Washington, DC: IEEE Computer Society.
18. Ranganathan, R. & Foster, I. 2004 Computation scheduling and data replication algorithms for data Grids. In Grid Resource Management: State of the Art and Future Trends (eds J. Nabrzyski, J. M. Schopf & J. Weglarz), pp. 359-373. Norwell, MA: Kluwer Academic Publishers.
19. Venugopal, S., Buyya, R. & Winton, L. 2004 A grid service broker for scheduling distributed data-oriented applications on global grids. In Proc. 2nd Workshop on Middleware for Grid Computing, Toronto, Canada, 18-22 October 2004, pp. 75-80. New York, NY: ACM.
20. ROOT. 2006 An object oriented data analysis framework. See http://root.cern.ch
21. Ravi, M. K., Cynthia, H. S. & William, E. A. 2002 Reliable file transfer in Grid environments. In Proc. 27th Annual IEEE Conf. on Local Computer Networks, Tampa, FL, 6-8 November 2002, pp. 737-738. Washington, DC: IEEE Computer Society.
22. Koranda, S. & Moe, M. 2007 Lightweight data replicator. See http://www.ligo.caltech.edu/ docs/G/G030623-00/G030623-00.pdf
23. Chervenak, A., Schuler, C., Kesselman, C., Koranda, S. & Moe, B. 2005 Wide area data replication for scientific collaborations. In Proc. 6th IEEE/ACM Int. Workshop on Grid Computing Seattle, WA, 13-14 November 2005. Washington, DC: IEEE Computer Society. (doi:10.1109/GRID.2005.1542717) (Pubitemid 44549195)
24. Beck, M., Elwasif, W. R., Plank, J. & Moore, T. 1999 The internet backplane protocol: Storage in the network. In Proc. 1999 Network Storage Symp. NetStore99, Seattle, WA, 14-15 October 1999. Washington, DC: IEEE Computer Society.
25. SRB. The Storage Resource Broker. See http://www.sdsc.edu/srb/.
26. iRODS. The Integrated Rule Oriented Data System. See http://www.irods. org/.
27. SRM. The Storage Resource Managers. See http://sdm.lbl.gov/srm.
28. Thain, D., Arpaci Dusseau, A., Bent, J. & Livny, M. 2004 Explicit control in a batch aware distributed file system. In Proc. 1st USENIX/ACM Conf. on Networked Systems Design and Implementation, San Francisco, CA, 29-31 March 2004. Berkeley, CA: USENIX.
29. Bent, J. 2005 Data-driven batch scheduling. PhD thesis, University of Wisconsin-Madison, Madison, USA.
30. Stockinger, H. 2005 Data management in data grids: Habilitation overview. Report by the Research Laboratory for Computational Technologies and Applications, University of Vienna.
31. Stockinger, H., Laure, E. & Stockinger, K. 2005 Performance engineering in data Grids. J. Concurrency Comput. Practice Exp. 17, 171-191. (doi:10.1002/cpe.923) (Pubitemid 40285774)
32. Stockinger, K., Schikuta, E., Stockinger, H. & Willers, I. 2001 Towards a cost model for distributed and replicated data stores. In Proc. 9th Euromicro Workshop on Parallel and Distributed Processing (PDP 2001), Mantova, Italy, 7-9 February 2001. Washington, DC: IEEE Computer Society.
33. Crowcroft, J. & Oechslin, P. 1998 Differentiated end-to-end Internet services using a weighted proportional fair sharing TCP. ACM SIGCOMM Comput. Commun. Rev. 28, 53-69. (doi:10.1145/293927.293930)
34. Hacker, T. J., Noble, B. D. & Atley, D. 2002 The end-to-end performance effects of parallel TCP sockets on a lossy wide area network. In Proc. 16th Int. Parallel and Distributed Processing Symposium, Fort Lauderdale, FL, 15-19 April 2002, pp. 314. Washington, DC: IEEE Computer Society.
35. Lu, D., Qiao, Y., Dinda, P. A. & Stamante, F. E. 2005 Modeling and taming parallel TCP on the wide area network. In Proc. 19th Int. Parallel and Distributed Processing Symposium, Denver, CO, 4-8 April 2005, pp. 682. Washington, DC: IEEE Computer Society.
36. Altman, E., Barman, D., Tuffin, T. & Voinovic, M. 2006 Parallel TCP sockets: Simple model, throughput and validation. In Proc. INFOCOM 2006 25th IEEE Int. Conf. Computer Communications, Joint Conference of the IEEE Computer and Communications Societies, Barcelona, Spain, 23-29 April 2006, pp. 1-12. Washington, DC: IEEE Computer Society.
37. Kosar, T. 2005 Data placement in widely distributed systems. PhD thesis, University of Wisconsin-Madison, Madison, USA.
38. Kosar, T., Kola, G., Livny, M., Brunner, R.J. & Remijan, M. 2005 Reliable, automatic transfer and processing of large scale astronomy data sets. In Proc. Astronomical Data Analysis Software and Systems (ADASS) XV, San Lorenzo de El Escorial, Spain, 2-5 October 2005.
39. Kola, G., Kosar, T. & Livny, M. 2004 A Fully automated fault-tolerant system for distributed video processing and off-site replication. In Proc. 14th ACM Int. Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV 2004), Cork, Ireland, 16-18 June 2004. New York, NY: ACM.
40. Ceyhan, E., Allen, G., White, C. & Kosar, T. 2008 A grid-enabled workflow system for reservoir uncertainty analysis. In Proc. 6th Int. Workshop Challenges of Large Applications in Distributed Environments (CLADE 2008), Boston, MA, 23-27 June 2008. New York, NY: ACM.
41. Yildirim, E., Yin, D. & Kosar, T. In press. Prediction of optimal parallelism level in wide area data transfers. IEEE Trans. Parallel Distrib. Syst.
42. LUSTRE Cluster File System, Inc. Lustre: A scalable, high performance file system. See .