Experiences with resource provisioning for scientific workflows using Corral

Scientific Programming

Volumn 18, Issue 2, 2010, Pages 77-92

  Juve, Gideon ^a   Deelman, Ewa ^a   Vahi, Karan ^a   Mehta, Gaurang ^a  

^a UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

Author keywords

distributed computing; glideins; grid computing; high throughput computing; pilot jobs; Scientific workflows; web services

Indexed keywords

DISTRIBUTED COMPUTER SYSTEMS; FENCES; SCHEDULING; WEB SERVICES;

DISTRIBUTED RESOURCES; EXECUTION ENVIRONMENTS; GLIDEINS; HIGH-THROUGHPUT COMPUTING; PILOT JOBS; SCIENTIFIC APPLICATIONS; SCIENTIFIC WORKFLOWS; WORKFLOW-BASED APPLICATIONS;

GRID COMPUTING;

EID: 77954738994     PISSN: 10589244     EISSN: None     Source Type: Journal    
DOI: 10.3233/SPR-2010-0300     Document Type: Article

References (38)

1. J. Blythe, S. Jain, E. Deelman, Y. Gil, K. Vahi, A. Mandal and K. Kennedy, Task scheduling strategies for workflow-based applications in grids, in: 5th IEEE International Symposium on Cluster Computing and the Grid (CCGrid '05), Vol.2, Cardiff, UK, 2005, pp. 759-767.
2. S. Callaghan, P. Maechling, E. Deelman, K. Vahi, G. Mehta, G. Juve, K. Milner, R. Graves, E. Field, D. Okaya and T. Jordan, Reducing time-to-solution using distributed highthroughput mega-workflows: experiences from SCEC Cyber- Shake, in: 4th IEEE International Conference on e-Science (e-Science 08), Indianapolis, IN, 2008.
3. H. Casanova, A. Legrand, D. Zagorodnov and F. Berman, Heuristics for scheduling parameter sweep applications in grid environments, in: 9th Heterogeneous Computing Workshop, Cancun, Mexico, 2000, pp. 349-363.
4. A. Chervenak, N. Palavalli, S. Bharathi, C. Kesselman and R. Schwartzkopf, Performance and scalability of a replica location service, in: 13th IEEE International Symposium on High- Performance Distributed Computing (HPDC '04), Honolulu, HI, 2004, pp. 182-191.
5. A. Chervenak, R. Schuler, M. Ripeanu, M.A. Amer, S. Bharathi, I. Foster, A. Iamnitchi and C. Kesselman, The globus replica location service: design and experience, IEEE Transactions on Parallel and Distributed Systems 20 (2005), 1260-1272.
6. Condor Connection Brokering (CCB), available at: http://www. cs.wisc.edu/condor/manual/v7.3/3-7Networking-includes. html#SECTION00473000000000000000.
7. Condor-glidein, available at: http://www.cs.wisc.edu/condor/ glidein.
8. E. Deelman, Grids and clouds: making workflow applications work in heterogeneous distributed environments, International Journal of High Performance Computing Applications, December 4, 2009, available at: http://hpc.sagepub. com/cgi/content/abstract/1094342009356432v1.
9. E. Deelman, S. Callaghan, E. Field, H. Francoeur, R. Graves, N. Gupta, V. Gupta, T.H. Jordan, C. Kesselman, P. Maechling, J. Mehringer, G. Mehta, D. Okaya, K. Vahi and L. Zhao, Managing large-scale workflow execution from resource provisioning to provenance tracking: the CyberShake example, in: 2nd IEEE International Conference on e-Science and Grid Computing (e-Science 06), Amsterdam, The Netherlands, 2006.
10. E. Deelman, C. Kesselman, G. Mehta, L. Meshkat, L. Pearlman, K. Blackburn, P. Ehrens, A. Lazzarini, R. Williams and S. Koranda, GriPhyN and LIGO, building a virtual data grid for gravitational wave scientists, in: 11th IEEE International Symposium on High Performance Distributed Computing (HPDC '02), Edinburgh, Scotland, UK, 2002, pp. 225-234.
11. E. Deelman, G. Singh, M. Su, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, G.B. Berriman, J. Good, A. Laity, J.C. Jacob and D.S. Katz, Pegasus: A framework for mapping complex scientific workflows onto distributed systems, Scientific Programming 13 (2005), 219-237.
12. R. Fielding, Architectural styles and the design of networkbased software architectures, PhD thesis, University of California, Irvine, 2000.
13. J. Frey, T. Tannenbaum, M. Livny, I. Foster and S. Tuecke, Condor-G: A computation management agent for multi-institutional grids, Cluster Computing 5 (2002), 237-246.
14. Generic Connection Brokering (GCB), available at: http://cs. wisc.edu/condor/gcb.
15. W. Gentzsch, Sun Grid Engine: towards creating a compute power grid, in: 1st IEEE/ACM International Symposium on Cluster Computing and the Grid (CCGrid '01), Brisbane, Australia, 2001, pp. 35-36.
16. G. Juve and E. Deelman, Resource provisioning options for large-scale scientific workflows, in: IEEE Fourth International Conference on e-Science (e-Science 08), Indianapolis, IN, 2008, pp. 608-613.
17. G. Juve, E. Deelman, K. Vahi and G. Mehta, Scientific workflow applications on Amazon EC2, in: Workshop on Cloud- Based Services and Applications in Conjunction with 5th IEEE International Conference on e-Science (e-Science 2009), Oxford, UK, 2009.
18. D.S. Katz, J.C. Jacob, E. Deelman, C. Kesselman, S. Gurmeet, S. Mei-Hui, G.B. Berriman, J. Good, A.C. Laity and T.A. Prince, A comparison of two methods for building astronomical image mosaics on a grid, in: 34th International Conference on Parallel Processing Workshops (ICPP '05 Workshops), Oslo, Norway, 2005, pp. 85-94.
19. Y. Kee, C. Kesselman, D. Nurmi and R. Wolski, Enabling personal clusters on demand for batch resources using commodity software, in: International Heterogeneity Computing Workshop (HCW 08) in Conjunction with IEEE IPDPS 08, Miami, FL, 2008.
20. M.J. Litzkow, M. Livny and M.W. Mutka, Condor: A hunter of idle workstations, in: 8th International Conference of Distributed Computing Systems (ICDCS '88), San Jose, CA, 1988, pp. 104-111.
21. M. Maheswaran, S. Ali, H.J. Siegel, D. Hensgen and R.F. Freund, Dynamic matching and scheduling of a class of independent tasks onto heterogeneous computing systems, in: 8th Heterogeneous Computing Workshop, San Juan, Puerto Rico, 1999.
22. Maui Cluster Scheduler, available at: http://www.supercluster. org/maui.
23. J. Moscicki, DIANE - distributed analysis environment for GRID-enabled simulation and analysis of physics data, in: IEEE Nuclear Science Symposium Conference Record, Portland, OR, 2003, pp. 1617-1620.
24. D. Nurmi, R. Wolski and J. Brevik, VARQ: virtual advance reservations for queues, in: 17th International Symposium on High Performance Distributed Computing (HPDC '08), Boston, MA, 2008.
25. Open Science Grid, available at: http://www.opensciencegrid. org.
26. PBSPro, available at: http://www.pbspro.com.
27. I. Raicu, Y. Zhao, C. Dumitrescu, I. Foster and M. Wilde, Falkon: a Fast and Light-weight tasK executiON framework, in: Supercomputing 2007, Reno, NV, 2007.
28. SDSC User Portal, available at: http://portal.sdsc.edu.
29. I. Sfiligoi, GlideinWMS - a generic pilot-based workload management system, Journal of Physics: Conference Series 119 (2008), 062044.
30. G. Singh, C. Kesselman and E. Deelman, Optimizing gridbased workflow execution, Journal of Grid Computing 3 (2005), 201-219.
31. G. Singh, C. Kesselman and E. Deelman, Performance impact of resource provisioning on workflows, Technical Report 05-850, University of Southern California, 2005.
32. G. Singh, M. Su, K. Vahi, E. Deelman, B. Berriman, J. Good, D.S. Katz and G. Mehta, Workflow task clustering for best effort systems with Pegasus, in: 15th ACM Mardi Gras Conference, Baton Rouge, LA, 2008.
33. Southern California Earthquake Center (SCEC), available at: http://www.scec.org.
34. I.J. Taylor, E. Deelman, D.B. Gannon and M. Shields, Workflows for e-Science: Scientific Workflows for Grids, Springer- Verlag, New York, 2006.
35. TeraGrid, available at: http://www.teragrid.org.
36. H. Topcuoglu, S. Hariri and W. Min-You, Performanceeffective and low-complexity task scheduling for heterogeneous computing, IEEE Transactions on Parallel and Distributed Systems 13 (2002), 260-274.
37. Torque, available at: http://supercluster.org/torque.
38. E. Walker, J.P. Gardner, V. Litvin and E. Turner, Creating personal adaptive clusters for managing scientific jobs in a distributed computing environment, in: IEEE International Workshop on Challenges of Large Applications in Distributed Environments (CLADE 06), Paris, France, 2006.