The performance of bags-of-tasks in large-scale distributed systems

Proceedings of the 17th International Symposium on High Performance Distributed Computing 2008, HPDC'08

Volumn , Issue , 2008, Pages 97-108

  Losup, Alexandru ^a   Sonmez, Ozan ^a   Anoep, Shanny ^a   Epema, Dick ^a  

^a DELFT UNIVERSITY OF TECHNOLOGY   (Netherlands)

Author keywords

Bags of tasks; Batches of jobs; Large scale distributed systems; Multi cluster systems; Performance evaluation; Prediction; Scheduling; Trace based simulation; Workload modeling

Indexed keywords

DIE CASTING; DISTRIBUTED COMPUTER SYSTEMS; HIGH PERFORMANCE LIQUID CHROMATOGRAPHY; SCHEDULING;

BAGS-OF-TASKS; BATCHES OF JOBS; LARGE-SCALE DISTRIBUTED SYSTEMS; MULTI-CLUSTER SYSTEMS; PERFORMANCE EVALUATION; PREDICTION; TRACE-BASED SIMULATION; WORKLOAD MODELING;

COMPUTER SYSTEMS;

EID: 57349116875     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1383422.1383435     Document Type: Conference Paper

References (39)

1. The Grid Workloads Archive. [Online] http://gwa.ewi.tudelft.nl., Mar 2008.
2. J. Aldrich. R. A. Fisher and the making of maximum likelihood 1912-1922. Statistical Science, 12(3):162-176, 1997.
3. O. Beaumont, L. Carter, J. Ferrante, A. Legrand, L. Marchal, and Y. Robert. Centralized versus distributed schedulers for multiple bag-of-task applications. In IPDPS. IEEE CS, 2006.
4. O. Beaumont, H. Casanova, A. Legrand, Y. Robert, and Y. Yang. Scheduling divisible loads on star and tree networks: Results and open problems. IEEE Trans. Parallel Distrib. Syst., 16(3):207-218, 2005.
5. F. Berman, R. Wolski, H. Casanova, W. Cirne, H. Dail, M. Faerman, S. M. Figueira, J. Hayes, G. Obertelli, J. M. Schopf, G. Shao, S. Smallen, N. T. Spring, A. Su, and D. Zagorodnov. Adaptive computing on the grid using apples. IEEE Trans. Parallel Distrib. Syst., 14(4):369-382, 2003.
6. V. Bharadwaj, D. Ghose, and T. G. Robertazzi. Divisible load theory: A new paradigm for load scheduling in distributed systems. Cluster Comput., 6(1):7-17, Jan. 2003.
7. P. J. Brockwell and R. A. Davis. Introduction to time series and forecasting. Springer, 1996.
8. T. E. Carroll and D. Grosu. A strategy proof mechanism for scheduling divisible loads in bus networks without control processors. In IPDPS. IEEE CS, 2006.
9. H. Casanova, A. Legrand, D. Zagorodnov, and F. Berman. Heuristics for scheduling parameter sweep applications in grid environments. In HCW, pages 349-363. IEEE CS, 2000.
10. H. Casanova, G. Obertelli, F. Berman, and R. Wolski. The apples parameter sweep template: User-level middleware for the grid. In SC, 2000.
11. W. Cirne, D. P. da Silva, L. Costa, E. Santos-Neto, F. V. Brasileiro, J. P. Sauvé, F. A. B. Silva, C. O. Barros, and C. Silveira. Running bag-of-tasks applications on computational grids: The mygrid approach. In ICPP, pages 407-. IEEE CS, 2003.
12. D. P. da Silva, W. Cirne, and F. V. Brasileiro. Trading cycles for information: Using replication to schedule bag-of-tasks applications on computational grids. In H. Kosch, L. Böszörményi, and H. Hellwagner, editors, Euro-Par, volume 2790 of LNCS, pages 169-180. Springer-Verlag, 2003.
13. A. B. Downey and D. G. Feitelson. The elusive goal of workload characterization. Performance Evaluation Review, 26(4):14-29, 1999.
14. D. Epema, M. Livny, R. Dantzig, X. Evers, and J. Pruyne. A worldwide flock of Condors: Load sharing among workstation clusters. Future Gener. Comput. Syst., 12:53-65.
15. C. Ernemann, B. Song, and R. Yahyapour. Scaling of workload traces. In D. G. Feitelson, L. Rudolph, and U. Schwiegelshohn, editors, JSSPP, volume 2862 of LNCS, pages 166-182. Springer-Verlag, 2003.
16. M. Evans, N. Hastings, and B. Peacock. Statistical Distributions. Wiley, 3rd edition, 2000.
17. D. G. Feitelson and L. Rudolph. Metrics and benchmarking for parallel job scheduling. In IPPS/SPDP, volume 1459 of LNCS, pages 1-24, 1998.
18. E. Frachtenberg and D. G. Feitelson. Pitfalls in parallel job scheduling evaluation. In D. G. Feitelson, E. Frachtenberg, L. Rudolph, and U. Schwiegelshohn, editors, JSSPP, volume 3834 of LNCS, pages 257-282. Springer-Verlag, 2005.
19. N. Fujimoto and K. Hagihara. Near-optimal dynamic task scheduling of independent coarse-grained tasks onto a computational grid. In ICPP, pages 391-398. IEEE CS, 2003.
20. N. Garg, D. Grosu, and V. Chaudhary. An antisocial strategy for scheduling mechanisms. In IPDPS. IEEE CS, 2005.
21. A. Iosup, C. Dumitrescu, D. H. Epema, H. Li, and L. Wolters. How are real grids used? The analysis of four grid traces and its implications. In GRID, pages 262-270. IEEE CS, 2006.
22. A. Iosup, D. Epema, T. Tannenbaum, M. Farrellee, and M. Livny. Inter-operating grids through delegated matchmaking. In SC. ACM Press, 2007.
23. A. Iosup, M. Jan, O. Sonmez, and D. Epema. The characteristics and performance of groups of jobs in grids. In Euro-Par, volume 4641 of LNCS, pages 382-393. Springer-Verlag, 2007.
24. A. Iosup, M. Jan, O. Sonmez, and D. H. Epema. On the dynamic resource availability in grids. In GRID, pages 26-33. IEEE CS, 2007.
25. A. Iosup, H. Li, M. Jan, S. Anoep, C. Dumitrescu, L. Wolters, and D. Epema. The grid workloads archive. Future Gener. Comput. Syst., Feb. 2008. (in print).
26. A. Iosup, O. Sonmez, and D. Epema. The DGSim trace-based simulator. TU Delft, Tech. Rep. PDS-2008-003, Mar 2008. ISSN 1387-2109.
27. D. Kondo, G. Fedak, F. Cappello, A. A. Chien, and H. Casanova. Characterizing resource availability in enterprise desktop grids. Future Gener. Comput. Syst.,23(7):888-903, 2007.
28. Y. C. Lee and A. Y. Zomaya. Practical scheduling of bag-of-tasks applications on grids with dynamic resilience. IEEE Trans. Computers, 56(6):815-825, 2007.
29. H. Li, M. Muskulus, and L. Wolters. Modeling job arrivals in a data-intensive grid. In E. Frachtenberg and U. Schwiegelshohn, editors, JSSPP, volume 4376 of LNCS, pages 210-231. Springer-Verlag, 2006.
30. H. W. Lilliefors. On the Kolmogorov-Smirnov test for the exponential distribution with mean unknown. Journal of the American Statistical Association, 64:387-389, 1969.
31. U. Lublin and D. G. Feitelson. The workload on parallel supercomputers: modeling the characteristics of rigid jobs. J. Parallel Distrib. Comput., 63(11):1105-1122, 2003.
32. M. Maheswaran, S. Ali, H. J. Siegel, D. A. Hensgen, and R. F. Freund. Dynamic matching and scheduling of a class of independent tasks onto heterogeneous computing systems. In Heterogeneous Computing Workshop, pages 30-, 1999.
33. D. A. Menascé, D. Saha, S. C. S. Porto, V. Almeida, and S. K. Tripathi. Static and dynamic processor scheduling disciplines in heterogeneous parallel architectures. J. Parallel Distrib. Comput., 28(1):1-18, 1995.
34. M. E. J. Newman. Power laws, pareto distributions and zipf's law. Contemporary Physics, 46:323, 2005.
35. A. K. Parekh and R. G. Gallager. A generalized processor-sharing approach to flow control in integrated services networks: the multiple node case. IEEE/ACM Trans. Netw., 2(2):137-150, 1994.
36. R. F. Rosin. Determining a computing center environment. Commun. ACM, 8(7):463-468, 1965.
37. R. K. Sahoo, A. J. Oliner, I. Rish, M. Gupta, J. E. Moreira, S. Ma, R,. Vilalta, and A. Sivasubramaniam. Critical event prediction for proactive management in large-scale computer clusters. In L. Getoor, T. E. Senator, P. Domingos, and C. Faloutsos, editors, KDD, pages 426-435. ACM, 2003.
38. SPECCPU Team. SPEC CPU2006. Standard Performance Evaluation Corporation, Mar 2008. [Online] Available: http://www.spec.org/cpu2006/.
39. D. Thain, T. Tannenbaum, and M. Livny. Distributed computing in practice: the condor experience. Elsevier Concurrency - Practice and Experience, 17(2-4):323-356, 2005.