Analyzing and adjusting user runtime estimates to improve job scheduling on the Blue Gene/P

Proceedings of the 2010 IEEE International Symposium on Parallel and Distributed Processing, IPDPS 2010

Volumn , Issue , 2010, Pages

  Tang, Wei ^a   Desai, Narayan ^b   Buettner, Daniel ^b   Lan, Zhiling ^a  

^a Illinois Institute of Technology   (United States)

^b ARGONNE NATIONAL LABORATORY   (United States)

Author keywords

Blue Gene; Job scheduling; Runtime estimates

Indexed keywords

ARGONNE NATIONAL LABORATORY; BLUE GENE; HISTORICAL DATA; JOB SCHEDULING; JOB SCHEDULING POLICIES; PRIORITIZATION; RUNTIME ESTIMATES; RUNTIMES; SCHEDULING POLICIES;

DISTRIBUTED PARAMETER NETWORKS;

SCHEDULING;

EID: 77953990599     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/IPDPS.2010.5470474     Document Type: Conference Paper

References (32)

1. Blue Gene Team, "Overview of the IBM Blue Gene/P project, " IBM Journal of Research and Development 52(1/2), 199-220, 2008.
2. Cobalt project. http://trac.mcs.anl.gov/projects/cobalt
3. DOE INCITE program. http://www.er.doe.gov/ascr/incite
4. TOP500 Supercomputing web site, http://www.top500.org
5. S.-H. Chiang, A. Arpaci-Dusseau, and M. Vernon, "The impact of more accurate requested runtimes on production job scheduling performance, " in Proc. of Job Scheduling Strategies for Parallel Processing, 2002.
6. S.-H. Chiang and M. Vernon, "Production job scheduling for parallel shared memory systems, " in Proc. of International Parallel and Distributed Processing Symposium, 2001.
7. W. Cirne and F. Berman, "A comprehensive model of the supercomputer workload." in Proc. of IEEE International Workshop on Workload Characterization, 2001.
8. A. Downey, "Predicting queue times on space-sharing parallel computers, " in Proc. of IEEE International Parallel Processing Symposium, 1997.
9. N. Desai, D. Buntinas, D. Buettner, P. Balaji, and A. Chan, "Improving Resource Availability By Relaxing Network Allocation Constraints on the Blue Gene/P, " in Proc. of International Conference on Parallel Processing, 2009.
10. Y. Etsion and D. Tsafrir, "A short survey of commercial cluster batch schedulers, " Technical Report 2005-13, the Hebrew University of Jerusalem, 2005.
11. E. Frachtenberg and D. Feitelson, "Pitfalls in parallel job scheduling evaluation, " in Proc. of Job Scheduling Strategies for Parallel Processing, 2005.
12. R. Gibbons, "A historical application profiler for use by parallel schedulers, " in Proc. of Job Scheduling Strategies for Parallel Processing, 1997.
13. C. Glansner and J. Volkert, "An architecture for an adaptive run-time prediction system, " International Symposium on Parallel and Distributed Computing, 2008.
14. C. Glansner and J. Volkert, "A three-phase adaptive prediction system of run-time of jobs based on user behaviour, " International Conference on Complex, Intelligent and Software Intensive Systems, 2009.
15. L. K. Goh and B. Veeravalli, "Design and performance evaluation of combined first-fit task allocation and migration strategies in mesh multiprocessor systems", Parallel Computing 34(9), 508-520, 2008.
16. J. Jones and B. Nitzberg, "Scheduling for parallel supercomputing: A historical perspective of achievable utilization, " in Proc. of Job Scheduling Strategies for Parallel Processing, 1999.
17. P. Krueger, T. Lai, and V. Dixit-Radiya, "Job scheduling is more important than processor allocation for hypercube computers, " IEEE Transactions on Parallel and Distributed Systems 5(5), 488-497, 1994.
18. S. Krishnaswamy, S. Loke, and A. Zaslavsky, "Estimating computation times of data-intensive applications, " IEEE Distributed Systems Online, vol.5, no.4, Apr. 2004.
19. C. Lee, Y. Schwartzman, J. Hardy, and A. Snavely, "Are user runtime estimates inherently inaccurate?" in Proc. of Job Scheduling Strategies for Parallel Processing, 2004.
20. A. Mu'alem and D. Feitelson, "Utilization, predictability, workloads, and user runtime estimates in scheduling the IBM SP2 with backfilling, " IEEE Transactions on Parallel and Distributed Systems 12(6), 529-543, 2001.
21. D. Perkovic and P. Keleher, "Randomization, speculation, and adaptation in batch schedulers, " IEEE/ACM Supercomputing Conference, 2000.
22. W. Smith, "Prediction services for distributed computing, " in Proc. of IEEE International Parallel and Distributed Processing Symposium, 2007.
23. W. Smith, I. Foster, and V. Taylor, "Predicting application runtimes with historical information, " Journal of Parallel and Distributed Computing 64(9), 1007-1016, 2004.
24. W. Smith, V. Taylor, and I. Fostor, "Using run-time predictions to estimate queue wait times and improve scheduler performance, " in Proc. of Job Scheduling Strategies for Parallel Processing, 1999.
25. S. Srinivasan, R. Kettimuthu, V. Subramani, and P. Sadayappan, "Characterization of backfilling strategies for parallel job scheduling, " in Proc. of the International Conference on Parallel Processing Workshops, 2002.
26. W. Tang, Z. Lan, N. Desai, and D. Buettner, "Fault-aware, utility-based job scheduling on Blue Gene/P systems, " in Proc. of IEEE International Conference on Cluster Computing, 2009.
27. D. Tsafrir, Y. Etsion, and D. Feitelson, "Backfilling using system-generated predictions rather than user runtime estimates, " IEEE Transactions on Parallel and Distributed Systems 18(6), 789-803, 2007.
28. D. Tsafrir and D. Feitelson, "The dynamics of backfilling: Solving the mystery of why increased inaccuracy may help, " in Proc. of IEEE International Symposium on Workload Characterization, 2006.
29. W. Ward, C. Mahood, and J. West, "Scheduling jobs on parallel systems using a relaxed backfill strategy, " in Proc. of Job Scheduling Strategies for Parallel Processing, 2002.
30. B. Yoo and C. Das, "A fast and efficient processor allocation scheme for mesh-connected multicomputers, " IEEE Transactions on Computers 51(1), 46-60, 2002.
31. D. Zotkin and P. Keleher, "Job-length estimation and performance in backfilling schedulers, " in Proc. of IEEE International Symposium on High Performance Distributed Computing, 1999.
32. Y. Zhang, H. Franke, J. Moreira, and A. Sivasubramaniam, "Improving parallel job scheduling by combining gang scheduling and backfilling techniques, " in Proc. of IEEE International Parallel and Distributed Processing Symposium, 2000.