Backfilling using system-generated predictions rather than user runtime estimates

IEEE Transactions on Parallel and Distributed Systems

Volumn 18, Issue 6, 2007, Pages 789-803

  Tsafrir, Dan ^a   Etsion, Yoav ^a   Feitelson, Dror G ^a  

^a HEBREW UNIVERSITY OF JERUSALEM   (Israel)

Author keywords

Backfilling; Dynamic prediction correction; EASY; EASY++; History based predictions; Parallel job scheduling; Performance metrics; Runtime estimates; SJBF; System generated predictions

Indexed keywords

ESTIMATION; PARALLEL PROCESSING SYSTEMS; PREDICTIVE CONTROL SYSTEMS; QUEUEING THEORY; SCHEDULING ALGORITHMS;

BACKFILLING; HISTORY-BASED PREDICTIONS; PARALLEL JOB SCHEDULING; PERFORMANCE METRICS; SYSTEM GENERATED PREDICTIONS;

SUPERCOMPUTERS;

EID: 34248674898     PISSN: 10459219     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPDS.2007.70606     Document Type: Article

References (39)

1. M.F. Arlitt and C.L. Williamson, "A Synthetic Workload Model for Internet Mosaic Traffic," Proc. Summer Computer Simulation Conf. (SCSC), pp. 852-857, July 1995.
2. S.-H. Chiang, A. Arpaci-Dusseau, and M.K. Vernon, "The Impact of More Accurate Requested Runtimes on Production Job Scheduling Performance," Job Scheduling Strategies for Parallel Processing (JSSPP), pp. 103-127, July 2002.
3. S.-H. Chiang and M.K. Vernon, "Production Job Scheduling for Parallel Shared Memory Systems," Proc. 15th IEEE Int'l Parallel and Distributed Processing Symp. (IPDPS), Apr. 2001.
4. W. Cirne and F. Berman, "Using Moldability to Improve the Performance of Supercomputer Jobs," J. Parallel and Distributed Computing, vol. 62, no. 10, pp. 1571-1601, Oct. 2002.
5. W. Cirne and F. Berman, "When the Herd Is Smart: Aggregate Behavior in the Selection of Job Request," IEEE Trans. Parallel and Distributed Systems, vol. 14, no. 2, pp. 181-192, Feb. 2003.
6. A.B. Downey, "Predicting Queue Times on Space-Sharing Parallel Computers," Proc. 11th IEEE Int'l Parallel Processing Symp. (IPPS) pp. 209-218, Apr. 1997.
7. Y. Etsion and D. Tsafrir, "A Short Survey of Commercial Cluster Batch Schedulers," Technical Report 2005-13, the Hebrew Univ. of Jerusalem, May 2005.
8. D.G. Feitelson, "Experimental Analysis of the Root Causes of Performance Evaluation Results: A Backfilling Case Study," IEEE Trans. Parallel and Distributed Systems, vol. 16, no. 2, pp. 175-182, Feb. 2005.
9. D.G. Feitelson and A. Mu'alem Weil, "Utilization and Predictability in Scheduling the IBM SP2 with Backfilling," Proc. 12th IEEE Int'l Parallel Processing Symp. (IPPS), pp. 542-546 Apr. 1998.
10. D.G. Feitelson and B. Nitzberg, "Job Characteristics of a Production Parallel Scientific Workload on the NASA Ames iPSC/860," Job Scheduling Strategies for Parallel Processing (JSSPP), pp. 337-360, Apr. 1995.
11. D.G. Feitelson, L. Rudolph, and U. Schwiegelshohn, "Parallel Job Scheduling - a Status Report," Job Scheduling Strategies for Parallel Processing (JSSPP), pp. 1-16, June 2004.
12. D.G. Feitelson and D. Tsafrir, "Workload Sanitation for Performance Evaluation," Proc. IEEE Int'l Symp. Performance Analysis of Systems and Software (ISPASS), Mar. 2006.
13. R. Gibbons, "A Historical Application Profiler for Use by Parallel Schedulers," Job Scheduling Strategies for Parallel Processing (JSSPP), pp. 58-77, Apr. 1997.
14. D. Jackson, "Maui/Moab Default Configuration," personal comm. with CTO of Cluster Resources, Jan. 2006.
15. D. Jackson, Q. Snell, and M. Clement, "Core Algorithms of the Maui Scheduler," Job Scheduling Strategies for Parallel Processing (JSSPP), pp. 87-102 June 2001.
16. R. Jain, The Art of Computer Systems Performance Analysis. John Wiley and Sons, 1991
17. J.P. Jones and B. Nitzberg, "Scheduling for Parallel Supercomputing: A Historical Perspective of Achievable Utilization," Job Scheduling Strategies for Parallel Processing (JSSPP), pp. 1-16, Apr. 1999.
18. S. Kannan, M. Roberts, P. Mayes, D. Brelsford, and J.F. Skovira, Workload Management with LoadLeveler, first ed. IBM, Nov. 2001.
19. N.H. Kapadia, J.A.B. Fortes, and C.E. Brodley, "Predictive Application-Performance Modeling in a Computational Grid Environment," Proc. IEEE Int'l Symp. High Performance Distributed Computing (HPDC) p. 6, Aug. 1999.
20. S. Krishnaswamy, S.W. Loke, and A. Zaslavsky, "Estimating Computation Times of Data-Intensive Applications," IEEE Distributed Systems Online, vol. 5, no. 4, Apr. 2004.
21. J. Labarta, J. Corbalán, F. Guim, and I. Rodero, "The eNANOS Grid," http://www.bsc.es/grid/enanos, 2006.
22. B.G. Lawson and E. Smirni, "Multiple-Queue Backfilling Scheduling with Priorities and Reservations for Parallel Systems," Job Scheduling Strategies for Parallel Processing (JSSPP), pp. 72-87, July 2002.
23. C.B. Lee, Y. Schwartzman, J. Hardy, and A. Snavely, "Are User Runtime Estimates Inherently Inaccurate?" Job Scheduling Strategies for Parallel Processing (JSSPP), pp. 253-263, June 2004.
24. H. Li, D. Groep, J. Templon, and L. Wolters, "Predicting Job Start Times on Clusters," Proc. Sixth IEEE Int'l Symp. Cluster Computing and the Grid (CCGrid), May 2004.
25. D. Lifka, "The ANL/IBM SP Scheduling System," Job Scheduling Strategies for Parallel Processing (JSSPP), pp. 295-303, Apr. 1995.
26. A.W. Mu'alem and D.G. Feitelson, "Utilization, Predictability, Workloads, and User Runtime Estimates in Scheduling the IBM SP2 with Backfilling," IEEE Trans. Parallel and Distributed Systems, vol. 12, no. 6, pp. 529-543, June 2001.
27. "Parallel Workloads Archive," http://www.cs.huji.ac.il/labs/parallel/workload/, 2006.
28. D. Perkovic and P.J. Keleher, "Randomization, Speculation, and Adaptation in Batch Schedulers," Supercomputing, p. 7, Sept. 2000.
29. E. Shmueli and D.G. Feitelson, "Backfilling with Lookahead to Optimize the Packing of Parallel Jobs," J. Parallel and Distributed Computing (JPDC), vol. 65, no. 9, pp. 1090-1107, Sept. 2005.
30. W. Smith, I. Foster, and V. Taylor, "Predicting Application Run Times Using Historical Information," Job Scheduling Strategies for Parallel Processing (JSSPP), pp. 122-142, Mar. 1998.
31. W. Smith, I. Foster, and V. Taylor, "Scheduling with Advanced Reservations," Proc. 14th IEEE Int'l Parallel and Distributed Processing Symp. (IPDPS), pp. 127-132, May 2000.
32. W. Smith, V. Taylor, and I. Foster, "Using Run-Time Predictions to Estimate Queue Wait Times and Improve Scheduler Performance," Job Scheduling Strategies for Parallel Processing (JSSPP), pp. 202-219, Apr. 1999.
33. S. Srinivasan, R. Kettimuthu, V. Subramani, and P. Sadayappan, "Selective Reservation Strategies for Backfill Job Scheduling," Job Scheduling Strategies for Parallel Processing (JSSPP), pp. 55-71, July 2002.
34. S. Srinivasan, R. Kettimuthu, V. Subrarnani, and P. Sadayappan, "Characterization of Backfilling Strategies for Parallel Job Scheduling," Proc. Int'l Conf. Parallel Processing (ICPP), pp. 514-522, Aug. 2002.
35. D. Tsafrir, Y. Etsion, and D.G. Feitelson, "Modeling User Runtime Estimates," Job Scheduling Strategies for Parallel Processing (JSSPP), pp. 1-35, June 2005.
36. D. Tsafrir and D.G. Feitelson, "The Dynamics of Backfilling: Solving the Mystery of Why Increased Inaccuracy May Help," Proc. IEEE Int'l Symp. Workload Characterization (IISWC), pp. 131-141, Oct. 2006.
37. D. Tsafrir and D.G. Feitelson, "Instability in Parallel Job Scheduling Simulation: The Role of Workload Flurries," Proc. 20th IEEE Int'l Parallel and Distributed Processing Symp. (IPDPS), p. 10, Apr. 2006.
38. Y. Zhang, H. Franke, J. Moreira, and A. Sivasubramaniam, "An Integrated Approach to Parallel Scheduling Using Gang-Scheduling, Backfilling, and Migration," IEEE Trans. Parallel and Distributed Systems, vol. 14, no. 3, pp. 236-247, Mar. 2003.
39. D. Zotkin and P.J. Keleher, "Job-Length Estimation and Performance in Backfilling Schedulers," Proc. IEEE Int'l Symp. High Performance Distributed Computing (HPDC), p. 39, Aug. 1999.