Online Task Resource Consumption Prediction for Scientific Workflows

Parallel Processing Letters

Volumn 25, Issue 3, 2015, Pages

  Da Silva, Rafael Ferreira ^a   Juve, Gideon ^a   Rynge, Mats ^a   Deelman, Ewa ^a   Livny, Miron ^b  

^a UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^b UNIVERSITY OF WISCONSIN MADISON   (United States)

Author keywords

MAPE K loop; online resource usage task estimation; Scientific workflow; workflow characterization

Indexed keywords

ACCURATE PREDICTION; CLUSTERING TECHNIQUES; MAPE; OFF-LINE APPROACHES; PEAK MEMORY CONSUMPTION; RESOURCE CONSUMPTION; SCIENTIFIC WORKFLOWS; TASK ESTIMATION;

DISTRIBUTION FUNCTIONS;

EID: 84942548951     PISSN: 01296264     EISSN: None     Source Type: Journal    
DOI: 10.1142/S0129626415410030     Document Type: Conference Paper

References (67)

1. I. Taylor, E. Deelman, D. Gannon, M. Shields, Workflows for e-Science, Springer, 2007.
2. J. D. Ullman, Np-complete scheduling problems, J. Comput. Syst. Sci. 10 (3) (1975) 384-393.
3. M. Maheswaran, S. Ali, H. J. Siegel, D. Hensgen, R. F. Freund, Dynamic matching and scheduling of a class of independent tasks onto heterogeneous computing systems, in: 8th Heterogeneous Computing Workshop, HCW '99, 1999, pp. 30-.
4. H. Topcuouglu, S. Hariri, M.-y. Wu, Performance-effective and low-complexity task scheduling for heterogeneous computing, IEEE Trans. Parallel Distrib. Syst. 13 (3) (2002) 260-274.
5. M. Rahman, R. Hassan, R. Ranjan, R. Buyya, Adaptive workflow scheduling for dynamic grid and cloud computing environment, Concurrency and Computation: Practice and Experience (2013) n/a-n/a.
6. S. Su, J. Li, Q. Huang, X. Huang, K. Shuang, J. Wang, Cost-efficient task scheduling for executing large programs in the cloud, Parallel Computing 39 (45) (2013) 177-188.
7. K. Bessai, S. Youcef, A. Oulamara, C. Godart, S. Nurcan, Bi-criteria workflow tasks allocation and scheduling in cloud computing environments, in: IEEE 5th Intern. Conference on Cloud Computing, 2012, pp. 638-645.
8. A. Wierman, M. Nuyens, Scheduling despite inexact job-size information, in: ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, 2008, pp. 25-36. doi:10.1145/1375457.1375461.
9. M. Dell'Amico, D. Carra, M. Pastorelli, P. Michiardi, Revisiting size-based scheduling with estimated job sizes, in: IEEE 22nd International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, 2014.
10. S. Verboven, P. Hellinckx, F. Arickx, J. Broeckhove, Runtime prediction based grid scheduling of parameter sweep jobs, in: Asia-Pacific Services Computing Conference, 2008. APSCC '08. IEEE, 2008, pp. 33-38. doi:10.1109/APSCC.2008.189.
11. C. Yan, H. Luo, Z. Hu, X. Li, Y. Zhang, Deadline guarantee enhanced scheduling of scientific workflow applications in grid, Journal of Computers 8 (4).
12. J. O. Gutierrez-Garcia, K. M. Sim, A family of heuristics for agent-based elastic cloud bag-of-tasks concurrent scheduling, Future Generation Computer Systems 29 (7) (2013) 1682-1699. doi:http://dx.doi.org/10.1016/j.future.2012.01.005.
13. G. Juve, A. Chervenak, E. Deelman, S. Bharathi, G. Mehta, K. Vahi, Characterizing and profiling scientific workflows, Future Generation Computer Systems 29 (3) (2013) 682-692, special Section: Recent Developments in High Performance Computing and Security. doi:10.1016/j.future.2012.08.015.
14. J.-S. Vöckler, G. Mehta, Y. Zhao, E. Deelman, M. Wilde, Kickstarting remote applications, in: 2nd International Workshop on Grid Computing Environments, 2006.
15. G. Juve, B. Tovar, R. Ferreira da Silva, C. Robinson, D. Thain, E. Deelman, W. Allcock, M. Livny, Practical resource monitoring for robust high throughput computing, Tech. rep., University of Southern California (2014).
16. G. Juve, A. Chervenak, E. Deelman, S. Bharathi, G. Mehta, K. Vahi, Characterizing and profiling scientific workflows, Future Generation Computer Systems 29 (3) (2014) 682-692.
17. F. Nadeem, M. Yousaf, R. Prodan, T. Fahringer, Soft benchmarks-based application performance prediction using a minimum training set, in: 2nd IEEE International Conference on e-Science and Grid Computing, 2006.
18. W. Tang, J. Bischof, N. Desai, K. Mahadik, W. Gerlach, T. Harrison, A. Wilke, F. Meyer, Workload characterization for mg-rast metagenomic data analytics service in the cloud, in: IEEE International Conference on Big Data, 2014.
19. T. Shibata, S. Choi, K. Taura, File-access patterns of data-intensive workflow applications and their implications to distributed filesystems, in: 19th ACM International Symposium on High Performance Distributed Computing (HPDC), 2010.
20. O. Sonmez, N. Yigitbasi, A. Iosup, D. Epema, Trace-based evaluation of job runtime and queue wait time predictions in grids, in: 18th ACM international symposium on High performance distributed computing, HPDC '09, ACM, 2009, pp. 111-120. doi:10.1145/1551609.1551632.
21. D. Martínez-Rego, M. Pontil, Multi-task averaging via task clustering, in:E. Hancock, M. Pelillo (Eds.), Similarity-Based Pattern Recognition, Vol. 7953 of Lecture Notes in Computer Science, Springer Berlin Heidelberg, 2013, pp. 148-159. doi:10.1007/978-3-642-39140-8-10.
22. J. Kephart, D. Chess, The vision of autonomic computing, Computer 36 (1) (2003) 41-50. doi:10.1109/MC.2003.1160055.
23. R. Ferreira da Silva, G. Juve, E. Deelman, T. Glatard, F. Desprez, D. Thain, B. Tovar, M. Livny, Toward fine-grained online task characteristics estimation in scientific workflows, in: Proceedings of the 8th Workshop on Workflows in Support of Large-Scale Science, WORKS '13, 2013, pp. 58-67. doi:10.1145/2534248.2534254.
24. E. Deelman, G. Singh, M.-H. Su, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, G. B. Berriman, J. Good, A. Laity, J. C. Jacob, D. S. Katz, Pegasus: A framework for mapping complex scientific workflows onto distributed systems, Sci. Program. 13 (3) (2005) 219-237.
25. E. Deelman, K. Vahi, G. Juve, M. Rynge, S. Callaghan, P. J. Maechling, R. Mayani, W. Chen, R. Ferreira da Silva, M. Livny, K. Wenger, Pegasus, a workflow management system for science automation, Future Generation Computer Systemsdoi:10.1016/j.future.2014.10.008.
26. M. Albrecht, P. Donnelly, P. Bui, D. Thain, Makeflow: a portable abstraction for data intensive computing on clusters, clouds, and grids, in: 1st ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies, SWEET '12, ACM, 2012, pp. 1:1-1:13. doi:10.1145/2443416.2443417.
27. T. Fahringer, A. Jugravu, S. Pllana, R. Prodan, C. Seragiotto, Jr., H.-L. Truong, Askalon: a tool set for cluster and grid computing: Research articles, Concurr. Comput. : Pract. Exper. 17 (2-4) (2005) 143-169.
28. T. Oinn, M. Greenwood, M. Addis, M. N. Alpdemir, J. Ferris, K. Glover, C. Goble, A. Goderis, D. Hull, D. Marvin, P. Li, P. Lord, M. R. Pocock, M. Senger, R. Stevens, A. Wipat, C. Wroe, Taverna: lessons in creating a workflow environment for the life sciences: Research articles, Concurr. Comput. : Pract. Exper. 18 (10) (2006) 1067-1100.
29. G. B. Berriman, E. Deelman, J. C. Good, J. C. Jacob, D. S. Katz, C. Kesselman, A. C. Laity, T. A. Prince, G. Singh, M.-H. Su, Montage: a grid-enabled engine for delivering custom science-grade mosaics on demand, SPIE Conference 5493 (2004) 221-232. doi:10.1117/12.550551.
30. USC epigenome center, http://epigenome.usc.edu/.
31. Periodogram workflow, http://portal.futuregrid.org/projects/77.
32. Rosetta Commons, http://www.rosettacommons.org.
33. T. Joshi, B. Valliyodan, S. M. Khan, Y. Liu, J. V. Maldonado dos Santos, et al., Next generation resequencing of soybean germplasm for trait discovery on xsede using pegasus workflows and iplant infrastructure, in: XSEDE 2014, 2014.
34. T. Joshi, M. R. Fitzpatrick, S. Chen, Y. Liu, H. Zhang, R. Z. Endacott, E. C. Gaudiello, G. Stacey, H. T. Nguyen, D. Xu, Soybean knowledge base (soykb): a web resource for integration of soybean translational genomics and molecular breeding, Nucleic Acids Research 42 (D1) (2014) D1245-D1252. doi:10.1093/nar/gkt905.
35. GATK, https://www.broadinstitute.org/gatk.
36. H.-P. Kriegel, P. Kröger, J. Sander, A. Zimek, Density-based clustering, Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 1 (3) (2011) 231-240. doi:10.1002/widm.30.
37. M. Ester, H. P. Kriegel, J. Sander, X. Xu, A density-based algorithm for discovering clusters in large spatial databases with noise, in: 2nd International Conference on Knowledge Discovery and Data Mining, 1996, pp. 226-231.
38. I. Chakravarty, J. Roy, R. Laha, Handbook of methods of applied statistics, McGraw-Hill, 1967.
39. G. Marsaglia, T. A. Bray, A convenient method for generating normal variables, Siam Review 6 (3) (1964) 260-264.
40. G. Marsaglia, W. W. Tsang, A simple method for generating gamma variables, ACM Trans. Math. Softw. 26 (3) (2000) 363-372. doi:10.1145/358407.358414.
41. R. Ferreira da Silva, T. Glatard, F. Desprez, Self-healing of operational workflow incidents on distributed computing infrastructures, in: 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid'12, 2012, pp. 318-325. doi:10.1109/CCGrid.2012.24.
42. R. Ferreira da Silva, T. Glatard, F. Desprez, Self-healing of workflow activity incidents on distributed computing infrastructures, Future Generation Computer Systems 29 (8) (2013) 2284-2294. doi:10.1016/j.future.2013.06.012.
43. Parallel workloads archive. www.cs.huji.ac.il/labs/parallel/workload/
44. A. Iosup, H. Li, M. Jan, S. Anoep, C. Dumitrescu, L. Wolters, D. H. J. Epema, The grid workloads archive, Future Gener. Comput. Syst. 24 (7) (2008) 672-686. doi:10.1016/j.future.2008.02.003.
45. D. Kondo, B. Javadi, A. Iosup, D. Epema, The failure trace archive: Enabling comparative analysis of failures in diverse distributed systems, in: 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing, CCGrid, IEEE, 2010, pp. 398-407.
46. C. Germain-Renaud, A. Cady, P. Gauron, M. Jouvin, C. Loomis, J. Martyniak, J. Nauroy, G. Philippon, M. Sebag, The grid observatory, IEEE International Symposium on Cluster Computing and the Grid (2011) 114-123.
47. R. Ferreira da Silva, W. Chen, G. Juve, K. Vahi, E. Deelman, Community resources for enabling and evaluating research on scientific workflows, in: 10th IEEE International Conference on e-Science, eScience'14, 2014, pp. 177-184. doi:10.1109/eScience.2014.44.
48. R. Ferreira da Silva, T. Glatard, A science-gateway workload archive to study pilot jobs, user activity, bag of tasks, task sub-steps, and workflow executions, in: Euro-Par 2012: Parallel Processing Workshops, Vol. 7640 of Lecture Notes in Computer Science, 2013, pp. 79-88. doi:10.1007/978-3-642-36949-0-10.
49. Workflow archive, http://workflowarchive.org.
50. L. Ramakrishnan, D. Gannon, A survey of distributed workflow characteristics and resource requirements, Indiana University.
51. S. Ostermann, R. Prodan, T. Fahringer, A. Iosup, D. Epema, On the characteristics of grid workflows, in: CoreGRID Symposium - Euro-Par 2008, 2008.
52. S. Ostermann, R. Prodan, T. Fahringer, A. Iosup, D. Epema, A trace-based investigation of the characteristics of grid workflows, in:T. Priol, M. Vanneschi (Eds.), From Grids to Service and Pervasive Computing, Springer US, 2008, pp. 191-203. doi:10.1007/978-0-387-09455-7-14.
53. A. Iosup, D. Epema, Grid computing workloads, Internet Computing, IEEE 15 (2) (2011) 19-26. doi:10.1109/MIC.2010.130.
54. D. L. Hart, Measuring teragrid: workload characterization for a high-performance computing federation, International Journal of High Performance Computing Applications 25 (4) (2011) 451-465.
55. Z. Ren, X. Xu, J. Wan, W. Shi, M. Zhou, Workload characterization on a production hadoop cluster: A case study on taobao, in: Workload Characterization (IISWC), 2012 IEEE International Symposium on, 2012, pp. 3-13. doi:10.1109/IISWC.2012.6402895.
56. S. Mahambre, P. Kulkarni, U. Bellur, G. Chafle, D. Deshpande, Workload characterization for capacity planning and performance management in iaas cloud, in: IEEE International Conference on Cloud Computing in Emerging Markets, 2012, pp. 1-7. doi:10.1109/CCEM.2012.6354624.
57. S. Madougou, S. Shahand, M. Santcroos, B. van Schaik, A. Benabdelkader, A. van Kampen, S. Olabarriaga, Characterizing workflow-based activity on a production e-infrastructure using provenance data, Future Generation Computer Systems 29 (8) (2013) 1931-1942. doi:http://dx.doi.org/10.1016/j.future.2013.04.019.
58. S. Pacheco-Sanchez, G. Casale, B. Scotney, S. McClean, G. Parr, S. Dawson, Markovian workload characterization for qos prediction in the cloud, in: IEEE International Conference on Cloud Computing, 2011, pp. 147-154. doi:10.1109/CLOUD.2011.100.
59. A. Khan, X. Yan, S. Tao, N. Anerousis, Workload characterization and prediction in the cloud: A multiple time series approach, in: Network Operations and Management Symposium (NOMS), 2012 IEEE, 2012, pp. 1287-1294. doi:10.1109/NOMS.2012.6212065.
60. R. Ferreira da Silva, M. Rynge, G. Juve, I. Sfiligoi, E. Deelman, J. Letts, F. Würthwein, M. Livny, Characterizing a high throughput computing workload: The compact muon solenoid (CMS) experiment at LHC, in: 2015 International Conference on Computational Science, ICCS 2015, 2015.
61. R. Duan, F. Nadeem, J. Wang, Y. Zhang, R. Prodan, T. Fahringer, A hybrid intelligent method for performance modeling and prediction of workflow activities in grids, in: 9th IEEE/ACM International Symposium on Cluster Computing and the Grid, IEEE Computer Society, 2009, pp. 339-347. doi:10.1109/CCGRID.2009.58.
62. E.-K. Byun, Y.-S. Kee, E. Deelman, K. Vahi, G. Mehta, J.-S. Kim, Estimating resource needs for time-constrained workflows, in: eScience, 2008. eScience '08. IEEE Fourth International Conference on, 2008, pp. 31-38. doi:10.1109/eScience.2008.18.
63. R. Huang, H. Casanova, A. A. Chien, Automatic resource specification generation for resource selection, in: 2007 ACM/IEEE conference on Supercomputing, ACM, 2007, pp. 11:1-11:11. doi:10.1145/1362622.1362638.
64. F. Nadeem, T. Fahringer, Using templates to predict execution time of scientific workflow applications in the grid, in: 9th IEEE/ACM International Symposium on Cluster Computing and the Grid, IEEE Computer Society, 2009, pp. 316-323. doi:10.1109/CCGRID.2009.77.
65. V. K. Pallipuram, J. DiMarco, M. Taufer, Applying frequency analysis techniques to dag-based workflows to benchmark and predict resource behavior on non-dedicated clusters, in: IEEE International Conference on Cluster Computing, CLUSTER, 2014, pp. 29-37. doi:10.1109/CLUSTER.2014.6968734.
66. A. M. Chirkin, A. S. Z. Belloum, S. V. Kovalchuk, M. X. Makkes, Execution time estimation for workflow scheduling, in: 9th Workshop on Workflows in Support of Large-Scale Science, WORKS'14, 2014, pp. 1-10. doi:10.1109/WORKS.2014.11.
67. I. Pietri, G. Juve, E. Deelman, R. Sakellariou, A performance model to estimate execution time of scientific workflows on the cloud, in: 9th Workshop on Workflows in Support of Large-Scale Science, WORKS'14, 2014, pp. 11-19. doi:10.1109/WORKS.2014.12.