Toward fine-grained online task characteristics estimation in scientific workflows

Proceedings of WORKS 2013: 8th Workshop on Workflows in Support of Large-Scale Science - Held in conjunction with SC 2013: The International Conference for High Performance Computing, Networking, Storage and Analysis

Volumn , Issue , 2013, Pages 58-67

  Da Silva, Rafael Ferreira ^a,b   Juve, Gideon ^b   Deelman, Ewa ^b   Glatard, Tristan ^a,c   Desprez, Frédéric ^d   Thain, Douglas ^e   Tovar, Benjamín ^e   Livny, Miron ^f  

^a UNIVERSITÉ DE LYON   (France)

^b UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^c MCGILL UNIVERSITY   (Canada)

^d UNIV LYON   (France)

^e UNIVERSITY OF NOTRE DAME   (United States)

^f UNIVERSITY OF WISCONSIN MADISON   (United States)

Author keywords

MAPE K loop; Online task estimation; Scientific workflow; Workflow characterization

Indexed keywords

DIGITAL STORAGE; SCHEDULING ALGORITHMS;

ACCURATE ESTIMATION; ACCURATE PREDICTION; CLUSTERING TECHNIQUES; MAPE; OFF-LINE APPROACHES; SCIENTIFIC WORKFLOWS; TASK CHARACTERISTICS; TASK ESTIMATION;

ESTIMATION;

EID: 84997796211     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2534248.2534254     Document Type: Conference Paper

References (43)

1. I. Taylor, E. Deelman, D. Gannon, and M. Shields, Workows for e-Science. Springer, 2007.
2. J. D. Ullman, \Np-complete scheduling problems," J. Comput. Syst. Sci., vol. 10, no. 3, pp. 384-393, Jun. 1975.
3. M. Maheswaran, S. Ali, H. J. Siegel, D. Hensgen, and R. F. Freund, \Dynamic matching and scheduling of a class of independent tasks onto heterogeneous computing systems," in Proceedings of the Eighth Heterogeneous Computing Workshop, ser. HCW '99. Washington, DC, USA: IEEE Computer Society, 1999, pp. 30-.
4. H. Topcuouglu, S. Hariri, and M.-y. Wu, \Performance-effective and low-complexity task scheduling for heterogeneous computing," IEEE Trans. Parallel Distrib. Syst., vol. 13, no. 3, pp. 260-274, Mar. 2002.
5. M. Rahman, R. Hassan, R. Ranjan, and R. Buyya, \Adaptive workow scheduling for dynamic grid and cloud computing environment," Concurrency and Computation: Practice and Experience, pp. n/a-n/a, 2013.
6. S. Su, J. Li, Q. Huang, X. Huang, K. Shuang, and J. Wang, \Cost-efficient task scheduling for executing large programs in the cloud," Parallel Computing, vol. 39,pp. 177 - 188, 2013.
7. K. Bessai, S. Youcef, A. Oulamara, C. Godart, and S. Nurcan, \Bi-criteria workow tasks allocation and scheduling in cloud computing environments," in Cloud Computing (CLOUD), 2012 IEEE 5th International Conference on, 2012, pp. 638-645.
8. S. Verboven, P. Hellinckx, F. Arickx, and J. Broeckhove, \Runtime prediction based grid scheduling of parameter sweep jobs," in Asia-Pacific Services Computing Conference, 2008. APSCC '08. IEEE, 2008, pp. 33-38.
9. C. Yan, H. Luo, Z. Hu, X. Li, and Y. Zhang, \Deadline guarantee enhanced scheduling of scientific workow applications in grid," Journal of Computers, vol. 8, no. 4, 2013.
10. J. O. Gutierrez-Garcia and K. M. Sim, \A family of heuristics for agent-based elastic cloud bag-of-tasks concurrent scheduling," Future Generation Computer Systems, vol. 29, no. 7, pp. 1682 - 1699, 2013.
11. G. Juve, A. Chervenak, E. Deelman, S. Bharathi, G. Mehta, and K. Vahi, \Characterizing and profiling scientific workows," Future Generation Computer Systems, vol. 29, no. 3, pp. 682 - 692, 2013, special Section: Recent Developments in High Performance Computing and Security.
12. O. Sonmez, N. Yigitbasi, A. Iosup, and D. Epema, \Trace-based evaluation of job runtime and queue wait time predictions in grids," in Proceedings of the 18th ACM international symposium on High performance distributed computing, ser. HPDC '09. New York, NY, USA: ACM, 2009, pp. 111-120.
13. D. Martfinez-Rego and M. Pontil, Multi-task averaging via task clustering," in Similarity-Based Pattern Recognition, ser. Lecture Notes in Computer Science, E. Hancock and M. Pelillo, Eds. Springer Berlin Heidelberg, 2013, vol. 7953, pp. 148-159.
14. J. Kephart and D. Chess, \The vision of autonomic computing," Computer, vol. 36, no. 1, pp. 41-50, 2003.
15. E. Deelman, G. Singh, M.-H. Su, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, G. B. Berriman, J. Good, A. Laity, J. C. Jacob, and D. S. Katz, \Pegasus: A framework for mapping complex scientific workows onto distributed systems," Sci. Program., vol. 13, no. 3, pp. 219-237, Jul. 2005.
16. J. s. VZckler, G. Mehta, Y. Zhao, E. Deelman, and M. Wilde, \Kickstarting remote applications," in 2nd International Workshop on Grid Computing Environments, 2006.
17. M. Albrecht, P. Donnelly, P. Bui, and D. Thain, \Makeow: A portable abstraction for data intensive computing on clusters, clouds, and grids," in Proceedings of the 1st ACM SIGMOD Workshop on Scalable Workow Execution Engines and Technologies, ser. SWEET '12. New York, NY, USA: ACM, 2012, pp. 1: 1-1: 13.
18. T. Fahringer, A. Jugravu, S. Pllana, R. Prodan, C. Seragiotto, Jr., and H.-L. Truong, \Askalon: A tool set for cluster and grid computing: Research articles," Concurr. Comput. : Pract. Exper., vol. 17, no. 2-4, pp. 143-169, Feb. 2005.
19. T. Oinn, M. Greenwood, M. Addis, M. N. Alpdemir, J. Ferris, K. Glover, C. Goble, A. Goderis, D. Hull, D. Marvin, P. Li, P. Lord, M. R. Pocock, M. Senger, R. Stevens, A. Wipat, and C. Wroe, \Taverna: Lessons in creating a workow environment for the life sciences: Research articles," Concurr. Comput. : Pract. Exper., vol. 18, no. 10, pp. 1067-1100, Aug. 2006.
20. G. B. Berriman, E. Deelman, J. C. Good, J. C. Jacob, D. S. Katz, C. Kesselman, A. C. Laity, T. A. Prince, G. Singh, and M.-H. Su, \Montage: A grid-enabled engine for delivering custom science-grade mosaics on demand," vol. 5493, pp. 221-232, 2004.
21. H.-P. Kriegel, P. Kröger, J. Sander, and A. Zimek, \Density-based clustering," Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 1, no. 3, pp. 231-240, 2011.
22. M. Ester, H. P. Kriegel, J. Sander, and X. Xu, \A density-based algorithm for discovering clusters in large spatial databases with noise," in Second International Conference on Knowledge Discovery and Data Mining, 1996, pp. 226-231.
23. R. Ferreira da Silva, T. Glatard, and F. Desprez, \Self-healing of workow activity incidents on distributed computing infrastructures," Future Generation Computer Systems, p. in press, 2013.
24. Parallel workloads archive. [Online]. Available: Www.cs.huji.ac.il/labs/parallel/workload/
25. A. Iosup, H. Li, M. Jan, S. Anoep, C. Dumitrescu, L. Wolters, and D. H. J. Epema, \The grid workloads archive," Future Gener. Comput. Syst., vol. 24, no. 7, pp. 672-686, 2008.
26. C. Germain-Renaud, A. Cady, P. Gauron, M. Jouvin, C. Loomis, J. Martyniak, J. Nauroy, G. Philippon, and M. Sebag, \The grid observatory," IEEE International Symposium on Cluster Computing and the Grid, pp. 114-123, 2011.
27. Workow gallery. [Online]. Available: Http://pegasus.isi.edu/workow gallery
28. R. Ferreira da Silva and T. Glatard, \A science-gateway workload archive to study pilot jobs, user activity, bag of tasks, task sub-steps, and workow executions," in Euro-Par 2012: Parallel Processing Workshops, ser. Lecture Notes in Computer Science, I. Caragiannis, M. Alexander, R. Badia, M. Cannataro, A. Costan, M. Danelutto, F. Desprez, B. Krammer, J. Sahuquillo, S. Scott, and J. Weidendorfer, Eds. Springer Berlin Heidelberg, 2013, vol. 7640, pp. 79-88.
29. Workow generator. [Online]. Available: Http://conuence.pegasus.isi.edu/display/pegasus/WorkowGenerator
30. L. Ramakrishnan and D. Gannon, \A survey of distributed workow characteristics and resource requirements," Indiana University, 2008.
31. S. Ostermann, R. Prodan, T. Fahringer, A. Iosup, and D. Epema, \On the characteristics of grid workows," in CoreGRID Symposium - Euro-Par 2008, 2008.
32. ||, \A trace-based investigation of the characteristics of grid workows," in From Grids to Service and Pervasive Computing, T. Priol and M. Vanneschi, Eds. Springer US, 2008, pp. 191-203.
33. A. Iosup and D. Epema, \Grid computing workloads," Internet Computing, IEEE, vol. 15, no. 2, pp. 19-26, 2011.
34. D. L. Hart, \Measuring teragrid: Workload characterization for a high-performance computing federation," International Journal of High Performance Computing Applications, vol. 25, no. 4, pp. 451-465, 2011.
35. Z. Ren, X. Xu, J. Wan, W. Shi, and M. Zhou, \Workload characterization on a production hadoop cluster: A case study on taobao," in Workload Characterization (IISWC), 2012 IEEE International Symposium on, 2012, pp. 3-13.
36. S. Mahambre, P. Kulkarni, U. Bellur, G. Chae, and D. Deshpande, \Workload characterization for capacity planning and performance management in iaas cloud," in Cloud Computing in Emerging Markets (CCEM), 2012 IEEE International Conference on, 2012, pp. 1-7.
37. S. Madougou, S. Shahand, M. Santcroos, B. van Schaik, A. Benabdelkader, A. van Kampen, and S. Olabarriaga, \Characterizing workow-based activity on a production e-infrastructure using provenance data," Future Generation Computer Systems, vol. 29, no. 8, pp. 1931 - 1942, 2013.
38. S. Pacheco-Sanchez, G. Casale, B. Scotney, S. McClean, G. Parr, and S. Dawson, \Markovian workload characterization for qos prediction in the cloud," in Cloud Computing (CLOUD), 2011 IEEE International Conference on, 2011, pp. 147-154.
39. A. Khan, X. Yan, S. Tao, and N. Anerousis, \Workload characterization and prediction in the cloud: A multiple time series approach," in Network Operations and Management Symposium (NOMS), 2012 IEEE, 2012, pp. 1287-1294.
40. R. Duan, F. Nadeem, J. Wang, Y. Zhang, R. Prodan, and T. Fahringer, \A hybrid intelligent method for performance modeling and prediction of workow activities in grids," in Proceedings of the 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid, ser. CCGRID '09. Washington, DC, USA: IEEE Computer Society, 2009, pp. 339-347.
41. E.-K. Byun, Y.-S. Kee, E. Deelman, K. Vahi, G. Mehta, and J.-S. Kim, \Estimating resource needs for time-constrained workows," in eScience, 2008. eScience '08. IEEE Fourth International Conference on, 2008, pp. 31-38.
42. R. Huang, H. Casanova, and A. A. Chien, \Automatic resource specification generation for resource selection," in Proceedings of the 2007 ACM/IEEE conference on Supercomputing, ser. SC '07. New York, NY, USA: ACM, 2007, pp. 11: 1-11: 11.
43. F. Nadeem and T. Fahringer, \Using templates to predict execution time of scientific workow applications in the grid," in Proceedings of the 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid, ser. CCGRID '09. Washington, DC, USA: IEEE Computer Society, 2009, pp. 316-323.