A General approach to real-time workflow monitoring

Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012

Volumn , Issue , 2012, Pages 108-118

  Vahi, Karan ^a   Harvey, Ian ^b   Samak, Taghrid ^c   Gunter, Daniel ^c   Evans, Kieran ^b   Rogers, Dave ^b   Taylor, Ian ^b   Goode, Monte ^c   Silva, Fabio ^d   Al Shkarchi, Eddie ^b   Mehta, Gaurang ^a   Jones, Andrew ^b   Deelman, Ewa ^a  

^a UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^b Welsh E Science Centre   (United Kingdom)

^c LAWRENCE BERKELEY NATIONAL LABORATORY   (United States)

^d UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

Author keywords

interoperability; log analysis; monitoring; scientific workflows; workflow performance statistics

Indexed keywords

EXTRACTING INFORMATION; HIGH-PERFORMANCE TOOLS; LOG ANALYSIS; PERFORMANCE STATISTICS; REAL TIME MONITORING; SCIENTIFIC WORKFLOW ENGINES; SCIENTIFIC WORKFLOWS; WORKFLOW MANAGEMENT SYSTEMS;

INTEROPERABILITY; MODELS; MONITORING;

AUTOMATION;

EID: 84876582748     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/SC.Companion.2012.26     Document Type: Conference Paper

References (45)

1. The EU Wf4Ever Project. http://www.wf4ever-project.org/.
2. Grid logging: Best practices guide, 2008.
3. David R. Owens 4. Adina Riposan, Ian J. Taylor, Omer Rana and Edward C. Conley. TRIACS Workflows Platform For Distributed Decision Support Processes. In In CBMS 2009, Albuquerque, 2009.
4. A.S. Ali, O.F. Rana, and I.J. Taylor. Web Services Composition for Distributed Data Mining. In ICPP 2005 Workshops, International Conference Workshops on Parallel Processing, pages 11-18. IEEE, New York, 2005.
5. I. Altintas, C. Berkley, E. Jaeger, M. Jones, B. Lud̈ascher, and S. Mock. Kepler: An Extensible System for Design and Execution of Scientific Workflows. In 16th International Conference on Scientific and Statistical Database Management (SSDBM), pages 423-424. IEEE Computer Society, New York, 2004.
6. Advanced message queuing protocol. Web: http://www.amqp.org.
7. Roger Barga, Jared Jackson, Nelson Araujo, Dean Guo, Nitin Gautam, and Yogesh Simmhan. The trident scientific workflow workbench. In Proceedings of the 2008 Fourth IEEE International Conference on eScience, pages 317-318, Washington, DC, USA, 2008. IEEE Computer Society.
8. Taylor I. Benson T, Conley EC, Harrison AB. Sintero Server Simplifying interoperability for distributed collaborative health care. In IHIC 2011 Conference, Orlando, May, 2011.
9. Scott Callaghan, Ewa Deelman, Dan Gunter, Gideon Juve, Philip Maechling, Christopher X. Brooks, Karan Vahi, Kevin Milner, Robert Graves, Edward Field, David Okaya, and Thomas Jordan. Scaling up workflowbased applications. J. Comput. Syst. Sci., 76(6):428-446, 2010.
10. Scott Callaghan, Philip Maechling, Patrick Small, Kevin Milner, Gideon Juve, Thomas Jordan, Ewa Deelman, Gaurang Mehta, Karan Vahi, Dan Gunter, Keith Beattie, and Christopher X. Brooks. Metrics for heterogeneous scientific workflows: A case study of an earthquake science application. IJHPCA, 25(3):274-285, 2011.
11. Data Mining Tools and Services for Grid Computing Environments (DataMiningGrid). http://www.datamininggrid.org/.
12. E. Deelman, G. Singh, M.-H. Su, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, G. B. Berriman, J. Good, A. Laity, J. C. Jacob, and D.S. Katz. Pegasus: a Framework for Mapping Complex Scientific Workflows onto Distributed Systems. Scientific Programming Journal, 13(3):219-237, 2005.
13. Ewa Deelman, Scott Callaghan, Edward Field, Hunter Francoeur, Robert Graves, Nitin Gupta, Vipin Gupta, Thomas H. Jordan, Carl Kesselman, Philip Maechling, John Mehringer, Gaurang Mehta, David Okaya, Karan Vahi, and Li Zhao. Managing large-scale workflow execution from resource provisioning to provenance tracking: The cybershake example. In Proceedings of the Second IEEE International Conference on e-Science and Grid Computing, E-SCIENCE '06, Washington, DC, USA, 2006. IEEE Computer Society.
14. Ewa Deelman, Dennis Gannon, Matthew Shields, and Ian Taylor. Workflows and e-science: An overview of workflow system features and capabilities. Future Gener. Comput. Syst., 25:528-540, May 2009.
15. Amazon Elastic Cloud. Web:http://aws.amazon.com/ec2.
16. T. Fahringer, R. Prodan, R. Duan, F. Nerieri, S. Podlipnig, J. Qin, M. Siddiqui, H.-L. Truong, A. Villazon, and M. Wieczorek. ASKALON: A Grid Application Development and Computing Environment. In 6th International Workshop on Grid Computing, pages 122-131. IEEE Computer Society Press, New York, 2005.
17. FutureGrid. Web:https://portal.futuregrid.org/.
18. I. Foster and C. Kesselman. Globus: A Metacomputing Infrastructure Toolkit. International Journal of Supercomputing Applications, 11(2):115-128, 1997.
19. J. Frey. Condor DAGMan: Handling inter-job dependencies, 2002.
20. James Frey, Todd Tannenbaum, Miron Livny, Ian Foster, and Steven Tuecke. Condor-G: A Computation Management Agent for Multi-Institutional Grids. In Proceedings of the 10th IEEE International Symposium on High Performance Distributed Computing (HPCD-'01). IEEE Computer Society, New York, 2001.
21. Tristan Glatard, Johan Montagnat, Diane Lingrand, and Xavier Pennec. Flexible and efficient workflow deployment of data-intensive applications on grids with moteur. IJHPCA, 22(3):347-360, 2008.
22. Dan Gunter, Ewa Deelman, Taghrid Samak, Christopher X. Brooks, Monte Goode, Gideon Juve, Gaurang Mehta, Priscilla Moraes, Fabio Silva, D. Martin Swany, and Karan Vahi. Online workflow management and performance analysis with stampede. In CNSM, pages 1-10. IEEE, 2011.
23. Dan Gunter and Brian Tierney. Netlogger: A toolkit for distributed system performance tuning and debugging. In Integrated Network Management, IFIP/IEEE Eighth International Symposium on Integrated Network Management (IM 2003), volume 246 of IFIP Conference Proceedings, pages 97-100. Kluwer, 2003.
24. Andrew Harrison, Ian Taylor, Ian Wang, and Matthew Shields. WSRF Workflow in Triana. International Journal of High Performance Computing Applications, 22(3):268-283, August 2008.
25. J. Huang, A. Kini, E. Paulson, C. Reilly, E. Robinson, S. Shankar, L. Shrinivas, D. DeWitt, and J. Naughton. An overview of Quill: A passive operational data logging system for Condor. Computer Sciences Technical Report, University of Wisconsin, 2007.
26. Peter Kacsuk. P-grade portal family for grid infrastructures. Concurr. Comput. : Pract. Exper., 23:235-245, March 2011.
27. Daniel S. Katz, Joseph C. Jacob, G. Bruce Berriman, John Good, Anastasia C. Laity, Ewa Deelman, Carl Kesselman, Gurmeet Singh, Mei-Hui Su, and Thomas A. Prince. A comparison of two methods for building astronomical image mosaics on a grid. In ICPP Workshops, pages 85-94. IEEE Computer Society, 2005.
28. P. Maechling, E. Deelman, L. Zhao, R. Graves, G. Mehta, N. Gupta, J. Mehringer, C. Kesselman, S. Callaghan, D. Okaya, H. Francoeur, V. Gupta, Y. Cui, K. Vahi, T. Jordan, and E. Field. Scec cybershake workflows-Automating probabilistic seismic hazard analysis calculations. In Ian Taylor, Ewa Deelman, Dennis Gannon, and Matthew Shield, editors, Worflows for e-Sciences. Springer, 2006.
29. Tom Oinn, Matthew Addis, Justin Ferris, Darren Marvin, Martin Senger, Mark Greenwood, Tim Carver, Kevin Glover, Matthew R. Pocock, Anil Wipat, and Peter Li. Taverna: A Tool for the Composition and Enactment of Bioinformatics Workflows. Bioinformatics, 20(17):3045-3054, November 2004.
30. The Open Provenance Model (OPM). http://openprovenance.org/.
31. The Open Provenance Model Vocabulary Specification (OPM-V). http://open-biomed.sourceforge.net/opmv/ns.html.
32. Simon Ostermann, Kassian Plankensteiner, Radu Prodan, Thomas Fahringer, and Alexandru Iosup. Workflow monitoring and analysis tool for askalon. In Ramin Yahyapour, Domenico Talia, and Norbert Meyer, editors, CoreGRID Workshop on Grid Middleware, pages 1-14, 2008.
33. PREservation Metadata Implementation Strategies (PREMIS). http://www.loc.gov/standards/premis/v2/premis-2-0.pdf.
34. PYANG-An extensible YANG validator and converter in python. Web:http://www.yangcentral. org/twiki/pub/Main/YangTools/pyang.1.html.
35. Yang Schema for Stampede Log Messages. Web:http://acs.lbl.gov/projects/ stampede/4.0/stampede-schema.html.
36. RabbitMQ. Web: http://www.rabbitmq.com.
37. Taghrid Samak, Dan Gunter, Monte Goode, Ewa Deelman, Gideon Juve, Gaurang Mehta, Fabio Silva, and Karan Vahi. Online fault and anomaly detection for large-scale scientific workflows. In Parimala Thulasiraman, Laurence Tianruo Yang, Qiwen Pan, Xingang Liu, Yaw-Chung Chen, Yo-Ping Huang, Lin huang Chang, Che-Lun Hung, Che-Rung Lee, Justin Y. Shi, and Ying Zhang, editors, HPCC, pages 373-381. IEEE, 2011.
38. The SHaring Interoperable Workflows for large-scale scientific simulations on Available DCIs Project . http://www.shiwa-workflow.eu/.
39. SQLAlchemy. Web: http://www.sqlalchemy.org.
40. Semantic Web Applications in Neuromedicine (SWAN). http://swan. mindinformatics.org/spec/1.2/pav.html.
41. Ian Taylor. Triana Generations. In Scientific Workflows and Business workflow standards in e-Science in conjunction with Second IEEE International Conference on e-Science, Amsterdam, Netherlands, December 2-4 2006.
42. Ian Taylor, Eddie Al-Shakarchi, and Stephen David Beck. Distributed Audio Retrieval using Triana (DART). In International Computer Music Conference (ICMC) 2006, November 6-11, at Tulane University, USA., pages 716-722, 2006.
43. B. Tierney and D. Gunter. NetLogger: A toolkit for distributed system performance, tuning and debugging. In Proceedings of the IFIP/IEEE Eighth International Symposium on Integrated Network Management (IM 2003), Vol. 246 of IFIP Conference Proceedings, pages 97-100. Kluwer, 2003.
44. Extreme science and engineering discovery environment. https://www. xsede.org/.
45. YANG-A Data Modeling Language for the Network Configuration Protocol. Web:http://tools.ietf.org/html/rfc6020.