Practical resource monitoring for robust high throughput computing

Proceedings - IEEE International Conference on Cluster Computing, ICCC

Volumn 2015-October, Issue , 2015, Pages 650-657

  Juve, Gideon ^a   Tovar, Benjamin ^b   Da Silva, Rafael Ferreira ^a   Król, Dariusz ^a,c   Thain, Douglas ^b   Deelman, Ewa ^a   Allcock, William ^d   Livny, Miron ^e  

^a UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^b UNIVERSITY OF NOTRE DAME   (United States)

^c AGH UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Poland)

^d ARGONNE NATIONAL LABORATORY   (United States)

^e UNIVERSITY OF WISCONSIN MADISON   (United States)

Author keywords

High Throughput Computing; Monitoring; Profiling

Indexed keywords

COMPUTER ARCHITECTURE; MONITORING; THROUGHPUT;

COMPUTATIONAL TASK; EXECUTION SYSTEMS; HIGH-THROUGHPUT COMPUTING; MONITORING TOOLS; NETWORK TRAFFIC; NON TRIVIAL PROBLEMS; PROFILING; RESOURCE MONITORING;

CLUSTER COMPUTING;

EID: 84959282843     PISSN: 15525244     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CLUSTER.2015.115     Document Type: Conference Paper

References (43)

1. R. Sobie et al., "Htc scientific computing in a distributed cloud environment, " in 4th ACM Workshop on Scientific Cloud Computing, 2013.
2. C. Hoffa et al., "On the use of cloud computing for scientific workflows, " in 3rd International Workshop on Scientific Workflows and Business Workflow Standards in e-Science (SWBES '08), 2008.
3. "Open science grid, " http://opensciencegrid. org.
4. I. Taylor et al., Workflows for e-Science: Scientific Workflows for Grids, 2007.
5. T. D. Braun et al., "A comparison of eleven static heuristics for mapping a class of independent tasks onto heterogeneous distributed computing systems, " Journal of Parallel and Distributed Computing, vol. 61, no. 6, pp. 810-837, jun 2001.
6. H. Casanova et al., "Heuristics for scheduling parameter sweep applications in grid environments, " in 9th Heterogeneous Computing Workshop, 2000.
7. J. Blythe et al., "Task scheduling strategies for workflow-based applications in grids, " in 5th IEEE International Symposium on Cluster Computing and the Grid (CCGrid'05), may 2005.
8. A. Mandal et al., "Scheduling strategies for mapping application workflows onto the grid, " in 14th IEEE International Symposium on High Performance Distributed Computing, 2005.
9. R. Ferreira da Silva et al., "Toward fine-grained online task characteristics estimation in scientific workflows, " in 8th Workshop on Workflows in Support of Large-Scale Science, 2013.
10. -, "Online task resource consumption prediction for scientific workflows, " Parallel Processing Letters, vol. Accepted, 2015.
11. "Performance application programming interface (papi), " http://icl. cs. utk. edu/papi.
12. "perf, " http://perf. wiki. kernel. org.
13. "Nvml, " https://developer. nvidia. com/nvidia-management-library-NVML.
14. "taskstats, " http://www. kernel. org/doc/Documentation/accounting/taskstats. Txt.
15. A. Mavinakayanahalli et al., "Probing the guts of kprobes, " in Proceedings of the Ottawa Linux Symposium, 2006.
16. J. Keniston et al., "Ptrace, utrace, uprobes: Lightweight, dynamic tracing of user apps, " in Ottowa Linux Symposium, 2007.
17. "Dtrace, " http://dtrace. org.
18. "Systemtap, " https://sourceware. org/systemtap.
19. M. P. I. Forum, "Mpi: A message-passing interface standard, " 2003.
20. N. Dun et al., "Paratrac: A fine-grained profiler for data-intensive workflows, " in Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, 2010.
21. "Dyninst, " http://www. dyninst. org.
22. "Intel pin, " http://software. intel. com/en-us/articles/pintool.
23. "Cctools, " http://www3. nd. edu/ ccl/software/download.
24. E. Deelman et al., "Kickstarting remote applications, " in International Workshop on Grid Computing Environments, 2006.
25. -, "Pegasus, a workflow management system for science automation, " Future Generation Computer Systems, vol. 46, no. 0, pp. 17-35, 2015.
26. S. Godard, "Sysstat, " http://sebastien. godard. pagesperso-orange. fr.
27. M. L. Massie et al., "The ganglia distributed monitoring system: design, implementation, and experience, " Parallel Computing, vol. 30, no. 7, pp. 817-840, jul 2004.
28. "Nagios, " http://nagios. org.
29. "Munin, " http://munin-monitoring. org.
30. C.-D. Lu et al., "Comprehensive job level resource usage measurement and analysis for xsede hpc systems, " in Proceedings of the Conference on Extreme Science and Engineering Discovery Environment: Gateway to Discovery (XSEDE), 2013.
31. D. D. Vento et al., "System-level monitoring of floating-point performance to improve effective system utilization, " in Supercomputing, 2011.
32. "strace, " http://sourceforge. net/projects/strace.
33. "ltrace, " http://ltrace. org.
34. "Lanl-trace, " http://institute. lanl. gov/data/software/#lanl-trace.
35. "Fuse: Filesystem in userspace, " http://fuse. sourceforge. net/.
36. O. Zaki et al., "Toward scalable performance visualization with jumpshot, " High Performance Computing Applications, vol. 13, pp. 277-288, 1999.
37. "mpip: Lightweight, scalable mpi profiling, " http://mpip. sourceforge. net.
38. "Fpmpi-2 fast profiling library for mpi, " http://www. mcs. Anl. gov/research/projects/fpmpi/WWW.
39. M. Geimer et al., "The scalasca performance toolset architecture, " Concurrency and Computation: Practice and Experience, vol. 22, no. 6, pp. 702-719, apr 2010.
40. P. Carns et al., "24/7 characterization of petascale i/o workloads, " in Workshop on Interfaces and Architectures for Scientific Data Storage, 2009.
41. P. C. Roth, "Characterizing the i/o behavior of scientific applications on the cray xt, " in Proceedings of the 2nd International Workshop on Petascale Data Storage, 2007.
42. S. Seelam et al., "Early experiences in application level i/o tracing on blue gene systems, " in IEEE International Symposium on Parallel and Distributed Processing IPDPS, 2008.
43. M. Litzkow et al., "Checkpoint and migration of unix processes in the condor distributed processing system, " University of Wisconsin-Madison Computer Sciences Technical Report #1346, Tech. Rep., 1997.