Predicting job completion times using system logs in supercomputing clusters

Proceedings of the International Conference on Dependable Systems and Networks

Volumn , Issue , 2013, Pages

  Chen, Xin ^a   Lu, Charng Da ^b   Pattabiraman, Karthik ^a  

^a UNIVERSITY OF BRITISH COLUMBIA   (Canada)

^b NONE

Author keywords

Hidden Markov Model; Log Analysis; Prediction

Indexed keywords

AUTOMATIC FAILURE; COMMERCIAL OFF-THE-SHELF COMPONENTS; COMPUTING CLUSTERS; FREQUENCY ANALYSIS; JOB COMPLETION; LOG ANALYSIS; SEMANTIC CONSISTENCY; SYSTEM ISSUES;

HIDDEN MARKOV MODELS; PERSONAL COMPUTERS; SEMANTICS;

FORECASTING;

EID: 84885982390     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/DSNW.2013.6615513     Document Type: Conference Paper

References (18)

1. Y. Zhang, M. S. Squillante, A. Sivasubramaniam, and R. K. Sahoo, "Performance implications of failures in large-scale cluster scheduling, " in International Workshop on Job Scheduling Strategies for Parallel Processing (JSSPP), 2004.
2. D. G. Feitelson, L. Rudolph, and U. Schwiegelshohn, "Parallel job scheduling-a status report, " in International Workshop on Job Scheduling Strategies for Parallel Processing (JSSPP), 2004.
3. G. Bronevetsky, R. Fernandes, D. Marques, K. Pingali, and P. Stodghill, "Recent advances in checkpoint/recovery systems, " in IEEE Parallel and Distributed Processing Symposium (IPDPS), 2006.
4. E. Elnozahy and J. Plank, "Checkpointing for peta-scale systems: A look into the future of practical rollback-recovery, " IEEE Transactions on Dependable and Secure Computing, vol. 1, no. 2, pp. 97-108, 2004.
5. A. Gainaru, F. Cappello, M. Snir, and W. Kramer, "Fault prediction under the microscope: A closer look into HPC systems, " in IEEE/ACM Conference on High Performance Computing Networking, Storage and Analysis (SC), 2012.
6. "http://www.infodrom.org/projects/sysklogd/."
7. A. Oliner and J. Stearley, "What supercomputers say: A study of five system logs, " in International Conference on Dependable Systems and Networks (DSN), 2007, pp. 575-584.
8. Y. Liang, Y. Zhang, M. Jette, A. Sivasubramaniam, and R. Sahoo, "BlueGene/L failure analysis and prediction models, " in International Conference on Dependable Systems and Networks (DSN), 2006, pp. 425-434.
9. L. Rabiner, "A tutorial on hidden Markov models and selected applications in speech recognition, " Proceedings of the IEEE, vol. 77, no. 2, pp. 257-286, 1989.
10. A. Oliner, G. Ganapathi, and W. Xu, "Advances and challenges in log analysis, " Communications of the ACM, vol. 55, no. 2, pp. 55-61, 2012.
11. Y. Zhang and A. Sivasubramaniam, "Failure prediction in IBM Blue-Gene/L event logs, " in IEEE Parallel and Distributed Processing Symposium (IPDPS), 2008.
12. J. Prewett, "Analyzing cluster log files using Logsurfer, " in The 4th Annual Conference on Linux Clusters, 2003.
13. J. Stearley and A. Oliner, "Bad words: Finding faults in Spirit's syslogs, " in IEEE International Symposium on Cluster Computing and the Grid (CCGrid), 2008.
14. A. Oliner, A. Aiken, and J. Stearley, "Alert detection in system logs, " in IEEE International Conference on Data Mining, 2008, pp. 959-964.
15. S. Sabato, E. Yom-Tov, A. Tsherniak, and S. Rosset, "Analyzing system logs: A new view of what's important, " in Workshop on Computer Systems with Machine Learning (SysML), 2007.
16. W. Xu, L. Huang, A. Fox, D. Patterson, and M. Jordan, "Experience mining Google's production console logs, " in Workshop on Managing Systems via Log Analysis and Machine Learning Techniques (SLAML), 2010.
17. K. Yamanishi and Y. Maruyama, "Dynamic syslog mining for network failure monitoring, " in Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, ser. KDD '05, 2005, pp. 499-508.
18. C. Basile, M. Gupta, Z. Kalbarczyk, and R. K. Iyer, "An approach for detecting and distinguishing errors versus attacks in sensor networks, " in Proceedings of the International Conference on Dependable Systems and Networks, ser. DSN '06, 2006, pp. 473-484.