Establishing hypothesis for recurrent system failures from cluster log files

Proceedings - IEEE 9th International Conference on Dependable, Autonomic and Secure Computing, DASC 2011

Volumn , Issue , 2011, Pages 15-22

  Chuah, Edward ^a   Lee, Gary ^a   Tjhi, William Chandra ^a   Kuo, Shyh Hao ^a   Hung, Terence ^a   Hammond, John ^b   Minyard, Tommy ^b   Browne, James C ^c  

^a INSTITUTE OF HIGH PERFORMANCE COMPUTING   (Singapore)

^b Texas Advanced Computing Center   (United States)

^c UNIVERSITY OF TEXAS AT AUSTIN   (United States)

Author keywords

Failure diagnosis; Hypothesis testing; Large cluster systems; Reliability; Syslogs

Indexed keywords

CAUSAL RELATIONSHIPS; CORRELATION ANALYSIS; DIAGNOSE SYSTEM; EVENT SEQUENCE; FAILURE DIAGNOSIS; FAILURE DIAGNOSTICS; FILE SYSTEMS; HIGH CONFIDENCE; HYPOTHESIS TESTING; LARGE CLUSTERS; LOG ANALYSIS; LOG FILE; OPEN SOURCE SOFTWARE; SECOND GENERATION; SYSLOGS; SYSTEM FAILURES; SYSTEMS ADMINISTRATOR; UNIVERSITY OF TEXAS;

CLUSTER COMPUTING; COMPUTER OPERATING SYSTEMS; EMBEDDED SOFTWARE; FAILURE ANALYSIS; FILE ORGANIZATION; RELIABILITY; SUPERCOMPUTERS; SYSTEMS ENGINEERING;

OPEN SYSTEMS;

EID: 84856109383     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/DASC.2011.27     Document Type: Conference Paper

References (30)

1. T.-T. Y. Lin and D. P. Siewiorek, "Error log analysis: Statistical modeling and heuristic trend analysis," IEEE Transactions on Reliability, vol. 39, no. 4, 1990.
2. M. M. Tsao and D. P. Siewiorek, "Trend analysis on system error files,"in Proceedings of FTCS '83, 1983.
3. C. Simache and M. Kaaniche, "Availability assessment of sunos/solaris unix systems based on syslogd and wtmpx log files: A case study," in Proceedings of IEEE PRDC, Dec 2005.
4. J. Stearley, "Towards informatic analysis of syslogs," in Proceedings of IEEE Cluster Computing, 2004, pp. 309-318.
5. B. Schroeder and G. Gibson, "A large-scale study of failures in high-performance computing systems," in Proceedings of IEEE/IFIP DSN, 2006, pp. 249-258.
6. A. Oliner and J. Stearley, "What supercomputers say: A study of five system logs," in Proceedings of IEEE/IFIP DSN, June 2007.
7. T. J. Hacker, F. Romero, and C. D. Carothers, "An analysis of clustered failures on large supercomputing systems," Journal of Parallel and Distributed Computing, vol. 69, no. 7, 2009.
8. S. Sabato, E. Yom-Tov, A. Tsherniak, and S. Rosset, "Analyzing system logs: A new view of what's important," in 2nd USENIX workshop on Tackling Computer Systems Problems with Machine Learning Techniques, 2007.
9. M. Aharon, G. Barash, I. Cohen, and E. Mordechai, "One graph is worth a thousand logs: Uncovering hidden structures in massive system event logs," in Proceedings of ECML PKDD, 2009.
10. J. Stearly and A. J. Oliner, "Bad words: Finding faults in spirit's syslogs," in Proceedings of IEEE CCGRID, 2008.
11. Z. Lan, Z. Zheng, and Y. Li, "Toward automated anomaly identification in large-scale systems," IEEE Transactions on Parallel and Distributed Systems, vol. 21, no. 2, 2010.
12. M. M. Tsao, "Trend analysis and fault prediction," PhD Thesis, Department of Electrical and Computer Engineering, Carnegie Mellon University, 1983.
13. Y. Liang, Y. Zhang, M. Jette, A. Sivasubramaniam, and R. Sahoo, "Bluegene/l failure analysis and prediction models," in Proceedings of IEEE/IFIP DSN, 2006.
14. Y. Liang, Y. Zhang, H. Xiong, and R. Sahoo, "Failure prediction in ibm bluegene/l event logs," in Proceedings of IEEE ICDM, 2007.
15. S. Fu and C.-Z. Xu, "Exploring event correlation for failure prediction in coalitions of clusters," in Proceedings of ACM/IEEE Supercomputing, no. 41, 2007.
16. Z. Zheng, Z. Lan, R. Gupta, S. Coghlan, and P. Beckman, "A practical failure prediction with location and lead time for blue gene/p," in 1st Workshop on Fault-Tolerance for HPC at Extreme Scale (in conjunction with IEEE/IFIP DSN 2010), 2010.
17. Z. Lan, J. Gu, Z. Zheng, R. Thakur, and S. Coghlan, "A study of dynamic meta-learning for failure prediction in large-scale systems," Journal of Parallel and Distributed Computing (JPDC), vol. 70, no. 6, 2010.
18. Z. Zheng, Z. Lan, B. Park, and A. Geist, "System log pre-processing to improve failure prediction," in Proceedings of IEEE/IFIP DSN, 2009.
19. Z. Lan, P. Gujrati, Y. Li, Z. Zheng, R. Thakur, and J. White, "A fault diagnosis and prognosis service for teragrid clusters," in Proceedings of ACM TeraGrid, 2007.
20. J. L. Hammond, T. Minyard, and J. Browne, "End-to-end framework for fault management for open source clusters: Ranger," in Proceedings of ACM TeraGrid, no. 9, 2010.
21. E. Chuah, S.-H. Kuo, P. Hiew, W.-C. Tjhi, G. Lee, J. Hammond, M. T. Michalewicz, T. Hung, and J. C. Browne, "Diagnosing the root-causes of failures from cluster log files," in Proceedings of IEEE HiPC, Dec 19-22 2010.
22. R. K. Iyer, L. T. Young, and V. Sridhar, "Recognition of error symptoms in large systems," in 1986 ACM Fall joint computer conference, 1986.
23. J. P. Hansen and D. P. Siewiorek, "Models for time coalescence in event logs," in Proceedings of FTCS '92, 1992.
24. S. E. Hansen and E. T. Atkins, "Automated system monitoring and notification with swatch," in USENIX LISA, 1993.
25. J. E. Prewett, "Listening to your cluster with logs," in 5th LCI International Conference on Linux Clusters, 2004.
26. M. Y. Chen, A. Accardi, E. Kiciman, J. Lloyd, D. Patterson, A. Fox, and E. Brewer, "Path-based faliure and evolution management," in Proceedings of NSDI, 2004.
27. M. Y. Chen, E. Kiciman, E. Fratkin, A. Fox, and E. Brewer, "Pinpoint: Problem determination in large, dynamic internet services," in Proceedings of IEEE/IFIP DSN, 2002.
28. J. Tan, X. Pan, E. Marinelli, S. Kavulya, R. Gandhi, and P. Narasimhan, "Problem diagnosis for mapreduce-based cloud computing environments, "in Proceedings of IEEE/IFIP NOMS, 2010.
29. J. Tan, S. Kavulya, R. Gandhi, and P. Narasimhan, "Visual, log-based causal tracing for performance debugging of mapreduce systems," in Proceedings of IEEE ICDCS, 2010.
30. A. J. Oliner, A. V. Kulkarni, and A. Aiken, "Using correlated surprise to infer shared influence," in Proceedings of IEEE/IFIP DSN, 2010.