Detecting large-scale system problems by mining console logs

ICML 2010 - Proceedings, 27th International Conference on Machine Learning

Volumn , Issue , 2010, Pages 37-44

  Xu, Wei ^a   Huang, Ling ^b   Fox, Armando ^a   Patterson, David ^a   Jordan, Michael I ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b INTEL CORPORATION   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DATA STREAM; GENERAL METHODOLOGIES; MACHINE-LEARNING; NUMERICAL FEATURES; ONLINE PROBLEMS; OPERATIONAL PROBLEMS; PROGRAM ANALYSIS; RUNTIMES; SOFTWARE COMPONENT;

DECISION TREES; LEARNING SYSTEMS;

INFORMATION RETRIEVAL;

EID: 77956513188     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (32)

1. Bodik, P, Goldszmidt, M., Fox, A., Woodard, D. B., and Andersen, H. Fingerprinting the datacenter: Automated classification of performance crises. In Proceedings of EuroSys'10, Paris, France, 2010.
2. Chen, M. Y. and Brewer, E. A. Path-based failure and evolution management. In Proceedings of NSDI'04, San Francisco, CA, 2004.
3. Chen, M. Y., Kidman, E., Fratkin, E., Fox, A., and Brewer, E. Pinpoint: Problem determination in large, dynamic internet services. In Proceedings of DSN'02, Washington, DC, 2002.
4. Cohen, I., Zhang, S., Goldszmidt, M., Symons, J., Kelly, T., and Fox, A. Capturing, indexing, clustering, and retrieving system history. In Proceedings of ACM SQSP'05, Brighton, UK, 2005.
5. Dunia, R. and Qin, S. J. Multi-dimensional fault diagnosis using a subspace approach. In Proceedings of American Control Conference (ACC'97), 1997.
6. Fisher, K., Walker, D., Zhu, K. Q., and White, P. From dirt to shovels: fully automatic tool generation from ad hoc data. In Proceedings of POPL '08, San Francisco, CA, 2008.
7. Fonseca, R., Porter, G., Katz, R. H., Shenker, S., and Stoica, I. Xtrace: A pervasive network tracing framework. In In Prococeedings of NSDI'07, Cambridge, MA, 2007.
8. Ganapathi, A., Kuno, H., Dayal, U., Wiener, J. L., Fox, A., Jordan, M., and Patterson, D. Predicting multiple performance metrics for queries: Better decisions enabled by machine learning. In Proceedings of ICDE'09, Shanghai, China, 2009.
9. Gulcu, C. Short introduction to log4j, March 2002. http://logging.apache. org/log4j.
10. Hansen, S. E. and Atkins, E. T. Automated system monitoring and notification with Swatch. In Proceedings of LISA '93, Monterey, CA, 1993.
11. Hellerstein, J., Ma, S., and Perng, C. Discovering actionable patterns in event data. IBM Sys. Jour, 41 (3), 2002.
12. Jiang, W., Hu, C., Pasupathy, S., Kanevsky, A., Li, Z., and Zhou, Y. Understanding customer problem troubleshooting from storage system logs. In Proceedings of FAST'09, San Francisco, California, 2009.
13. Karagiannis, T., Papagiannaki, K., Taft, N., and Faloutsos, M. Profiling the end hos. In Proceedings of Passive and Active Measurement Workshop (PAM'07), Belgium, 2007.
14. Lakhina, A., Crovella, M., and Diot, C. Diagnosing network-wide traffic anomalies. In Proceedings of ACM SIGCOMM, Portland, OR, 2004.
15. Li, Z., Lu, S., Myagmar, S., and Zhou, Y. Cp-miner: A tool for finding copy-paste and related bugs in operating system code. In Proceedings of OSDI'04, San Francisco, CA, 2004.
16. Lim, C., Singh, N., and Yajnik, S. A log mining approach to failure analysis of enterprise telephony systems. In Proceedings of DSN'08, June 2008.
17. Ma, S. and Hellerstein, J. L. Mining partially periodic event patterns with unknown periods. In Proceedings of ICDE'01, Washington, DC, 2001.
18. Makanju, A. A.O., Zincir-Heywood, A. N., and Milios, E. E. Clustering event logs using iterative partitioning. In Proceedings of KDD '09, Paris, France, 2009.
19. Oliner, A. and Stearley, J. What supercomputers say: A study of five system logs. In Proceedings of DSN'07, 2007.
20. Prewett, J. E. Analyzing cluster log files using log-surfer. In Proceedings of Annual Conf. on Linux Clusters, 2003.
21. Splunk. Splunk user guide. Splunk Inc., Sept 2008.
22. Stearley, J. Towards informatic analysis of syslogs. In Proceedings of IEEE CLUSTER, Washington, DC, 2004.
23. Vaarandi, R. A data clustering algorithm for mining patterns from event logs. Proceedings of IEEE IPOM'03, 2003.
24. Vaarandi, R. A breadth-first algorithm for mining frequent patterns from event logs. In INTELLCOMM, volume 3283. Springer, 2004.
25. Wang, H. J., Piatt, J. C., Chen, Y., Zhang, R., and Wang, Y. Automatic misconfiguration troubleshooting with peerpressure. In Proceedings of OSDI'04, San Francisco, CA, 2004.
26. Witten, I. H. and Frank, E. Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations. M. Kaufmann, 2000.
27. Xu, K., Zhang, Z., and Bhattacharyya, S. Profiling internet backbonetraffic: Behavior models and applications. In Proceedings of ACM SIGCOMM, 2005.
28. Xu, W., Huang, L., Fox, A., Patterson, D., and Jordan, M. Online system problem detection by mining patterns of console logs. In Proceedings of ICDM'09, Miami, FL, 2009a.
29. Xu, W., Huang, L., Fox, A., Patterson, D., and Jordan, M. Large-scale system problems detection by mining console logs. In Proceedings of SOSP '09, Big Sky, MT, 2009b.
30. Yamanishi, K. and Maruyama, Y. Dynamic syslog mining for network failure monitoring. In Proceedings of ACM KDD'05, Chicago, IL, 2005.
31. Ye, Y., Wang, D., Li, T., and Ye, D. IMDS: Intelligent malware detection system. In Proceedings of ACM KDD'07, 2007.
32. Zheng, A. X., Jordan, M. I., Liblit, B., Naik, M., and Aiken, A. Statistical debugging: Simultaneous isolation of multiple bugs. In Proceedings of ICML '06, 2006.