LogSig: Generating system events from raw textual logs

International Conference on Information and Knowledge Management, Proceedings

Volumn , Issue , 2011, Pages 785-794

  Tang, Liang ^a   Li, Tao ^a   Perng, Chang Shing ^b  

^a The Department of Electrical and Computer Engineering   (United States)

^b IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

event generation; message signature; system logs

Indexed keywords

ALTERNATIVE ALGORITHMS; COMPUTING SYSTEM; DOMAIN EXPERTS; DOMAIN KNOWLEDGE; EVENT GENERATION; GENERATING SYSTEM; LARGE VOCABULARY; LOG ANALYSIS; LOG DATA; MESSAGE SIGNATURE; POOR PERFORMANCE; REAL SYSTEMS; SYSTEM ADMINISTRATORS; SYSTEM BEHAVIORS; SYSTEM LOGS; TEXT CLUSTERING; TEXT MESSAGES;

ALGORITHMS; EXPERIMENTS; KNOWLEDGE MANAGEMENT; TELEPHONE SYSTEMS;

NETWORK SECURITY;

EID: 83055187032     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2063576.2063690     Document Type: Conference Paper

References (30)

1. Apache HTTP Server : An Open-Source HTTP Web Server. http://httpd.apache. org/.
2. FileZilla: An open-source and free FTP/SFTP solution. http://filezilla-project.org.
3. Hadoop : An Open-Source MapReduce computing platform. http://hadoop.apache.org/.
4. PVFS2 : The state-of-the-art parallel I/O and high performance virtual file system. http://pvfs.org.
5. ThunderBird: A supercomputer in Sandia National Laboratories. http://www.cs.sandia.gov/~jrstear/logs/.
6. M. Aharon, G. Barash, I. Cohen, and E. Mordechai. One graph is worth a thousand logs: Uncovering hidden structures in massive system event logs. In Proceedings of ECML/PKDD, pages 227-243, Bled, Slovenia, September 2009.
7. S. Bereg, M. Kubica, T. Walen, and B. Zhu. RNA multiple structural alignment with longest common subsequences. Journal of Combinatorial Optimization, 13(2):179-188, 2007. (Pubitemid 46034292)
8. M. Bilenko, S. Basu, and R. J. Mooney. Integrating constraints and metric learning in semi-supervised clustering. In Proceedings of ICML, Alberta, Canada, July 2004.
9. K. Fisher, D. Walker, and K. Q. Zhu. Incremental learning of system log formats. SIGOPS Oper. Syst. Rev., 44(1):85-90, 2010.
10. J. Gao, G. Jiang, H. Chen, and J. Han. Modeling probabilistic measurement correlations for problem determination in large-scale distributed systems. In Proceedings of the 29th International Conference on Distributed Computing Systems (ICDCS'09), pages 623-630, 2009.
11. G. Hamerly and C. Elkan. Learning the k in k-means. In Proceedings of NIPS, Vancouver, British Columbia, Canada, December 2003.
12. J. Han, M. Kamber, and J. Pei. Data Mining: Concepts and Techniques, 2ed. Morgan Kaufmann, 2005.
13. J. L. Hellerstein, S. Ma, and C.-S. Perng. Discovering actionable patterns in event data. IBM Systems Journal, 43(3):475-493, 2002.
14. J. Kiernan and E. Terzi. Constructing comprehensive summaries of large event sequences. In Proceedings of ACM KDD, pages 417-425, Las Vegas, Nevada, USA, August 2008.
15. T. Li, F. Liang, S. Ma, and W. Peng. An integrated framework on mining logs files for computing system management. In Proceedings of ACM KDD, pages 776-781, Chicago, Illinois, USA, August 2005.
16. H. Lodhi, C. Saunders, J. Shawe-Taylor, N. Cristianini, and C. Watkins. Text classification using string kernels. The Journal of Machine Learning Research, 2:419-444, March 2002. (Pubitemid 135712568)
17. D. Maier. The complexity of some problems on subsequences and supersequences. Journal of the ACM, 25:322-336, April 1978.
18. A. Makanju, A. N. Zincir-Heywood, and E. E. Milios. Clustering event logs using iterative partitioning. In Proceedings of ACM KDD, pages 1255-1264, Paris, France, June 2009.
19. K. Ning, H. K. Ng, and H. W. Leong. Finding patterns in biological sequences by longest common subsequencesand shortest common supersequences. In Proceedings of BIBE, pages 53-60, Arlington, Virginia, USA, 2006.
20. A. J. Oliner, A. Aiken, and J. Stearley. Alert detection in system logs. In Proccedings of ICDM, pages 959-964, Pisa, Italy, December 2008.
21. A. J. Oliner and J. Stearley:. What supercomputers say: A study of five system logs. In Proceedings of DSN 2007, pages 575-584, Edinburgh, UK, June 2007.
22. W. Peng, C. Perng, T. Li, and H. Wang. Event summarization for system management. In Proceedings of ACM KDD, pages 1028-1032, San Jose, California, USA, August 2007.
23. G. Salton and M. McGill. Introduction to Modern Information Retrieval. McGraw-Hill, 1984.
24. J. Stearley. Towards informatic analysis of syslogs. In Proceedings of IEEE International Conference on Cluster Computing, pages 309-318, San Diego, California, USA, September 2004.
25. P.-N. Tan, M. Steinbach, and V. Kumar. Introduction to Data Mining. Addison Wesley, 2005.
26. L. Tang and T. Li. LogTree: A framework for generating system events from raw textual logs. In Proceedings of ICDM, pages 491-500, Sydney, Australia, December 2010.
27. S. Theodoridis and r. e. Konstantinos Koutroumbas. Pattern Recognition. Academic Press, 2006.
28. K. Wagstaff, C. Cardie, S. Rogers, and S. Schrodl:. Constrained k-means clustering with background knowledge. In Proceedings of ICML, pages 577-584, June 2001.
29. P. Wang, H. Wang, M. Liu, and W. Wang. An algorithmic approach to event summarization. In Proceedings of ACM SIGMOD, pages 183-194, Indianapolis, Indiana, USA, June 2010.
30. W. Xu, L. Huang, A. Fox, D. A. Patterson, and M. I. Jordan. Mining console logs for large-scale system problem detection. In SysML, San Diego, CA, USA, December 2008.