Adaptive system anomaly prediction for large-scale hosting infrastructures

Proceedings of the Annual ACM Symposium on Principles of Distributed Computing

Volumn , Issue , 2010, Pages 173-182

  Tan, Yongmin ^a   Gu, Xiaohui ^a   Wang, Haixun ^b  

^a NORTH CAROLINA STATE UNIVERSITY   (United States)

^b MICROSOFT RESEARCH ASIA   (China)

Author keywords

Anomaly prediction; Context aware prediction model

Indexed keywords

ALERT SYSTEMS; ANOMALY DETECTION; ANOMALY PREDICTION; AUTOMATIC SYSTEMS; CONTEXT-AWARE; CONTINUOUS SYSTEM; JUST IN TIME; LOW OVERHEAD; PLANETLAB; PREDICTION ACCURACY; PREDICTION SCHEMES; PREDICTION SYSTEMS; RUNTIMES; STREAM PROCESSING; SYSTEM S;

ADAPTIVE SYSTEMS; MATHEMATICAL MODELS;

FORECASTING;

EID: 77956256340     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1835698.1835741     Document Type: Conference Paper

References (45)

1. Amazon Elastic Compute Cloud. http://aws.amazon.com/ec2/.
2. C4.5 Release 8. http://www.rulequest.com/Personal/.
3. CoMon. http://comon.cs.princeton.edu/.
4. InfoScope Distributed Monitoring System. http://dance.csc.ncsu.edu/ projects/infoscope/index.html.
5. PlanetLab. https://www.planet-lab.org/.
6. The STREAM Group, STREAM: The Stanford Stream Data Manager. IEEE Data Engineering Bulletin, 26(1):19-26, Mar. 2003.
7. D. J. Abadi and et al. The Design of the Borealis Stream Processing Engine. In Proc. of CIDR, 2005.
8. M. K. Aguilera, J. Mogul, J. Wiener, P. Reynolds, and A. Muthitacharoen. Performance debugging for distributed systems of black boxes. In Proc. of ACM SOSP, 2003.
9. P. Barham, A. Donnelly, R. Isaacs, and R. Mortier. Using Magpie for request extraction and workload modelling. In Proc. of OSDI, 2004.
10. S. Bhatia, A. Kumar, M. E. Fiuczynski, and L. L. Peterson. Lightweight, high-resolution monitoring for troubleshooting production systems. In Proc. of OSDI, pages 103-116, 2008.
11. J. Breese and R. Blake. Automating computer bottleneck detection with belief nets. In Proc. of UAI, pages 36-45, San Francisco, CA, 1995. Morgan Kaufmann.
12. G. Candea, S. Kawamoto, Y. Fujiki, G. Friedman, and A. Fox. Microreboot- A Technique for Cheap Recovery. In Proc. of OSDI, Dec. 2004.
13. M. Y. Chen, A. Accardi, E. Kiciman, D. Patterson, A. Fox, and E. Brewer. Path-Based Failure and Evolution Management. In Proc. of NSDI, 2004.
14. S. Chen, H. Wang, S. Zhou, and P. S. Yu. Stop Chasing Trends: Discovering High Order Models in Evolving Data. In Proc. of ICDE, 2008.
15. L. Cherkasova, K. M. Ozonat, N. Mi, J. Symons, and E. Smirni. Anomaly? application change? or workload change? towards automated detection of application performance anomaly and change. In Proc. of DSN, pages 452-461, 2008.
16. I. Cohen, M. Goldszmidt, T. Kelly, J. Symons, and J. S. Chase. Correlating Instrumentation Data to System States: A Building Block for Automated Diagnosis and Control. In Proc. of OSDI, 2004.
17. I. Cohen, S. Zhang, M. Goldszmidt, J. Symons, T. Kelly, and A. Fox. Capturing, indexing, clustering, and retrieving system history. In Proc. of SOSP, 2005.
18. J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. In Proc. of OSDI, Dec. 2004.
19. S. Duan, S. Babu, and K. Munagala. Fa: A System for Automating Failure Diagnosis. In Proc. of ICDE, 2009.
20. G. W. Dunlap, S. T. King, S. Cinar, M. A. Basrai, and P. M. Chen. Detecting Past and Present Intrusions Through Vulnerability-Specific Predicates. In Proc. of SOSP, Oct. 2005.
21. K.-L. W. et al. Challenges and Experience in Prototyping a Multi-Modal Stream Analytic and Monitoring Application on System S. In Proc. of VLDB, 2007.
22. B. Gedik, H. Andrade, K.-L. Wu, P. S. Yu, and M. Doo. SPADE: the system s declarative stream processing engine. In Proc. of SIGMOD, 2008.
23. X. Gu, S. Papadimitriou, P. S. Yu, and S. P. Chang. Toward Predictive Failure Management for Distributed Stream Processing Systems. In Proc. of ICDCS, 2008.
24. X. Gu and H. Wang. Online Anomaly Prediction for Robust Cluster Systems. In Proc. of IEEE ICDE, 2009.
25. X. Gu, P. S. Yu, and H. Wang. Adaptive load diffusion for multiway windowed stream joins. In Proc. of ICDE, 2007.
26. Z. Guo, G. Jiang, H. Chen, and K. Yoshihira. Tracking probabilistic correlation of monitoring data for fault detection in complex systems. In Proc. of DSN, pages 259-268, 2006.
27. R. Jin and G. Agrawal. Efficient decision tree construction on streaming data. In Proc. of KDD, 2003.
28. E. Kiciman and A. Fox. Detecting Application-Level Failures in Component-based Internet Services. IEEE Transactions on Neural Networks, 2005.
29. Y. Liang, Y. Zhang, H. Xiong, and R. Sahoo. Failure Prediction in IBM BlueGene/L Event Logs. In Proc. of ICDM, 2007.
30. J. F. Murray, G. F. Hughes, and K. Kreutz-Delgado. Comparison of machine learning methods for predicting failures in hard drives. Journal of Machine Learning Research, 2005.
31. C. C. Noble and D. J. Cook. Graph-based anomaly detection. In Proc. of KDD, pages 631-636, Aug. 24-27 2003.
32. K. M. Ozonat. An information-theoretic approach to detecting performance anomalies and changes for large-scale distributed web services. In Proc. of DSN, pages 522-531, 2008.
33. L. Peterson, T. Anderson, D. Culler, and T. Roscoe. A blueprint for introducing disruptive technology into the internet. In Proc. of HotNets-I, Princeton, New Jersey, October 2002.
34. R. Powers, M. Goldszmidt, and I. Cohen. Short term performance forecasting in enterprise systems. In Proc. of KDD, pages 801-807, 2005.
35. R. K. Sahoo and et al. Critical event prediction for proactive management in large-scale computer clusters. In Proc. of ACM SIGKDD, 2003.
36. B. Schroeder and G. Gibson. Disk failures in the real world: What does an MTTF of 1,000,000 hours mean too you? In Proc. of FAST, 2007.
37. K. Shen, C. Stewart, C. Li, and X. Li. Reference-driven performance anomaly identification. In Proc. of SIGMETRICS/Performance, pages 85-96, 2009.
38. K. Shen, M. Zhong, and C. Li. I/o system performance debugging using model-driven anomaly characterization. In Proc. of FAST, 2005.
39. J. Tucek, S. Lu, C. Huang, S. Xanthos, and Y. Zhou. Triage: Diagnosing Production Run Failures at the User's Site. In Proc. of SOSP, 2007.
40. R. Vilalta, C. V. Apte, J. L. Hellerstein, S. Ma, and S. M. Weiss. Predictive algorithms in the management of computer systems. IBM Systems Journal, 2002.
41. H. J. Wang, J. C. Platt, Y. Chen, R. Zhang, and Y.-M. Wang. Automatic misconfiguration troubleshooting with peerpressure. In Proc. of OSDI, pages 245-258, 2004.
42. A. W. Williams, S. M. Pertet, and P. Narasimhan. Tiresias: Black-box failure prediction in distributed systems. In Proc. of IPDPS, 2007.
43. I. H. Witten and E. Frank. Data Mining : Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann, 1999.
44. W. Xu, L. Huang, A. Fox, D. Patterson, and M. Jordan. Large-scale system problems detection by mining console logs. In Proc. of SOSP, 2009.
45. S. Zhang, I. Cohen, M. Goldszmidt, J. Symons, and A. Fox. Ensemble of models for automated diagnosis of system performance problems. In Proc. of DSN, 2005.