Understanding customer problem troubleshooting from storage system logs

Proceedings of the 7th USENIX Conference on File and Storage Technologies, FAST 2009

Volumn , Issue , 2009, Pages 43-56

  Jiang, Weihang ^a   Hu, Chongfeng ^a   Pasupathy, Shankar ^b   Kanevsky, Arkady ^b   Li, Zhenmin ^c   Zhou, Yuanyuan ^a  

^a UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

^b Network Appliance Inc   (United States)

^c Pattern Insight Inc   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CUSTOMER PROBLEMS; HARDWARE FAILURES; LOG ANALYSIS; MISCONFIGURATIONS; REAL-WORLD; ROOT CAUSE; SOFTWARE FAILURE; STORAGE SYSTEMS;

SALES;

EID: 85077345110     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (58)

1. C. Abad, J. Taylor, C. Sengul, W. Yurcik, Y. Zhou, and K. Rowe. Log correlation for intrusion detection: A proof of concept. In ACSAC’03: Proceedings of the 19th Annual Computer Security Applications Conference, page 255, Washington, DC, USA, 2003. IEEE Computer Society.
2. M. K. Aguilera, J. C. Mogul, J. L. Wiener, P. Reynolds, and A. Muthitacharoen. Performance debugging for distributed systems of black boxes. In Proceedings of SOSP, Bolton Landing, NY, Oct. 2003.
3. P. Bahl, R. Chandra, A. Greenberg, S. Kandula, D. A. Maltz, and M. Zhang. Towards highly reliable enterprise network services via inference of multi-level dependencies. In SIGCOMM’07: Proceedings of the 2007 conference on Applications, technologies, architectures, and protocols for computer communications, pages 13–24, New York, NY, USA, 2007. ACM.
4. L. N. Bairavasundaram, G. R. Goodson, S. Pasupathy, and J. Schindler. An analysis of latent sector errors in disk drives. SIGMETRICS Perform. Eval. Rev., 35(1):289–300, 2007.
5. L. N. Bairavasundaram, G. R. Goodson, B. Schroeder, A. C. Arpaci-Dusseau, and R. H. Arpaci-Dusseau. An Analysis of Data Corruption in the Storage Stack. In FAST’08: Proceedings of the 6th USENIX Conference on File and Storage Technologies, San Jose, CA, Feb. 2008.
6. G. Banga. Auto-diagnosis of field problems in an appliance operating system. In ATEC’00: Proceedings of the annual conference on USENIX Annual Technical Conference, pages 24–24, Berkeley, CA, USA, 2000. USENIX Association.
7. P. Barham, A. Donnelly, R. Isaacs, and R. Mortier. Using magpie for request extraction and workload modelling. In OSDI’04: Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation, pages 18–18, Berkeley, CA, USA, 2004. USENIX Association.
8. A. Bondavalli, S. Chiaradonna, F. D. Giandomenico, and F. Grandoni. Threshold-based mechanisms to discriminate transient from intermittent faults. IEEE Transactions on Computers, 49(3):230–245, 2000.
9. L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone. Classification and Regression Trees. Statistics/Probability Series. Wadsworth Publishing Company, Belmont, California, U.S.A., 1984.
10. M. F. Buckley and D. P. Siewiorek. A comparative analysis of event tupling schemes. In FTCS’96: Proceedings of the The Twenty-Sixth Annual International Symposium on Fault-Tolerant Computing (FTCS’96), page 294, Washington, DC, USA, 1996. IEEE Computer Society.
11. Bug-buddy GNOME bug-reporting utility, 2004. http://directory.fsf.org/All_Packages_in_Directory/bugbuddy.html.
12. B. Chelf and A. Chou. The next generation of static analysis: Boolean satisfiability and path simulation — a perfect match. In Coverity White Paper, 2007.
13. M. Y. Chen, A. Accardi, E. Kiciman, J. Lloyd, D. Patterson, A. Fox, and E. Brewer. Path-based faliure and evolution management. In NSDI’04: Proceedings of the 1st conference on Symposium on Networked Systems Design and Implementation, pages 23–23, Berkeley, CA, USA, 2004. USENIX Association.
14. M. Y. Chen, E. Kiciman, E. Fratkin, A. Fox, and E. Brewer. Pinpoint: Problem determination in large, dynamic internet services. In DSN’02: Proceedings of the 2002 International Conference on Dependable Systems and Networks, pages 595–604, Washington, DC, USA, 2002. IEEE Computer Society.
15. I. Cohen, S. Zhang, M. Goldszmidt, J. Symons, T. Kelly, and A. Fox. Capturing, indexing, clustering, and retrieving system history. In SOSP’05: Proceedings of the twentieth ACM symposium on Operating systems principles, pages 105–118, New York, NY, USA, 2005. ACM.
16. P. Corbett, B. English, A. Goel, T. Grcanac, S. Kleiman, J. Leong, and S. Sankar. Row-diagonal parity for double disk failure correction. In FAST’04: Proceedings of the 3rd USENIX Conference on File and Storage Technologies, pages 1–14, 2004.
17. Crimson Consulting Group. The solaris 10 advantage: Understanding the real cost of ownership of red hat enterprise linux. In Crimson Consulting Group Business White Paper, 2007.
18. Microsoft Corporation. Dr. Watson Overview, 2002. http://www.microsoft.com/TechNet/prodtechnol/ winxppro/proddocs/drwatson%_overview.asp.
19. R. Finlayson and D. Cheriton. Log files: an extended file service exploiting write-once storage. In SOSP’87: Proceedings of the eleventh ACM Symposium on Operating systems principles, pages 139–148, New York, NY, USA, 1987. ACM.
20. A. Ganapathi, Y.-M. Wang, N. Lao, and J.-R. Wen. Why pcs are fragile and what we can do about it: A study of windows registry problems. In DSN’04: Proceedings of the 2004 International Conference on Dependable Systems and Networks, page 561, Washington, DC, USA, 2004. IEEE Computer Society.
21. J. Gray. Why do computers stop and what can be done about it? In Symposium on Reliability in Distributed Software and Database Systems, 1986.
22. S. Hallem, B. Chelf, Y. Xie, and D. Engler. A system and language for building system-specific, static analyses. In PLDI’02: Proceedings of the ACM SIGPLAN 2002 Conference on Programming language design and implementation, pages 69–82, New York, NY, USA, 2002. ACM.
23. J. P. Hansen. Trend analysis and modeling of uni/multi-processor event logs. In Master Thesis, Dept. Electrical and Computer Engineering, Carnegie Mellon University, 1998.
24. H. Huang, I. Raymond Jennings, Y. Ruan, R. Sahoo, S. Sahu, and A. Shaikh. PDA: a tool for automated problem determination. In LISA’07: Proceedings of the 21st conference on 21st Large Installation System Administration Conference, pages 1–14, Berkeley, CA, USA, 2007. USENIX Association.
25. S. Inc. Why splunk? In Splunk White Paper, 2006.
26. R. K. Iyer, L. T. Young, and P. V. K. Iyer. Automatic recognition of intermittent failures: An experimental study of field data. IEEE Trans. Comput., 39(4):525–537, 1990.
27. W. Jiang, C. Hu, Y. Zhou, and A. Kanevsky. Are disks the dominant contributor for storage failures? a comprehensive study of storage subsystem failure characteristics. In FAST’08: Proceedings of the 5th conference on USENIX Conference on File and Storage Technologies, 2008.
28. W. Jiang, C. Hu, Y. Zhou, and A. Kanevsky. Don’t blame disks for every storage subsystem failure. In NetApp Technical Journal, 2008.
29. S. Johnson. Lint, a c program checker, 1978.
30. S. Kandula, D. Katabi, and J.-P. Vasseur. Shrink: a tool for failure diagnosis in ip networks. In MineNet’05: Proceedings of the 2005 ACM SIGCOMM workshop on Mining network data, pages 173–178, New York, NY, USA, 2005. ACM.
31. M. P. Kasick, P. Narasimhan, K. Atkinson, and J. Lepreau. Towards fingerpointing in the emulab dynamic distributed system. In WORLDS’06: Proceedings of the 3rd conference on USENIX Workshop on Real, Large Distributed Systems, pages 7–7, Berkeley, CA, USA, 2006. USENIX Association.
32. R. R. Kompella, J. Yates, A. Greenberg, and A. C. Snoeren. Ip fault localization via risk modeling. In NSDI’05: Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation, pages 57–70, Berkeley, CA, USA, 2005. USENIX Association.
33. L. Lancaster and A. Rowe. Measuring real-world data availability. In LISA’01: Proceedings of the 15th USENIX conference on System administration, pages 93–100, Berkeley, CA, USA, 2001. USENIX Association.
34. N. Lao, J.-R. Wen, W.-Y. Ma, and Y.-M. Wang. Combining high level symptom descriptions and low level state information for configuration fault diagnosis. In LISA’04: Proceedings of the 18th USENIX conference on System administration, pages 151–158, Berkeley, CA, USA, 2004. USENIX Association.
35. Z. Li, S. Lu, S. Myagmar, and Y. Zhou. Cp-miner: a tool for finding copy-paste and related bugs in operating system code. In OSDI’04: Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation, pages 20–20, Berkeley, CA, USA, 2004. USENIX Association.
36. Z. Li and Y. Zhou. Pr-miner: automatically extracting implicit programming rules and detecting violations in large software code. In ESEC/FSE-13: Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIG-SOFT international symposium on Foundations of software engineering, pages 306–315, New York, NY, USA, 2005. ACM.
37. Y. Liang, Y. Zhang, H. Xiong, and R. K. Sahoo. An adaptive semantic filter for blue gene/l failure log analysis. In IPDPS, pages 1–8, 2007.
38. T. T. Y. Lin and D. P. Siewiorek. Error log analysis: statistical modeling and heuristic trend analysis. Reliability, IEEE Transactions on, 39(4):419–432, 1990.
39. I. LogLogic. Logs: Data warehouse style. In LogLogic White Paper, 2007.
40. G. Lohman, J. Champlin, and P. Sohn. Quickly finding known software problems via automated symptom matching. In ICAC ’05: Proceedings of the Second International Conference on Automatic Computing, pages 101–110, Washington, DC, USA, 2005. IEEE Computer Society.
41. T. Mitchell. Machine learning. McGraw Hill, 1997.
42. B. Murphy and T. Gent. Measuring system and software reliability using an automated data collection process. Quality and Reliability Engineering International, 11, 1995.
43. K. Nagaraja, F. Oliveira, R. Bianchini, R. P. Martin, and T. D. Nguyen. Understanding and dealing with operator mistakes in internet services. In OSDI’04: Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation, pages 5–5, Berkeley, CA, USA, 2004. USENIX Association.
44. I. Network Appliance. Proactive health management with autosupport. In NetApp White Paper, 2007.
45. D. Oppenheimer, A. Ganapathi, and D. A. Patterson. Why do internet services fail, and what can be done about it? In USITS’03: Proceedings of the 4th conference on USENIX Symposium on Internet Technologies and Systems, pages 1–1, Berkeley, CA, USA, 2003. USENIX Association.
46. D. Patterson, A. Brown, P. Broadwell, G. Candea, M. Chen, J. Cutler, P. Enriquez, A. Fox, E. Kiciman, M. Merzbacher, D. Oppenheimer, N. Sastry, W. Tetzlaff, J. Traupman, and N. Treuhaft. Recovery oriented computing (roc): Motivation, definition, techniques,. Technical report, Berkeley, CA, USA, 2002.
47. E. Pinheiro, W.-D. Weber, and L. A. Barroso. Failure trends in a large disk drive population. In FAST’07: Proceedings of the 5th USENIX conference on File and Storage Technologies, pages 2–2, Berkeley, CA, USA, 2007. USENIX Association.
48. J. A. Redstone, M. M. Swift, and B. N. Bershad. Using computers to diagnose computer problems. In HOTOS’03: Proceedings of the 9th conference on Hot Topics in Operating Systems, pages 16–16, Berkeley, CA, USA, 2003. USENIX Association.
49. C. J. V. Rijsbergen. Information Retrieval. Butterworth-Heinemann, Newton, MA, USA, 1979.
50. M. Steinle, K. Aberer, S. Girdzijauskas, and C. Lovis. Mapping moving landscapes by mining mountains of logs: novel techniques for dependency model generation. In VLDB’06: Proceedings of the 32nd international conference on Very large data bases, pages 1093–1102. VLDB Endowment, 2006.
51. Linux system logging utilities. Linux Man Page: SYSKLOGD(8).
52. The Association of Support Professionals. Technical Support Cost Ratios. 2000.
53. M. M. Tsao. Trend analysis and fault prediction. In PhD Dissertation, Dept. Electrical and Computer Engineering, Carnegie Mellon University, 1998.
54. C. Verbowski, E. Kiciman, A. Kumar, B. Daniels, S. Lu, J. Lee, Y.-M. Wang, and R. Roussev. Flight data recorder: monitoring persistent-state interactions to improve systems management. In OSDI’06: Proceedings of the 7th symposium on Operating systems design and implementation, pages 117–130, Berkeley, CA, USA, 2006. USENIX Association.
55. H. J. Wang, J. C. Platt, Y. Chen, R. Zhang, and Y.-M. Wang. Automatic misconfiguration troubleshooting with peerpressure. In OSDI’04: Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation, pages 17–17, Berkeley, CA, USA, 2004. USENIX Association.
56. Y.-M. Wang, C. Verbowski, J. Dunagan, Y. Chen, H. J. Wang, C. Yuan, and Z. Zhang. Strider: A black-box, state-based approach to change and configuration management and support. In LISA’03: Proceedings of the 17th USENIX conference on System administration, pages 159–172, Berkeley, CA, USA, 2003. USENIX Association.
57. A. C. Xansa, A. Hanna, C. Rudd, I. Macfarlane, J. Windebank, and S. Rance. An Introductory Overview of ITIL v3. http://www.itsmfi.org/, 2007.
58. C. Yuan, N. Lao, J.-R. Wen, J. Li, Z. Zhang, Y.-M. Wang, and W.Y. Ma. Automated known problem diagnosis with event traces. SIGOPS Oper. Syst. Rev., 40(4):375–388, 2006.