Simple testing can prevent most critical failures: An analysis of production failures in distributed data-intensive systems

Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014

Volumn , Issue , 2014, Pages 249-265

  Yuan, Ding ^a   Luo, Yu ^a   Zhuang, Xin ^a   Rodrigues, Guilherme Renna ^a   Zhao, Xu ^a   Zhang, Yongle ^a   Jain, Pranay U ^a   Stumm, Michael ^a  

^a UNIVERSITY OF TORONTO   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

CELL PROLIFERATION; CODES (SYMBOLS); DISTRIBUTED COMPUTER SYSTEMS; ERRORS; FILE ORGANIZATION; PROGRAM DEBUGGING; SOFTWARE DESIGN; SOFTWARE TESTING; SYSTEMS ANALYSIS;

CATASTROPHIC FAILURES; CRITICAL FAILURES; DISTRIBUTED SYSTEMS; ERROR HANDLING CODES; HADOOP DISTRIBUTED FILE SYSTEM (HDFS); HADOOP MAPREDUCE; PRODUCTION FAILURE; PRODUCTION QUALITY;

FAILURE (MECHANICAL);

EID: 85020616835     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (67)

1. Why Amazon's cloud titanic went down. http://money.cnn.com/2011/04/22/technology/a mazon ec2 cloud outage/index.htm.
2. Apache Cassandra. http://cassandra.apache.o rg.
3. Apache HBase. http://hbase.apache.org.
4. T. Bergan, N. Hunt, L. Ceze, and S. D. Gribble. Deterministic process groups in dOS. In Proceedings of the 9th USENIX Conference on Operating Systems Design and Implementation, OSDI'10, 2010.
5. I. Beschastnikh, Y. Brun,, J. Abrahamson, M. D. Ernst, and A. Krishnamurthy. Unifying FSM-inference algorithms through declarative specification. In Proceedings of The International Conference on Software Engineering, ICSE'13, 2013.
6. I. Beschastnikh, Y. Brun, M. D. Ernst, A. Krishnamurthy, and T. E. Anderson. Mining temporal invariants from partially ordered logs. In Managing Large-scale Systems via the Analysis of System Logs and the Application of Machine Learning Techniques, SLAML'11, pages 3:1-3:10, 2011.
7. J. Bloch. Effective Java (2nd Edition). Prentice Hall, 2008.
8. C. Cadar, D. Dunbar, and D. Engler. Klee: Unassisted and automatic generation of high-coverage tests for complex systems programs. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation, OSDI'08, pages 209-224, 2008.
9. Chaos monkey. https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey.
10. V. Chipounov, V. Kuznetsov, and G. Candea. The S2E platform: Design, implementation, and applications. ACM Trans. Comput. Syst., 30(1):2:1-2:49, Feb. 2012.
11. Chord: A program analysis platform for Java. http://pag.gatech.edu/chord.
12. A. Chou, J. Yang, B. Chelf, S. Hallem, and D. Engler. An empirical study of operating systems errors. In Proceedings of the 18th ACM Symposium on Operating Systems Principles, SOSP'01, pages 73-88, 2001.
13. H. Cui, J. Simsa, Y.-H. Lin, H. Li, B. Blum, X. Xu, J. Yang, G. A. Gibson, and R. E. Bryant. Parrot: A practical runtime for deterministic, stable, and reliable threads. In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, SOSP'13, pages 388-405, 2013.
14. J. Dean. Underneath the covers at Google: current systems and future directions. In Google I/O, 2008.
15. G. W. Dunlap, S. T. King, S. Cinar, M. A. Basrai, and P. M. Chen. ReVirt: enabling intrusion analysis through virtual-machine logging and replay. In Proceedings of the Fifth Symposium on Operating Systems Design and Implementation, OSDI'02, 2002.
16. D. Engler, B. Chelf, A. Chou, and S. Hallem. Checking system rules using system-specific, programmer-written compiler extensions. In Proceedings of the 4th conference on Symposium on Operating System Design and Implementation, OSDI'00, pages 1-16, 2000.
17. Facebook: More details on today's outage. https://www.facebook.com/note.php?note id= 431441338919&id=9445547199&ref=mf.
18. Hadoop team. Fault injection framework: How to use it, test using artificial faults, and develop new faults. http://wiki.apache.org/hadoop/HowT oUseInjectionFramework.
19. N. Feamster and H. Balakrishnan. Detecting BGP configuration faults with static analysis. In Proceedings of the 2nd USENIX Symposium on Networked System Design and Implementation, NSDI'05, 2005.
20. C. Fu and G. B. Ryder. Exception-chain analysis: Revealing exception handling architecture in java server applications. In 29th International Conference on Software Engineering, ICSE'07, pages 230-239, 2007.
21. Google outage reportedly caused big drop in global traffic. http://www.cnet.com/news/googleoutage-reportedly-caused-big-drop-in-global-traffic/.
22. J. Gray. Why do computers stop and what can be done about it? In Proceedings of the Symposium on Reliability in Distributed Software and Database Systems, 1986.
23. H. S. Gunawi, T. Do, P. Joshi, P. Alvaro, J. M. Hellerstein, A. C. Arpaci-Dusseau, R. H. Arpaci-Dusseau, K. Sen, and D. Borthakur. FATE and DESTINI: a framework for cloud recovery testing. In Proceedings of the 8th USENIX conference on Networked Systems Design and Implementation, NSDI'11, 2011.
24. H. S. Gunawi, C. Rubio-González, A. C. Arpaci-Dusseau, R. H. Arpaci-Dussea, and B. Liblit. EIO: Error handling is occasionally correct. In Proceedings of the 6th USENIX Conference on File and Storage Technologies, FAST'08, 2008.
25. H. Guo, M. Wu, L. Zhou, G. Hu, J. Yang, and L. Zhang. Practical software model checking via dynamic interface reduction. In Proceedings of the 23rd ACM Symposium on Operating Systems Principles, pages 265-278, October 2011.
26. Z. Guo, X. Wang, J. Tang, X. Liu, Z. Xu, M. Wu, M. F. Kaashoek, and Z. Zhang. R2: An application-level kernel for record and replay. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation, OSDI'08, pages 193-208, Berkeley, CA, USA, 2008. USENIX Association.
27. Hadoop Distributed File System (HDFS) architecture guide. http://hadoop.apache.org/docs/stable/hdfs design.html.
28. Hadoop MapReduce. http://hadoop.apache.org/docs/stable/mapred tutorial.html.
29. Hadoop market is expected to reach usd 20.9 billion globally in 2018. http://www.prnewswire.com/news-releases/hadoop-market-is-expectedto-reach-usd-209-billion-globallyin-2018-transparency-market-research-217735621.html.
30. DB-Engines ranking of wide column stores. http://db-engines.com/en/ranking/wi de+column+store.
31. HBase bug report 10452 - Fix bugs in exception handler. https://issues.apache.org/jira/browse/HBASE-10452.
32. D. Hovemeyer and W. Pugh. Finding bugs is easy. SIGPLAN Notice, 39(12):92-106, Dec. 2004.
33. S. Kandula, R. Mahajan, P. Verkaik, S. Agarwal, J. Padhye, and P. Bahl. Detailed diagnosis in enterprise networks. In Proceedings of the ACM SIGCOMM 2009 conference, SIGCOMM'09, pages 243-254, 2009.
34. C. Killian, J. W. Anderson, R. Jhala, and A. Vahdat. Life, death, and the critical transition: finding liveness bugs in systems code. In Proceedings of the Fourth Symposium on Networked Systems Design and Implementation, pages 243-256, April 2007.
35. O. Laadan, N. Viennot, and J. Nieh. Transparent, lightweight application execution replay on commodity multiprocessor operating systems. In Proceedings of the ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS'10, pages 155-166, 2010.
36. J.-C. Laprie. Dependable computing: concepts, limits, challenges. In Proceedings of the 25th International Conference on Fault-tolerant Computing, FTCS'95, pages 42-54, 1995.
37. T. Leesatapornwongsa, M. Hao, P. Joshi, J. F. Lukman, and H. S. Gunawi. Samc: Semantic-aware model checking for fast discovery of deep bugs in cloud systems. In Proceedings of the 11th USENIX Symposium on Operating System Design and Implementation, OSDI'14, 2014.
38. S. Li, T. Xiao, H. Zhou, H. Lin, H. Lin, W. Lin, and T. Xie. A characteristic study on failures of production distributed data-parallel programs. In Proc. International Conference on Software Engineering (ICSE 2013), Software Engineering in Practice (SEIP) track, May 2013.
39. Z. Li, S. Lu, S. Myagmar, and Y. Zhou. Cp-miner: A tool for finding copy-paste and related bugs in operating system code. In Proceedings of the 6th Conference on Symposium on Opearting Systems Design and Implementation, OSDI'04, 2004.
40. G. C. Lorenzo Keller, Prasang Upadhyaya. ConfErr: A tool for assessing resilience to human configuration errors. In Proceedings International Conference on Dependable Systems and Networks, DSN'08, 2008.
41. L. Lu, A. C. Arpaci-Dusseau, R. H. Arpaci-Dusseau, and S. Lu. A study of Linux file system evolution. In Proceedings of the 11th USENIX Conference on File and Storage Technologies, FAST'13, pages 31-44, 2013.
42. S. Lu, S. Park, E. Seo, and Y. Zhou. Learning from mistakes: a comprehensive study on real world concurrency bug characteristics. In Proceedings of the 13th international conference on Architectural support for programming languages and operating systems, ASPLOS'08, pages 329-339, 2008.
43. R. Mahajan, D. Wetherall, and T. Anderson. Understanding BGP misconfiguration. In Proceedings of the ACM SIGCOMM 2002 conference, SIGCOMM'02, pages 3-16, 2002.
44. P. D. Marinescu, R. Banabic, and G. Candea. An extensible technique for high-precision testing of recovery code. In Proceedings of the 2010 USENIX annual technical conference, USENIX ATC'10, 2010.
45. P. D. Marinescu and G. Candea. Efficient testing of recovery code using fault injection. ACM Trans. Comput. Syst., 29(4):11:1-11:38, Dec. 2011.
46. Missing dot drops Sweden off the internet. http://www.networkworld.com/community/node/46115.
47. P. Montesinos, L. Ceze, and J. Torrellas. Delorean: Recording and deterministically replaying shared-memory multiprocessor execution effiently. In Proceedings of the 35th Annual International Symposium on Computer Architecture, ISCA'08, pages 289-300, Washington, DC, USA, 2008. IEEE Computer Society.
48. K. Nagaraj, C. Killian, and J. Neville. Structured comparative analysis of systems logs to diagnose performance problems. In Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation, NSDI'12, 2012.
49. K. Nagaraja, F. Oliveira, R. Bianchini, R. P. Martin, and T. D. Nguyen. Understanding and dealing with operator mistakes in internet services. In Proceedings of the 6th conference on Symposium on Opearting Systems Design and Implementation, OSDI'04, 2004.
50. D. Oppenheimer, A. Ganapathi, and D. A. Patterson. Why do Internet services fail, and what can be done about it? In Proceedings of the 4th conference on USENIX Symposium on Internet Technologies and Systems, USITS'03, pages 1-15, 2003.
51. N. Palix, G. Thomas, S. Saha, C. Calvès, J. Lawall, and G. Muller. Faults in Linux: ten years later. In Proceedings of the 16th International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS'11, pages 305-318, 2011.
52. A. Rabkin and R. Katz. How Hadoop clusters break. Software, IEEE, 30(4):88-94, 2013.
53. DB-Engines ranking of key-value stores. http://dbengines.com/en/ranking/key-value+store.
54. Redis: an open source, advanced key-value store. http://redis.io/.
55. C. Rubio-González, H. S. Gunawi, B. Liblit, R. H. Arpaci-Dusseau, and A. C. Arpaci-Dusseau. Error propagation analysis for file systems. In Proceedings of the 2009 ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI'09, pages 270-280, 2009.
56. B. Schroeder and G. Gibson. A large-scale study of failures in high-performance computing systems. IEEE Transactions on Dependable and Secure Computing, 7(4):337-350, 2010.
57. C. Spatz. Basic statistics, 1981.
58. M. Sullivan and R. Chillarege. Software defects and their impact on system availability - A study of field failures in operating systems. In Twenty-First International Symposium on Fault-Tolerant Computing, FTCS'91, pages 2-9, 1991.
59. Summary of the Amazon EC2 and RDS service disruption. http://aws.amazon.com/message/65648/.
60. The curse of the swallowed exception. http://michaelscharf.blogspot.ca/2006/09/dontswallow-interruptedexception-call.html.
61. K. Veeraraghavan, D. Lee, B. Wester, J. Ouyang, P. M. Chen, J. Flinn, and S. Narayanasamy. DoublePlay: Parallelizing sequential logging and replay. In Proceedings of the 16th International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS'11, 2011.
62. K. V. Vishwanath and N. Nagappan. Characterizing cloud computing hardware reliability. In Proceedings of the 1st ACM symposium on Cloud computing, SoCC'10, pages 193-204, New York, NY, USA, 2010. ACM.
63. T. Xu, J. Zhang, P. Huang, J. Zheng, T. Sheng, D. Yuan, Y. Zhou, and S. Pasupathy. Do not blame users for misconfigurations. In Proceedings of the 24th ACM Symposium on Operating Systems Principles, SOSP'13, 2013.
64. W. Xu, L. Huang, A. Fox, D. Patterson, and M. I. Jordan. Detecting large-scale system problems by mining console logs. In Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, SOSP'09, pages 117-132, 2009.
65. J. Yang, T. Chen, M. Wu, Z. Xu, X. Liu, H. Lin, M. Yang, F. Long, L. Zhang, and L. Zhou. MODIST: Transparent model checking of unmodified distributed systems. In Proceedings of the Sixth Symposium on Networked Systems Design and Implementation (NSDI'09), pages 213-228, April 2009.
66. Z. Yin, X. Ma, J. Zheng, Y. Zhou, L. N. Bairavasundaram, and S. Pasupathy. An empirical study on configuration errors in commercial and open source systems. In Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles, SOSP'11, pages 159-172, 2011.
67. D. Yuan, S. Park, P. Huang, Y. Liu, M. Lee, Y. Zhou, and S. Savage. Be conservative: Enhancing failure diagnosis with proactive logging. In Proceedings of the 10th USENIX Symposium on Operating System Design and Implementation, OSDI'12, pages 293-306, 2012.