Lessons learned from the analysis of system failures at petascale: The case of blue waters

Proceedings of the International Conference on Dependable Systems and Networks

Volumn , Issue , 2014, Pages 610-621

  Di Martino, Catello ^a   Kalbarczyk, Zbigniew ^a   Iyer, Ravishankar K ^a   Baccanico, Fabio ^b   Fullop, Joseph ^a   Kramer, William ^a  

^a UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

^b UNIVERSITY OF NAPLES FEDERICO II   (Italy)

Author keywords

Cray XE6; Cray XK7; Failure Analysis; Failure Reports; Machine Check; Nvidia GPU errors; Supercomputer

Indexed keywords

ERRORS; FAILURE ANALYSIS; FILE ORGANIZATION; GRAPHICS PROCESSING UNIT; REPAIR; SUPERCOMPUTERS;

AUTOMATICALLY GENERATED; CRAY XE6; CRAY XK7; FAILURE REPORTS; NVIDIA GPU; PROTECTION MECHANISMS; SINGLE NODE FAILURE; UNIVERSITY OF ILLINOIS;

FAILURE (MECHANICAL);

EID: 84912075762     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/DSN.2014.62     Document Type: Conference Paper

References (24)

1. R. K. Sahoo, A. Sivasubramaniam, M. S. Squillante, and Y. Zhang. Failure data analysis of a large-scale heterogeneous server environment. In DSN '04: Proc. of the 2004 Int. Conference on Dependable Systems and Networks, pages 772-781, 2004.
2. Y. Liang, A. Sivasubramaniam, J. Moreira, Y. Zhang, R.K. Sahoo, and M. Jette. Filtering failure logs for a bluegene/l prototype. In DSN '05: Proc. of the 2005 Int. Conference on Dependable Systems and Netw orks, pages 476-485, 2005.
3. Y. Liang, Y. Zhang, M. Jette, Anand Sivasubramaniam, and R. Sahoo. Bluegene/l failure analysis and prediction models. In Dependable Systems and Networks, 2006. DSN 2006. International Conference on, pages 425-434, 2006.
4. B. Schroeder and G.A. Gibson. A large-scale study of failures in highperformance computing systems. Dependable and Secure Computing, IEEE Transactions on, 7(4):337-350, 2010.
5. B. Schroeder and G. A. Gibson. Disk failures in the real world: what does an mttf of 1,000,000 hours mean to you? In Proceedings of the 5th USENIX conference on File and Storage Technologies, FAST '07, Berkeley, CA, USA, 2007. USENIX Associa tion.
6. A. Oliner and J. Stearley. What supercomputers say: A study of five system logs. Dependable Systems and Networks, 2007. DSN '07. 37th Annual IEEE/IFIP Int. Conference on, pages 575-584, June 2007.
7. C. Di Martino, M. Cinque, and D. Cotroneo. Assessing time coalescence techniques for the analysis of supercomputer logs. In In Proc. of 42nd Annual IEEE/IFIP Int. Conf. on Dependable Systems and Networks (DSN), 2012, pages 1-12, 2012.
8. A. Pecchia, d. Cotroneo, Z. Kalbarczyk, and R. K. Iyer. Improving logbased field failure data analysis of multi-node computing systems. In Proceedings of the 2011 IEEE/IFIP 41st International Conference on Dependable Systems&Networks, DSN '11, pages 97-108, Washington, DC, USA, 2011. IEEE Computer Society.
9. B. Schroeder, E. Pinheiro, and W. Weber. Dram errors in the wild: a largescale field study. SIGMETRICS Perform. Eval. Rev., 37(1):193-204, June 2009.
10. A. Gainaru, F. Cappello, M. Snir, and W. Kramer. Fault prediction under the microscope: A closer look into hpc systems. In High Performance Co mputing, Networking, Storage and Analysis (SC), 2012 International Conference for, pages 1-11, 2012.
11. AMD Inc. Bios and kernel developers guide, for amd family 16th.
12. T. Dell. IBM Micr oelectronics Division. A white paper on the benefits of chipkill-correct ecc for pc server main memory, 1997.
13. M. Karo, R. Lagerstrom, M. Kohnke, and C. Albing. The application level placement scheduler. In Cray User Group-CUG, 2008.
14. http://www.adaptivecomputing.com/products/hpc-products/moab-hpc-suite-enterprise-edition.
15. http://www.cray.com/Products/Storage/Sonexion/Specifications.aspx.
16. J. Stearley, R. Ballance, and L. Bauman. A state-machine approach to disambiguating supercomputer event logs. In proc. of Workshop on Managing System Automatically and Dynamically 2, 155-192, Berkeley, CA, 2012. USENIX.
17. V. Sridharan, J. Stearley, N. DeBardeleben, S. Blanchard, and S. Gurumurthi. Feng shui of supercomputer memory: Positional effects in dram and sram faults. In Proceedings of SC13: International Conference for High Performance Computing, Networking, Storage and Analysis, SC '13, pages 22:1-22:11, New York, NY, USA, 2013. ACM.
18. http://www.olcf.ornl.gov/titan/,number2ontop500.org.
19. M. R. Lyu editor. Handbook of software reliability engineering. McGraw-Hill, Hightstown, NJ, USA 1996
20. E. Heien, D. Kondo, A. Gainaru, A. LaPine, W. Kramer, and F. Cappello. Modeling and tolerating heterogeneous failures in large parallel systems. In Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, SC '11, pages 45:1-45:11, New York, NY, USA, 2011. ACM.
21. C. Di Martino. One size does not fit all: Clustering supercomputer failures using a multiple time window approach. In JulianMartin Kunkel, Thomas Ludwig, and HansWerner Me uer, editors, International Supercomputing Conference-Supercomputing, volume 7905 of Lecture Notes in Computer Science, pages 302-316. Springer Berlin Heidelberg, 2013.
22. Xin Chen, Charng-Da Lu, and K. Pattabiraman. Predicting job completion times using system logs in supercomputing clusters. In Dependable Systems and Networks Workshop (DSN-W), 2013 43rd Annual IEEE/IFIP Conference on, pages 1-8, June 2013.
23. A. Gainaru, F. Cappello, and W. Kramer. Taming of the shrew: Modeling the normal and faulty behaviour of large-scale hpc systems. In Parallel Distributed Processing Symposium (IPDPS), 2012 IEEE 26th International , pages 1168-1179, 2012.
24. D. Oppenheimer and D. A. Patterson. Studying and using failure data from large-scale internet services. In Proceedings of the 10th workshop on ACM SIGOPS European workshop, EW 10, pages 255-258, New York, NY, USA, 2002. ACM.