Understanding GPU errors on large-scale HPC systems and the implications for system design and operation

2015 IEEE 21st International Symposium on High Performance Computer Architecture, HPCA 2015

Volumn , Issue , 2015, Pages 331-342

  Tiwari, Devesh ^d   Gupta, Saurabh ^d   Rogers, James ^d   Maxwell, Don ^d   Rech, Paolo ^c   Vazhkudai, Sudharshan ^d   Oliveira, Daniel ^c   Londo, Dave ^a   Debardeleben, Nathan ^b   Navaux, Philippe ^c   Carro, Luigi ^c   Bland, Arthur ^d  

^a CRAY INC   (United States)

^b LOS ALAMOS NATIONAL LABORATORY   (United States)

^c FEDERAL UNIVERSITY OF RIO GRANDE DO SUL   (Brazil)

^d OAK RIDGE NATIONAL LABORATORY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ASTROPHYSICS; CLUSTER COMPUTING; COMPUTER GRAPHICS EQUIPMENT; GRAPHICS PROCESSING UNIT; NEUTRON BEAMS; NEUTRONS; PROGRAM PROCESSORS; SUPERCOMPUTERS;

COMPUTING FACILITIES; DESIGN AND OPERATIONS; GENERAL-PURPOSE COMPUTING; GRAPHICS HARDWARE; LARGE SCALE SIMULATIONS; LARGE-SCALE COMPUTING SYSTEMS; LOS ALAMOS NATIONAL LABORATORY; PERFORMANCE EFFICIENCY;

COMPUTER ARCHITECTURE;

EID: 84934300860     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/HPCA.2015.7056044     Document Type: Conference Paper

References (43)

1. "Computational science requirements for leadership computing. 2007, hnp:/Iwww. olcf. ornl. gov/wp-contcntJuploads/20 IO/03/0RNL TM-2007-44. pdf:"
2. '7esla k20 gpu accelerator. board specification. " http://www. nvidia. comlconlentIPOF/keplerffesla-K20-Passive-80-06455-00 1-v07. pdf.
3. "Understanding xid errors. " hup:/ldocs. nvidia. com/deploy/xid-errorsl index. hlnll.
4. M. Bagatin. S. Gerardin. A. Paccagnella, C. Andreani. G. Gorini. and C. Frost, "Temperature dependence of neutron-induced soft errors in scams," Microelectronics Reliability, vol. 52, no. 1. pp. 289-293. 2012.
5. R. Baumann. "Radiation-induced soft errors in advanced semiconductor technologies," Device and Materials Reliability, IEEE Transactions on,voI. 5,no. 3,pp. 305-316,Sept2005.
6. G. Bruni. "Temperature effects on soft error rate due to atmospheric neutrons on 28 nm fpgas:' 2014.
7. A. Danalis, G. Marin, C. McCurdy, J. S. Meredith, P. C. Roth, K. Spafford. V. 1ipparaju, and J. S. Vetter, The scalable heterogeneous computing (shoe) benchmark suite," in Proceeding.,. of the 3rd Workshop on General-Purpose Computation on Graphics Processing Units, sera GPGPU ' 10. New York, NY, USA: ACM, 2010, pp. 63-74. . Available: http://doi. acm. org/l0. 1145/17356H~t1735702
8. N. DeBardeleben, S. Blanchard, L. Monroe, P. Romero, D. Grunau, C. Idler, and C. Wright, "GPU Behavior on a Large HPC Cluster:' 6th Workshop on Resiliency in High Perfonnance Computing (Resilience) in Clusters, Clouds, and Grids in conjunction with the 19th International European Conference on Parallel and Distributed Computing (Euro-Par 2013), Aachen, Gennan)'. . August 26-30 2013.
9. C. Di Martino, F. Baccanico, W. Kramer, J. Fullop, Z. Kalbarczyk, and R. Iyer, "Lessons learned from the analysis of system failures at petascale: The case of blue waters," 44th international.
10. Ding C. Karlsson C. Liu H. Davies T. Chen Z. Matrix multiplication on gpus with on-line fault tolerance Parallel and Distributed Processing with Applications (ISPA), 201 J IEEE 9th International Symposium on 311-317 May 2001.
11. A. Dixit and A. Wood, ''The impact of new technology on soft error rates," in Reliability Physics SYlnposium (IRPS), 201 I IEEE International. IEEE, 2011, pp. 5B-4.
12. N. EI-Sayed and B. Schroeder, ~'Readjng between the lines of failure logs: Understanding how hpc systems fail, DSN," 2013.
13. B. Fang, K. Pattabiraman. M. Ripeanu, and S. Gurumurthi, uGpu-qin: A methodology for evaluating the error resilience of gpgpu applications," 2014.
14. B. Fang, J. Wei, K. Pattabiraman, and M. Ripeanu, UPoster: Evaluating error resiliency of gpgpu applications," in High Peiformance Computing, Networking, Storage and Analysis (SCC), 2012 SC Companion:, Nov 2012, pp. 1504-1504.
15. L. A. B. Gomez, F. CapPello, L. Carro, N. DeBardeleben, B. Fang, S. Gurumurthi. S. Keckler, K. Pattabiraman, R. Rech, and M. S. Reorda, "Gpgpus: How to combine high computational power with high reliability," in 2014 Design Automation and Test in Europe Conference and E. thihition, Dresden. Germany, 2014.
16. D. M. H. and M. J. Schervish, "Probability and statistics. 3rd ed. boston, rna: Addison-wesley, 2002. "
17. I. Haque and V. Pande, "Hard Data on Soft Errors: A Large-Scale Assessment of Real-World Error Rates in GPGPU. " in Cluster. Cloud and Grid Computing (CCGrid), 2010 10th IEEE/ACM International Conference on, 2010, pp. 691-696.
18. A. A. Hwang, I. A. Stefanovici, and B. Schroeder. ''Cosmic rays don't strike twice: understanding the nature of dram errors and the implications for system design," ACM SIGPLAN Notices, vol. 47, no. 4, pp. 111-122. 2012.
19. E. lbe, S. Chung, S. Wen, H. Yamaguchi, Y. Yahagi, H. Kameyama. S. Yamamoto, and T. Akioka, "Spreading diversity in multi-cell neutron-induced upsets with device scaling," in Custom Integrated Circuits Conference, 2006. CICC'06. IEEE. IEEE, 2006, pp. 437444.
20. JEDEC. '"Mea~urement and Reporting of Alpha Particle and Terrestrial Cosmic Ray-Induced Soft Errors in Semiconductor Devices," JEDEC Standard, Tech. Rep. JESD89A, 2006.
21. Y. Liang, Y. Zhang, M. Jette, A. Sivasubramaniam, and R. Sahoo, UBluegenell failure analysis and prediction models," in Dependable Systelns and Networks, 2006. DSN 2006. International Conference on. IEEE, 2006, pp. 425-434.
22. Y. Liang, Y. Zhang, A. Sivasubramaniam, R. K. Sahoo, J. Moreira, and M. Gupta, "Filtering failure logs for a bluegenell prototype. " in Dependable Systelns and Networks, 2005. DSN 2005. Proceedings. International Conference on. IEEE, 2005, pp. 476-485.
23. R. Lucas, '70p ten exascale research challenges:' in DOE ASCAC Subcommillee Report, 2014.
24. NVIDIA. "uBLAS Library User Guide," http://docs. nvidia. comlcudal pdf/CUBLAS-Library. pdf. 2014.
25. A. Oliner and J. Stearley, "What supercomputers say: A study of five system logs," in Dependable Systems and Networks, 2007. DSN'07. 37th Annual IEEElIFIP International Conference on. IEEE, 2007, pp. 575-584.
26. D. A. G. Oliveira, P. Rech, L. L. Pill~ P. O. A. Navaux, and L. Carro, "Gpgpus ecc efficiency and efficacy," in International Symposium on Defect and Failit Tolerance ill VLSI and Nanotechnology Systems (DFT 2014), 2014.
27. "Preparing for Exascale: ORNL Leadership Computing Facility Application Requirements and Strategy, 2009, http://www. olcf. oml. gov/ wp-contentluploads/20 IO/03/olcf-requirenlents. pdf. "
28. A. Pecchia, D. Cotroneo, Z. Kalbarczyk, and R. K. Iyer, "Improving log-based field failure data analysis of multi-node computing systems," in Dependable Systems &: Networks (DSN), 2011 IEEE/IFIP 41st International Conference on. IEEE, 2011, pp. 97-108.
29. L. Pilla, P. Rech, F. Silvestri, C. Frost. P. Navaux, M. Reorda, and L. Carro, "Software-based hardening strategies for neutron sensitive fft algorithms on gpus," Nuclear Science, IEEE Transactions on, vol. PP. no. 99,pp. 1-7,2014.
30. P. Rech, C. Aguiar, R. Ferreira, C. Frost, and L. Carro, "Neutron radiation test of graphic processing units," in On-Line Testing Symposium (fOLTS), 2012 IEEE 18th International, June 2012, pp. 55-60.
31. P. Rech, C. Aguiar, C. Frost, and L. Carro, "An Efficient and Experimentally Tuned Software-Based Hardening Strategy for Matrix Multiplication on GPUs," Nuclear Science, IEEE Transactions on, vol. 6O,no. 4,pp. 2797-2804,2013.
32. -, "Experimental evaluation of thread distribution effects on multiple output errors in gpus," in Test Symposium (ETS), 2013 18th IEEE European. IEEE, 2013, pp. 1-6.
33. P. Rech, L. Carro, N. Wang, T. Tsai, S. K. S. Hari, and S. W. Keckler. "Measuring the Radiation Reliability of SRAM Structures in OPUS Designed for HPC," in IEEE 10th Workshop on Silicon Errors in Logic-Systeln Effects (SELSE), 2014.
34. D. Sabena, M. Sonza Reorda, L. Sterpone, P. Rech, and L. Carro, "On the evaluation of soft-errors detection techniques for gpgpus," in Design and Test Symposillln (IDT), 2013 8th International, Dec 2013, pp. 1-6.
35. R. K. Sahoo, M. S. Squillante, A. Sivasubramaniam, and Y. Zhang, "Failure data analysis of a large-scale heterogeneous server environment:' in Dependable Systems and Nehvorks, 2004 International Conference on. IEEE, 2004, pp. 772-781.
36. B. Schroeder and G. Gibson, "A large-scale study of failures in highperformance computing systems. " Dependable and Secure Computing, IEEE Transactions on, vol. 7, no. 4, pp. 337-350, 2010.
37. B. Schroeder and G. A. Gibson, "Disk failures in the real world: What does an muf of 1,000,000 hours mean to you" in FAST, vol. 7,2007, pp. I-16.
38. B. Schroeder, E. Pinheiro, and W.-D. Weber, "Dram errors in the wild: a large-scale field study," in ACM SIGMEfRICS Perfonnance Eltaluation Review, vol. 37, no. I. ACM, 2009, pp. 193-204.
39. M. Snir, R. W. Wisniewski, J. A. Abraham, S. V. Adve, S. Bagchi, P. Balaji, J. Belak, P. Bose. F. Cappello, B. Carlson et al., "Addressing failures in exascale computing," International Journal of High Peifonnance Computing Applications. p. 1094342014522573. 2014.
40. V. Sridharan, J. Stearley, N. DeBardeleben, S. Blanchard, and S. Gurumurthi, "Feng shui of supercomputer memory: positional effects in dram and sram faults," in Proceedings ofSC13: International Conference for High Perfonnance Computing, Networking, Storage and Analysis. ACM. 2013, p. 22.
41. J. Tan, N. Goswami, T. Li, and X. Fu, "Analyzing soft-error vulnerability on gpgpu microarchitecture," in Workload Characterization (lISWC), 2011 IEEE International Symposiuln on, Nov 2011, pp. 226235.
42. D. Tiwari, S. Gupta, and S. S. Vazhkudai, ULazy checkpointing: Exploiting temporal locality in failures to mitigate checkpointing overheads on extreme-scale systems," International Conference on Dependable Systems and Nehl'orks (DSN), 2014.
43. M. Violante, L. Sterpone, A. Manuzzato, S. Gerardin, P. Rech, M. Bagatin, A. Paccagnella, C. Andreani, G. Gorini, A. Pietropaolo, G. Cardarilli, S. Pontarelli, and C. Frost, "A New Hardware/Software Platform and a New lIE Neutron Source for Soft Error Studies: Testing FPGAs at the ISIS Facility," Nuclear Science, IEEE Transactions on, voI. 54,no. 4,pp. 1184-1189,2oo7. (44) J. F. Ziegler and H. Puchner, SER-history, Trends and Challenges: A Guide for Designing with Memory les. Cypress, 2010.