Proactive process-level live migration and back migration in HPC environments

Journal of Parallel and Distributed Computing

Volumn 72, Issue 2, 2012, Pages 254-267

  Wang, Chao ^a   Mueller, Frank ^a   Engelmann, Christian ^b   Scott, Stephen L ^b  

^a North Carolina State University   (United States)

^b OAK RIDGE NATIONAL LABORATORY   (United States)

Author keywords

Back migration; Fault tolerance; Health monitoring; High performance computing; Live migration

Indexed keywords

BACK MIGRATION; EXECUTION ENVIRONMENTS; HEALTH MONITORING; HIGH-PERFORMANCE COMPUTING; LIVE MIGRATIONS; LOAD IMBALANCE; NODE FAILURE; PROCESS LEVELS; PROCESS MIGRATION; SELF-HEALING; VIRTUALIZATIONS;

COMPUTER SOFTWARE SELECTION AND EVALUATION; EXPERIMENTS; FAULT TOLERANT COMPUTER SYSTEMS; HEALTH;

FAULT TOLERANCE;

EID: 84855350553     PISSN: 07437315     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jpdc.2011.10.009     Document Type: Article

References (80)

1. Advanced configuration & power interface, http://www.acpi.info.
2. N. Adiga An overview of the BlueGene/L supercomputer Supercomputing 2002
3. B. Allen, Monitoring hard disk with smart, Linux Journal, 2004.
4. J. Ansel, K. Arya, G. Cooperman, Dmtcp: Transparent checkpointing for cluster computations and the desktop, in: 23rd IEEE International Parallel and Distributed Processing Symposium, 2009.
5. I. T. Association, Infiniband, http://www.infinibandta.org/.
6. R.T. Aulwes, D.J. Daniel, N.N. Desai, R.L. Graham, L.D. Risinger, M.A. Taylor, T. Woodall, M. Sukalski, Architecture of LA-MPI, a network-fault- tolerant MPI, in: IPDPS, 2004.
7. A. Avritzer, A. Bondi, M. Grottke, K.S. Trivedi, E.J. Weyuker, Performance assurance via software rejuvenation: Monitoring, statistics and algorithms, in: Proc. International Conference on Dependable Systems and Networks, 2006, pp. 435444.
8. A. Barak, R. Wheeler, MOSIX: An integrated multiprocessor UNIX, in: Proceedings of the Winter 1989 USENIX Conference, USENIX, Berkeley, CA, USA, 1989, pp. 101112.
9. G. Bosilca, A. Boutellier, and F. Cappello MPICH-V: toward a scalable fault tolerant MPI for volatile nodes Supercomputing 2002
10. J. Brevik, D. Nurmi, R. Wolski, Automatic methods for predicting machine availability in desktop grid and peer-to-peer systems, in: IEEE International Symposium on Cluster Computing and the Grid, 2004, pp. 190199.
11. J. Cao, Y. Li, M. Guo, Process migration for MPI applications based on coordinated checkpoint, in: ICPADS, 2005, pp. 306312.
12. S. Chakravorty, C. Mendes, L. Kale, Proactive fault tolerance in large systems, in: HPCRI: 1st Workshop on High Performance Computing Reliability Issues, in: Proceedings of HPCA-11, 2005.
13. S. Chakravorty, C. Mendes, L. Kale, Proactive fault tolerance in MPI applications via task migration, in: HiPC, 2006.
14. S. Chakravorty, C. Mendes, L. Kale, A fault tolerance protocol with fast fault recovery, in: IPDPS, 2007.
15. C. Clark, K. Fraser, S. Hand, J. Hansem, E. Jul, C. Limpach, I. Pratt, A. Warfield, Live migration of virtual machines, in: NSDI, 2005.
16. F. Douglis, and J.K. Ousterhout Transparent process migration: Design alternatives and the sprite implementation Softw. - Pract. Exp. 21 8 1991 757 785 (Pubitemid 21697317)
17. J. Duell, The design and implementation of berkeley lab's linux checkpoint/restart, Tech. rep., Lawrence Berkeley National Laboratory (2000).
18. C. Du, X.-H. Sun, MPI-Mitten: Enabling migration technology in MPI, in: IEEE CCGrid, 2006.
19. C. Du, X.-H. Sun, K. Chanchio, HPCM: A pre-compiler aided middleware for the mobility of legacy code, in: IEEE Cluster, 2003.
20. C. Du, X.-H. Sun, M. Wu, Dynamic scheduling with process migration, in: IEEE CCGrid, 2007.
21. E.N. Elnozahy, and W. Zwaenepoel Manetho: Transparent roll back-recovery with low overhead, limited rollback, and fast output commit IEEE Trans. Comput. 41 5 1992 526 531
22. G.E. Fagg, J.J. Dongarra, FT-MPI: Fault Tolerant MPI, supporting dynamic applications in a dynamic world, in: Euro PVM/MPI User's Group Meeting, vol. 1908, 2000, pp. 346353.
23. A. Faraj, X. Yuan, Communication characteristics in the nas parallel benchmarks, in: IASTED PDCS, 2002, pp. 724729.
24. G. Florez, Z. Liu, S.M. Bridges, A. Skjellum, and R.B. Vaughn Lightweight monitoring of mpi programs in real time Concurr. Comput.: Pract. Exper. 2005
25. Ganglia, http://ganglia.sourceforge.net/.
26. S. Ghemawat, H. Gobioff, S.-T. Leung, The Google file system, in: SOSP'03, 2003, pp. 2943.
27. R. Gioiosa, J.C. Sancho, S. Jiang, F. Petrini, Transparent, incremental checkpointing at kernel level: a foundation for fault tolerance for parallel computers, in: Supercomputing, 2005.
28. P. Gujrati, Y. Li, Z. Lan, R. Thakur, J. White, A meta-learning failure predictor for BlueGene/L systems, in: ICPP, 2007.
29. X. Gu, S. Papadimitriou, P.S. Yu, S.-P. Chang, Toward predictive failure management for distributed stream processing systems, in: IEEE ICDCS, 2008.
30. C.-H. Hsu, and W.-C. Feng A power-aware run-time system for high-performance computing Supercomputing 2005
31. htop, http://htop.sourceforge.net/.
32. C. Huang, G. Zheng, L. Kalé, S. Kumar, Performance evaluation of adaptive mpi, in: ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, 2006, pp. 1221.
33. M.T. Huda, H.W. Schmidt, I.D. Peake, An agent oriented proactive fault-tolerant framework for grid computing, in: International Conference on e-Science and Grid Computing, 2005.
34. J. Hursey, J.M. Squyres, A. Lumsdaine, A checkpoint and restart service specification for Open MPI, Technical report, Indiana University, Computer Science Department (2006).
35. J. Hursey, J.M. Squyres, T.I. Mattox, A. Lumsdaine, The design and implementation of checkpoint/restart process fault tolerance for Open MPI, in: DPDNS, 2007.
36. E. Jul, H.M. Levy, N.C. Hutchinson, and A.P. Black Fine-grained mobility in the emerald system ACM Trans. Comput. Syst. 6 1 1988 109 133
37. O.R.N. Laboratory, Resources - national center for computational sciences (nccs), Jun. 2007. http://info.nccs.gov/resources/jaguar.
38. Z. Lan, and Y. Li Adaptive fault management of parallel applications for high-performance computing IEEE Trans. Comput. 57 2008 1647 1660
39. T. LeBlanc, R. An, E. Gabriel, J. Subhlok, Volpexmpi: an mpi library for execution of parallel applications on volatile nodes, in: European PVM/MPI Users' Group Meeting, 2009, pp. 124133.
40. Y. Li, P. Gujrati, Z. Lan, X.-H. Sun, Fault-driven re-scheduling for improving system-level fault resilience, in: ICPP, 2007.
41. Y. Li, Z. Lan, P. Gujrati, and X.-H. Sun Fault-aware runtime strategies for high-performance computing IEEE Trans. Parallel Distrib. Syst. 20 2009 460 473
42. M. Litzkow, Remote unix - turning idle workstations into cycle servers, in: Usenix Summer Conference, 1987, pp. 381384.
43. M. Litzkow, T. Tannenbaum, J. Basney, M. Livny, Checkpoint and migration of UNIX processes in the Condor distributed processing system, Tech. Rep. UW-CS-TR-1346, University of Wisconsin - Madison Computer Sciences Department, April 1997.
44. J. Liu, W. Huang, B. Abali, D. Panda, High performance vmm-bypass I/O in virtual machines, in: USENIX Conference, 2006.
45. J. Mehnert-Spahn, E. Feller, M. Schoettner, Incremental checkpointing for grids, in: Linux Symposium, 2009.
46. A. Menon, A. Cox, W. Zwaenepoel, Optimizing network virtualization in Xen, in: USENIX Conference, 2006.
47. D.S. Milojicic, F. Douglis, Y. Paindaveine, R. Wheeler, and S. Zhou Process migration ACM Comput. Surv. (CSUR) 32 3 2000 241 299
48. mpip: Lightweight, scalable mpi profiling, http://mpip.sourceforge.net/.
49. A.B. Nagarajan, F. Mueller, Proactive fault tolerance for HPC with Xen virtualization, Tech. Rep. TR 2007-1, Dept. of Computer Science, North Carolina State University (2007).
50. A.B. Nagarajan, F. Mueller, Proactive fault tolerance for HPC with Xen virtualization, in: ICS, 2007.
51. A. Oliner, R. Sahoo, J. Moreira, M. Gupta, A. Sivasubramaniam, Fault-aware job scheduling for BlueGene/L systems, in: IPDPS, 2004.
52. X. Ouyang, S. Marcarelli, R. Rajachandrasekar, D. Panda, Rdma-based job migration framework for mpi over infiniband, in: Cluster, 2010.
53. Performance application programming interface, http://icl.cs.utk.edu/ papi/.
54. I. Philp Software failures and the road to a petaflop machine HPCRI: 1st Workshop on High Performance Computing Reliability Issues Proceedings of HPCA-11 2005 IEEE Computer Society
55. M. Pietro, P. Russek, K. Wiatr, Loop Profiling Tool For HPC Code Inspection as An Efficient Method of FPGA Based Acceleration, Int. J. Appl. Math. Comput. Sci., 1010.
56. J.S. Plank, M. Beck, G. Kingsley, K. Li, Libckpt: Transparent checkpointing under Unix, in: Usenix Winter Technical Conference, 1995, pp. 213223.
57. M.L. Powell, B.P. Miller, Process migration in DEMOS/MP, in: Symposium on Operating Systems Principles, 1983.
58. H. Quinn, P. Graham, Terrestrial-based radiation upsets: A cautionary tale, in: FCCM 05, 2005.
59. S. Rani, C. Leangsuksun, A. Tikotekar, V. Rampure, S. Scott, Toward efficient failure detection and recovery in HPC, in: High Availability and Performance Computing Workshop, 2006.
60. Readable dirty-bits for IA64 linux, https://www.gelato.unsw.edu.au/ archives/gelato-technical/2005-November/001080.html.
61. S.H. Russ, R. Jean-Baptiste, T.S. Kumar, and M. Harmon Transparent real-time monitoring in mpi Springer 1999
62. R. Sahoo, A. Oliner, I. Rish, M. Gupta, J. Moreira, S. Ma, R. Vilalta, A. Sivasubramaniam, Critical event prediction for proactive management in large-scale computer clusters, in: KDD '03, 2003.
63. S. Sankaran, J.M. Squyres, B. Barrett, A. Lumsdaine, J. Duell, P. Hargrove, E. Roman, The LAM/MPI checkpoint/restart framework: System-initiated checkpointing, in: LACSI, 2003.
64. B. Schroeder, E. Pinheiro, W.-D. Weber, Dram errors in the wild: a large-scale field study, SIGMETRICS'09: Proceedings of the eleventh international joint conference on Measurement and modeling of computer systems.
65. H. Song, C. Leangsuksun, R. Nassar, Availability modeling and analysis on high performance cluster computing systems., in: ARES, 2006, pp. 305313.
66. J.M. Squyres, and A. Lumsdaine A Component Architecture for LAM/MPI Lecture Notes in Computer Science vol. 2840 2003 Springer-Verlag Venice, Italy 379 387
67. G. Stellner, CoCheck: checkpointing and process migration for MPI, in: Proceedings of IPPS '96, 1996.
68. X.-H. Sun, Z. Lan, Y. Li, H. Jin, Z. Zheng, Towards a fault-aware computing environment, in: HAPCW, 2008.
69. M. Theimer, K.A. Lantz, D.R. Cheriton, Preemptable remote execution facilities for the V-System., in: SOSP, 1985, pp. 212.
70. A. Tikotekar, C. Leangsuksun, S.L. Scott, On the survivability of standard MPI applications, in: LCI International Conference on Linux Clusters: The HPC Revolution, 2006.
71. A. Tikotekar, G. Vallée, T. Naughton, S.L. Scott, C. Leangsuksun, Evaluation of fault-tolerant policies using simulation, in: IEEE Cluster, 2007.
72. V. Tipparaju, M. Krishnan, B. Palmer, F. Petrini, J. Nieplocha, Towards fault resilient global arrays, in: Parallel computing: architectures, algorithms, and applications, 2008.
73. Top500 supercomputer sites, http://www.top500.org/.
74. G. Vallée, K. Charoenpornwattana, C. Engelmann, A. Tikotekar, C.B. Leangsuksun, T. Naughton, S.L. Scott, A framework for proactive fault tolerance, in: ARES, 2007, pp. 659664.
75. G. Vallee, R. Lottiaux, D. Margery, C. Morin, J.-Y. Berthou, Ghost process: a sound basis to implement process duplication, migration and checkpoint/restart in linux clusters, in: ISPDC, 2005.
76. J. Varma, C. Wang, F. Mueller, C. Engelmann, S.L. Scott, Scalable, fault-tolerant membership for MPI tasks on hpc systems, in: International Conference on Supercomputing, 2006, pp. 219228.
77. M. Vasavada, Innovative schemes to support incremental checkpointing, Master's thesis, Dept. of CS, North Carolina State University, Aug. 2010.
78. C. Wang, F. Mueller, C. Engelmann, S. Scott, A job pause service under LAM/MPI+BLCR for transparent fault tolerance, in: IPDPS, 2007.
79. F. Wong, R. Martin, R. Arpaci-Dusseau, D. Culler, Architectural requirements and scalability of the NAS parallel benchmarks, in: Supercomputing, 1999.
80. J.W. Young A first order approximation to the optimum checkpoint interval Commun. ACM 17 9 1974 530 531 10.1145/361147.361115