Just in time: Adding value to the IO pipelines of high performance applications with JITStaging

Proceedings of the IEEE International Symposium on High Performance Distributed Computing

Volumn , Issue , 2011, Pages 27-36

  Abbasi, Hasan ^a   Eisenhauer, Greg ^a   Wolf, Matthew ^a   Schwan, Karsten ^a   Klasky, Scott ^b  

^a Georgia Institute of Technology   (United States)

^b OAK RIDGE NATIONAL LABORATORY   (United States)

Author keywords

datatap; gtc; i o; infiniband; staging; warp; xt3; xt4

Indexed keywords

DATATAP; GTC; INFINIBAND; STAGING; WARP; XT3; XT4;

PIPELINE CODES; PIPELINES; VISUALIZATION;

DATA VISUALIZATION;

EID: 79960533277     PISSN: 10828907     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1996130.1996137     Document Type: Conference Paper

References (40)

1. ABBASI, H., WOLF, M., AND SCHWAN, K. LIVE Data Workspace: A Flexible, Dynamic and Extensible Platform for Petascale Applications. Cluster Computing, 2007. IEEE International (Sept. 2007).
2. ABBASI, H., WOLF, M., SCHWAN, K., EISENHAUER, G., AND HILTON, A. Xchange: coupling parallel applications in a dynamic environment. In CLUSTER (2004), IEEE Computer Society, pp. 471-480.
3. ABBASI, H., WOLF, M., ZHENG, F., EISENHAUER, G., KLASKY, S., AND SCHWAN, K. Scalable data staging services for petascale applications. In hpdc2009 (2009).
4. ALI, N., CARNS, P. H., ISKRA, K., KIMPE, D., LANG, S., LATHAM, R., ROSS, R. B., WARD, L., AND SADAYAPPAN, P. Scalable i/o forwarding framework for high-performance computing systems. In CLUSTER (2009), IEEE, pp. 1-10.
5. BUSTAMANTE, F. E., EISENHAUER, G., SCHWAN, K., AND WIDENER, P. Efficient wire formats for high performance computing. In Proceedings of Supercomputing '00 (SC2000) (November 4-10 2000).
6. Lustre: A scalable, high-performance file system. Cluster File Systems Inc. white paper, version 1.0, November 2002. http://www.lustre.org/docs/ whitepaper.pdf.
7. CHACÓN, L. A non-staggered, conservative, ∇ṡB →= 0, finite-volume scheme for 3D implicit extended magnetohydrodynamics in curvilinear geometries. Computer Physics Communications 163 (Nov. 2004), 143-171.
8. CHANG, C., KLASKY, S., CUMMINGS, J., SAMTANEY, R., SHOSHANI, A., SUGIYAMA, L., KEYES, D., KU, S., PARK, G., PARKER, S., ET AL. Toward a first-principles integrated simulation of tokamak edge plasmas. In Journal of Physics: Conference Series (2008), vol. 125, Institute of Physics Publishing, p. 012042.
9. CHANG, C. S., AND KU, S. Spontaneous Rotation Sources in a Quiescent Tokamak Edge Plasma. Physics of Plasmas 15, 6 (2008).
10. COMMITTEE, T. M.-I. Mpi-io: A parallel file i/o interface for mpi, April 1996. Version 0.5. MPI-2 standard is at http://www.mcs.anl.gov/mpi.
11. DEAN, J., AND GHEMAWAT, S. Mapreduce: simplified data processing on large clusters. In OSDI'04: Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation (Berkeley, CA, USA, 2004), USENIX Association, pp. 10-10.
12. DOCAN, C., PARASHAR, M., CUMMINGS, J., PODHORSZKI, N., AND KLASKY, S. Experiments with Memory-to-Memory Coupling for End-to-End Fusion Simulation Workflows. Tech. Rep. TR-104, Center for Autonomic Computing (CAC), Rutgers University, Piscataway, NJ, USA, July 2009.
13. DOCAN, C., PARASHAR, M., AND KLASKY, S. Dataspaces: An interaction and coordination framework for coupled simulation workflows. In Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing (HPDC 2010) (June 2010).
14. EKANAYAKE, J., PALLICKARA, S., AND FOX, G. Mapreduce for data intensive scientific analyses. In ESCIENCE '08: Proceedings of the 2008 Fourth IEEE International Conference on eScience (Washington, DC, USA, 2008), IEEE Computer Society, pp. 277-284.
15. GANEV, I., EISENHAUER, G., AND SCHWAN, K. Kernel Plugins: When a VM is Too Much. In Proceedings of the 3rd Virtual Machine Research and Technology Symposium (VM'04) (San Jose, CA, May 2004).
16. GUPTA, V., GAVRILOVSKA, A., SCHWAN, K., KHARCHE, H., TOLIA, N., TALWAR, V., AND RANGANATHAN, P. Gvim: Gpu-accelerated virtual machines. In HPCVirt '09: Proceedings of the 3rd ACM Workshop on System-level Virtualization for High Performance Computing (New York, NY, USA, 2009), ACM, pp. 17-24.
17. INTEL. Virtualization technology (vt). http://www.intel.com/technology/ computing/vptech/.
18. ISKRA, K., ROMEIN, J., YOSHII, K., AND BECKMAN, P. ZOID: I/O-forwarding infrastructure for petascale architectures. In Proceedings of the 13th ACM SIGPLAN Symposium on Principles and practice of parallel programming (2008), ACM, pp. 153-162.
19. JIANG, N., QUIROZ, A., SCHMIDT, C., AND PARASHAR, M. Meteor: a middleware infrastructure for content-based decoupled interactions in pervasive grid environments. Concurrency and Computation: Practice and Experience 20, 12 (2008), 1455-1484.
20. KENG LIAO, W., AND CHOUDHARY, A. N. Dynamically adapting file domain partitioning methods for collective i/o based on underlying parallel file system locking protocols. In Proceedings of the ACM/IEEE Conference on High Performance Computing, SC 2008, November 15-21, 2008, Austin, Texas, USA (2008), IEEE/ACM, p. 3.
21. KLASKY, S., ETHIER, S., LIN, Z., MARTINS, K., MCCUNE, D., AND SAMTANEY, R. Grid -based parallel data streaming implemented for the gyrokinetic toroidal code. In SC '03: Proceedings of the 2003 ACM/IEEE conference on Supercomputing (Washington, DC, USA, 2003), IEEE Computer Society, p. 24.
22. KUTARE, M., EISENHAUER, G., WANG, C., SCHWAN, K., TALWAR, V., AND WOLF, M. Monalytics: Online monitoring and analytics for managing large scale data centers. In The 7th IEEE/ACM International Conference on Autonomic Computing and Communications (June 2010).
23. LATHAM, R., MILLER, N., ROSS, R., AND CARNS, P. A next-generation parallel file system for linux clusters. LinuxWorld 2, 1 (January 2004).
24. LEE, J., ROSS, R. B., ATCHLEY, S., BECK, M., AND THAKUR, R. Mpi-io/l: efficient remote i/o for mpi-io via logistical networking. In IPDPS (2006), IEEE.
25. LOFSTEAD, J., ZHENG, F., KLASKY, S., AND SCHWAN, K. Adaptable, metadata rich io methods for portable high performance io. In In Proceedings of IPDPS'09, May 25-29, Rome, Italy (2009).
26. LOFSTEAD, J., ZHENG, F., KLASKY, S., AND SCHWAN, K. Adaptable, metadata rich io methods for portable high performance io. In In IPDPS'09, May 25-29, Rome, Italy (2009).
27. LOFSTEAD, J., ZHENG, F., LIU, Q., KLASKY, S., OLDFIELD, R., KORDENBROCK, T., SCHWAN, K., AND WOLF, M. Managing variability in the io performance of petascale storage systems. In SC '10: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis (New York, NY, USA, 2010), ACM.
28. LUDSCHER, B., ALTINTAS, I., BERKLEY, C., HIGGINS, D., JAEGER, E., JONES, M., LEE, E. A., TAO, J., AND ZHAO, Y. Scientific workflow management and the kepler system: Research articles. Concurr. Comput. : Pract. Exper. 18, 10 (2006), 1039-1065.
29. MILLER, B. P., CALLAGHAN, M. D., CARGILLE, J. M., HOLLINGSWORTH, J. K., IRVIN, R. B., KARAVANIC, K. L., KUNCHITHAPADAM, K., AND NEWHALL, T. The paradyn parallel performance measurement tool. IEEE Computer 28, 11 (1995), 37-46.
30. NISAR, A., KENG LIAO, W., AND CHOUDHARY, A. N. Scaling parallel i/o performance through i/o delegate and caching system. In Proceedings of the ACM/IEEE Conference on High Performance Computing, SC 2008, November 15-21, 2008, Austin, Texas, USA (2008), p. 9.
31. OLIKER, L., CARTER, J., MICHAEL WEHNER, CANNING, A., ETHIER, S., MIRIN, A., BALA, G., PARKS, D., PATRICK WORLEY SHIGEMUNE KITAWAKI, AND TSUDA, Y. Leading computational methods on scalar and vector hec platforms. In Proceedings of SuperComputing 2005 (2005).
32. PLIMPTON, S. Fast parallel algorithms for short-range molecular dynamics. Journal of Computational Physics 117, 1 (1995), 1-19. http://lammps.sandia.gov/ index.html.
33. POLTE, M., SIMSA, J., TANTISIRIROJ, W., AND GIBSON, G. Fast log-based concurrent writing of checkpoints. In In Proceedings of Petascale Data Storage Workshop 2008 at Supercomputing 2008 (2008).
34. ROTH, P., ARNOLD, D., AND MILLER, B. Mrnet: A software-based multicast/reduction network for scalable tools. In Proceedings of the 2003 ACM/IEEE conference on Supercomputing (2003), IEEE Computer Society, p. 21.
35. SCHMUCK, F., AND HASKIN, R. GPFS: A Shared-Disk File System for Large Computing Clusters. Proceedings of the 1st USENIX Conference on File and Storage Technologies (2002).
36. TITZER, B. L., WÜRTHINGER, T., SIMON, D., AND CINTRA, M. Improving compiler-runtime separation with xir. In VEE '10: Proceedings of the 6th ACM SIGPLAN/SIGOPS international conference on Virtual execution environments (New York, NY, USA, 2010), ACM, pp. 39-50.
37. TU, T., RENDLEMAN, C. A., MILLER, P. J., SACERDOTI, F., DROR, R. O., AND SHAW, D. E. Accelerating parallel analysis of scientific simulation data via zazen. In Proceedings of the 8th USENIX Conference on File and Storage Technologies (FAST '10) (February 23-26 2010). San Jose, California.
38. WIDENER, P. M., WOLF, M., ABBASI, H., MCMANUS, S., PAYNE, M., BRIDGES, P. G., AND SCHWAN, K. Exploiting latent i/o asynchrony in petascale science applications. In In Proceedings of the 2nd International Workshop on Parallel Programming Models and Systems Software for High-End Computing (P2S2) (September 2009).
39. YU, W., VETTER, J. S., AND ORAL, S. Performance characterization and optimization of parallel i/o on the cray xt. In IPDPS (2008), IEEE, pp. 1-11.
40. ZHENG, F., ABBASI, H., DOCAN, C., LOFSTEAD, J., LIU, Q., KLASKY, S., PARASHAR, M., PODHORSZKI, N., SCHWAN, K., AND WOLF, M. Predata - preparatory data analytics on peta-scale machines. In Proceedings of the 24th IEEE International Parallel and Distributed Processing Symposium (IPDPS 2010) (April 2010).