Timely offloading of result-data in HPC centers

Proceedings of the International Conference on Supercomputing

Volumn , Issue , 2008, Pages 124-133

  Monti, Henry M ^a   Butt, Ali R ^a   Vazhkudai, Sudharshan S ^b  

^a Virginia Polytechnic Institute and State University   (United States)

^b OAK RIDGE NATIONAL LABORATORY   (United States)

Author keywords

End user data delivery; High performance data management; HPC center serviceability; Offloading

Indexed keywords

COMPUTERS; DATA TRANSFER; HIGH PERFORMANCE LIQUID CHROMATOGRAPHY; INTELLIGENT CONTROL; MANAGEMENT INFORMATION SYSTEMS;

BIT TORRENTS; CHANGING DYNAMICS; DATA PATHS; DATA STAGINGS; DATASET SIZES; DELIVERY DEADLINES; END-USER DATA DELIVERY; EXPONENTIAL GROWTHS; FAULT-TOLERANT; FLOW PATHS; HIGH PERFORMANCE COMPUTING; HIGH PERFORMANCE DATA MANAGEMENT; HPC CENTER SERVICEABILITY; INTERMEDIATE NODES; LANDMARK NODES; MULTIPLE DATUMS; OFFLOADING; PRODUCTION JOBS; RESOURCE CONSUMPTIONS; SERVICE ABILITIES; SERVICE LEVEL AGREEMENTS; SUPERCOMPUTING CENTERS; USER DATUMS;

DATA HANDLING;

EID: 57349191500     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1375527.1375548     Document Type: Conference Paper

References (32)

1. UC/ANL Teragrid Guide, 2004. http://www.uc. teragrid.org/tg-docs/user- guide.html\#disk.
2. NCCS. GOV File Systems, 2007. http://info.nccs. gov/computing-resources/ jaguar/file-systems.
3. User Support and Assistance, 2007. http://www.nccs.gov/user-support/.
4. S. Annapureddy, M. J. Freedman, and D. Mazires. Shark: Scaling file servers via cooperative caching. In Proc. 2nd USENIX NSDI, 2005.
5. A. Bayucan, R. L. Henderson, C. Lesiak, B. Mann, T. Proett, and D. Tweten. Portable Batch System: External reference specification, November 1999. http://www-unix.mcs.anl.gov/openpbs/docs/v2-2-ers.pdf.
6. J. Bester, I. Foster, C. Kesselman, J. Tedesco, and S. Tuecke. GASS: A data movement and access service for wide area computing systems. In Proc. 6th Workshop on I/O in Parallel and Distributed Systems, 1999.
7. M. Christie and S. Marru. The lead portal: a teragrid gateway and application service architecture: Research articles. Concurrency and Computation: Practice and Experience, 19(6):767-781, 2007.
8. J. W. Cobb, A. Geist, J. A. Kohl, S. D. Miller, P. F. Peterson, G. G. Pike, M. A. Renter, T. Swain, S. S. Vazhkudai, and N. N. Vijayakumar. The neutron science teragrid gateway: a teragrid science gateway to support the spallation neutron source: Research articles. Concurrency and Computation: Practice and Experience., 19(6):809-826, 2007.
9. B. Cohen. BitTorrent Protocol Specification, May 2007. http://www.bittorrent.org/protocol.html.
10. R. L. Collins and J. S. Plank. Downloading replicated, wide-area files - a framework and empirical evaluation. In Proc. 3rd IEEE International Symposium on Network Computing, 2004.
11. Department of Energ, Office of Science. Innovative and Novel Computational Impact on Theory and Experiment (INCITE), January 2008. http://www.er.doe.gov/ascr/incite/.
12. Druschel et. al. Freepastry, 2004 http://freepastry.rice.edu/.
13. M. Gleicher. HSI: Hierarchical storage interface for HPSS. http://www.hpss-collaboration.org/hpss/HSI/.
14. S. Kiswany, M. Ripeanu, A. Iamnitchi, and S. Vazhkudai. Are peer-to-peer data dissemination techniques viable in todays data intensive scientific collaborations? In Proc. 13th International Euro-Par Conference: European Conference on Parallel and Distributed Computing, 2007.
15. D. Kostic, A. Rodriguez, J. Albrecht, A. Bhirud, and A. M. Validat. Using random subsets to build scalable network services. In Proc. 4th USENIX USITS, 2003.
16. D. Kostic, A. Rodriguez, J. Albrecht, and A. Vahdat. Bullet: High bandwidth data dissemination using an overlay mesh. In Proc. 19th ACM SOSP, 2003.
17. P. Maymounkov. Online Codes. Technical Report TR2003-883, New York University, New York, Nov. 2002.
18. K. Park and V. S. Pai. Scale and performance in the CoBlitz large-file distribution service. In Proc. 3rd USENIX NSDI, 2006.
19. L. Peterson, T. Anderson, D. Culler, and T. Roscoe. A Blueprint for Introducing Disruptive Technology into the Internet. In Proc. 1st ACM Workshop on Hot Topics in Networks (HotNets-I), 2002.
20. J. Plank, M. Beck, W. Elwasif, T. Moore, M. Swany, and R. Wolski. The Internet Backplane Protocol: Storage in the network. In Proc. Network Storage Symposium, 1999.
21. J. S. Plank. A tutorial on Reed-Solomon coding for fault-tolerance in RAID-like systems. Software - Practice & Experience, 27(9):995-1012, 1997.
22. J. S. Plank. Erasure codes for storage applications, 2005. Tutorial Slides, presented at USENIX FAST, http://www.cs.utk.edu/~plank/plank/ papers/FAST-2005.html.
23. J. S. Plank, S. Atchley, Y. Ding, and M. Beck. Algorithms for high performance, wide-area distributed file downloads. Parallel Processing Letters, 13(2):207-224, 2003.
24. P. Rodriguez, A. Kirpal, and E. W. Biersack. Parallel-access for mirror sites in the internet. In Proc. IEEE Infocom, 2000.
25. A. Rowstron and P. Druschel. Pastry: Scalable, distributed object location and routing for large-scale peer-to-peer systems. In Proc. IFIP/ACM Middleware, 2001.
26. D. Thain, S. S. J. Basney, and M. Livny. The kangaroo approach to data movement on the grid. In Proc. 10th IEEE Symposium on High Performance Distributed Computing (HPDC10), 2001.
27. S. Vazhkudai and J. Schopf, Predicting sporadic grid data transfers. In Proc. 11th IEEE Int'l Symposium on High Performance Distributed Computing (HPDC-11), 2002.
28. S. Vazhkudai, J. Schopf, and I. Foster. Predicting the performance of wide-area data transfers. In Proc. 16th Int'l Parallel and Distributed Processing Symposium (IPDPS 2002), 2002.
29. S. Vazhkudai, S. Tuecke, and I. Foster. Replica selection in the globus data grid. In Proc. IEEE International Conference on Cluster Computing and the Grid (CCGRID 2001), 2001.
30. L. Wang, K. Park, R. Pang, V. Pai, and L. Peterson. Reliability and security in the CoDeeN content distribution network. In Proc. USENIX ATC, 2004.
31. R. Wolski, N. Spring, and J. Hayes. The Network Weather Service: A distributed resource performance forecasting service for metacomputing. Future Generation Computing Systems, 15(5):757-768, 1999.
32. Z. Zhang, C. Wang, S. S. Vazhkudai, X. Ma, G. Pike, J. Cobb, and F. Mueller. Optimizing center performance through coordinated data staging, scheduling and recovery. In Proc. Supercomputing 2007 (SC07): Int'l Conference on High Performance Computing, Networking, Storage and Analysis, 2007.