Reconciling scratch space consumption, exposure, and volatility to achieve timely staging of job input data

Proceedings of the 2010 IEEE International Symposium on Parallel and Distributed Processing, IPDPS 2010

Volumn , Issue , 2010, Pages

  Monti, Henry M ^a   Butt, Ali R ^a   Vazhkudai, Sudharshan S ^b  

^a Virginia Polytechnic Institute and State University   (United States)

^b OAK RIDGE NATIONAL LABORATORY   (United States)

Author keywords

Data staging; End user data delivery; High performance data management; HPC center serviceability

Indexed keywords

CURRENT PRACTICES; DATA DELIVERY; DATA SOURCE; DIRECT TRANSFER; END USERS; HIGH-END COMPUTING; HIGH-SPEED; INPUT DATAS; INTERMEDIATE NODE; JOB THROUGHPUT; PERFORMANCE DATA; PLANETLAB; SCIENTIFIC APPLICATIONS; SPACE CONSUMPTION; START-UPS; STARTUP TIME;

DISTRIBUTED PARAMETER NETWORKS; SUPERCOMPUTERS;

INPUT OUTPUT PROGRAMS;

EID: 77953972798     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/IPDPS.2010.5470367     Document Type: Conference Paper

References (49)

1. LHC-the large hadron collider. http://lhc.web.cern.ch/lhc/, 2009
2. Spallation Neutron Source. http://www.sns.gov/, 2008.
3. Sloan digital sky survey. http://www.sdss.org, 2005.
4. Laser Interferometer Gravitational-Wave Observatory. http://www.ligo. caltech.edu/, 2008.
5. R.A. Coyne and R.W. Watson. The parallel i/o architecture of the high-performance storage system (hpss). In Proc. IEEE MSS Symposium, 1995.
6. National center for biotechnology information. http://www.ncbi.nlm.nih. gov/, 2009.
7. National Center for Computational Sciences. http://www.nccs.gov/, 2008.
8. Bianca Schroeder and Garth A. Gibson. Disk failures in the real world: what does an mttf of 1,000,000 hours mean to you? In Proc. USENIX FAST, 2007.
9. Eduardo Pinheiro, Wolf-Dietrich Weber, and Luiz André Barroso. Failure trends in a large disk drive population. In Proc. USENIX FAST, 2007.
10. S. Shah and J.G. Elerath. Reliability analysis of disk drive failure mechanisms. In Proc. RAMS, 2005.
11. W. Smith, V.E. Taylor, and I.T. Foster. Using run-time predictions to estimate queue wait times and improve scheduler performance. In Proc. JSSPP, 1997.
12. A. Downey. Using queue time predictions for processor allocation. In Proc. JSSPP, 1997.
13. Pbs pro technical overview: Scheduling and file staging. https://secure.altair.com/sched-staging.html, 2008.
14. Cluster resources. http://www.clusterresources.com/, 2008.
15. Bram Cohen. BitTorrent Protocol Specification. http://www.bittorrent.org/ protocol.html, 2007.
16. Rich Wolski, Neil Spring, and Jim Hayes. The Network Weather Service: A distributed resource performance forecasting service for metacomputing. Future Generation Computing Systems, 15(5):757-768, 1999.
17. D. Kostic, A. Rodriguez, J. Albrecht, and A. Vahdat. Bullet: High bandwidth data dissemination using an overlay mesh. In Proc. ACM SOSP, 2003.
18. Dejan Kostic, Adolfo Rodriguez, Jeannie Albrecht, Abhijeet Bhirud, and Amin M. Vahdat. Using random subsets to build scalable network services. In Proc. 4th USENIX USITS, 2003.
19. S. Annapureddy, M. J. Freedman, and D. Mazires. Shark: Scaling file servers via cooperative caching. In Proc. 2nd USENIX NSDI, 2005.
20. T. Kosar and M. Livny. Stork: Making data placement a first class citizen in the grid. In Proc. ICDCS, 2004.
21. KyoungSoo Park and Vivek S. Pai. Scale and performance in the CoBlitz large-file distribution service. In Proc. 3rd USENIX NSDI, 2006.
22. W. Allcock, J. Bresnahan, R. Kettimuthu, and M. Link. The Globus Striped GridFTP framework and server. In Proc. of Supercomputing, 2005.
23. M. Gleicher. HSI: Hierarchical storage interface for HPSS. http://www.hpss-collaboration.org/hpss/HSI/, 2009.
24. Larry Peterson, Tom Anderson, David Culler, and Timothy Roscoe. A Blueprint for Introducing Disruptive Technology into the Internet. In Proc. ACM HotNets, 2002.
25. C. Hsu and W. Feng. A power-aware run-time system for high-performance computing. In Proc. SC, 2005.
26. Department of Energy, Office of Science. Innovative and Novel Computational Impact on Theory and Experiment (INCITE), Jan 2008. http://www.er.doe.gov/ascr/incite/.
27. TeraGrid. http://www.teragrid.org/, 2009.
28. Internet2. http://www.internet2.edu/, 2008.
29. Reddnet. http://www.reddnet.org/, 2009.
30. National Lambda Rail. http://www.nlr.net/, 2008.
31. Batch Queue Prediction. http://nws.cs.ucsb.edu/ewiki/nws.php?id= Batch+Queue+Prediction, 2008.
32. Z. Zhang, C. Wang, S. S. Vazhkudai, X. Ma, G. Pike, J. Cobb, and F. Mueller. Optimizing center performance through coordinated data staging, scheduling and recovery. In Proc. SC, 2007.
33. H. Monti, A.R. Butt, and S.S. Vazhkudai. /scratch as a cache: Rethinking hpc center scratch storage. In Proc. ICS, 2009.
34. Lakshmi N. Bairavasundaram, Garth R. Goodson, Shankar Pasupathy, and Jiri Schindler. An Analysis of Latent Sector Errors in Disk Drives. In Proc. SIGMETRICS, 2007.
35. Ilias Iliadis, Robert Haas, Xiao-Yu Hu, and Evangelos Eleftheriou. Disk scrubbing versus intra-disk redundancy for high-reliability raid storage systems. In Proc. SIGMETRICS, 2008.
36. Alma Riska and Erik Riedel. Idle read after write: Iraw. In Proc. USENIX ATC, 2008.
37. Ali R. Butt, Troy A. Johnson, Yili Zheng, and Y. Charlie Hu. Kosha: A peer-to-peer enhancement for the network file system. Journal of Grid Computing: Special issue on Global and Peer-to-Peer Computing, 4(3):323-341, 2006.
38. Druschel et. al. Freepastry. http://freepastry.rice.edu/, 2004.
39. Albeaus Bayucan, Robert L. Henderson, Casimir Lesiak, Bhroam Mann, Tom Proett, and Dave Tweten. Portable Batch System: External reference specification. November 1999. http://www-unix.mcs.anl.gov/openpbs/docs/v2-2-ers. pdf.
40. J. S. Plank. A tutorial on Reed-Solomon coding for fault- tolerance in RAID-like systems. Software - Practice & Experience, 27(9):995-1012, 1997.
41. M. J M. J. Litzkow, M Livny, and M. W Mutka. Condor - A hunter of idle workstations. In Proc. ICDCS, 1988.
42. Directed acyclic graph manager. http://www.cs.wisc.edu/condor/dagman/, 2007.
43. J. Bent, D. Thain, A. Arpaci-Dusseau, R. Arpaci-Dusseau, and M. Livny. Explicit control in a batch aware distributed file system. In Proc. NSDI, 2004.
44. D. Thain, S. Son J. Basney, and M. Livny. The kangaroo approach to data movement on the grid. In Proc. HPDC, 2001.
45. J. Plank, M. Beck, W. Elwasif, T. Moore, M. Swany, and R. Wolski. The Internet Backplane Protocol: Storage in the network. In Proc. Network Storage Symposium, 1999.
46. L. Wang, K. Park, R. Pang, V. Pai, and L. Peterson. Reliability and security in the CoDeeN content distribution network. In Proc. USENIX ATC, 2004.
47. P. Rodriguez, A. Kirpal, and E. W. Biersack. Parallel-access for mirror sites in the internet. In Proc. IEEE Infocom, 2000.
48. James S. Plank, Scott Atchley, Ying Ding, and Micah Beck. Algorithms for high performance, wide-area distributed file downloads. Parallel Processing Letters, 13(2):207-224, 2003.
49. Rebecca L. Collins and James S. Plank. Downloading replicated, wide-area files - a framework and empirical evaluation. In Proc. Symposium on Network Computing, 2004.