Accelerating parallel analysis of scientific simulation data via ZazEn

Proceedings of FAST 2010: 8th USENIX Conference on File and Storage Technologies

Volumn , Issue , 2010, Pages 129-142

  Tu, Tiankai ^a   Rendleman, Charles A ^a   Miller, Patrick J ^a   Sacerdoti, Federico ^a   Dror, Ron O ^a   Shaw, David E ^a,b  

^a D E SHAW RESEARCH   (United States)

^b Och Spine at New York Presbyterian Hospitals   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BANDWIDTH; COMPUTER OPERATING SYSTEMS; METADATA; MOLECULAR DYNAMICS; SUPERCOMPUTERS;

MASSIVE DATA SETS; METADATA SERVERS; MOLECULAR DYNAMICS SIMULATIONS; PARALLEL ANALYSIS; PARALLEL DISKS; PARALLEL SUPERCOMPUTER; SCIENTIFIC SIMULATIONS; SIMULATION OUTPUTS;

DIGITAL STORAGE;

EID: 80955127570     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (49)

1. R. Ananthanarayanan, M. Eshel, R. Haskin, M. Naik, F. Schmuck, and R. Tewari. Panache: a parallel WAN cache for clustered filesystems. ACM SIGOPS Operating Systems Review, 42(1):48-53, January 2008.
2. H. Balakrishnan, M. F. Kaashoek, D. Karger, R. Morris, and I. Stoica. Looking up data in P2P systems. Communications of the ACM, 46(2):43-48, February 2003.
3. G. Bell, T. Hey, and A. Szaley. Beyond the data deluge. Science, 323(5919):1297-1298, March 2009.
4. J. Bent, G. Gibson, G. Grider, B. McClelland, P. Nowoczynski, et al. PLFS: a checkpoint filesystem for parallel applications. In Proceedings of the 2009 ACM/IEEE Conference on Supercomputing (SC09), Portland, OR, November 2009.
5. J. Bent, D. Thain, A. C. Arpaci-Dusseau, R. H. Arpaci-Dusseau, and M. Livny. Explicit control in a batch-aware distributed file system. In Proceedings of the 1st USENIX Symposium on Networked Systems Design and Implementation (NSDI'04), San Francisco, CA, March 2004.
6. J. Bruck, C-T. Ho, S. Kipnis, E. Upfal, and D. Weathersby. Efficient algorithms for all-to-all communications in multiport message-passing systems. IEEE Transactions on Parallel and Distributed Systems, 8(11):1143-1156, November 1997.
7. A. R. Butt, T. A. Johnson, Y. Zheng, and Y. C. Yu. Kosha: a peer-to-peer enhancement for the network file system. In Proceedings of the 2004 ACM/IEEE Conference on Supercomputing (SC04), Pittsburgh, PA, November 2004.
8. P. H. Carns, W. B. Ligon III, R. B. Ross, and R. Thakur. PVFS: a parallel file system for Linux clusters. In Proceedings of the 4th Annual Linux Showcase and Conference, pages 317-327, Atlanta, GA, October 2000.
9. F. Dabek, M. F. Kaashoek, D. Karger, R. Morris, and I. Stoica. Wide-area cooperative storage with CFS. In Proceedings of the 18th ACM Symposium on Operating Systems Principles (SOSP'01), pages 202-215, Banff, Alberta, Canada, October 2001.
10. J. Dean and S. Ghemawat. MapReduce: simplified data processing on large clusters. Communications of the ACM, 51(1):107-113, January 2008.
11. G. E. Fagg, J. Pjesivac-Grbovic, G. Bosilca, T. Angskun, J. J. Dongarra, and E. Jeannot. Flexible collective communication tuning architecture applied to Open MPI. In Proceedings of the 13th European PVM/MPI Users' Group Meeting (Euro PVM/MPI 2006), Bonn, Germany, September 2006.
12. I. Foster, D. Kohr, R. Krishnaiyer, and J. Mogill. Remote I/O: fast access to distant storage. In Proceedings of the 5th Workshop on Input/Output in Parallel and Distributed Systems, pages 14-25, San Jose, CA, November 1997.
13. S. Ghemawat, H. Gobioff, and S-T. Leung. The Google file system. In Proceedings of the 19th ACM Symposium on Operating Systems Principles (SOSP'03), Bolton Landing, NY, October 2003.
14. GridFTP. http://www.globus.org/grid_software/data/gridftp.php/.
15. Hadoop. http://hadoop.apache.org/.
16. Hadoop/HDFS small files problem. http://www.cloudera.com/blog/2009/02/02/the-small-files-problem/.
17. J. H. Howard, M. L. Kazar, S. G. Menees, D. A. Nichols, M. Satyanarayanan, et al. Scale and performance in a distributed file system. ACM Transactions on Computer Systems, 6(1):51-81, February 1988.
18. M. A. Kozuch, M. P. Ryan, R. Gass, S. W. Schlosser, D. R. O'Hallaron, et al. Tashi: location-aware cluster management. In Proceedings of the 1st Workshop on Automated Control for Datacenters and Clouds (ACDC09), Barcelona, Spain, June 2009.
19. T. Kurc, Ü Çatalyürek, C. Chang, A. Sussman, and J. Saltz. Visualization of Large Data Sets with the Active Data Repository. IEEE Computer Graphics and Applications, 21(4):24-33, July/August 2001.
20. J. Lofstead, S. Klasky, K. Schwan, N. Podhorszki, and C. Jin. Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS). In Proceedings of the 6th ACM/IEEE International Workshop on Challenges of Large Applications in Distributed Environments (CLADE.2008), Boston, MA, June 2008.
21. Lustre. http://www.sun.com/software/products/lustre/.
22. H. M. Monti, A. R. Butt, and S. S. Vazhkudai. Just-in-time staging of large input data for supercomputing jobs. In Proceedings of the 3rd Petascale Data Storage Workshop, Austin, TX, November 2008.
23. H. M. Monti, A. R. Butt, and S. S. Vazhkudai. /scratch as a cache: rethinking HPC center scratch storage. In Proceedings of the 23rd International Conference on Supercomputing, Yorktown Height, NY, June 2009.
24. A. Muthitacharoen, R. Morris, T. M. Gil, and B. Chen. Ivy: a read/write peer-to-peer file system. In Proceedings of the 5th Symposium on Operating Systems Design and Implementation (OSDI'02), Boston, MA, November 2002.
25. P. Nowoczynski, N. Stone, J. Yanovich, and J. Sommerfield. Zest: checkpoint storage system for large supercomputers. In Proceedings of the 3rd Petascale Data Storage Workshop, Austin, TX, November 2008.
26. Open MPI. http://www.open-mpi.org/.
27. D. S. Parker, G. J. Popek, G. Rudisin, A. Stoughton, B. J. Walker, et al. Detection of mutual inconsistency in distributed systems. IEEE Trascations on Software Engineering, 9(3):240-247, May 1983.
28. Petascale Data Storage Institute. http://www.pdsiscidac.org/.
29. Parallel NFS. http://www.pnfs.com/.
30. D. H. Porter, P. R. Woodward, and A. Iyer. Initial experiences with grid-based volume visualization of fluid flow simulations on PC clusters. In Proceedings of Visualization and Data Analysis 2005 (VDA2005), San Jose, CA, January 2005.
31. PVFS. http://www.pvfs.org/.
32. S. Rhea, P. Eaton, D. Geels, H. Weatherspoon, B. Zhao, and J. Kubiatowicz. Pond: the OceanStore prototype. In Proceedings of the 2nd USENIX Conference on File and Storage Technologies (FAST'03), San Francisco, CA, March 2003.
33. ROMIO. http://www.mcs.anl.gov/research/projects/romio/
34. A. Rowstron and P. Druschel. Storage management and caching in PAST, a large-scale, persistent peer-to-peer storage utility. In Proceedings of the 18th ACM Symposium on Operating Systems Principles (SOSP'01), Banff, Alberta, Canada, November 2001.
35. M. Satyanarayanan, J. H. Howard, D. A. Nichols, R. N. Sidebotham, A. Z. Spector, and M. J.West. The ITC distributed file system: principles and design. In Proceedings of the 10th ACM symposium on Operating Systems Principles, Orcas Island, WA, 1985.
36. M. Satyanarayanan, J. J. Kistler, P. Kumar, M. E. Okasaki, E. H. Siegel, and D. C. Steere. Coda: a highly available file system for a distributed workstation environment. IEEE Transactions on Computers, 39(4):447-459, April 1990.
37. F. Schmuck and R. Haskin. GPFS: A shared-disk file system for large computing clusters. In Proceedings of the 1st USENIX Conference on File and Storage Technologies (FAST'02), Monterey, CA, January 2002.
38. D. E. Shaw, R. O. Dror, J. K. Salmon, J. P. Grossman, K. M. Mackenzie, et al. Millisecond-scale molecular dynamics simulation on Anton. In Proceedings of the 2009 ACM/IEEE Conference on Supercomputing (SC09), Portland, OR, November 2009.
39. SLURM. https://computing.llnl.gov/linux/slurm/slurm.html/.
40. N. T. B. Stone, D. Balog, B. Gill, B. Johanson, J. Marsteller, et al. PDIO: high-performance remote file I/O for Portals enabled compute nodes. In Proceedings of the 2006 Conference on Parallel and Distributed Processing Techniques and Applications, Las Vegas, NV, June 2006.
41. O. Tatebe, Y. Morita, S. Matsuoka, N. Soda, and S. Sekiguchi. Grid datafarm architecture for petascale data intensive computing. In Proceedings of the 2nd IEEE/ACM Internaiontal Symposium on Cluster Computing and the Grid (CCGrid2002), Berlin, Germany, May 2002.
42. O. Tatebe, N. Soda, Y. Morita, S. Matsuoka, and S. Sekiguchi. Gfarm v2: A grid file system that supports high-performance distributed and parallel data computing. In Proceedings of the 2004 Computing in High Energy and Nuclear Physics, Interlaken, Switzerland, September 2004.
43. R. Thakur, W. Gropp, and E. Lusk. An abstract-device interface for implementing portable parallel-I/O interfaces. In Proceedings of the 6th Symposium on the Frontiers of Massively Parallel Computation, Annapolis, MD, October 1996.
44. R. Thakur, R. Rabenseifner, and W. Gropp. Optimization of collective communication operations in MPICH. International Journal of High Performance Computing Applications, 19(1):49-66, 2005.
45. B. L. Tierney, J. Lee, B. Crowley, M. Holding, J. Hylton, and F. L. Drake Jr. A network-aware distributed storage cache for data intensive environments. In Proceedings of the 8th IEEE International Symposium on High Performance Distributed Computing (HPDC-8), Redondo Beach, CA, August 1999.
46. T. Tu, C. A. Rendleman, D. W. Borhani, R. O. Dror, J. Gullingsrud, et al. A Scalable Parallel Framework for Analyzing Terascale Molecular Dynamics Simulation Trajectories. In Proceedings of the ACM/IEEE Conference on Supercomputing (SC08), Austin, Texas, November 15-21, 2008.
47. S. S. Vazhkudai, X. Ma, V.W. Freeh, J.W. Strickland, N. Tammineedi, and S. L. Scott. FreeLoader: scavenging desktop storage resources for scientific data. In Proceedings of the 2005 ACM/IEEE Conference on Supercomputing (SC05), Settle, WA, November 2005.
48. B. Walker, G. Popek, R. English, C. Kline, and G. Thiel. The LOCUS distributed operating system. In Proceedings of the 9th ACM Symposium on Operating Systems Principles, Bretton Woods, MA, October 1983.
49. A. Yoo, M. Jette, and M. Grondona. SLURM: simple Linux utility for resource management. In Lecture Notes in Computer Science, volume 2862 of Job Scheduling Strategies for Parallel Processing, pages 44-60. Springer Berlin/Heidelberg, 2003.