Adaptable, metadata rich IO methods for portable high performance IO

IPDPS 2009 - Proceedings of the 2009 IEEE International Parallel and Distributed Processing Symposium

Volumn , Issue , 2009, Pages

  Lofstead, Jay ^a   Zheng, Fang ^a   Klasky, Scott ^b   Schwan, Karsten ^a  

^a Georgia Institute of Technology   (United States)

^b OAK RIDGE NATIONAL LABORATORY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALTERNATIVE METHODS; ANALYSIS TOOLS; CODE EXECUTION; DATA ANALYSIS; DATA CHARACTERIZATION; DATA CONTENTS; DATA SETS; EFFICIENT METHOD; END USERS; FILE FORMATS; METADATA SERVERS; ON CURRENTS; ON-MACHINES; PARALLEL DATA; PERFORMANCE BASED; PERFORMANCE-DRIVEN; PETASCALE;

DISTRIBUTED PARAMETER NETWORKS; METADATA;

DATA HANDLING;

EID: 70450091356     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/IPDPS.2009.5161052     Document Type: Conference Paper

References (30)

1. HDF-5, "http://hdf.ncsa.uiuc.edu/products/hdf5/index.html."
2. J. Li, W. keng Liao, A. Choudhary, R. Ross, R. Thakur, and R. Latham, "Parallel netcdf: A scientific high-performance i/o interface," 2003.
3. P. M. Widener, M. Wolf, H. Abbasi, M. Barrick, J. Lofstead, J. Pullikottil, G. Eisenhauer, A. Gavrilovska, S. Klasky, R. Oldfield, P. G. Bridges, A. B. Maccabe, and K. Schwan, "Structured streams: Data services for petascale science environments," University of New Mexico, Albuquerque, NM, Tech. Rep. TR-CS-2007-17, November 2007. [Online]. Available: http://www.cs.unm.edu/ treport/tr/07-11/Peta-Scale.pdf
4. P. M. Widener, M. Barrick, J. Pullikottil, P. G. Bridges, and A. B. Maccabe, "Metabots: A framework for out-of-band processing in largescale data flows," in Proc. 2007 International Conference on Grid Computing (Grid 2007), Austin, Texas, September 2007.
5. B. Ludäscher, I. Altintas, C. Berkley, D. Higgins, E. Jaeger, M. Jones, E. A. Lee, J. Tao, and Y. Zhao, "Scientific workflow management and the kepler system: Research articles," Concurr. Comput. : Pract. Exper., vol. 18, no. 10, pp. 1039-1065, 2006.
6. E. Deelman, G. Singh, M.-H. Su, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, G. B. Berriman, J. Good, A. Laity, J. C. Jacob, and D. S. Katz, "Pegasus: A framework for mapping complex scientific workflows onto distributed systems," Sci. Program., vol. 13, no. 3, pp. 219-237, 2005.
7. G. Malewicz, I. Foster, A. Rosenberg, and M. Wilde, "A tool for prioritizing dagman jobs and its evaluation," High Performance Distributed Computing, 2006 15th IEEE International Symposium on, pp. 156-168, 0-0 2006.
8. R. R. Sinha and M. Winslett, "Multi-resolution bitmap indexes for scientific data," ACM Trans. Database Syst., vol. 32, no. 3, p. 16, 2007.
9. L. Gosink, J. Shalf, K. Stockinger, K. Wu, and W. Bethel, "Hdf5-fastquery: Accelerating complex queries on hdf datasets using fast bitmap indices," in In SSDBM, 2006, pp. 149-158.
10. H.-. P. Indices, "http://www.hdfgroup.uiuc.edu/rfc/hdf5/ hdf5indexing/pis.html."
11. H.-. P. Tables, "http://www.carabos.com/docs/opsi-indexes.pdf."
12. NetCDF, "http://epp.eps.nagoya-u.ac.jp/num-analysis/guidec/netcdf-c- 14.html."
13. R. W. Watson and R. A. Coyne, "The parallel i/o architecture of the high-performance storage system (hpss)," in MSS '95: Proceedings of the 14th IEEE Symposium on Mass Storage Systems. Washington, DC, USA: IEEE Computer Society, 1995, p. 27.
14. NFS, "http://www.ietf.org/rfc/rfc3010.txt."
15. R. Oldfield, L. Ward, R. Riesen, A. Maccabe, P. Widener, and T. Kordenbrock, "Lightweight i/o for scientific applications," Cluster Computing, 2006 IEEE International Conference on, pp. 1-11, 25-28 Sept. 2006.
16. P. J. Braam, "Lustre: a scalable high-performance file system," Nov. 2002. [Online]. Available: http://www.lustre.org/docs/whitepaper.pdf
17. Panasas, "http://www.panasas.com/."
18. F. Schmuck and R. Haskin, "Gpfs: A shared-disk file system for large computing clusters," in In Proceedings of the 2002 Conference on File and Storage Technologies (FAST, 2002, pp. 231-244.
19. P. NFS, "http://tools.ietf.org/wg/nfsv4/."
20. T. E. Anderson, M. D. Dahlin, J. M. Neefe, D. A. Patterson, D. S. Roselli, and R. Y. Wang, "Serverless network file systems," ACM Trans. Comput. Syst., vol. 14, no. 1, pp. 41-79, 1996.
21. M. C. Miller, J. F. Reus, R. P. Matzke, W. J. Arrighi, L. A. Schoof, R. T. Hitt, and P. K. Espen, "Enabling interoperation of high performance, scientific computing applications: Modeling scientific data with the sets & fields (saf) modeling system," in International Conference on Computational Science (2), 2001, pp. 158-170.
22. P. D. System, "http://pds.jpl.nasa.gov/documents/sr/index.html. "
23. G. in Binary, "http://www.sesp.cse.clrc.ac.uk/publications/ datamanagement/report/node12.html."
24. H. D. System, "http://www.sesp.cse.clrc.ac.uk/publications/ datamanagement/report/node14.html."
25. N. Operators, "http://nco.sourceforge.net/."
26. N. Podhorszki, B. Ludaescher, and S. A. Klasky, "Workflow automation for processing plasma fusion simulation data," in WORKS '07: Proceedings of the 2nd workshop on Workflows in support of large-scale science. New York, NY, USA: ACM, 2007, pp. 35-44.
27. B. Ludäscher, I. Altintas, C. Berkley, D. Higgins, E. Jaeger, M. Jones, E. A. Lee, J. Tao, and Y. Zhao, "Scientific workflow management and the kepler system: Research articles," Concurr. Comput. : Pract. Exper., vol. 18, no. 10, pp. 1039-1065, 2006.
28. J. Lofstead, S. Klasky, K. Schwan, N. Podhorszki, and C. Jin, "Flexible io and integration for scientific codes through the adaptable io system (adios)," in CLADE 2008 at HPDC. Boston, Massachusetts: ACM, June 2008.
29. C. Jin, S. Klasky, S. Hodson, W. Yu, J. Lofstead, H. Abbasi, K. Schwan, M. Wolf, W. Liao, A. Choudhary, M. Parashar, C. Docan, and R. Oldfield, "Adaptive io system (adios)." Cray User's Group, 2008.
30. J. Lofstead, F. Zheng, S. Klasky, and K. Schwan, "Input/output apis and data organization for high performance scientific computing," in In Proceedings of Petascale Data Storage Workshop 2008 at Supercomputing 2008, 2008.