Input/output APIs and data organization for high performance scientific computing

Proceedings of the 2008 3rd Petascale Data Storage Workshop, PDSW 2008

Volumn , Issue , 2008, Pages

  Lofstead, Jay ^a   Zheng, Fang ^a   Klasky, Scott ^b   Schwan, Karsten ^a  

^a Georgia Institute of Technology   (United States)

^b OAK RIDGE NATIONAL LABORATORY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DATA CHARACTERIZATION; DATA CONSISTENCY; DATA FORMAT; DATA ORGANIZATION; DATA OUTPUT; END USERS; FAILURE PROBABILITY; FILE FORMATS; FILE SYSTEMS; HIGH PERFORMANCE SCIENTIFIC COMPUTING; INFORMED DECISION; INPUT/OUTPUT; MACHINE RESOURCES; OUTPUT DATA; PARALLEL DATA; PERFORMANCE EVALUATION; PETASCALE; RUNNING APPLICATIONS; RUNTIMES; SCIENTIFIC DATA MANAGEMENT;

INDEXING (OF INFORMATION); METADATA;

DATA STORAGE EQUIPMENT;

EID: 70149087347     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/PDSW.2008.4811881     Document Type: Conference Paper

References (15)

1. L. Gosink, J. Shalf, K. Stockinger, K. Wu, and W. Bethel, "Hdf5- fastquery: Accelerating complex queries on hdf datasets using fast bitmap indices," in In SSDBM, 2006, pp. 149-158.
2. H.-. P. Tables, "http://www.carabos.com/docs/opsi-indexes.pdf."
3. P. M. Widener, M. Wolf, H. Abbasi, M. Barrick, J. Lofstead, J. Pullikottil, G. Eisenhauer, A. Gavrilovska, S. Klasky, R. Oldfield, P. G. Bridges, A. B. Maccabe, and K. Schwan, "Structured streams: Data services for petascale science environments," University of New Mexico, Albuquerque, NM, Tech. Rep. TR-CS-2007-17, November 2007. [Online]. Available: http://www.cs.unm.edu/ treport/tr/07-11/Peta- Scale.pdf
4. P. M. Widener, M. Barrick, J. Pullikottil, P. G. Bridges, and A. B. Maccabe, "Metabots: A framework for out-of-band processing in largescale data flows," in Proc. 2007 International Conference on Grid Computing (Grid 2007), Austin, Texas, September 2007.
5. B. Ludäscher, I. Altintas, C. Berkley, D. Higgins, E. Jaeger, M. Jones, E. A. Lee, J. Tao, and Y. Zhao, "Scientific workflow management and the kepler system: Research articles," Concurr. Comput. : Pract. Exper., vol. 18, no. 10, pp. 1039-1065, 2006.
6. E. Deelman, G. Singh, M.-H. Su, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, G. B. Berriman, J. Good, A. Laity, J. C. Jacob, and D. S. Katz, "Pegasus: A framework for mapping complex scientific workflows onto distributed systems," Sci. Program., vol. 13, no. 3, pp. 219-237, 2005.
7. G. Malewicz, I. Foster, A. Rosenberg, and M. Wilde, "A tool for prioritizing dagman jobs and its evaluation," High Performance Distributed Computing, 2006 15th IEEE International Symposium on, pp. 156-168, 0-0 2006.
8. R. W. Watson and R. A. Coyne, "The parallel i/o architecture of the high-performance storage system (hpss)," in MSS '95: Proceedings of the 14th IEEE Symposium on Mass Storage Systems. Washington, DC, USA: IEEE Computer Society, 1995, p. 27.
9. S. Ghemawat, H. Gobioff, and S.-T. Leung, "The google file system," SIGOPS Oper. Syst. Rev., vol. 37, no. 5, pp. 29-43, 2003.
10. J. Lofstead, S. Klasky, K. Schwan, N. Podhorszki, and C. Jin, "Flexible io and integration for scientific codes through the adaptable io system (adios)," in CLADE 2008 at HPDC. Boston, Massachusetts: ACM, June 2008.
11. R. Oldfield, L. Ward, R. Riesen, A. Maccabe, P. Widener, and T. Kordenbrock, "Lightweight i/o for scientific applications," Cluster Computing, 2006 IEEE International Conference on, pp. 1-11, 25-28 Sept. 2006.
12. T. E. Anderson, M. D. Dahlin, J. M. Neefe, D. A. Patterson, D. S. Roselli, and R. Y. Wang, "Serverless network file systems," ACM Trans. Comput. Syst., vol. 14, no. 1, pp. 41-79, 1996.
13. NFS, "http://www.ietf.org/rfc/rfc3010.txt."
14. W. Yu, J. Vetter, and H. Oral, "Performance characterization and optimization of parallel i/o on the cray xt," Parallel and Distributed Processing, 2008. IPDPS 2008. IEEE International Symposium on, pp. 1-11, April 2008.
15. F. Bustamante, G. Eisenhauer, K. Schwan, and P. Widener, "Efficient wire formats for high performance computing," in In Proceedings of Supercomputing 2000, 2000.