Multicollective I/O: A technique for exploiting inter-file access patterns

ACM Transactions on Storage

Volumn 2, Issue 3, 2006, Pages 349-369

  Memik, Gokhan ^a,c   Kandemir, Mahmut T ^b,d   Liao, Wei Keng ^a   Choudhary, Alok ^a  

^a NORTHWESTERN UNIVERSITY   (United States)

^b PENNSYLVANIA STATE UNIVERSITY   (United States)

^c University of California   (United States)

^d UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

Collective I O; File accesses; Parallel I O; Runtime library; Software optimizations

Indexed keywords

COLLECTIVE I/O; FILE ACCESSES; PARALLEL I/O; RUNTIME LIBRARY; SOFTWARE OPTIMIZATIONS;

COMPUTATIONAL METHODS; DIGITAL LIBRARIES; HEURISTIC METHODS; INPUT OUTPUT PROGRAMS; OPTIMIZATION; PROBLEM SOLVING;

DATA STORAGE EQUIPMENT;

EID: 33750476657     PISSN: 15533077     EISSN: 15533093     Source Type: Journal    
DOI: 10.1145/1168910.1168915     Document Type: Article

References (33)

1. BAGRODIA, R., CHIEN, A., HSU, Y., AND REED, D. 1994. Input/output: Instrumentation, characterization, modeling and management policy. In Proceedings of the Scalable I/O Initiative. Caltech Concurrent Supercomputing Facilities.
2. BARU, S. 1997. Storage Resource Broker (SRB) Reference Manual. Enabling Technologies, San Diego Supercomputer Center. La Jolla, CA.
3. BEYNON, M., KURC, T., SUSSMAN, A., AND SALTZ, J. 2000. Design of a framework for data-intensive wide-area applications. In Proceedings of the 9th Heterogeneous Computing Workshop (HCW). Cancun, Mexico.
4. CHOUDHARY, A., BORDAWEKAR, R., HARRY, M., KRISHNAIYER, R., PONNUSAMY, R., SINGH, T., AND THAKUR, R. 1994. Passion: Parallel and scalable software for input-output. Tech. Rep. SCCS-636, NPAC.
5. CORBETT, P., FEITELSON, D., FINEBERG, S., HSU, Y., NITZBERG, B., PROST, J., SNIR, M., TRAVERSAT, B., AND WONG, P. 1995. Overview of the MPI-IO parallel I/O interface. In Proceedings of the Workshop on I/O in Parallel and Distributed Systems (IPPS '95). Santa Barbara, CA.
6. CORBETT, P., FEITELSON, D., PROST, J.-P., ALMASI, G., BAYLOR, S. J., BOLMAHCICH, A., HSU, Y., SATRAN, J., SNIR, M., COLAO, R., HERR, B., KAVAKY, J., MORGAN, T., AND ZLOTEK, A. 1996. Parallel file systems for the IBM SP computers. IBM Syst. J. 34, 222-248.
7. COYNE, R. A., HULEN, H., AND WATSON, R. 1993. The high performance storage system. In Proceedings of the Supercomputing Conference. Portland, OR.
8. CRANDALL, P., AYDT, R., CHIEN, A., AND REED, D. 1995. Input/Output characteristics of scalable parallel applications. In Proceedings of the Supercomputing Conference.
9. CYPHER, R., HO, A., KONSTANTINIDOU, S., AND MESSINA, P. 1993. Architectural requirements of parallel scientific applications with explicit communication. In Proceedings of the 20th International Symposium on Computer Architecture.
10. DIMACS. 2000. Url: ftp://dimacs.rutgers.edu/pub/netflow/.
11. ELLIS, C. S. AND KOTZ, D. 1989. Prefetching in file systems for mimd multiprocessors. In Proceedings of the International Conference on Parallel Processing. 306-314.
12. FORUM, M. P. I. 1997. Mpi-2: Extension to the message passing interface. Ph.D. thesis, University of Tennessee.
13. GABOV, H. 1973. Implementation of algorithms for maximum matching on non-bipartite graphs. Ph.D. thesis, Stanford University, Palo Alto, CA.
14. GAREY, M. R. AND JOHNSON, D. S. 1979. Computers and Intractability, A Guide to the Theory of NP-Completeness. W. H. Freeman and Company, New York.
15. KARPOVICH, J. F., GRIMSHAW, A. S., AND FRENCH, J. C. 1994. Extensible file systems (elfs): An object-oriented approach to high performance file I/O. In Proceedings of the 9th Annual Conference on Object-Oriented Programming Systems, Languages, and Applications. 191-204.
16. KOTZ, D. 1993. Multiprocessor file system interfaces. In Proceedings of the 2nd International Conference on Parallel and Distributed Information Systems. 194-201.
17. KOTZ, D. 1994. Disk-Directed I/O for mimd multiprocessors. In Proceedings of the Symposium on Operating Systems Design and Implementation. 61-74.
18. KROEGER, T. M. AND LONG, D. E. 1996. Predicting file system actions from prior events. In Proceedings of the USENIX Annual Technical Conference. 319-328.
19. MADHYASTHA, T. M. 1997. Automatic classification of input/output access patterns. Ph.D. thesis, University of Illinois, Department of Computer Science, Urbana, IL.
20. MALAGOLI, A. 1998. Astro3d - 2.0. the asci flash center. Chicago, IL.
21. MEMIK, G., KANDEMIR, M. T., AND CHOUDHARY, A. 2000. April: A run-time library for tape resident data. In Proceedings of the 8th NASA Goddard Conference on Mass Storage Systems and Technologies. Baltimore, MD.
22. MEMIK, G., KANDEMIR, M. T., AND CHOUDHARY, A. 2002. Exploiting inter-file access patterns using multi-collective I/O. In Proceedings of the USENIX Conference on File and Storage Technologies (FAST). Monterey, CA.
23. NIEPLOCHA, J. AND FOSTER, J. 1996. Disk resident arrays: An array-oriented I/O library for out-of-core computations. In Proceedings of the Frontiers of Massively Parallel Processing Conference.
24. NIEUWEJAAR, N., KOTZ, D., PURAKAYASTHA, A., ELLIS, C., AND BEST, M. 1996. File-Access characteristics of parallel scientific workloads. IEEE Trans. Parall. Distribut. Syst., 1075-1089.
25. NlTZBERG, B. 1992. Performance of the ipsc/860 concurrent file system. Tech. rep. RND-92-020, NAS Systems Division, NASA Ames.
26. NITZBERG, B. 1995. Collective parallel I/O. Ph.D. thesis, Department of Computer and Information Science, University of Oregon.
27. ROSARIO, J., BORDAWEKAR, R., AND CHOUDHARY, A. 1993. Improved parallel I/O via a two-phase run-time access strategy. In Proceedings of the IPPS Workshop on Input / Output in Parallel Computer Systems.
28. SEAMONS, K. E., CHEN, Y., JONES, P., JOZWIAK, J., AND WINSLETT, M. 1995. Server-Directed collective I/O in panda. In Proceedings of the Supercomputing Conference.
29. SEAMONS, K. E. AND WINSLETT, M. 2000. Efficient sub-image extraction using panda. http://drl.cs.uiuc.edu/panda/demo/.
30. SHEN, X., LIAO, W., CHOUDHARY, A., MEMIK, G., KANDEMIR, M. T., MORE, S., THIRUVATHUKAL, G., AND SINGH, A. 2000. A novel application development environment for large-scale scientific computations. In Proceedings of the International Conference on Supercomputing.
31. SMITH, P. H. AND VAN ROSENDALE, J. 1998. Data and visualization corridors. Tech. Rep. CACR-164, CACR, Caltech.
32. THAKUR, R., LUSK, E., AND GROPP, W. 1998. Users guide for romio: A high-performance, portable MPI-IO implementation. Tech. Memo. ANL/MCS-TM-234, Mathematics and Computer Science Division, Argonne National Laboratory.
33. TOLEDO, S. AND GUSTAVSON, F. G. 1996. The design and implementation of solar, a portable library for scalable out-of-core linear algebra computations. In Proceedings of the 4th Annual Workshop on I / O in Parallel and Distributed Systems.