Design and analysis of data management in scalable parallel scripting

International Conference for High Performance Computing, Networking, Storage and Analysis, SC

Volumn , Issue , 2012, Pages

  Zhang, Zhao ^a   Katz, Daniel S ^b,c   Wozniak, Justin M ^c   Espinosa, Allan ^a   Foster, Ian ^a,b,c  

^a THE UNIVERSITY OF CHICAGO   (United States)

^b UNIVERSITY OF CHICAGO   (United States)

^c ARGONNE NATIONAL LABORATORY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ASTRONOMY DATA ANALYSIS; AUTOMATIC OPTIMIZATION; DATA MANAGEMENT SYSTEM; DATAFLOW PATTERNS; DESIGN AND ANALYSIS; DESIGN AND IMPLEMENTS; MANY-TASK COMPUTING; PARALLEL EXECUTIONS;

DESIGN; DIGITAL STORAGE; INFORMATION MANAGEMENT; OPTIMIZATION;

DATA FLOW ANALYSIS;

EID: 84877692499     PISSN: 21674329     EISSN: 21674337     Source Type: Conference Proceeding    
DOI: 10.1109/SC.2012.44     Document Type: Conference Paper

References (34)

1. D. Borthakur. HDFS architecture. http://hadoop.apache.org/hdfs/docs/ current/hdfs-design.pdf.
2. S. Brin and L. Page. The anatomy of a large-scale hypertextual web search engine. Computer Networks and ISDN Systems, 30(1-7):107-117, 1998.
3. A. Chakrabarti, R. Dheepak, and S. Sengupta. Integration of scheduling and replication in data grids. In L. Bougé and V. Prasanna, editors, High Performance Computing - HiPC 2004, volume 3296 of Lect. Notes in Comp. Sci., pages 85-101. Springer, 2005.
4. F. Desprez and A. Vernois. Simultaneous scheduling of replication and computation for data-intensive applications on the grid. J. of Grid Comp., 4:19-31, 2006. (Pubitemid 43559860)
5. I. Foster. Designing and Building Parallel Programs: Concepts and Tools for Parallel Software Engineering. Addison-Wesley Longman Publishing Co., Boston, MA, 1995.
6. R. Graves, T. Jordan, S. Callaghan, E. Deelman, E. Field, G. Juve, C. Kesselman, P. Maechling, G. Mehta, K. Milner, D. Okaya, P. Small, and K. Vahi. CyberShake: A physics-based seismic hazard model for southern California. Pure and Applied Geophysics, Online Fir:1-15, May 2010.
7. J. C. Jacob, D. S. Katz, G. B. Berriman, J. C. Good, A. C. Laity, E. Deelman, C. Kesselman, G. Singh, M.-H. Su, T. A. Prince, and R. Williams. Montage: a grid portal and software toolkit for science-grade astronomical image mosaicking. Intl. J. of Comp. Sci. and Eng., 4(2):73-87, 2009.
8. D. S. Katz, T. Armstrong, Z. Zhang, M. Wilde, and J. Wozniak. Many task computing and Blue Waters. Technical Report CI-TR-13-0911, Computation Institute, University of Chicago, November 2011.
9. D. S. Katz, J. C. Jacob, G. B. Berriman, J. Good, A. C. Laity, E. Deelman, C. Kesselman, and G. Singh. A comparison of two methods for building astronomical image mosaics on a grid. In Proc. 2005 Intl. Conf. on Par. Proc. Works., pages 85-94, 2005.
10. H. Lin, P. Balaji, R. Poole, C. Sosa, X. Ma, and W. Feng. Massively parallel genomic sequence search on the Blue Gene/P architecture. In Proc. IEEE/ACM Supercomputing 2008, November 2008.
11. P. Maechling, E. Deelman, L. Zhao, R. Graves, G. Mehta, N. Gupta, J. Mehringer, C. Kesselman, S. Callaghãn, D. Okaya, H. Francoeur, V. Gupta, Y. Cui, K. Vahi, T. Jordan, and E. Field. SCEC CyberShake workflows - automating probabilistic seismic hazard analysis calculations. In I. J. Taylor, E. Deelman, D. B. Gannon, and M. Shields, editors, Workflows for e-Science, pages 143-163. Springer, 2007.
12. D. R. Mathog. Parallel BLAST on split databases. Bioinformatics, 19(14):1865-1866, 2003.
13. A. Matsunaga, M. Tsugawa, and J. Fortes. CloudBLAST: Combining MapReduce and virtualization on distributed resources for bioinformatics applications. In Fourth IEEE International Conference on eScience (eScience '08), pages 222-229, dec. 2008.
14. J. Ousterhout. Scripting: Higher level programming for the 21st Century. IEEE Computer, 31:23-30, 1998. (Pubitemid 128547778)
15. Partitioned Global Address Space languages. http://pgas.org.
16. R. Preissl, M. Schulz, D. Kranzlmüller, B. R. de Supinski, and D. J. Quinlan. Transforming MPI source code based on communication patterns. Future Gener. Comput. Syst., 26(1):147-154, Jan. 2010.
17. I. Raicu, I. Foster, and Y. Zhao. Many-task computing for grids and supercomputers. In Proc. of Many-Task Comp. on Grids and Supercomputers, 2008, 2008.
18. I. Raicu, Y. Zhao, C. Dumitrescu, I. Foster, and M. Wilde. Falkon: a fast and light-weight task execution framework for grid environments. In SC'2007, 2007.
19. I. Raicu, Y. Zhao, I. T. Foster, and A. Szalay. Accelerating large-scale data exploration through data diffusion. In Proc. of 2008 Intl. Work. on Data-Aware Dist. Comp. (DADC '08), pages 9-18. ACM, 2008.
20. K. Ranganathan and I. Foster. Simulation studies of computation and data scheduling algorithms for data grids. J. of Grid Comp., 1(1):53-62, 2003.
21. G. Singh, M.-H. Su, K. Vahi, E. Deelman, B. Berriman, J. Good, D. S. Katz, and G. Mehta. Workflow task clustering for best effort systems with pegasus. In 15th ACM Mardi Gras Conference: From lightweight mash-ups to lambda grids: Understanding the spectrum of distributed computing requirements, applications, tools, infrastructures, interoperability, and the incremental adoption of key capabilities (MG'08), pages 9:1-9:8, New York, 2008. ACM.
22. D. Thain, C. Moretti, and J. Hemmes. Chirp: a practical global filesystem for cluster and grid computing. J. of Grid Comp., 7(1):51-72, 2009.
23. R. Thakur and W. Gropp. Improving the performance of collective operations in MPICH. In J. Dongarra, D. Laforenza, and S. Orlando, editors, Recent Advances in Parallel Virtual Machine and Message Passing Interface, volume 2840 of Lect. Notes in Comp. Sci., pages 257-267. Springer, 2003.
24. R. Thakur, W. Gropp, and E. Lusk. Data sieving and collective I/O in ROMIO. Symp. on Frontiers of Massively Par. Proc., page 182, 1999.
25. R. Thakur, W. Gropp, and E. Lusk. On implementing MPI-IO portably and with high performance. In Proc. of 6th Work. on I/O in Par. and Dist. Systems (IOPADS'99), pages 23-32. ACM, 1999.
26. R. Thakur and R. Rabenseifner. Optimization of collective communication operations in MPICH. Intl. J. of High Perf. Comp. Applications, 19:49-66, 2005. (Pubitemid 40329106)
27. E. Vairavanathan, S. Al-Kiswany, L. Costa, M. Ripeanu, Z. Zhang, D. S. Katz, and M. Wilde. A workflow-aware storage system: An opportunity study. In Proceedings of the 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), 2012.
28. M. Wilde, I. Foster, K. Iskra, P. Beckman, Z. Zhang, A. Espinosa, M. Hategan, B. Clifford, and I. Raicu. Parallel scripting for applications at the petascale and beyond. Computer, 42:50-60, 2009.
29. M. Wilde, M. Hategan, J. M. Wozniak, B. Clifford, D. S. Katz, and I. Foster. Swift: A language for distributed parallel scripting. Par. Comp., pages 633-652, September 2011.
30. J. Wozniak and M. Wilde. http://www.ci.uchicago.edu/swift/guides/release- 0.93/userguide/userguide.html#-collective-data-management.
31. J. M. Wozniak and M. Wilde. Case studies in storage access by loosely coupled petascale applications. In Proc. 4th Annual Work. on Petascale Data Storage, pages 16-20, 2009.
32. K. Yelick, D. Bonachea, W.-Y. Chen, P. Colella, K. Datta, J. Duell, S. L. Graham, P. Hargrove, P. Hilfinger, P. Husbands, C. Iancu, A. Kamil, R. Nishtala, J. Su, M. Welcome, and T. Wen. Productivity and performance using partitioned global address space languages. In 2007 International Workshop on Parallel Symbolic Computation (PASCO '07), pages 24-32, New York, 2007. ACM.
33. Z. Zhang, A. Espinosa, K. Iskra, I. Raicu, I. Foster, and M. Wilde. Design and evaluation of a collective I/O model for loosely coupled petascale programming. In Proc. of Many-Task Comp. on Grids and Supercomputers, 2008, 2008.
34. Z. Zhang, D. S. Katz, M. Ripeanu, M. Wilde, and I. Foster. AME: An anyscale many-task computing engine. In 6th Work. on Workflows in Support of Large-Scale Science (WORKS11), 2011.