PreDatA - Preparatory data analytics on peta-scale machines

Proceedings of the 2010 IEEE International Symposium on Parallel and Distributed Processing, IPDPS 2010

Volumn , Issue , 2010, Pages

  Zheng, Fang ^a   Abbasi, Hasan ^a   Docan, Ciprian ^b   Lofstead, Jay ^a   Liu, Qing ^c   Klasky, Scott ^c   Parashar, Manish ^a   Podhorszki, Norbert ^c   Schwan, Karsten ^a   Wolf, Matthew ^a,c  

^a Georgia Institute of Technology   (United States)

^b RUTGERS UNIVERSITY   (United States)

^c OAK RIDGE NATIONAL LABORATORY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

APPLICATION-SPECIFIC; ASYNCHRONOUS DATA; COMPUTATIONAL POWER; DATA ACCESS; DATA ANALYTICS; DATA CHARACTERISTICS; DATA EXCHANGE; DATA MANIPULATIONS; DATA PREPROCESSING; DATA PRESENTATION; DATA REORGANIZATION; END USERS; FILE SYSTEMS; GAIN INSIGHT; HIGH END COMPUTING; LARGE SCALE SIMULATIONS; MASSIVE DATA SETS; OUTPUT DATA; RUN-TIME DATA; RUNNING SIMULATIONS; SCIENTIFIC APPLICATIONS; SPEED-UPS; STREAMING DATA;

DATA REDUCTION; DISTRIBUTED PARAMETER NETWORKS; METADATA; MIDDLEWARE; VISUALIZATION;

DATA VISUALIZATION;

EID: 77953972291     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/IPDPS.2010.5470454     Document Type: Conference Paper

References (53)

1. H. Abbasi, J. Lofstead, F. Zheng, S. Klasky, K. Schwan, and M. Wolf. Extending i/o through high performance data services. In CLUSTER, 2009.
2. H. Abbasi, M. Wolf, G. Eisenhauer, S. Klasky, K. Schwan, and F. Zheng. Datastager: scalable data staging services for petascale applications. In HPDC, 2009.
3. H. Abbasi, M. Wolf, K. Schwan, G. Eisenhauer, and A. Hilton. Xchange: coupling parallel applications in a dynamic environment. In CLUSTER, 2004.
4. N. Ali, P. Carns, K. Iskra, D. Kimpe, S. Lang, R. Latham, R. Ross, L. Ward, and P. Sadayappan. Scalable i/o forwarding framework for high-performance computing systems. In CLUSTER, 2009.
5. K. Amiri, D. Petrou, G. R. Ganger, and G. A. Gibson. Dynamic function placement for data-intensive cluster computing. In USENIX Annual Technical Conference, 2000.
6. M. D. Beynon, R. Ferreira, T. M. Kurc, A. Sussman, and J. H. Saltz. Datacutter: Middleware for filtering very large scientific datasets on archival storage systems. In MSST, 2000.
7. R. Brightwell, T. Hudson, K. T. Pedretti, R. Riesen, and K. D. Underwood. Implementation and performance of portals 3.3 on the cray xt3. In CLUSTER, 2005.
8. P. Carns, S. Lang, R. Ross, M. Vilayannur, J. Kunkel, and T. Ludwig. Small-file access in parallel file systems. In IPDPS, 2009.
9. S. Center. Scidac scientific data management center. https://sdm.lbl.gov/ sdmcenter/, September 2009.
10. L. Chacon. A non-staggered, conservative, δ ṡ B →=0, finite-volume scheme for 3D implicit extended magnetohydrodynamics in curvilinear geometries. Computer Physics Communications, 163:143-171, Nov. 2004.
11. T. Condie, N. Conway, P. Alvaro, J. M. Hellerstein, K. Elmeleegy, and R. Sears. Mapreduce online. Technical Report UCB/EECS-2009-2136, EECS Department, University of California, Berkeley, Oct 2009.
12. J. Dean and S. Ghemawat. Mapreduce: simplified data processing on large clusters. In OSDI, 2004.
13. E. Deelman and A. Chervenak. Data management challenges of data-intensive scientific workflows. In CCGRID, 2008.
14. E. Deelman, G. Singh, M. hui Su, J. Blythe, A. Gil, C. Kesselman, G. Mehta, K. Vahi, G. B. Berriman, J. Good, A. Laity, J. C. Jacob, and D. S. Katz. Pegasus: a framework for mapping complex scientific workflows onto distributed systems. Scientific Programming Journal, 13:219-237, 2005.
15. C. Docan, M. Parashar, J. Cummings, N. Podhorszki, and S. Klasky. Experiments with Memory-to-Memory Coupling for End-to-End Fusion Simulation Workflows. Technical Report TR-104, Center for Autonomic Computing (CAC), Rutgers University, July 2009.
16. C. Docan, M. Parashar, and S. Klasky. Dart: a substrate for high speed asynchronous data io. In HPDC, 2008.
17. G. Eisenhauer. Evpath: event transport middleware layer. http://www.cc.gatech.edu/systems/projects/EVPath/, September 2009.
18. G. Eisenhauer, F. E. Bustamante, and K. Schwan. Native data representation: An efficient wire format for high-performance distributed computing. IEEE Trans. Parallel Distrib. Syst., 13(12):1234-1246, 2002.
19. J. Gray, D. T. Liu, M. Nieto-Santisteban, A. Szalay, D. J. DeWitt, and G. Heber. Scientific data management in the coming decade. SIGMOD Rec., 34(4):34-41, 2005.
20. W. Gu, G. Eisenhauer, K. Schwan, and J. S. Vetter. Falcon: Online monitoring for steering parallel programs. Concurrency - Practice and Experience, 10(9):699-736, 1998.
21. C. Jones, K.-L. Ma, A. Sanderson, and L. R. M. Jr. Visual interrogation of gyrokinetic particle simulations. J. Phys.: Conf. Ser., 78(012033):6, 2007.
22. W. keng Liao and A. N. Choudhary. Dynamically adapting file domain partitioning methods for collective i/o based on underlying parallel file system locking protocols. In SC, 2008.
23. S. Klasky, S. Ethier, Z. Lin, K. Martins, D. McCune, and R. Samtaney. Grid -based parallel data streaming implemented for the gyrokinetic toroidal code. In SC, 2003.
24. J. Lee, R. B. Ross, S. Atchley, M. Beck, and R. Thakur. Mpi-io/l: efficient remote i/o for mpi-io via logistical networking. In IPDPS, 2006.
25. J. Lee, R. B. Ross, R. Thakur, X. Ma, and M. Winslett. Rfs: efficient and flexible remote file access for mpi-io. In CLUSTER, 2004.
26. J.-Y. Lee and A. Sussman. High performance communication between parallel programs. In IPDPS, 2005.
27. Y. Liu, N. Vijayakumar, and B. Plale. Stream processing in data-driven computational science. In GRID, 2006.
28. J. Lofstead, S. Klasky, K. Schwan, N. Podhorszki, and C. Jin. Flexible io and integration for scientific codes through the adaptable io system (adios). In CLADE at HPDC, 2008.
29. J. Lofstead, F. Zheng, S. Klasky, and K. Schwan. Input/output apis and data organization for high performance scientific computing. In PDSW at SC, 2008.
30. J. Lofstead, F. Zheng, S. Klasky, and K. Schwan. Adaptable, metadata rich io methods for portable high performance io. In In IPDPS, 2009.
31. B. Ludascher, I. Altintas, C. Berkley, D. Higgins, E. Jaeger, M. Jones, E. A. Lee, J. Tao, and Y. Zhao. Scientific workflow management and the kepler system. Concurrency and Computation: Practice and Experience, 18(10):1039-1065, 2006.
32. X. Ma, J. Lee, and M. Winslett. High-level buffering for hiding periodic output cost in scientific simulations. IEEE Trans. Parallel Distrib. Syst., 17(3):193-204, 2006.
33. C. Moretti, J. Bulosan, D. Thain, and P. J. Flynn. All-pairs: An abstraction for data-intensive cloud computing. In IPDPS, 2008.
34. A. Nisar, W. keng Liao, and A. N. Choudhary. Scaling parallel i/o performance through i/o delegate and caching system. In SC, 2008.
35. R. Oldfield, L. Ward, R. Riesen, A. B. Maccabe, P. Widener, and T. Kordenbrock. Lightweight i/o for scientific applications. In CLUSTER, 2006.
36. PDSI. Scidac petascale data storage institute. http://www.pdsi-scidac. org/, September 2009.
37. J. Piernas, J. Nieplocha, and E. J. Felix. Evaluation of active storage strategies for the lustre parallel file system. In SC, 2007.
38. M. Polte, J. Simsa, W. Tantisiriroj, and G. Gibson. Fast log-based concurrent writing of checkpoints. In PDSW at SC, 2008.
39. O. Rubel, Prabhat, K. Wu, H. Childs, J. Meredith, C. G. R. Geddes, E. Cormier- Michel, S. Ahern, G. H. Weber, P. Messmer, H. Hagen, B. Hamann, and E. W. Bethel. High performance multivariate visual data exploration for extremely large data. In SC, 2008.
40. S. W. Schlosser, M. P. Ryan, R. Taborda-Rios, J. Lopez, D. R. O'Hallaron, and J. Bielak. Materialized community ground models for large-scale earthquake simulation. In SC, 2008.
41. K. E. Seamons, Y. Chen, P. Jones, J. Jozwiak, and M. Winslett. Server-directed collective i/o in panda. In SC, 1995.
42. R. R. Sinha and M. Winslett. Multi-resolution bitmap indexes for scientific data. ACM Trans. Database Syst., 32(3):16, 2007.
43. K. Stockinger, J. Shalf, E. W. Bethel, and K. Wu. Dex: Increasing the capability of scientific data analysis pipelines by using efficient bitmap indices to accelerate scientific visualization. In SSDBM, 2005.
44. T. Tu, C. A. Rendleman, D. W. Borhani, R. O. Dror, J. Gullingsrud, M. O. Jensen, J. L. Klepeis, P. Maragakis, P. Miller, K. A. Stafford, and D. E. Shaw. A scalable parallel framework for analyzing terascale molecular dynamics simulation trajectories. In SC, 2008.
45. T. Tu, H. Yu, J. Bielak, O. Ghattas, J. C. Lopez, K.-L. Ma, D. R. O'Hallaron, L. Ramirez-Guzman, N. Stone, R. Taborda-Rios, and J. Urbanic. Analytics challenge - remote runtime steering of integrated terascale simulation and visualization. In SC, 2006.
46. T. Tu, H. Yu, L. Ramirez-Guzman, J. Bielak, O. Ghattas, K.-L. Ma, and D. R. O'Hallaron. Scalable systems sof Tware - from mesh generation to scientific visualization: an end-to-end approach to parallel supercomputing. In SC, 2006.
47. H. Wang, S. Parthasarathy, A. Ghoting, S. Tatikonda, G. Buehrer, T. M. Kurc, and J. H. Saltz. Design of a next generation sampling service for large scale data analysis applications. In ICS, 2005.
48. B. Welch, M. Unangst, Z. Abbasi, G. A. Gibson, B. Mueller, J. Small, J. Zelenka, and B. Zhou. Scalable performance of the panasas parallel file system. In FAST, 2008.
49. H. Yu and K.-L. Ma. A study of i/o methods for parallel visualization of large-scale data. Parallel Comput., 31(2):167-183, 2005.
50. W. Yu, J. S. Vetter, and S. Oral. Performance characterization and optimization of parallel i/o on the cray xt. In IPDPS, 2008.
51. L. Zhang and M. Parashar. Seine: a dynamic geometry-based shared-space interaction framework for parallel scientific applications. Concurrency and Computation: Practice and Experience, 18(15):1951-1973, 2006.
52. X. Zhang, S. Jiang, and K. Davis. Making resonance a common case: A high- performance implementation of collective i/o on parallel file systems. In IPDPS, 2009.
53. F. Zheng, H. Abbasi, C. Docan, J. Lofstead, Q. Liu, S. Klasky, M. Parashar, N. Podhorszki, K. Schwan, and M. Wolf. Predata - preparatory data analytics on peta-scale machines. Technical Report GIT-CERCS-10-101, Georgia Institute of Technology, January 2010.