Parallel I/O, analysis, and visualization of a trillion particle simulation

International Conference for High Performance Computing, Networking, Storage and Analysis, SC

Volumn , Issue , 2012, Pages

  Byna, Surendra ^a   Chou, Jerry ^b   Rübel, Oliver ^a   Prabhat, ^a   Karimabadi, Homa ^c   Daughter, William S ^d   Roytershteyn, Vadim ^c   Bethel, E Wes ^a   Howison, Mark ^e   Hsu, Ke Jou ^b   Lin, Kuan Wu ^b   Shoshani, Arie ^a   Uselton, Andrew ^a   Wu, Kesheng ^a  

^a LAWRENCE BERKELEY NATIONAL LABORATORY   (United States)

^b NATIONAL TSING HUA UNIVERSITY   (Taiwan)

^c UNIVERSITY OF CALIFORNIA   (United States)

^d LOS ALAMOS NATIONAL LABORATORY   (United States)

^e BROWN UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DATA EXTENSION; DISTRIBUTED MEMORY; EFFICIENT ANALYSIS; MULTI-CORE CPUS; PARTICLE SIMULATIONS; PLASMA PHYSICS; SYSTEM PEAKS; VISUALIZATION RESULTS;

PROGRAM PROCESSORS; VISUALIZATION;

DIGITAL STORAGE;

EID: 84877691569     PISSN: 21674329     EISSN: 21674337     Source Type: Conference Proceeding    
DOI: 10.1109/SC.2012.92     Document Type: Conference Paper

References (49)

1. K. Antypas and A. Uselton. MPI-I/O on Franklin XT4 System at NERSC. In 52nd Cray User Group Conference, Edinburgh, UK, 2010.
2. IPCC Fifth Assessment Report. http://en.wikipedia.org/wiki/IPCC-Fifth- Assessment-Report.
3. P. Balaji, A. Chan, W. Gropp, R. Thakur, and E. L. Lusk. Non-datacommunication overheads in MPI: Analysis on blue gene/P. In A. L. Lastovetsky, M. T. Kechadi, and J. Dongarra, editors, PVM/MPI, volume 5205 of Lecture Notes in Computer Science, pages 13-22. Springer, 2008.
4. K. J. Bowers, B. J. Albright, L. Yin, B. Bergen, and T. J. T. Kwan. Ultrahigh performance three-dimensional electromagnetic relativistic kinetic plasma simulation. Physics of Plasmas, 15(5):7, 2008.
5. H. Childs, D. Pugmire, S. Ahern, B. Whitlock, M. Howison, Prabhat, G. H. Weber, and E. W. Bethel. Extreme scaling of production visualization software on diverse architectures. IEEE Computer Graphics and Applications, 30:22-31, 2010.
6. J. Chou, K. Wu, and Prabhat. FastQuery: A general indexing and querying system for scientific data. In SSDBM, pages 573-574, 2011. http://dx.doi.org/10. 1007/978-3-642-22351-8-42.
7. J. Chou, K. Wu, and Prabhat. FastQuery: A parallel indexing system for scientific data. In IASDS. IEEE, 2011.
8. J. Chou, K. Wu, O. Rübel, M. Howison, J. Qiang, Prabhat, B. Austin, E. W. Bethel, R. D. Ryne, and A. Shoshani. Parallel index and query for large scale data analysis. In SC11, 2011.
9. Getting Started with MPI I/O. http://docs.cray.com/books/S-2490-40/S- 2490-40.pdf.
10. W. Daughton, V. Roytershteyn, H. Karimabadi, L. Yin, B. J. Albright, B. Bergen, and K. J. Bowers. Role of electron physics in the development of turbulent magnetic reconnection in collisionless plasmas. Nature Physics, 7:539542, 2011.
11. W. Daughton, V. Roytershteyn, H. Karimabadi, L. Yin, B. J. Albright, B. Bergen, and K. J. Bowers. Role of electron physics in the development of turbulent magnetic reconnection in collisionless plasmas. Nature Physics, 7(7):539-542, July 2011.
12. W. Daughton, J. D. Scudder, and H. Karimabadi. Fully kinetic simulations of undriven magnetic reconnection with open boundary conditions. Physics of Plasmas, 13, 2006.
13. J. Egedal, W. Daughton, and A. Le. Large-scale electron acceleration by parallel electric fields during magnetic reconnection. Nature Physics, 8:321324, 2012.
14. K. Gao, W. keng Liao, A. Nisar, A. Choudhary, R. Ross, and R. Latham. Using subfiling to improve programming flexibility and performance of parallel shared-file I/O. In Proceedings of the 2009 International Conference on Parallel Processing, ICPP '09, pages 470-477, Washington, DC, USA, 2009. IEEE Computer Society.
15. D. S. Henty. Performance of hybrid message-passing and shared-memory parallelism for discrete element modeling. In SC'00, Washington, DC, USA, 2000. IEEE Computer Society.
16. C. M. Herb Wartens, Jim Garlick. LMT - The Lustre Monitoring Tool. https://github.com/chaos/lmt/wiki. Developed at Lawrence Livermore National Lab.
17. T. Hey, S. Tansley, and K. Tolle, editors. The Fourth Paradigm: Data- Intensive Scientific Discovery. Microsoft, Oct. 2009.
18. H. Hochheiser and B. Shneiderman. Visual specification of queries for finding patterns in time-series data. In Proceedings of Discovery Science, pages 441-446, 2001.
19. M. Howison, A. Adelmann, E. W. Bethel, A. Gsell, B. Oswald, and Prabhat. H5hut: A High-Performance I/O Library for Particle-Based Simulations. In Proceedings of 2010 Workshop on Interfaces and Abstractions for Scientific Data Storage (IASDS10), Heraklion, Crete, Greece, Sept. 2010. LBNL-4021E.
20. M. Howison, E. W. Bethel, and H. Childs. MPI-hybrid parallelism for volume rendering on large, multi-core systems. In Eurographics Symposium on Parallel Graphics and Visualization, pages 1-10, 2010.
21. M. Howison, Q. Koziol, D. Knaak, J. Mainzer, and J. Shalf. Tuning HDF5 for Lustre File Systems. In Proceedings of 2010 Workshop on Interfaces and Abstractions for Scientific Data Storage (IASDS10), Heraklion, Crete, Greece, Sept. 2010. LBNL-4803E.
22. In-situ Visualization: State-of-the-art and Some Use Cases. http://www.prace-project.eu/IMG/pdf/In-situ-Visualization-State-of-the-art-and- Some-Use-Cases.pdf.
23. C. R. Johnson and J. Huang. Distribution-driven visualization of volume data. IEEE Transactions on Visualization and Computer Graphics, 15(5):734-746, Sept. 2009.
24. H. Karimabadi, W. Daughton, and J. Scudder. Multi-scale structure of the electron diffusion region. Geophys. Res. Lett., 34, 2007.
25. H. Karimabadi, B. Loring, A. Majumdar, and M. Tatineni. I/O strategies for massively parallel kinetic simulations, 2010.
26. D. Keim and H.-P. Kriegel. VisDB: Database exploration using multidimensional visualization. IEEE Computer Graphics and Applications, 14(4):40-49, 1994.
27. ADIOS. http://www.nccs.gov/user-support/center-projects/adios/.
28. A. Klimas, M. Hesse, and S. Zenitani. Particle-in-cell simulation of collisionless reconnection with open outflow boundary conditions. Physics of Plasmas, pages 082102-082102-9, 2008.
29. J. Li, W. keng Liao, A. Choudhary, R. Ross, R. Thakur, W. Gropp, R. Latham, A. Siegel, B. Gallagher, and M. Zingale. Parallel netCDF: A high-performance scientific I/O interface. In SC'03, page 39, New York, NY, USA, 2003. ACM.
30. J. Lofstead, F. Zheng, S. Klasky, and K. Schwan. Adaptable, metadata rich IO methods for portable high performance IO. In Proceedings of the 2009 IEEE International Symposium on Parallel&Distributed Processing, IPDPS '09, pages 1-10, Washington, DC, USA, 2009. IEEE Computer Society.
31. J. F. Lofstead, S. Klasky, K. Schwan, N. Podhorszki, and C. Jin. Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS). In CLADE'08, pages 15-24, New York, NY, USA, 2008. ACM.
32. O. Rübel, C. G. R. Geddes, E. Cormier-Michel, K. Wu, Prabhat, G. H. Weber, D. M. Ushizima, P. Messmer, H. Hagen, B. Hamann, and W. Bethel. Automatic beam path analysis of laser wakefield particle acceleration data. IOP Computational Science & Discovery, 2(015005 (38pp)), November 2009.
33. O. Rübel, Prabhat, K. Wu, H. Childs, J. Meredith, C. G. R. Geddes, E. Cormier-Michel, S. Ahern, G. H. weber, P. Messmer, H. Hagen, B. Hamann, and E. W. Bethel. High Performance Multivariate Visual Data Exploration for Extemely Large Data. In SuperComputing 2008 (SC08), Austin, Texas, USA, Nov. 2008.
34. J. D. Scudder, R. D. Holdaway, W. S. Daughton, H. Karimabadi, V. Roytershteyn, C. T. Russell, and J. Y. Lopez. First resolved observations of the demagnetized electron-diffusion region of an astrophysical magnetic- reconnection site. Phys. Rev. Lett., 108:225005, Jun 2012.
35. M. Shay, J. Drake, and M. Swisdak. Two-scale structure of the electron dissipation region during collisionless magnetic reconnection. Phys. Rev. Lett., 99, 2007.
36. A. Shoshani and D. Rotem, editors. Scientific Data Management: Challenges, Technology, and Deployment. Chapman & Hall/CRC Press, 2010.
37. G. Smith, M. Czerwinski, B. Meyers, D. Robbins, G. Robertson, and D. S. Tan. Facetmap: A scalable search and browse visualization. IEEE Transactions on Visualization and Computer Graphics, 12(5):797-804, Sept. 2006. (Pubitemid 44955606)
38. K. Stockinger, E. W. Bethel, S. Campbell, E. Dart, and K. Wu. Detecting Distributed Scans Using High-Performance Query-Driven Visualization. In SC '06: Proceedings of the 2006 ACM/IEEE Conference on High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society Press, Nov. 2006.
39. K. Stockinger, J. Shalf, W. Bethel, and K. Wu. Query-driven visualization of large data sets. In IEEE Visualization 2005, Minneapolis, MN, October 23-28, 2005, page 22, 2005. http://doi.ieeecomputersociety.org/10.1109/VIS.2005.84.
40. The HDF Group. HDF5 user guide. http://hdf.ncsa.uiuc.edu/HDF5/doc/H5. user.html, 2010.
41. Unidata. The NetCDF users' guide. http://www.unidata.ucar.edu/software/ netcdf/docs/netcdf/, 2010.
42. A. Uselton. Deploying server-side file system monitoring at NERSC. In Cray User Group Conference, Atlanta, GA, 2009.
43. A. Uselton and B. Behlendorf. Visualizing I/O performance during the BGL deployment. In 8th LCI Conference on High-Performance Clustered Computing, South Lake Tahoe, CA, 2007.
44. R. V., W. Daughton, H. Karimabadi, and F. S. Mozer. Influence of the lower-hybrid drift instability on magnetic reconnection in asymmetric configurations. Phys. Rev. Lett., 108:185001, 2012.
45. B. Whitlock, J. M. Favre, and J. S. Meredith. Parallel in situ coupling of simulation with a fully featured visualization system. In EGPGV, pages 101-109, 2011.
46. K. Wu. FastBit: an efficient indexing technology for accelerating data-intensive science. Journal of Physics: Conference Series, 16:556-560, 2005. http://dx.doi.org/10.1088/1742-6596/16/1/077. (Pubitemid 41259399)
47. K. Wu, S. Ahern, E. W. Bethel, J. Chen, H. Childs, E. Cormier-Michel, C. Geddes, J. Gu, H. Hagen, B. Hamann, W. Koegler, J. Lauret, J. Meredith, P. Messmer, E. Otoo, V. Perevoztchikov, A. Poskanzer, Prabhat, O. Rubel, A. Shoshani, A. Sim, K. Stockinger, G. Weber, and W.-M. Zhang. FastBit: Interactively searching massive data. In SciDAC, 2009.
48. K. Wu, A. Shoshani, and K. Stockinger. Analyses of multi-level and multi-component compressed bitmap indexes. ACM Transactions on Database Systems, pages 1-52, 2010.
49. H. Yu, C. Wang, R. W. Grout, J. H. Chen, and K.-L. Ma. In situ visualization for large-scale combustion simulations. IEEE Comput. Graph. Appl., 30(3):45-57, May 2010.