Data management and analysis at the Large Scale Data Facility

2011 6th International Conference on Digital Information Management, ICDIM 2011

Volumn , Issue , 2011, Pages 235-241

  Garcia A O ^a   Bourov, S ^a   Hammad, A ^a   Jejkal, T ^a   Otte, J C ^a   Pfeiffer, S ^a   Schenker, T ^a   Schmidt, C ^a   Van Wezel, J ^a   Neumair, B ^a   Streit, A ^a  

^a KARLSRUHE INSTITUTE OF TECHNOLOGY   (Germany)

Author keywords

data intensive computing; Hadoop; Large Scale Data Facility; OpenNebula; storage

Indexed keywords

BIOMEDICAL PROCESS; DATA COLLECTION; DATA-INTENSIVE COMPUTING; GRAPHICAL TOOLS; HADOOP; HIGH THROUGHPUT; ITS DATA; LARGE SCALE DATA; META DATA REPOSITORIES; MICROSCOPY IMAGES; OPENNEBULA; SCIENTIFIC COMMUNITY; STORAGE SERVICES; VALUE ADDED SERVICE; WORK-FLOWS;

APPLICATION PROGRAMMING INTERFACES (API); DATA HANDLING; ENERGY STORAGE;

INFORMATION MANAGEMENT;

EID: 83755162633     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICDIM.2011.6093357     Document Type: Conference Paper

References (31)

1. Gordon E. Moore, Cramming more components onto integrated circuits, Electronics 38, nr. 8, 1965.
2. T. Hey, S. Tansley, and K. Tolle (Editors), The Fourth Paradigm: Data-Intensive Scientific Discovery, Microsoft Research, 2009; http://research.microsoft.com/en-us/collaboration/fourthparadigm
3. http://public.web.cern. ch/public/en/lhc/lhc-en.html
4. Douglas B. Kothe, Science Prospects and Benefits with Exascale Computing, Report ORNL/TM-2007/232, Oak Ridge National Laboratory, USA, 2007; http://www.nccs.gov/wp-content/media/nccs-reports/Science%20Case%20-012808%20v3- final.pdf
5. Vivek Sarkar et. al., Exascale Software Study: Software Challenges in Extreme Scale Systems, Report DARPA/IPTO, USA, 2009; http://users.ece.gatech. edu/mrichard/ExascaleComputingStudyReports/ECSS%20report%20101909.pdf
6. Large Scale Data Management and Analysis, http://www.scc.kit.edu/ forschung/7113.php
7. F. Schmuck, R. Haskin, GPFS: A Shared-Disk File System for Large Computing Clusters, FAST'02:1st USENIX Conference on File and Storage Technologies, 2002.
8. S. Oehme, J. Deicke, J.-P. Akelbein, R. Sahlberg, A. Tridgell, R. L. Haskin, IBM Scale out File Services: Reinventing network-attached storage, IBM J. Res. & Dev. 52, nr. 4/5, 2008.
9. http://www.bioquant.uni-heidelberg.de/
10. A. Rajasekar, M. Wan, R. Moore, W. Schroeder, A Prototype Rulebased Distributed Data Management System, HPDC workshop on Next Generation Distributed Data Management, Paris, France, 2006.
11. A. Rajasekar, M. Wan, R. Moore, Event Processing in Policy Oriented Data Grids, Intelligent Event Processing AAAI Spring Symposium, Stanford, USA, 2009, pp. 61-66.
12. J. R. Swedlow, I. G. Goldberg, K. W. Eliceiri, and the OME Consortium, Bioimage Informatics for Experimental Biology, Annual Review of Biophysics 38, 2009, pp. 327-346.
13. B. Sotomayor, R. S. Montero, I. M. Llorente, I. Foster, Virtual Infrastructure Management in Private and Hybrid Clouds, IEEE Internet Computing 13, nr. 5, 2009, pp. 14-22.
14. http://aws.amazon.com/ec2/
15. http://hadoop. apache.org/
16. J. Dean, S. Ghemawat, MapReduce: Simplified Data Processing on Large Clusters, OSDI'04: Sixth Symposium on Operating System Design and Implementation, San Francisco, USA, 2004.
17. Konstantin V. Shvachko, HDFS Scalability: The Limits to Growth,;login: The USENIX Magazine 35, nr. 2, 2010.
18. http://fuse.sourceforge.net/
19. http://tools.ietf.org/html/rfc5280
20. R. Stotzka, V. Hartmann, T. Jejkal, M. Sutter, J. van Wezel, M. Hardt, A. García, R. Kupsch, S. Bourov, Perspective of the Large Scale Data Facility (LSDF) supporting nuclear fusion applications, Proceedings of The 19th Euromicro International Conference on Parallel, Distributed and Network-Based Computing (PDP2011), 2011, pp. 373-379.
21. Michael Goetter, Design of abstract access methods for various distributed file systems, master thesis at Duale Hochschule Baden- Württemberg Karlsruhe, Germany, 2010; http://www.ipe.fzk.de/~stotzka/ publications/publications/goetter2010.1.pdf
22. T. P. Randall, Chemical Biology and the Limits of Reductionism, Nature Chemical Biology 4, 2008, pp. 635-638.
23. J. Gehrig, M. Reischl, E. Kalmar, M. Ferg, Y. Hadzhiev, A. Zaucker, C. Song, S. Schindler, U. Liebel, and F. Müller, Automated high-throughput mapping of promoter-enhancer interactions in zebrafish embryos, Nature Methods 6, 2009, pp. 911-916; http://www.nature.com/nmeth/journal/v6/n12/abs/nmeth.1396. html
24. S. Pfeiffer, M. Mai, and J. Calliess, On the Computational Benefit of Tensor Separation for High-Dimensional Discrete Convolutions, Multidimensional Systems and Signal Processing 31, 2010, pp. 1-25.
25. Christian Schmidt, Konzept zum Datenmanagement Großskaliger Biologischer Datensätze mit der Integration eines Skalierbaren Rechen-und Dateisystems, master thesis at the Hochschule Karlsruhe and Karlsruhe Institute of Technology, Germany, 2011.
26. C. Trapnell and S. L. Salzberg, How to map billions of short reads onto genomes, Nature Biotechnology 27, 2009, pp. 455-457; http://www.nature.com/nbt/ journal/v27/n5/full/nbt0509-455.html
27. http://www.euforia-project.eu/F. Castejon, M. Cárdenas-Montes, A. Gómez-Iglesias, E. Morales-Ramos, B. Guillerminet, D. P. Coster, E. Sonnendrücker, I. Campos Plasencia, J. øAström, J. Westerholm, J. María Cela, L. Kos, L. Smith, M. Plociennik, M. Hardt, M. Aspnäs, P. Strand, R. Stotzka, EUFORIA: Grid and High Performance Computing at the Service of Fusion Modelling, 2nd Iberian Grid Infrastructure Conference, Porto, Portugal, 2008, ISBN 978-84-9745-288-5, pp. 115-127.
28. T. McPhillips, S. Bowers, D. Zinn, and B. Ludaescher, Scientific workflow design for mere mortals, Future Generation Computer Systems, Vol. 25, 2008, pp. 541-551.
29. http://www.datadirectnet.com/products/web-object-scaler-wos
30. http://nebula.nasa.gov/
31. E. Pourmal, M. Folk, HDF5 Data Model, File Format and Library - HDF5 1.6, ESDS-RFC-007v1 Recommended Standard, 2007; http://www.esdswg.org/spg/rfc/ese- rfc-007/ESDS-RFC-007v1.pdf