Using virtual clusters to decouple computation and data management in high throughput analysis applications

Proceedings of the 18th Euromicro Conference on Parallel, Distributed and Network-Based Processing, PDP 2010

Volumn , Issue , 2010, Pages 411-415

  Leo, Simone ^a   Anedda, Paolo ^a   Gaggero, Massimo ^a   Zanetti, Gianluigi ^a  

^a CRS4 ^*  (Italy)

Author keywords

[No Author keywords available]

Indexed keywords

COST RATIO; DATA MANAGEMENT; DATA SETS; DATA-INTENSIVE APPLICATION; DECOUPLE COMPUTATION; EXPERIMENTAL DATA; HIGH-THROUGHPUT ANALYSIS; INTERNAL STRUCTURE; MULTIPLE RESEARCH; RAPID GROWTH; SCIENTIFIC DATA; SOFTWARE STACKS; STORAGE INFRASTRUCTURE; VIRTUAL CLUSTERS;

IMAGE ANALYSIS; INTERCONNECTION NETWORKS; THROUGHPUT;

DISTRIBUTED COMPUTER SYSTEMS;

EID: 77952632769     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/PDP.2010.29     Document Type: Conference Paper

References (19)

1. P. t Hoen, Y. Ariyurek, H. Thygesen, E. Vreugdenhil, R. Vossen, R. de Menezes, J. Boer, G. van Ommen, and J. den Dunnen, "Deep sequencing-based expression analysis shows major advances in robustness, resolution and inter-lab portability over five microarray platforms," Nucleic acids research, vol. 36, no. 21, 2008.
2. C. L. Borgman, J. C. Wallis, M. S. Mayernik, and A. Pepe, "Drowning in data: digital library architecture to support scientific use of embedded sensor networks," in JCDL '07: Proceedings of the 7th ACM/IEEE-CS joint conference on Digital libraries, 2007, pp. 269-277.
3. I. Foster, T. Freeman, K. Keahey, D. Scheftner, B. Sotomayor, and X. Zhang, "Virtual clusters for grid communities," in CCGRID '06: Sixth IEEE International Symposium on Cluster Computing and the Grid, 2006.
4. J. Dean and S. Ghemawat, "MapReduce: simplified data processing on large clusters," in Proceedings of the 6th symposium on operating system design and implementation (OSDI 2004), 2004.
5. P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield, "Xen and the art of virtualization," SIGOPS Oper. Syst. Rev., vol. 37, no. 5, pp. 164-177, 2003.
6. Hadoop. [Online]. Available: http://hadoop.apache.org
7. H. Nishimura, N. Maruyama, and S. Matsuoka, "Virtual clusters on the fly - fast, scalable, and flexible installation," in CCGRID '07: 7th IEEE International Symposium on Cluster Computing and the Grid, 2007.
8. K. Begnum and M. Disney, "Scalable deployment and configuration of high-performance virtual clusters," in CISE/CGCS 2006: 3rd International Conference on Cluster and Grid Computing Systems, 2006.
9. D. Wolinsky, A. Agrawal, P. Boykin, J. Davis, A. Ganguly, V. Paramygin, P. Sheng, , and R. Figueiredo, "On the design of virtual machine sandboxes for distributed computing in wide-area overlays of virtual workstations," in First IEEE workshop on virtualization technologies in distributed computing (VTDC), 2006, pp. 8-8
10. VDOOP. [Online]. Available: http://www.vforvdoop.com
11. Amazon web services. [Online]. Available: http://aws.amazon.com
12. S. Ibrahim, H. Jin, B. Cheng, H. Cao, S. Wu, and L. Qi, "CLOUDLET: towards MapReduce implementation on virtual machines," in Proceedings of the 18th ACM international symposium on High performance distributed computing, 2009, pp. 65-66.
13. Illumina. [Online]. Available: http://www.illumina.com
14. D. Bentley, "Whole-genome re-sequencing," Current opinion in genetics & development, vol. 16, no. 6, pp. 545-552, 2006.
15. Pydoop: a Python MapReduce and HDFS API for Hadoop. [Online]. Available: http://pydoop.sourceforge.net
16. D. Abrahams and R. Grosse-Kunstleve, "Building hybrid systems with Boost.Python," C/C++ Users Journal, vol. 21, no. 7, pp. 29-36, 2003.
17. R. M. I. Llorente, R. Moreno-Vozmediano, "Cloud computing for on-demand grid resource provisioning," Advances in Parallel Computing, 2009, in press.
18. W. Gentzsch, "Sun grid engine: towards creating a compute power grid," in Proceedings of the first IEEE/ACM International Symposium on Cluster Computing and the Grid, 2001, 2001, pp. 35-36.
19. Hadoop on demand. [Online]. Available: http://hadoop.apache.org/core/ docs/current/hod-user-guide.html