Stream as you go: The case for incremental data access and processing in the cloud

Proceedings - 2012 IEEE 28th International Conference on Data Engineering Workshops, ICDEW 2012

Volumn , Issue , 2012, Pages 159-166

  Kienzler, Romeo ^a   Bruggmann, Rémy ^b   Ranganathan, Anand ^c   Tatbul, Nesime ^a  

^a ETH ZURICH   (Switzerland)

^b UNIVERSITY OF BERN   (Switzerland)

^c IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AMAZON EC2; COMPUTING PLATFORM; COST-EFFECTIVE SOLUTIONS; DATA-INTENSIVE APPLICATION; DNA SEQUENCE ANALYSIS; INCREMENTAL DATA; INFOSPHERE; MAP-REDUCE; PROCESSING PROBLEMS; STATE OF THE ART; STREAM DATA; TOTAL PROCESSING TIME; TRANSFER LATENCY;

DATA TRANSFER; DNA SEQUENCES; INFORMATION MANAGEMENT; TECHNICAL PRESENTATIONS;

DATA HANDLING;

EID: 84869007066     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICDEW.2012.69     Document Type: Conference Paper

References (36)

1. G. H. Fernald, E. Capriotti, R. Daneshjou, K. J. Karczewski, and R. B. Altman, "Bioinformatics Challenges for Personalized Medicine," Bioinformatics, vol. 27, no. 13, 2011.
2. G. Graefe, "Query Evaluation Techniques for Large Databases," ACM Computing Surveys, vol. 25, no. 2, 1993.
3. R. Kienzler, R. Bruggmann, A. Ranganathan, and N. Tatbul, "Largescale DNA Sequence Analysis in the Cloud: A Stream-based Approach," in Euro-Par 6th Workshop on Virtualization in High-Performance Cloud Computing (VHPC'11), August 2011.
4. B. Langmead, C. Trapnell, M. Pop, and S. Salzberg, "Ultrafast and Memory-efficient Alignment of Short DNA Sequences to the Human Genome," Genome Biology, vol. 10, no. 3, 2009.
5. R. Li, Y. Li, X. Fang, H. Yang, J. Wang, K. Kristiansen, and J. Wang, "SNP Detection for Massively Parallel Whole-genome Resequencing," Genome Research, vol. 19, no. 6, 2009.
6. S. M. Rumble, P. Lacroute, A. V. Dalca, M. Fiume, A. Sidow, and M. Brudno, "SHRiMP: Accurate Mapping of Short Color-space Reads," PLoS Computational Biology, vol. 5, no. 5, 2009.
7. B. Langmead, M. C. Schatz, J. Lin, M. Pop, and S. L. Salzberg, "Searching for SNPs with Cloud Computing," Genome Biology, vol. 10, no. 11, 2009.
8. "IBM InfoSphere Streams," http://www.ibm.com/software/data/ infosphere/streams/.
9. "Amazon Elastic Compute Cloud," http://aws.amazon.com/ec2/.
10. K. V. Voelkerding, S. A. Dames, and J. D. Durtschi, "Next Generation Sequencing: From Basic Research to Diagnostics," Clinical Chemistry, vol. 55, no. 4, 2009.
11. H. Li and N. Homer, "A Survey of Sequence Alignment Algorithms for Next Generation Sequencing," Briefings in Bioinformatics, vol. 11, no. 5, 2010.
12. F. S. Collins, M. Guyer, and A. Chakravarti, "Variations on a Theme: Cataloging Human DNA Sequence Variation," Science, vol. 278, no. 5343, 1997.
13. M. C. Schatz, "CloudBurst: Highly Sensitive Read Mapping with MapReduce," Bioinformatics, vol. 25, no. 11, 2009.
14. "FASTQ Format Specification," http://maq.sourceforge.net/fastq. shtml.
15. "The Hadoop Distributed File System: Architecture and Design," http://hadoop.apache.org/common/docs/r0.18.3/hdfsdesign.pdf.
16. J.-H. Boese, A. Andrzejak, and M. Hoegqvist, "Beyond Online Aggregation: Parallel and Incremental Data Mining with Online Map-Reduce," in WWW Workshop on Massive Data Analytics over the Cloud (MDAC'10), April 2010.
17. T. Condie, N. Conway, P. Alvaro, J. Hellerstein, J. Gerth, J. Talbot, K. Elmeleegy, and R. Sears, "Online Aggregation and Continuous Query Support in MapReduce," in ACM SIGMOD Conference, June 2010.
18. J. Dittrich, J. A. Quiané-Ruiz, A. Jindal, Y. Kargin, V. Setty, and J. Schad, "Hadoop++: Making a Yellow Elephant Run Like a Cheetah (Without It Even Noticing)," PVLDB, vol. 3, no. 1, 2010.
19. R. Chen, H. Chen, and B. Zang, "Tiled-MapReduce: Optimizing Resource Usages of Data-parallel Applications on Multicore with Tiling," in International Conference on Parallel Architectures and Compilation Techniques (PACT'10), September 2010.
20. A. Verma, N. Zea, B. Cho, I. Gupta, and R. Campbell, "Breaking the MapReduce Stage Barrier," in IEEE International Conference on Cluster Computing (CLUSTER'10), October 2010.
21. M. Elteir, H. Lin, and W. chun Feng, "Enhancing MapReduce via Asynchronous Data Processing," in IEEE International Conference on Parallel and Distributed Systems (ICPADS'10), December 2010.
22. Y. Zhang, Q. Gao, L. Gao, and C. Wang, "iMapReduce: A Distributed Computing Framework for Iterative Computation," in IPDPS DataCloud Workshop, May 2011.
23. C. Olston, G. Chiou, L. Chitnis, F. Liu, Y. Han, M. Larsson, A. Neumann, V. Rao, V. Sankarasubramanian, S. Seth, C. Tian, T. ZiCornell, and X. Wang, "Nova: Continuous Pig/Hadoop Workflows," in ACM SIGMOD Conference, June 2011.
24. N. Pansare, V. Borkar, C. Jermaine, and T. Condie, "Online Aggregation for Large MapReduce Jobs," PVLDB, vol. 4, no. 11, 2011.
25. B. Li, E. Mazur, Y. Diao, A. McGregor, and P. Shenoy, "A Platform for Scalable One-pass Analytics using MapReduce," in ACM SIGMOD Conference, June 2011.
26. R. Lammel and D. Saile, "MapReduce with Deltas," in International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA'11), July 2011.
27. P. Bhatotia, A. Wieder, R. Rodrigues, U. A. Acar, and R. Pasquini, "Incoop: MapReduce for Incremental Computations," in ACM Symposium on Cloud Computing (SOCC'11), October 2011.
28. N. Backman, K. Pattabiraman, and U. Cetintemel, "C-MR: A Continuous MapReduce Processing Model for Low-Latency Stream Processing on Multi-Core Architectures," Brown University, Tech. Rep. CS-10-01, 2010.
29. D. Logothetis, C. Trezzo, K. Webb, and K. Yocum, "In-situ MapReduce for Log Processing," in USENIX Annual Technical Conference, June 2011.
30. "Zebedee Secure Tunnel," http://sourceforge.net/projects/ zebedee.
31. "Whole Genome Sequencing," http://en.wikipedia.org/wiki/ Fullgenomesequencing.
32. "Whole Exome Sequencing," http://en.wikipedia.org/wiki/ Exomesequencing.
33. "RNA-Seq," http://en.wikipedia.org/wiki/RNA-Seq.
34. "Red-Black Trees," http://en.wikipedia.org/wiki/Red-blacktree.
35. "Crossbow," http://bowtie-bio.sourceforge.net/crossbow/.
36. "Hadoop Streaming," http://hadoop.apache.org/common/docs/r0.15. 2/streaming.html.