Comet: Batched stream processing for data intensive distributed computing

Proceedings of the 1st ACM Symposium on Cloud Computing, SoCC '10

Volumn , Issue , 2010, Pages 63-74

  He, Bingsheng ^a   Yang, Mao ^a   Guo, Zhenyu ^a   Chen, Rishan ^b   Su, Bing ^a   Lin, Wei ^c   Zhou, Lidong ^a  

^a MICROSOFT RESEARCH ASIA   (China)

^b PEKING UNIVERSITY   (China)

^c MICROSOFT   (United States)

Author keywords

Batched stream processing; Data intensive scalable computing; Query series; Resource management

Indexed keywords

BATCH PROCESSING; COMPLEMENTARY METHODS; DATA INTENSIVE; DATA STREAM; DISTRIBUTED COMPUTING; DISTRIBUTED DATA PROCESSING; EMPIRICAL STUDIES; LARGE-SCALE PRODUCTION; PROCESSING CLUSTERS; PROTOTYPE SYSTEM; QUERY OPTIMIZATION; QUERY PROCESSING SYSTEM; RESOURCE MANAGEMENT; SCALABLE COMPUTING; STREAM PROCESSING;

DISTRIBUTED COMPUTER SYSTEMS; HYDRAULICS; NATURAL RESOURCES MANAGEMENT; RESOURCE ALLOCATION;

BATCH DATA PROCESSING;

EID: 77954920597     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1807128.1807139     Document Type: Conference Paper

References (32)

1. P. Agrawal, D. Kifer, and C. Olston. Scheduling shared scans of large data files. Proc. VLDB Endow., 1(1):958-969, 2008.
2. S. Agrawal, S. Chaudhuri, and V. R. Narasayya. Automated selection of materialized views and indexes in SQL databases. In VLDB, 2000.
3. B. Babcock, S. Babu, M. Datar, R. Motwani, and J. Widom. Models and issues in data stream systems. In ACM PODS, 2002.
4. J. A. Blakeley, P.-A. Larson, and F. W. Tompa. Efficiently updating materialized views. SIGMOD Rec., 15(2):61-71, 1986.
5. H. Boral, W. Alexander, L. Clay, G. Copeland, S. Danforth, M. Franklin, B. Hart, M. Smith, and P. Valduriez. Prototyping Bubba, a highly parallel database system. IEEE Trans. on Knowl. and Data Eng., 2(1):4-24, 1990.
6. R. Chaiken, B. Jenkins, P.-A. Larson, B. Ramsey, D. Shakib, S. Weaver, and J. Zhou. SCOPE: easy and efficient parallel processing of massive data sets. Proc. VLDB Endow., 1(2), 2008.
7. J. Chen, D. J. Dewitt, F. Tian, and Y. Wang. NiagaraCQ: A scalable continuous query system for internet databases. In ACM SIGMOD, 2000.
8. L. Chen, C. Olston, and R. Ramakrishnan. Parallel evaluation of composite aggregate queries. In ICDE, 2008.
9. L. S. Colby, R. L. Cole, E. Haslam, N. Jazayeri, G. Johnson, W. J. McKenna, L. Schumacher, and D. Wilhite. Redbrick Vista: Aggregate computation and management. In ICDE, 1998.
10. J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. In OSDI, 2004.
11. D. DeWitt and J. Gray. Parallel database systems: the future of high performance database systems. Commun. ACM, 35(6):85-98, 1992.
12. D. J. Dewitt, S. Ghandeharizadeh, D. A. Schneider, A. Bricker, H. I. Hsiao, and R. Rasmussen. The Gamma database machine project. IEEE Trans. on Knowl. and Data Eng., 2(1):44-62, 1990.
13. D. J. DeWitt and M. Stonebraker. Mapreduce: A major step backwards. The Database Column, 1, 2008.
14. S. Ghemawat, H. Gobioff, and S.-T. Leung. The Google file system. SIGOPS Oper. Syst. Rev., 37(5):29-43, 2003.
15. G. Graefe. Query evaluation techniques for large databases. ACM Comput. Surv., 25(2):73-169, 1993.
16. Hadoop. http://hadoop.apache.org/.
17. B. He, M. Yang, Z. Guo, R. Chen, W. Lin, B. Su, H. Wang, and L. Zhou. Wave computing in the cloud. In HotOS, 2009.
18. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: distributed data-parallel programs from sequential building blocks. SIGOPS Oper. Syst. Rev., 41(3):59-72, 2007.
19. C. Olston, B. Reed, A. Silberstein, and U. Srivastava. Automatic optimization of parallel dataflow programs. In USENIX ATC, 2008.
20. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig Latin: a not-so-foreign language for data processing. In ACM SIGMOD, 2008.
21. R. Pike, S. Dorward, R. Griesemer, and S. Quinlan. Interpreting the data: Parallel analysis with Sawzall. Sci. Program., 13(4), 2005.
22. L. Popa, M. Budiu, Y. Yu, and M. Isard. DryadInc: Reusing work in large-scale computations. In HotCloud, 2009.
23. G. Ramalingam and T. Reps. A categorized bibliography on incremental computation. In POPL, 1993.
24. P. Roy, S. Seshadri, S. Sudarshan, and S. Bhobe. Efficient and extensible algorithms for multi query optimization. SIGMOD Rec., 29(2), 2000.
25. P. G. Selinger, M. M. Astrahan, D. D. Chamberlin, R. A. Lorie, and T. G. Price. Access path selection in a relational database management system. In ACM SIGMOD, 1979.
26. T. K. Sellis. Multiple-query optimization. ACM Trans. Database Syst., 13(1):23-52, 1988.
27. D. Terry, D. Goldberg, D. Nichols, and B. Oki. Continuous queries over append-only databases. In ACM SIGMOD, 1992.
28. The Stanford STREAM Group. STREAM: The stanford stream data manager. IEEE Data Engineering Bulletin, 26(1), 2003.
29. X. Wang, R. Burns, and T. Malik. Liferaft: Data-driven, batch processing for the exploration of scientific databases. In CIDR, 2009.
30. Y. Yu, M. Isard, D. Fetterly, M. Budiu, U. Erlingsson, P. K. Gunda, and J. Currey. DryadLINQ: A system for general-purpose distributed data-parallel computing using a high-level language. In OSDI, 2008.
31. J. Zhou, P.-A. Larson, J.-C. Freytag, and W. Lehner. Efficient exploitation of similar subexpressions for query processing. In ACM SIGMOD, 2007.
32. M. Zukowski, S. Héman, N. Nes, and P. Boncz. Cooperative scans: dynamic bandwidth sharing in a dbms. In VLDB, 2007.