Stateful bulk processing for incrementalincremental analytics

Proceedings of the 1st ACM Symposium on Cloud Computing, SoCC '10

Volumn , Issue , 2010, Pages 51-62

  Logothetis, Dionysios ^a   Olston, Christopher ^b   Reed, Benjamin ^b   Webb, Kevin C ^a   Yocum, Ken ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b YAHOO RESEARCH   (United States)

Author keywords

Algorithms; Design; Performance

Indexed keywords

CURRENT PRACTICES; DATA MOVEMENTS; DATA PARALLEL; DATA SETS; DATA-INTENSIVE APPLICATION; DATAFLOW; DESIGN PERFORMANCE; EVOLVING GRAPHS; HARDWARE RESOURCES; LEVEL OF ABSTRACTION; LOG FILE; MULTI-STEP; PAGERANK; PROCESSING SYSTEMS; REAL WORLD DATA; RUNNING TIME; SOCIAL NETWORKS;

DATA FLOW ANALYSIS; DATA PROCESSING; DISTRIBUTED COMPUTER SYSTEMS; MATHEMATICAL OPERATORS; PARALLEL ARCHITECTURES;

DATA REDUCTION;

EID: 77954926935     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1807128.1807138     Document Type: Conference Paper

References (28)

1. The Hive project. http://hadoop.apache.org/hive/.
2. B. Babcock, S. Babu, M. Datar, R. Motwani, and J. Widom. Models and issues in data stream systems. In Proceedings of PODS, March 2002.
3. J. A. Blakeley, P.-A. Larson, and F. W. Tompa. Efficiently updating materialized views. ACM SIGMOD Record, 15(2):61-71, 1986.
4. P. Boldi and S. Vigna. The WebGraph framework I: Compression techniques. In Proceedings of WWW'04, pages 595-601, Manhattan, USA, 2004. ACM Press.
5. S. Brin and L. Page. The anatomy of a large-scale hypertextual (web) search engine. 30(7):107-117, 1998.
6. R. Chaiken, B. Jenkins, P.-A. Larson, B. Ramsey, D. Shakib, S. Weaver, and J. Zhou. SCOPE: Easy and efficient parallel processing of massive data sets. In Proceedings of VLDB, August 2008.
7. F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Burrows, T. Ch, A. Fikes, and R. E. Gruber. Bigtable: A Distributed Storage System for Structured Data. In Proceedings of OSDI, pages 205-218, Seattle, WA, November 2006.
8. S. Chien, C. Dwork, R. Kumar, D. R. Simon, and D. Sivakumar. Link evolution: Analysis and algorithms. Internet Mathematics, 1(3), November 2004.
9. T. Condie, N. Conway, P. Alvaro, and J. M. Hellerstein. MapReduce online. In Proceedings of NSDI, April 2010.
10. J. Dean and S. Ghemawat. Mapreduce: Simplified data processing on large clusters. In Proceedings of OSDI, San Francisco, CA, December 2004.
11. J. Ekanayake and G. Fox. High performance parallel computing with clouds and cloud technologies. In 1st International Conference on Cloud Computing (CloudComp09), 2009.
12. M. Ester, H. Kriegal, J. Sander, M. Wimmer, and X. Xu. Incremental clustering and mining in a data warehousing environment. In Proceedings of VLDB, 1998.
13. S. Ghemawat, H. Gogioff, and S. Leung. The Google file system. In Proceedings of SOSP, December 2003.
14. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: Distributed data-parallel programs from sequential building blocks. In Proceedings of EuroSys, March 2007.
15. D. Logothetis, C. Olston, B. Reed, and K. Yocum. Programming bulk-incremental dataflows. Technical Report CS2009-0944, UCSD, June 2009.
16. D. Logothetis and K. Yocum. Data indexing for stateful, large-scale data processing. In Proceedings of NetDB, October 2009.
17. G. Malewicz, M. H. Austern, A. J. C. Bik, J. C. Dehnert, I. Horn, N. Leiser, and G. Czajkowski. Pregel: A system for large-scale graph processing. In Talk abstract in the Proceedings of PODC, August 2009.
18. G. S. Manku and R. Motwani. Approximate frequency counts over data streams. In Proceedings of VLDB, 2002.
19. M. F. Mokbel, X. Xiong, and W. G. Aref. SINA: Scalable incremental processing of continuous queries in a spatio-temporal databases. In Proc. of SIGMOD, June 2004.
20. C. Monash. Facebook, Hadoop, and Hive, May 2009. http://www.dbms2.com/ 2009/05/11/facebook-hadoop-and-hive.
21. H. Ning, W. Xu, Y. Chi, Y. Gong, and T. Huang. Incremental spectral clustering with application to monitoring of evolving blog communities. In SIAM Int. Conf. on Data Mining, 2007.
22. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig Latin: A not-so-foreign language for data processing. In Proceedings of SIGMOD, June 2008.
23. B. Pariseau. IDC unstructured data will become the primary task for storage, October 2008. http://itknowledgeexchange.techtarget.com/storage-soup/ idc-unstructured-data-willbecome-the-primary-task-for-storage/.
24. L. Popa, M. Budiu, Y. Yu, and M. Isard. DryadInc:reusing work in large-scale computations. In HotCloud Workshop, June 2009.
25. X. Qian and G. Wiederhold. Incremental recomputation of active relational expressions. IEEE Transactions on Knowledge and Data Engineering, 3(3), September 1991.
26. C. Wilson, B. Boe, A. Sala, K. P. N. Puttaswamy, and B. Y. Zhao. User interactions in social networks and their implications. In Proceedings of EuroSys), April 2009.
27. Windows Azure and Facebook teams. Personal communications, August 2008.
28. Y. Yu, M. Isard, D. Fetterly, M. Budiu, U. Erlingsson, P. K. Gunda,, and J. Currey. DryadLINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language. In Proceedings of OSDI, San Diego, CA, December 2008.