Summingbird: A framework for integrating batch and online MapReduce computations

Proceedings of the VLDB Endowment

Volumn 7, Issue 13, 2014, Pages 1441-1451

  Boykin, Oscar ^a   Ritchie, Sam ^a   O'Connell, Ian ^a   Lin, Jimmy ^a  

^a TWITTER INC   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ABSTRACTING; PROBLEM ORIENTED LANGUAGES; SOCIAL NETWORKING (ONLINE);

ALGEBRAIC STRUCTURES; DOMAIN SPECIFIC LANGUAGES; EXECUTION PLATFORMS; HYBRID PROCESSING; MESSAGE QUEUES; ONLINE PROCESSING; PROGRAM LOGIC; THEORETICAL FOUNDATIONS;

BATCH DATA PROCESSING;

EID: 84905858499     PISSN: None     EISSN: 21508097     Source Type: Conference Proceeding    
DOI: 10.14778/2733004.2733016     Document Type: Conference Paper

References (40)

1. T. Akidau, A. Balikov, K. Bekiroǧlu, S. Chernyak, J. Haberman, R. Lax, S. McVeety, D. Mills, P. Nordstrom, and S. Whittle. MillWheel: Fault-tolerant stream processing at internet scale. VLDB, 2013.
2. R. Ananthanarayanan, V. Basker, S. Das, A. Gupta, H. Jiang, T. Qiu, A. Reznichenko, D. Ryabkov, M. Singh, and S. Venkataraman. Photon: Fault-tolerant and scalable joining of continuous data streams. SIGMOD, 2013.
3. P. Bhatotia, A. Wieder, R. Rodrigues, U. A. Acar, and R. Pasquini. Incoop: MapReduce for incremental computations. SoCC, 2011.
4. B. Bloom. Space/time trade-offs in hash coding with allowable errors. Communications of the ACM, 13(7):422-426, 1970.
5. L. Bottou. Large-scale machine learning with stochastic gradient descent. COMPSTAT, 2010.
6. A. Z. Broder. On the resemblance and containment of documents. SEQUENCES, 1997.
7. D. Carney, U. Çetintemel, M. Cherniack, C. Convey, S. Lee, G. Seidman, M. Stonebraker, N. Tatbul, and S. Zdonik. Monitoring streams-a new class of data management applications. VLDB, 2002.
8. T. Condie, N. Conway, P. Alvaro, J. M. Hellerstein, K. Elmeleegy, and R. Sears. MapReduce online. NSDI, 2010.
9. G. Cormode and S. Muthukrishnan. An improved data stream summary: The count-min sketch and its applications. Journal of Algorithms, 55(1):58-75, 2005.
10. P. Flajolet, É. Fusy, O. Gandouet, and F. Meunier. Hyperloglog: The analysis of a near-optimal cardinality estimation algorithm. Conference on Analysis of Algorithms, 2007.
11. B. Gedik, H. Andrade, K.-L. Wu, P. Yu, and M. Doo. SPADE: The System S declarative stream processing engine. SIGMOD, 2008.
12. J. Gehrke. Special issue on data stream processing. Bulletin of the Technical Committee on Data Engineering, 26(1):2, 2003.
13. M. Grinev, M. Grineva, M. Hentschel, and D. Kossmann. Analytics for the real-time web. VLDB, 2011.
14. M. Hayes and S. Shah. Hourglass: A library for incremental processing on Hadoop. Big Data, 2013.
15. S. Heule, M. Nunkesser, and A. Hall. HyperLogLog in practice: Algorithmic engineering of a state of the art cardinality estimation algorithm. EDBT, 2013.
16. B. Hindman, A. Konwinski, M. Zaharia, A. Ghodsi, A. D. Joseph, R. Katz, S. Shenker, and I. Stoica. Mesos: A platform for fine-grained resource sharing in the data center. NSDI, 2011.
17. M. Izbicki. Algebraic classifiers: A generic approach to fast cross-validation, online training, and parallel training. ICML, 2013.
18. J. Kreps, N. Narkhede, and J. Rao. Kafka: A distributed messaging system for log processing. NetDB, 2011.
19. S. Krishnamurthy, M. Franklin, J. Davis, D. Farina, P. Golovko, A. Li, and N. Thombre. Continuous analytics over discontinuous streams. SIGMOD, 2010.
20. V. Kumar, H. Andrade, B. Gedik, and K.-L. Wu. DEDUCE: At the intersection of MapReduce and stream processing. EDBT, 2010.
21. W. Lam, L. Liu, S. Prasad, A. Rajaraman, Z. Vacheri, and A. Doan. Muppet: MapReduce-style processing of fast data. VLDB, 2012.
22. B. Lampson. Hints for computer system design. SOSP, 1983.
23. G. Lee, J. Lin, C. Liu, A. Lorek, and D. Ryaboy. The unified logging infrastructure for data analytics at Twitter. VLDB, 2012.
24. J. Lin and A. Kolcz. Large-scale machine learning at Twitter. SIGMOD, 2012.
25. J. Lin and D. Ryaboy. Scaling big data mining infrastructure: The Twitter experience. SIGKDD Explorations, 14(2):6-19, 2012.
26. D. Logothetis, C. Olston, B. Reed, K. C. Webb, and K. Yocum. Stateful bulk processing for incremental analytics. SoCC, 2010.
27. E. Meijer and G. Bierman. A co-relational model of data for large shared data banks. Communications of the ACM, 54(4):49-58, 2011.
28. G. Mishne, J. Dalton, Z. Li, A. Sharma, and J. Lin. Fast data in the era of big data: Twitter's real-time related query suggestion architecture. SIGMOD, 2013.
29. S. A. Myers, A. Sharma, P. Gupta, and J. Lin. Information network or social network? The structure of the Twitter follow graph. WWW Companion, 2014.
30. D. Navalho, S. Duarte, N. Preguiça, and M. Shapiro. Incremental stream processing using computational conflict-free replicated data types. CloudDP, 2013.
31. L. Neumeyer, B. Robbins, A. Nair, and A. Kesari. S4: Distributed stream computing platform. KDCloud Workshop at ICDM, 2010.
32. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig Latin: A not-so-foreign language for data processing. SIGMOD, 2008.
33. R. Pike, S. Dorward, R. Griesemer, and S. Quinlan. Interpreting the data: Parallel analysis with Sawzall. Scientific Programming Journal, 13(4):277-298, 2005.
34. Z. Qian, Y. He, C. Su, Z. Wu, H. Zhu, T. Zhang, L. Zhou, Y. Yu, and Z. Zhang. TimeStream: Reliable stream computation in the cloud. EuroSys, 2013.
35. M. Shapiro, N. Preguiça, C. Baquero, and M. Zawirski. A comprehensive study of Convergent and Commutative Replicated Data Types. Technical report, INRIA, 2011.
36. D. Simoncelli, M. Dusi, F. Gringoli, and S. Niccolini. Stream-monitoring with BlockMon: Convergence of network measurements and data analytics platforms. ACM SIGCOMM Computer Communication Review, 43(2):30-35, 2013.
37. A. Toshniwal, S. Taneja, A. Shukla, K. Ramasamy, J. M. Patel, S. Kulkarni, J. Jackson, K. Gade, M. Fu, J. Donham, N. Bhagat, S. Mittal, and D. Ryaboy. Storm @Twitter. SIGMOD, 2014.
38. Y. Yu, M. Isard, D. Fetterly, M. Budiu, Ú. Erlingsson, P. K. Gunda, and J. Currey. DryadLINQ: A system for general-purpose distributed data-parallel computing using a high-level language. OSDI, 2008.
39. M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J. Franklin, S. Shenker, and I. Stoica. Resilient Distributed Datasets: A fault-tolerant abstraction for in-memory cluster computing. NSDI, 2012.
40. M. Zaharia, T. Das, H. Li, S. Shenker, and I. Stoica. Discretized streams: An efficient and fault-tolerant model for stream processing on large clusters. HotCloud, 2012.