Discretized streams: Fault-tolerant streaming computation at scale

SOSP 2013 - Proceedings of the 24th ACM Symposium on Operating Systems Principles

Volumn , Issue , 2013, Pages 423-438

  Zaharia, Matei ^a   Das, Tathagata ^a   Li, Haoyuan ^a   Hunter, Timothy ^a   Shenker, Scott ^a   Stoica, Ion ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DISTRIBUTED STREAM PROCESSING; FAULT-TOLERANT; INTERACTIVE QUERIES; LINEAR SCALING; PARALLEL PLATFORMS; PROCESSING MODEL; RECOVERY MECHANISMS; STREAMING COMPUTATIONS;

DISTRIBUTED PARAMETER CONTROL SYSTEMS; RECOVERY;

FAULT TOLERANT COMPUTER SYSTEMS;

EID: 84889637396     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2517349.2522737     Document Type: Conference Paper

References (42)

1. T. Akidau, A. Balikov, K. Bekiroglu, S. Chernyak, J. Haberman, R. Lax, S. McVeety, D. Mills, P. Nordstrom, and S. Whittle. MillWheel: Fault-tolerant stream processing at internet scale. In VLDB, 2013.
2. M. H. Ali, C. Gerea, B. S. Raman, B. Sezgin, T. Tarnavski, T. Verona, P. Wang, P. Zabback, A. Ananthanarayan, A. Kirilov, M. Lu, A. Raizman, R. Krishnan, R. Schindlauer, T. Grabs, S. Bjeletich, B. Chandramouli, J. Goldstein, S. Bhat, Y. Li, V. Di Nicola, X. Wang, D. Maier, S. Grell, O. Nano, and I. Santos. Microsoft CEP server and online behavioral targeting. Proc. VLDB Endow., 2(2):1558, Aug. 2009.
3. Apache Flume. http://incubator.apache.org/flume/.
4. A. Arasu, B. Babcock, S. Babu, M. Datar, K. Ito, I. Nishizawa, J. Rosenstein, and J. Widom. STREAM: The Stanford stream data management system. SIGMOD 2003.
5. M. Balazinska, H. Balakrishnan, S. R. Madden, and M. Stonebraker. Fault-tolerance in the Borealis distributed stream processing system. ACM Trans. Database Syst., 2008.
6. P. Bhatotia, A. Wieder, R. Rodrigues, U. A. Acar, and R. Pasquin. Incoop: MapReduce for incremental computations. In SOCC '11, 2011.
7. D. Carney, U. Çetintemel, M. Cherniack, C. Convey, S. Lee, G. Seidman, M. Stonebraker, N. Tatbul, and S. Zdonik. Monitoring streams: a new class of data management applications. In VLDB '02, 2002.
8. S. Chandrasekaran, O. Cooper, A. Deshpande, M. J. Franklin, J. M. Hellerstein, W. Hong, S. Krishnamurthy, S. Madden, V. Raman, F. Reiss, and M. Shah. TelegraphCQ: Continuous dataflow processing for an uncertain world. In CIDR, 2003.
9. M. Cherniack, H. Balakrishnan, M. Balazinska, D. Carney, U. Cetintemel, Y. Xing, and S. B. Zdonik. Scalable distributed stream processing. In CIDR, 2003.
10. T. Condie, N. Conway, P. Alvaro, and J. M. Hellerstein. MapReduce online. NSDI, 2010.
11. J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. In OSDI, 2004.
12. EsperTech. Performance-related information. http://esper.codehaus.org/ esper/performance/performance.html, Retrieved March 2013.
13. EsperTech. Tutorial. http://esper.codehaus.org/tutorials/tutorial/ tutorial.html, Retrieved March 2013.
14. M. Franklin, S. Krishnamurthy, N. Conway, A. Li, A. Russakovsky, and N. Thombre. Continuous analytics: Rethinking query processing in a networkeffect world. CIDR, 2009.
15. S. Ghemawat, H. Gobioff, and S.-T. Leung. The Google File System. In Proceedings of SOSP '03, 2003.
16. J. Hammerbacher. Who is using flume in production? http://www.quora.com/ Flume/Who-is-using-Flume-in-production/answer/Jeff-Hammerbacher.
17. B. He, M. Yang, Z. Guo, R. Chen, B. Su, W. Lin, and L. Zhou. Comet: batched stream processing for data intensive distributed computing. In SoCC, 2010.
18. T. Hunter, T. Moldovan, M. Zaharia, S. Merzgui, J. Ma, M. J. Franklin, P. Abbeel, and A. M. Bayen. Scaling the Mobile Millennium system in the cloud. In SOCC '11, 2011.
19. J.-H. Hwang, M. Balazinska, A. Rasin, U. Cetintemel, M. Stonebraker, and S. Zdonik. High-availability algorithms for distributed stream processing. In ICDE, 2005.
20. J. hyon Hwang, Y. Xing, and S. Zdonik. A cooperative, self-configuring high-availability solution for stream processing. In ICDE, 2007.
21. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: distributed data-parallel programs from sequential building blocks. In EuroSys 07, 2007.
22. S. Krishnamurthy, M. Franklin, J. Davis, D. Farina, P. Golovko, A. Li, and N. Thombre. Continuous analytics over discontinuous streams. In SIGMOD, 2010.
23. D. Logothetis, C. Olston, B. Reed, K. C.Webb, and K. Yocum. Stateful bulk processing for incremental analytics. SoCC, 2010.
24. D. Logothetis, C. Trezzo, K. C. Webb, and K. Yocum. In-situ MapReduce for log processing. In USENIX ATC, 2011.
25. N. Marz. Trident: a high-level abstraction for realtime computation. http://engineering.twitter.com/2012/08/trident-high-level-abstraction-for.html.
26. F. McSherry, D. G. Murray, R. Isaacs, and M. Isard. Differential dataflow. In Conference on Innovative Data Systems Research (CIDR), 2013.
27. D. Murray, F. McSherry, R. Isaacs, M. Isard, P. Barham, and M. Abadi. Naiad: A timely dataflow system. In SOSP '13, 2013.
28. L. Neumeyer, B. Robbins, A. Nair, and A. Kesari. S4: Distributed stream computing platform. In Intl. Workshop on Knowledge Discovery Using Cloud and Distributed Computing Platforms (KDCloud), 2010.
29. D. Ongaro, S. M. Rumble, R. Stutsman, J. K. Ousterhout, and M. Rosenblum. Fast crash recovery in RAMCloud. In SOSP, 2011.
30. Oracle. Oracle complex event processing performance. http://www.oracle. com/technetwork/middleware/complex-event-processing/overview/ cepperformancewhitepaper-128060.pdf, 2008.
31. D. Peng and F. Dabek. Large-scale incremental processing using distributed transactions and notifications. In OSDI 2010.
32. Z. Qian, Y. He, C. Su, Z. Wu, H. Zhu, T. Zhang, L. Zhou, Y. Yu, and Z. Zhang. Timestream: Reliable stream computation in the cloud. In EuroSys '13, 2013.
33. M. Shah, J. Hellerstein, and E. Brewer. Highly available, fault-tolerant, parallel dataflows. SIGMOD, 2004.
34. Z. Shao. Real-time analytics at Facebook. XLDB 2011, http://www-conf. slac.stanford.edu/xldb2011/talks/xldb2011-tue-0940-facebookrealtimeanalytics. pdf.
35. U. Srivastava and J. Widom. Flexible time management in data stream systems. In PODS, 2004.
36. Storm. https://github.com/nathanmarz/storm/wiki.
37. Guaranteed message processing (Storm wiki). https://github.com/ nathanmarz/storm/wiki/Guaranteeing-message-processing.
38. K. Thomas, C. Grier, J. Ma, V. Paxson, and D. Song. Design and evaluation of a real-time URL spam filtering service. In IEEE Symposium on Security and Privacy, 2011.
39. R. Tibbetts. Streambase performance & scalability characterization. http://www.streambase.com/wp-content/uploads/downloads/StreamBase-White-Paper- Performance-and-Scalability-Characterization.pdf, 2009.
40. H. Wang, L.-S. Peh, E. Koukoumidis, S. Tao, and M. C. Chan. Meteor shower: A reliable stream processing system for commodity data centers. In IPDPS '12, 2012.
41. Y. Yu, M. Isard, D. Fetterly, M. Budiu, Ú. Erlingsson, P. K. Gunda, and J. Currey. DryadLINQ: A system for general-purpose distributed dataparallel computing using a high-level language. In OSDI '08, 2008.
42. M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. Franklin, S. Shenker, and I. Stoica. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In NSDI, 2012.