Tutorial: Stream processing optimizations

DEBS 2013 - Proceedings of the 7th ACM International Conference on Distributed Event-Based Systems

Volumn , Issue , 2013, Pages 249-258

  Schneider, Scott ^a   Hirzel, Martin ^a   Gedik, Buǧra ^b  

^a IBM T J WATSON RESEARCH CENTER   (United States)

^b BILKENT UNIVERSITY   (Turkey)

Author keywords

Data parallelism; Fission; Optimizaition; Stream processing

Indexed keywords

DATA PARALLELISM; DISTRIBUTED STREAMING; FISSION; MULTIPLE MACHINE; OPTIMIZAITION; RESEARCH CHALLENGES; STREAM PROCESSING; STREAMING APPLICATIONS;

INFORMATION MANAGEMENT; SOFTWARE ARCHITECTURE; SURVEYS;

OPTIMIZATION;

EID: 84881175336     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2488222.2488268     Document Type: Conference Paper

References (37)

1. Daniel J. Abadi, Yanif Ahmad, Magdalena Balazinska, Ugur Cetintemel, Mitch Cherniack, Jeong-Hyon Hwang, Wolfgang Lindner, Anurag S. Maskey, Alexander Rasin, Esther Ryvkina, Nesime Tatbul, Ying Xing, and Stan Zdonik. The design of the Borealis stream processing engine. In Conference on Innovative Data Systems Research (CIDR), pages 277-289, 2005.
2. Lisa Amini, Henrique Andrade, Ranjita Bhagwan, Frank Eskesen, Richard King, Philippe Selo, Yoonho Park, and Chitra Venkatramani. SPC: A distributed, scalable platform for data mining. In Workshop on Data Mining Standards, Services and Platforms (DM-SSP), pages 27-37, 2006.
3. Arvind Arasu, Shivnath Babu, and Jennifer Widom. The CQL continuous query language: semantic foundations and query execution. Journal on Very Large Data Bases (VLDB J.), 15(2): 121-142, 2006.
4. Arvind Arasu, Mitch Cherniack, Eduardo Galvez, David Maier, Anurag S. Maskey, Esther Ryvkina, Michael Stonebraker, and Richard Tibbetts. Linear road: A stream data management benchmark. In Very Large Data Bases (VLDB), pages 480-491, 2004.
5. Remzi H. Arpaci-Dusseau, Eric Anderson, Noah Treuhaft, David E. Culler, Joseph M. Hellerstein, David Patterson, and Kathy Yelick. Cluster I/O with River: Making the fast case common. In Workshop on I/O in Parallel and Distributed Systems (IOPADS), pages 10-22, 1999.
6. Ron Avnur and Joseph M. Hellerstein. Eddies: Continuously adaptive query processing. In International Conference on Management of Data (SIGMOD), pages 261-272, 2000.
7. Andrey Brito, Christof Fetzer, Heiko Sturzrehm, and Pascal Felber. Speculative out-of-order event processing with software transaction memory. In Conference on Distributed Event-Based Systems (DEBS), pages 265-275, 2008.
8. Don Carney, Ugur Cetintemel, Alex Rasin, Stan Zdonik, Mitch Cherniack, and Mike Stonebraker. Operator scheduling in a data stream manager. In Very Large Data Bases (VLDB), pages 838-849, 2003.
9. Jianjun Chen, David J. DeWitt, Feng Tian, and Yuan Wang. NiagaraCQ: A scalable continuous query system for internet databases. In International Conference on Management of Data (SIGMOD), pages 379-390, 2000.
10. Jeffrey Dean and Sanjay Ghemawat. MapReduce: Simplified data processing on large clusters. In Operating Systems Design and Implementation (OSDI), pages 137-150, 2004.
11. Charles L. Forgy. Rete: A fast algorithm for the many pattern/many object pattern match problem. Artificial Intelligence, 19: 17-37, 1982.
12. Hector Garcia-Molina, Jeffrey D. Ullman, and Jennifer Widom. Database Systems: The Complete Book. Pearson / Prentice Hall, second edition, 2008.
13. Bugra Gedik, Kun-Lung Wu, and Philip S. Yu. Efficient construction of compact shedding filters for data stream processing. In International Conference on Data Engineering (ICDE), pages 396-405, 2008.
14. Michael I. Gordon, William Thies, and Saman Amarasinghe. Exploiting coarse-grained task, data, and pipeline parallelism in stream programs. In Architectural Support for Programming Languages and Operating Systems (ASPLOS), pages 151-162, 2006.
15. Michael I. Gordon, William Thies, Michal Karczmarek, Jasper Lin, Ali S. Meli, Andrew A. Lamb, Chris Leger, Jeremy Wong, Henry Hoffmann, David Maze, and Saman Amarasinghe. A stream compiler for communication-exposed architectures. In Architectural Support for Programming Languages and Operating Systems (ASPLOS), pages 291-303, 2002.
16. Goetz Graefe. Encapsulation of parallelism in the Volcano query processing system. In International Conference on Management of Data (SIGMOD), pages 102-111, 1990.
17. Martin Hirzel, Henrique Andrade, Bugra Gedik, Vibhore Kumar, Giuliano Losa, Mark Mendell, Howard Nasgaard, Robert Soule, and Kun-Lung Wu. Streams processing language specification. Research Report RC24897, IBM, 2009.
18. Martin Hirzel, Robert Soule, Scott Schneider, Bugra Gedik, and Robert Grimm. A catalog of stream processing optimizations. Research Report RC25215, IBM, 2011.
19. Amir Hormati, Yoonseo Choi, Manjunath Kudlur, Rodric M. Rabbah, Trevor N. Mudge, and Scott A. Mahlke. Flextream: Adaptive compilation of streaming applications for heterogeneous architectures. In Parallel Architectures and Compilation Techniques (PACT), pages 214-223, 2009.
20. Fabian Hueske, Mathias Peters, Matthias J. Sax, Astrid Rheinlander, Rico Bergmann, Aljoscha Krettek, and Kostas Tzoumas. Opening the black boxes in data flow optimization. In Very Large Data Bases (VLDB), pages 1256-1267, 2012.
21. D. Karger, E. Lehman, T. Leighton, R. Panigrahy, M. Levine, and D. Lewin. Consistent hashing and random trees: Distributed caching protocols for relieving hot spots on the world wide web. 1997.
22. Rohit Khandekar, Kirsten Hildrum, Sujay Parekh, Deepak Rajan, Joel Wolf, Lung Kun-Wu, Henrique Andrade, and Bugra Gedik. COLA: Optimizing stream processing applications via graph partitioning. In International Conference on Middleware, pages 308-327, 2009.
23. Francois Labonte, Peter Mattson, William Thies, Ian Buck, Christos Kozyrakis, and Mark Horowitz. The Stream virtual machine. In Parallel Architectures and Compilation Techniques (PACT), pages 267-277, 2004.
24. E. A. Lee and Messerschmitt D. G. Synchronous data flow. Proceedings of the IEEE, 75(9), 1987.
25. Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques. A performance study of event processing systems. In TPC Technology Conference on Performance Evaluation & Benchmarking (TPC TC), pages 221-236, 2009.
26. David Mosberger and Larry L. Peterson. Making paths explicit in the Scout operating system. In Operating Systems Design and Implementation (OSDI), pages 153-167, 1996.
27. Leonardo Neumeyer, Bruce Robbins, Anish Nair, and Anand Kesari. S4: Distributed stream processing platform. In Workshop on Knowledge Discovery Using Cloud and Distributed Computing Platforms (KDCloud), 2010.
28. Guilherme Ottoni, Ram Rangan, Adam Stoler, and David I. August. Automatic thread extraction with decoupled software pipelining. In International Symposium on Microarchitecture (MICRO), pages 105-118, 2005.
29. Peter Pietzuch, Jonathan Ledlie, Jeffrey Shneidman, Mema Roussopoulos, Matt Welsh, and Margo Seltzer. Network-aware operator placement for stream-processing systems. In International Conference on Data Engineering (ICDE), pages 49-61, 2006.
30. Scott Schneider, Martin Hirzel, Bugra Gedik, and Kun-Lung Wu. Auto-parallelizing stateful distributed streaming applications. In International Conference on Parallel Architectures and Compilation Techniques (PACT), pages 53-64, 2012.
31. Janis Sermulins, William Thies, Rodric Rabbah, and Saman Amarasinghe. Cache aware optimization of stream programs. In Languages, Compiler, and Tool Support for Embedded Systems (LCTES), pages 115-126, 2005.
32. Storm. http://storm-project.net/. Retrieved April, 2013.
33. Nesime Tatbul, Ugur Cetintemel, Stan Zdonik, Mitch Cherniack, and Michael Stonebraker. Load shedding in a data stream manager. In Very Large Data Bases (VLDB), pages 309-320, 2003.
34. William Thies and Saman Amarasinghe. An empirical characterization of stream programs and its implications for language and compiler design. In Parallel Architectures and Compilation Techniques (PACT), pages 365-376, 2010.
35. Matt Welsh, David Culler, and Eric Brewer. SEDA: An architecture for well-conditioned, scalable internet services. In Symposium on Operating Systems Principles (SOSP), pages 230-243, 2001.
36. Joel Wolf, Nikhil Bansal, Kirsten Hildrum, Sujay Parekh, Deepak Rajan, Rohit Wagle, Kun-Lung Wu, and Lisa Fleischer. SODA: an optimizing scheduler for large-scale stream-based distributed computer systems. In International Conference on Middleware, pages 306-325, 2008.
37. Yuan Yu, Pradeep Kumar Gunda, and Michael Isard. Distributed aggregation for data-parallel computing: Interfaces and implementations. In Symposium on Operating Systems Principles (SOSP), pages 247-260, 2009.