Auto-parallelizing stateful distributed streaming applications

Parallel Architectures and Compilation Techniques - Conference Proceedings, PACT

Volumn , Issue , 2012, Pages 53-63

  Schneider, Scott ^a   Hirzel, Martin ^a   Gedik, Bǔgra G ^b   Wu, Kun Lung ^a  

^a IBM T J WATSON RESEARCH CENTER   (United States)

^b BILKENT UNIVERSITY   (Turkey)

Author keywords

Automatic parallelization; Distributed stream processing

Indexed keywords

AUTO-PARALLELIZING; AUTOMATIC PARALLELIZATION; DATA PARALLELISM; DATA TUPLES; DISTRIBUTED STREAMING; EFFICIENT STRATEGY; HIGH THROUGHPUT; LARGE CLUSTERS; LOW LATENCY; PARALLEL SYSTEM; PROGRAMMING MODELS; RUNTIME SYSTEMS; STREAM PROCESSING; STREAMING APPLICATIONS;

DISTRIBUTED PARAMETER CONTROL SYSTEMS; PARALLEL ARCHITECTURES;

PROGRAM COMPILERS;

EID: 84867568822     PISSN: 1089795X     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2370816.2370826     Document Type: Conference Paper

References (26)

1. L. Amini, H. Andrade, R. Bhagwan, F. Eskesen, R. King, P. Selo, Y. Park, and C. Venkatramani. SPC: A distributed, scalable platform for data mining. In International Workshop on Data Mining Standards, Services, and Platforms, 2006.
2. R. H. Arpaci-Dusseau, E. Anderson, N. Treuhaft, D. E. Culler, J. M. Hellerstein, D. Patterson, and K. Yelick. Cluster I/O with River: Making the fast case common. In Workshop on I/O in Parallel and Distributed Systems (IOPADS), 1999.
3. A. Brito, C. Fetzer, H. Sturzrehm, and P. Felber. Speculative out-of-order event processing with software transaction memory. In Conference on Distributed Event-Based Systems (DEBS), 2008.
4. C. Chambers, A. Raniwala, F. Perry, S. Adams, R. R. Henry, R. Bradshaw, and N. Weizenbaum. FlumeJava: easy, efficient data-parallel pipelines. In Proceedings of the 2010 ACM SIGPLAN conference on Programming Language Design and Implementation (PLDI), 2010.
5. L. Chiticariu, R. Krishnamurthy, Y. Li, S. Raghavan, F. R. Reiss, and S. Vaithyanathan. SystemT: an algebraic approach to declarative information extraction. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL), 2010.
6. R. L. Collins and L. P. Carloni. Flexible filters: Load balancing through backpressure for stream programs. In International Conference on Embedded Software (EMSOFT), 2009.
7. T. Condie, N. Conway, P. Alvaro, J. M. Hellerstein, K. Elmeleegy, and R. Sears. MapReduce online. In Networked Systems Design and Implementation, 2010.
8. J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. In Operating Systems Design and Implementation (OSDI), 2004.
9. D. J. Dewitt, S. Ghandeharizadeh, D. A. Schneider, A. Bricker, H. I. Hsiao, and R. Rasmussen. The Gamma database machine project. Transactions on Knowledge and Data Engineering, 2(1), 1990.
10. M. I. Gordon, W. Thies, and S. Amarasinghe. Exploiting coarse-grained task, data, and pipeline parallelism in stream programs. In Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2006.
11. G. Graefe. Encapsulation of parallelism in the Volcano query processing system. In International Conference on Management of Data (SIGMOD), 1990.
12. M. Hirzel. Partition and compose: Parallel complex event processing. In Conference on Distributed Event-Based Systems (DEBS), 2012.
13. M. Hirzel, H. Andrade, B. Gedik, V. Kumar, G. Losa, M. Mendell, H. Nasgaard, R. Soulé, and K.-L. Wu. Streams processing language specification. Research Report RC24897, IBM, 2009.
14. M. Isard, M. B. Y. Yu, A. Birrell, and D. Fetterly. Dryad: Distributed data-parallel program from sequential building blocks. In European Conference on Computer Systems (EuroSys), 2007.
15. R. Khandekar, I. Hildrum, S. Parekh, D. Rajan, J. Wolf, K.-L. Wu, H. Andrade, and B. Gedik. COLA: Optimizing stream processing applications via graph partitioning. In International Conference on Middleware, 2009.
16. E. A. Lee and D. G. Messerschmitt. Synchronous data flow. Proceedings of the IEEE, 75(9), 1987.
17. L. Neumeyer, B. Robbins, A. Nair, and A. Kesari. S4: Distributed stream processing platform. In KDCloud, 2010.
18. L. Page, S. Brin, R. Motwani, and T. Winograd. The pagerank citation ranking: Bringing order to the web. Technical Report 1999-66, Stanford InfoLab, November 1999.
19. D. Peng and F. Dabek. Large-scale incremental processing using distributed transactions and notifications. In Operating Systems Design and Implementation (OSDI), 2010.
20. Pig. http://pig.apache.org/Retrieved July, 2012.
21. E. Raman, G. Ottoni, A. Raman, M. J. Bridges, and D. I. August. Parallel-stage decoupled software pipelining. In Code Generation and Optimization (CGO), 2008.
22. S. Schneider, H. Andrade, B. Gedik, A. Biem, and K.-L. Wu. Elastic scaling of data parallel operators in stream processing. In International Parallel & Distributed Processing Symposium (IPDPS), 2009.
23. M. A. Shah, J. M. Hellerstein, and E. Brewer. Highly Available, Fault-Tolerant, Parallel Dataflows. In International Conference on Management of Data (SIGMOD), 2004.
24. Storm. http://storm-project.net/. Retrieved July, 2012.
25. A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony, H. Liu, P. Wyckoff, and R. Murthy. Hive: a warehousing solution over a map-reduce framework. Proc. VLDB Endow., 2(2):1626-1629, Aug. 2009.
26. D. Turaga, H. Andrade, B. Gedik, C. Venkatramani, O. Verscheure, J. D. Harris, J. Cox, W. Szewczyk, and P. Jones. Design principles for developing stream processing applications. Software-Practice and Experience (SP&E), 40(12), Nov. 2010.