Safe data parallelism for general streaming

IEEE Transactions on Computers

Volumn 64, Issue 2, 2015, Pages 504-517

  Schneider, Scott ^a   Hirzel, Martin ^a   Gedik, Bugra ^b   Wu, Kun Lung ^a  

^a IBM T J WATSON RESEARCH CENTER   (United States)

^b BILKENT UNIVERSITY   (Turkey)

Author keywords

Data processing; distributed computing; parallel programming

Indexed keywords

DATA HANDLING; DATA PROCESSING; DISTRIBUTED COMPUTER SYSTEMS; PARALLEL PROGRAMMING; SCALABILITY; SEMANTICS;

DATA PARALLELISM; DATA PARALLELIZATION; DISTRIBUTED RUNTIME; EFFICIENT STRATEGY; PIPELINE PARALLELISMS; STREAM PROCESSING; STREAM PROGRAMMING; STREAMING APPLICATIONS;

PROGRAM COMPILERS;

EID: 84921326768     PISSN: 00189340     EISSN: None     Source Type: Journal    
DOI: 10.1109/TC.2013.221     Document Type: Article

References (34)

1. M. Hirzel, R. Soulé, S. Schneider, B. Gedik, and R. Grimm, "Acatalog of stream processing optimizations," IBM, Res. Rep. RC25215, 2011.
2. L. Neumeyer, B. Robbins, A. Nair, and A. Kesari, "S4: Distributed stream processing platform," in Workshop Knowl. Discov. Using Cloud Distrib. Comput. Platforms (KDCloud), 2010, pp. 170-177.
3. (Oct. 2013) Storm [Online]. Available: http://storm-project.net/
4. M. I. Gordon, W. Thies, and S. Amarasinghe, "Exploiting coarsegrained task, data, and pipeline parallelism in stream programs," in Proc. 12th Int. Conf. Archit. Support Program. Lang. Operat. Syst. (ASPLOS), 2006, pp. 151-162.
5. A. Brito, C. Fetzer, H. Sturzrehm, and P. Felber, "Speculative out-oforder event processing with software transaction memory," in Proc. 2nd Int. Conf. Distrib. Event-Based Syst. (DEBS), 2008, pp. 265-275.
6. M. Hirzel, H. Andrade, B. Gedik, G. Jacques-Silva, R. Khandekar, V. Kumar, M. Mendell, H. Nasgaard, S. Schneider, R. Soulé, and K.-L. Wu, "IBM Streams Processing Language: Analyzing big data in motion," IBM J. Res. Dev. (IBMRD), vol. 57, no. 3/4, 2013.
7. (Oct. 2013) IBM InfoSphere Streams [Online]. Available: http://www. ibm.com/software/data/infosphere/streams
8. S. Schneider, M. Hirzel, B. Gedik, and K.-L. Wu, "Auto-parallelizing stateful distributed streaming applications," in Proc. Int. Conf. Parallel Archit. Compil. Tech. (PACT), 2012, pp. 53-64.
9. M. Hirzel, H. Andrade, B. Gedik, V. Kumar, G. Losa, M. Mendell, H. Nasgaard, R. Soulé, and K.-L. Wu, "Streams processing language specification," IBM, Res. Rep. RC24897, 2009.
10. E. A. Lee and D. G. Messerschmitt, "Synchronous data flow," Proc. IEEE, vol. 75, no. 9, pp. 1235-1245, 1987.
11. M. Engels, G. Bilson, R. Lauwereins, and J. Peperstraete, "Cyclestatic dataflow: Model and implementation," in Proc. Conf. Signals Syst. Comput. (SSC), 1994, pp. 503-507.
12. R. Khandekar, I. Hildrum, S. Parekh, D. Rajan, J. Wolf, K.-L. Wu, H. Andrade, and B. Gedik, "COLA: Optimizing stream processing applications via graph partitioning," in Proc. Int. Conf. Middleware, 2009, pp. 308-327.
13. M. Finifter, A. Mettler, N. Sastry, and D. Wagner, "Verifiable functional purity in Java," in Proc. 15thACMConf. Comput. Commun. Security (CCS), 2008, pp. 161-174.
14. P. Li, K. Agrawal, J. Buhler, R. D. Chamberlain, and J. M. Lancaster, "Deadlock-avoidance for streaming applications with split-join structure: Two case studies," in Appl.-Proc. 21st IEEE Conf. Specific Syst. Archit. Processors (ASAP), 2010, pp. 333-336.
15. L. Page, S. Brin, R. Motwani, and T. Winograd, "The PageRank citation ranking: Bringing order to the web," Stanford InfoLab, Tech. Rep. 1999-66, Nov. 1999.
16. L. Chiticariu, R. Krishnamurthy, Y. Li, S. Raghavan, F. Reiss, and S. Vaithyanathan, "SystemT: An algebraic approach to declarative information extraction," in Proc. 48th Annu. Meet. Assoc. Comput. Linguistics (ACL), 2010, pp. 128-137.
17. M.Hirzel, "Partition and compose: Parallel complex event processing," in Proc. 6th Int. Conf.Distrib.Event-Based Syst. (DEBS), 2012, pp. 191-200.
18. D. J. Dewitt, S. Ghandeharizadeh, D. A. Schneider, A. Bricker, H. I. Hsiao, and R. Rasmussen, "The Gamma database machine project," IEEE Trans. Knowl. Data Eng., vol. 2, no. 1, pp. 44-62, 1990.
19. G. Graefe, "Encapsulation of parallelism in the volcano query processing system," in Proc. Int. Conf. Manage. Data (SIGMOD), 1990, pp. 102-111.
20. J. Dean and S. Ghemawat, "MapReduce: Simplified data processing on large clusters," in Proc. 6th Conf. Symp. Operat. Syst. Des. Implement. (OSDI), 2004, pp. 313-328.
21. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, "Dryad: Distributed data-parallel program from sequential building blocks," in Proc. 2nd Eur. Conf. Comput. Syst. (EuroSys), 2007, pp. 59-72.
22. A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony, H. Liu, P. Wyckoff, and R. Murthy, "Hive: A warehousing solution over a map-reduce framework," in Proc. Very Large Data Bases (VLDB), 2009, pp. 1626-1629.
23. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins, "Pig Latin: A not-so-foreign language for data processing," in Proc. Int. Conf. Manage. Data (SIGMOD), 2008, pp. 1099-1110.
24. C. Chambers, A. Raniwala, F. Perry, S. Adams, R. R. Henry, R. Bradshaw, and N. Weizenbaum, "FlumeJava: Easy, efficient data-parallel pipelines," in Proc. Program. Lang. Des. Implement. (PLDI), 2010, pp. 363-375.
25. T. Condie, N. Conway, P. Alvaro, J. M. Hellerstein, K. Elmeleegy, and R. Sears, "MapReduce online," in Proc. Netw. Syst. Des. Implement. (NSDI), 2010, pp. 313-328.
26. D. Peng and F. Dabek, "Large-scale incremental processing using distributed transactions and notifications," in Proc. Operat. Syst. Des. Implement. (OSDI), 2010, pp. 251-264.
27. M. Zaharia, T. Das, H. Li, T. Hunter, S. Shenker, and I. Stoica, "Discretized streams: Fault-tolerant streaming computation at scale," in Proc. 24thACMSymp.Operat. Syst. Principles (SOSP), 2013, pp. 827-838.
28. L. Amini, H. Andrade, R. Bhagwan, F. Eskesen, R. King, P. Selo, Y. Park, and C. Venkatramani, "SPC: A distributed, scalable platform for data mining," in Proc. Workshop Data Mining Stand. Serv. Platforms (DM-SSP), 2006, pp. 27-37.
29. R. H. Arpaci-Dusseau, E. Anderson, N. Treuhaft, D. E. Culler, J. M. Hellerstein, D. Patterson, and K. Yelick, "Cluster I/O with river: Making the fast case common," in Proc. Workshop I/O Parallel Distrib. Syst. (IOPADS), 1999, pp. 10-22.
30. M. A. Shah, J. M. Hellerstein, S. Chandrasekaran, and M. J. Franklin, "Flux: An adaptive partitioning operator for continuous query systems," in Proc. Int. Conf. Data Eng. (ICDE), 2003, pp. 25-36.
31. M. H. Ali et al., "Microsoft CEP server and online behavioral targeting," in Proc. Very Large Data Bases (VLDB), 2009, pp. 1558-1561.
32. S. Schneider, H. Andrade, B. Gedik, A. Biem, and K.-L. Wu, "Elastic scaling of data parallel operators in stream processing," in Proc. Int. Parallel Distrib. Process. Symp. (IPDPS), 2009, pp. 1-12.
33. R. L. Collins and L. P. Carloni, "Flexible filters: Load balancing through backpressure for stream programs," in Proc. Int. Conf. Embedded Softw. (EMSOFT), 2009, pp. 205-214.
34. M. A. Suleman, M. K. Qureshi, Khubaib, and Y. N. Patt, "Feedbackdirected pipeline parallelism," in Proc. Parallel Archit. Compil. Tech. (PACT), 2010, pp. 147-156.