Pipelined fission for stream programs with dynamic selectivity and partitioned state

Journal of Parallel and Distributed Computing

Volumn 96, Issue , 2016, Pages 106-120

  Gedik, B ^a   Ozsema H G ^a   Ozturk O ^a  

^a BILKENT UNIVERSITY   (Turkey)

Author keywords

Auto parallelization; Data stream processing; Fission; Pipelining

Indexed keywords

APPLICATION PROGRAMS; DATA COMMUNICATION SYSTEMS; DATA FLOW ANALYSIS; HEURISTIC ALGORITHMS; OPTIMIZATION; PIPE LININGS; PROBLEM SOLVING;

APPLICATION DEVELOPERS; AUTO-PARALLELIZATION; COMPUTATIONAL PARADIGM; DATA STREAM PROCESSING; FISSION; OPTIMIZATION PROBLEMS; PIPELINE PARALLELISMS; SEQUENTIAL PROGRAMS;

DATA HANDLING;

EID: 84975458327     PISSN: 07437315     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jpdc.2016.05.003     Document Type: Article

References (42)

1. D. Abadi, Y. Ahmad, M. Balazinska, U. Çetintemel, M. Cherniack, J.-H. Hwang, W. Lindner, A. Maskey, A. Rasin, E. Ryvkina, N. Tatbul, Y. Xing, S. Zdonik, The design of the Borealis stream processing engine, in: Innovative Data Systems Research Conference, CIDR, 2005.
2. D.J. Abadi, D. Carney, U. Çetintemel, M. Cherniack, C. Convey, S. Lee, M. Stonebraker, N. Tatbul, and S. Zdonik Aurora: A new model and architecture for data stream management VLDB J. 12 2 2003 120 139
3. H. Andrade, B. Gedik, and D. Turaga Fundamentals of Stream Processing: Application Design, Systems, Analytics first ed. 2014 Cambridge Press Cambridge, UK (Chapter 12.4) - The Semiconductor Process Control application
4. H. Andrade, B. Gedik, K.-L. Wu, and P.S. Yu Processing high data rate streams in systems J. Parallel Distrib. Comput. (JPDC) 71 2011 145 156 Special Issue on Data Intensive Computing
5. A. Arasu, B. Babcock, S. Babu, M. Datar, K. Ito, R. Motwani, I. Nishizawa, U. Srivastava, D. Thomas, R. Varma, J. Widom, STREAM: The Stanford stream data manager, IEEE Data Eng. Bull., Vol. 26, no. 1.
6. M.D. Beynon, T.M. Kurç, Ü.V. Çatalyürek, C. Chang, A. Sussman, and J.H. Saltz Distributed processing of very large datasets with DataCutter Parallel Comput. J. 27 11 2001 1457 1478
7. E. Bouillet, R. Kothari, V. Kumar, L. Mignet, S. Nathan, A. Ranganathan, D.S. Turaga, O. Udrea, O. Verscheure, Processing 6 billion CDRs/day: from research to production (experience report), in: International Conference on Distributed Event Based Systems, DEBS, 2012.
8. P. Charles, C. Grothoff, V.A. Saraswat, C. Donawa, A. Kielstra, K. Ebcioglu, C. von Praun, V. Sarkar, X10: An object-oriented approach to non-uniform cluster computing, in: International Conference on Object-Oriented Programming, Systems, Languages & Applications, OOPSLA, 2005.
9. J. Dean, S. Ghemawat, Mapreduce: Simplified data processing on large clusters, in: USENIX Symposium on Operating Systems Design and Implementation, OSDI, 2004, pp. 137-150, 2008.
10. W. Du, R. Ferreira, G. Agrawal, Compiler support for exploiting coarse-grained pipelined parallelism, in: Supercomputing Conference, SC, 2003, p. 8.
11. S. Garcia, D. Jeon, C.M. Louie, M.B. Taylor, Kremlin: Rethinking and rebooting gprof for the multicore age, in: International Conference on Programming Language Design and Implementation, PLDI, 2011.
12. B. Gedik, and H. Andrade A model-based framework for building extensible, high performance stream processing middleware and programming language for IBM InfoSphere Streams Software: Practice and Experience 42 11 2012 1363 1391
13. B. Gedik, H. Andrade, K.-L. Wu, A code generation approach to optimizing high-performance distributed data stream processing, in: ACM International Conference on Information and Knowledge Management, CIKM, 2009.
14. B. Gedik, H. Andrade, K.-L. Wu, P.S. Yu, M. Doo, SPADE: The systems declarative stream processing engine, in: ACM International Conference on Management of Data, SIGMOD, 2008, pp. 1123-1134.
15. B. Gedik, S. Schneider, K.-L.W.M. Hirzel, Elastic scaling for data stream processing, IEEE Trans. Parallel Distrib. Syst. (TPDS). http://dx.doi.org/10.1109/TPDS.2013.295.
16. M.I. Gordon, W. Thies, S. Amarasinghe, Exploiting coarse-grained task, data, and pipeline parallelism in stream programs, in: International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS, 2006.
17. M.I. Gordon, W. Thies, S. Amarasinghe, Exploiting coarse-grained task, data, and pipeline parallelism in stream programs, in: International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS, 2006, pp. 151-162.
18. M.I. Gordon, W. Thies, M. Karczmarek, J. Lin, A.S. Meli, A.A. Lamb, C. Leger, J. Wong, H. Hoffmann, D. Maze, S. Amarasinghe, A stream compiler for communication-exposed architectures, in: International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS, 2002, pp. 291-303.
19. Y. He, C.E. Leiserson, W.M. Leiserson, The cilkview scalability analyzer, in: ACM Symposium on Parallelism in Algorithms and Architectures, SPAA, 2010.
20. M. Hirzel, H. Andrade, B. Gedik, G. Jacques-Silva, R. Khandekar, V. Kumar, M. Mendell, H. Nasgaard, S. Schneider, R. Soule, and K.-L. Wu Streams processing language: Analyzing big data in motion IBM J. Res. Dev. 57 2013 7:1 7:11
21. M. Hirzel, H. Andrade, B. Gedik, V. Kumar, G. Losa, M. Mendell, H. Nasgaard, R. Soule, and K.-L. Wu SPL language spec., Tech. Rep. RC24897 2009 IBM
22. M. Hirzel, B. Gedik, Streams that compose using macros that oblige, in: ACM Workshop on Partial Evaluation and Program Manipulation, PEPM, 2012.
23. M. Hirzel, R. Soule, S. Schneider, B. Gedik, R. Grimm, A catalog of stream processing optimizations, ACM Comput. Surv., Vol. 46, no. 4.
24. Intel cilk++. http://software.intel.com/en-us/articles/intel-cilk-plus/ (retrieved October, 2014).
25. R. Khandekar, K. Hildrum, S. Parekh, D. Rajan, J. Wolf, K.-L. Wu, H. Andrade, B. Gedik, COLA: Optimizing stream processing applications via graph partitioning, in: ACM/IFIP/USENIX Middleware Conference, Middleware, 2009.
26. T. Klug, M. Ott, J. Weidendorfer, and C. Trinitis Autopin: Automated optimization of thread-to-core pinning on multicore systems Trans. High-Perform. Embedded Archit. Compil. (HiPEAC) 3 2011 219 235
27. S.M. Krishnamurthy A brief survey of papers on scheduling for pipelined processors ACM SIGPLAN Not. 25 7 1990 97 106
28. M. Kudlur, S.A. Mahlke, Orchestrating the execution of stream programs on multicore platforms, in: International Conference on Programming Language Design and Implementation, PLDI, 2008, pp. 114-124.
29. S. Kulkarni, N. Bhagat, M. Fu, V. Kedigehalli, C. Kellogg, S. Mittal, J.M. Patel, K. Ramasamy, S. Taneja, Twitter heron: Stream processing at scale, in: ACM International Conference on Management of Data, SIGMOD, 2105, pp. 239-250.
30. Openmp. http://www.openmp.org (retrieved October, 2014).
31. Samza project. http://samza.apache.org/ (retrieved Nov., 2014).
32. S. Schneider, H. Andrade, B. Gedik, A. Biem, K.-L. Wu, Elastic scaling of data parallel operators in stream processing, in: IEEE International Parallel and Distributed Processing Symposium, IPDPS, 2009.
33. S. Schneider, M. Hirzel, B. Gedik, K.-L. Wu, Safe data parallelism for general streaming, IEEE Trans. Comput. (TC), Vol. 64, no. 2.
34. M.A. Shah, J.M. Hellerstein, S. Chandrasekaran, M.J. Franklin, Flux: An adaptive partitioning operator for continuous query systems, in: IEEE International Conference on Data Engineering, ICDE, 2003.
35. D.M. Sow, A. Biem, M. Blount, M. Ebling, O. Verscheure, Body sensor data processing using stream computing.
36. Storm project. http://storm-project.net/ (retrieved Nov., 2013).
37. StreamBase Systems. http://www.streambase.com (retrieved Nov., 2013).
38. S4 distributed stream computing platform. http://www.s4.io/ (retrieved May, 2012).
39. Y. Tang, and B. Gedik Auto-pipelining for data stream processing IEEE Trans. Parallel Distrib. Syst. (TPDS) 24 12 2013 2344 2354
40. M. Welsh, D. Culler, E. Brewer, SEDA: An architecture for well-conditioned, scalable Internet services, in: ACM Symposium on Operating Systems Principles (SOSP), 2001, pp. 230-243.
41. X.J. Zhang, H. Andrade, B. Gedik, R. King, J.F. Morar, S. Nathan, Y. Park, R. Pavuluri, E. Pring, R. Schnier, P. Selo, M. Spicer, V. Uhlig, C. Venkatramani, Implementing a high-volume, low-latency market data processing system on commodity hardware using IBM middleware, in: Workshop on High Performance Computational Finance, SC-WHPCF, 2009.
42. X. Zhang, A. Navabi, S. Jagannathan, Alchemist: A transparent dependence distance profiling infrastructure, in: International Symposium on Code Generation and Optimization, CGO, 2009, pp. 47-58.