Elastic scaling for data stream processing

IEEE Transactions on Parallel and Distributed Systems

Volumn 25, Issue 6, 2014, Pages 1447-1463

  Gedik, Buǧra ^a   Schneider, Scott ^b   Hirzel, Martin ^b   Wu, Kun Lung ^b  

^a BILKENT UNIVERSITY   (Turkey)

^b IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

Data stream processing; elasticity; parallelization

Indexed keywords

DATA FLOW ANALYSIS; DATA FLOW GRAPHS; ELASTICITY; PROFITABILITY;

APPLICATION DEVELOPERS; AUTO-PARALLELIZATION; DATA PARTITIONING; DATA STREAM PROCESSING; DISTRIBUTED DATA STREAM PROCESSING; PARALLEL CHANNEL; PARALLELIZATIONS; RESOURCE AVAILABILITY;

DATA COMMUNICATION SYSTEMS;

EID: 84901049742     PISSN: 10459219     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPDS.2013.295     Document Type: Article

References (34)

1. D. Abadi, Y. Ahmad, M. Balazinska, U. Çetintemel, M. Cherniack, J.-H. Hwang, W. Lindner, A. Maskey, A. Rasin, E. Ryvkina, N. Tatbul, Y. Xing, and S. Zdonik, "The Design of the Borealis Stream Processing Engine," in Proc. CIDR, 2005, pp. 277-289.
2. A. Alexandrov, D. Battre, S. Ewen, M. Heimel, F. Hueske, O. Kao, V. Markl, E. Nijkamp, and D. Warneke, "Massively Parallel Data Analysis with PACTs on Nephele," in Proc. VLDB, 2010, pp. 1625-1628.
3. D. Alves, P. Bizarro, and P. Marques, "Flood: Elastic Streaming MapReduce," in Proc. Int'l Conf. ACM DEBS, 2010, pp. 113-114.
4. H. Andrade, B. Gedik, K.-L. Wu, and P.S. Yu, "Processing High Data Rate Streams in System S," J. Parallel Distrib. Comput., vol. 71, no. 2, pp. 145-156, Feb. 2011.
5. A. Arasu, B. Babcock, S. Babu, M. Datar, K. Ito, R. Motwani, I. Nishizawa, U. Srivastava, D. Thomas, R. Varma, and J. Widom, "STREAM: The Stanford Stream Data Manager," IEEE Data Eng. Bull., vol. 26, no. 1, pp. 1-8, 2003.
6. R. Avnurand and J.M. Hellerstein, "Eddies: Continuously Adaptive Query Processing," in Proc. Int'l Conf. ACMSIGMOD, 2000, pp. 261-272.
7. V.R. Borkar, M.J. Carey, R. Grover, N. Onose, and R. Vernica, "Hyracks: A Flexible and Extensible Foundation for Data-Intensive Computing," in Proc. IEEE ICDE, 2011, pp. 1151-1162.
8. S. Brin and L. Page, "The Anatomy of a Large-Scale Hypertextual Web Search Engine," Comput. Netw. ISDN Syst., vol. 30, no. 1-7, pp. 107-117, Apr. 1998.
9. J. Dean and S. Ghemawat, "MapReduce: A Flexible Data Processing Tool," Commun. ACM, vol. 53, no. 1, pp. 72-77, Jan. 2010.
10. A. Deshpande, Z.G. Ives, and V. Raman, "Adaptive Query Processing," Found. Trends Databases, vol. 1, no. 1, pp. 1-140, Jan. 2007.
11. M.I. Gordon, W. Thies, and S. Amarasinghe, "Exploiting Coarse-Grained Task, Data, and Pipeline Parallelism in Stream Programs," in Proc. Int'l Conf. ASPLOS, 2006, pp. 151-162.
12. J.L. Hellerstein, Y. Diao, S. Parekh, and D.M. Tilbury, Feedback Control of Computing Systems. Hoboken, NJ, USA: Wiley, 2004.
13. M. Hirzel, H. Andrade, B. Gedik, G. Jacques-Silva, R. Khandekar, V. Kumar, M. Mendell, H. Nasgaard, S. Schneider, R. Soulé, and K.-L. Wu, "IBM Streams Processing Language: Analyzing Big Data in Motion," IBM J. Res. Develop., vol. 57, no. 3/4, pp. 7:1-7:11, May-July 2013.
14. M. Hirzel, R. Soulé, S. Schneider, B. Gedik, and R. Grimm, "A CatAlOg of Stream Processing Optimizations," IBM, New York, NY, USA, Research Report RC25215, 2011.
15. A.H. Hormati, Y. Choi, M. Kudlur, R. Rabbah, T. Mudge, and S. Mahlke, "Flextream: Adaptive Compilation of Streaming Applications for Heterogeneous Architectures," in Proc. Int'l Conf. PACT, 2009, pp. 214-223.
16. IBM InfoSphere Streams. [Online]. Available: http://www-01. ibm.com/software/data/infosphere/streams/Retrieved, Nov. 2012.
17. M. Isard, M.B.Y. Yu, A. Birrell, and D. Fetterly, "Dryad: Distributed Data-Parallel Program from Sequential Building Blocks," in Proc. EuroSys, 2007, pp. 59-72. (Pubitemid 47281575)
18. N. Jain, L. Amini, H. Andrade, R. King, Y. Park, P. Selo, and C. Venkatramani, "Design, Implementation, and Evaluation of the Linear Road Benchmark on the Stream Processing Core," in Proc. Int'l Conf. ACM SIGMOD, 2006, pp. 431-442. (Pubitemid 46946529)
19. D. Karger, E. Lehman, T. Leighton, R. Panigrahy, M. Levine, and D. Lewin, "Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web," in Proc. Int'l ACM STOC, 1997, pp. 654-663.
20. A. Martin, C. Fetzer, and A. Brito, "Active Replication at (Almost) No Cost," in Proc. Int'l SRDS, 2011, pp. 21-30.
21. D.S. Milojicic, F. Douglis, Y. Paindaveine, R. Wheeler, and S. Zhou, "Process Migration," ACM Comput. Surveys, vol. 32, no. 3, pp. 241-299, Sept. 2000.
22. N.W. Paton, J.B. Chavez, M. Chen, V. Raman, G. Swart, I. Narang, D.M. Yellin, and A.A.A. Fernandes, "Autonomic Query Parallelization Using Non-Dedicated Computers: An Evaluation of Adaptivity Options," Very Large Data Bases J., vol. 18, no. 1, pp. 119-140, Jan. 2009.
23. S. Schneider, H. Andrade, B. Gedik, A. Biem, and K.-L. Wu, "Elastic Scaling of Data Parallel Operators in Stream Processing," in Proc. IEEE IPDPS, 2009, pp. 1-12.
24. S. Schneider, M. Hirzel, B. Gedik, and K.-L. Wu, "Automatic Exploitation of Data Parallelism in Stateful Streaming Applications," in Proc. Int'l Conf. PACT, 2012, pp. 53-64.
25. M.A. Shah, J.M. Hellerstein, S. Chandrasekaran, and M.J. Franklin, "Flux: An Adaptive Partitioning Operator for Continuous Query Systems," in Proc. IEEE ICDE, 2003, pp. 25-36.
26. M. Stonebraker, U. Çetintemel, and S.B. Zdonik, "The 8 Requirements of Real-Time Stream Processing," SIGMOD Rec., vol. 34, no. 4, pp. 42-47, Dec. 2005. (Pubitemid 43152547)
27. Storm Project, Oct. 2013. [Online]. Available: http://stormproject. net/.
28. StreamBase, May 2012. [Online]. Available: http://www.streambase.com.
29. M.A. Suleman, M.K. Qureshi, Khubaib, and Y.N. Patt, "Feedback- Directed Pipeline Parallelism," in Proc. Int'l Conf. PACT, 2010, pp. 147-156.
30. S. Viglas, J.F. Naughton, and J. Burger, "Maximizing the Output Rate of Multi-Way Join Queries over Streaming Information Sources," in Proc. VLDB Conf., 2003, pp. 285-296.
31. M. Welsh, D. Culler, and E. Brewer, "SEDA: An Architecture for Well-Conditioned, Scalable Internet Services," in Proc. Symp. OSDI, 2001, pp. 230-243. (Pubitemid 33612617)
32. S. Wu, V. Kumar, K.-L. Wu, and B.C. Ooi, "Parallelizing Stateful Operators in a DSPS: How, Should You and How Much?" Proc. Int'l Conf. ACM DEBS, 2012, pp. 278-289.
33. Z. Xu, M. Hirzel, G. Rothermel, and K.-L. Wu, "Testing Properties of Dataflow Program Operators," in Proc. Conf. ASE, 2013, pp. 103-113.
34. S4 Distributed Stream Computing Platform, Oct. 2013. [Online]. Available: http://incubator.apache.org/s4.