Elastic stream processing in the Cloud

Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery

Volumn 3, Issue 5, 2013, Pages 333-345

  Hummer, Waldemar ^a   Satzger, Benjamin ^a   Dustdar, Schahram ^a  

^a VIENNA UNIVERSITY OF TECHNOLOGY   (Austria)

Author keywords

[No Author keywords available]

Indexed keywords

APPLICATION AREA; CLOUD-BASED; COMPUTING PARADIGM; CONTINUOUS QUERIES; HIGH VOLUMES; HISTORICAL EVOLUTIONS; PATTERN DETECTION; STREAM PROCESSING;

DATA MINING;

ELASTICITY;

EID: 84882695642     PISSN: 19424787     EISSN: 19424795     Source Type: Journal    
DOI: 10.1002/widm.1100     Document Type: Review

References (80)

1. Armbrust M, Fox A, Griffith R, Joseph AD, Katz R, Konwinski A, Lee G, Patterson D, Rabkin A, Stoica I, et al. A view of cloud computing. Commun ACM 2010, 53:50-58.
2. Hilley D. Cloud computing: a taxonomy of platform and infrastructure-level offerings. Georgia Institute of Technology, Tech. Rep. GIT-CERCS-09-13; 2009.
3. Amazon. Amazon Elastic Compute Cloud (Amazon EC2). Available at: http://aws.amazon.com/ec2; 2012. (Accessed May 1, 2013).
4. Openstack. Openstack open source cloud computing software. http://www.openstack.org/; 2012. (Accessed May 1, 2013).
5. Voorsluys W, Broberg J, Buyya R. Cloud computing: Principles and paradigms. Hoboken, NJ: John Wiley & Sons; 2011.
6. Dustdar S, Guo Y, Satzger B, Truong HL. Principles of elastic processes. IEEE Internet Comput 2011, 15:66-71, doi: 10.1109/MIC.2011.121.
7. Dustdar S, Guo Y, Han R, Satzger B, Truong HL. Programming directives for elastic computing. IEEE Internet Comput 2012, 16:72-77.
8. Moxey C, Edwards M, Etzion O, Ibrahim M, Iyer S, Lalanne H, Monze M, Peters M, Rabinovich Y, Sharon G, et al. A conceptual model for event processing systems. IBM Redguide publication; 2010. Available at: http://www.redbooks.ibm.com/abstracts/redp4642.html. (Accessed May 1, 2013).
9. Luckham D, Schulte R. Event processing glossary v1.1. Event Proces Tech Soc 2008, 2:5-19.
10. Sharon G, Etzion O. Event-processing network model and implementation. IBM Syst J 2008, 47:321-334.
11. Hummer W, Inzinger C, Satzger B, Leitner P, Dustdar S. Deriving a unified fault taxonomy for event-based systems. 6th ACM International Conference on Distributed Event-Based Systems (DEBS'12). Berlin, Germany: ACM; 2012, 167-178.
12. Babcock B, Babu S, Datar M, Motwani R, Widom J. Models and issues in data stream systems. 21st ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems, PODS '02. New York: ACM; 2002, 1-16.
13. Patroumpas K, Sellis T. Window specification over data streams. Current Trends in Database Technology, EDBT 2006. Munich, Germany: Springer; 2006, 445-464.
14. Botan I, Kossmann D, Fischer PM, Kraska T, Florescu D, Tamosevicius R. Extending XQuery with window functions. 33rd International Conference on Very Large Data Bases (VLDB). Dallas, TX: ACM; 2007, 75-86.
15. Chen J, DeWitt D, Tian F, Wang Y. Niagaracq: a scalable continuous query system for Internet databases. ACM SIGMOD Rec 2000, 29:379-390.
16. Chandrasekaran S, Cooper O, Deshpande A, Franklin MJ, Hellerstein JM, Hong W, Krishnamurthy S, Madden S, Raman V, Reiss F, et al. Telegraphcq: continuous dataflow processing for an uncertain world. First Biennial Conference on Innovative Data Systems Research (CIDR 2003), Asilomar, CA; January 5-8, 2003.
17. Carney D, Çetintemel U, Cherniack M, Convey C, Lee S, Seidman G, Stonebraker M, Tatbul N, Zdonik S. Monitoring streams: a new class of data management applications. 28th International Conference on Very Large Data Bases (VLDB). Hong Kong, China: ACM; 2002, 215-226.
18. Abadi D, Carney D, Çetintemel U, Cherniack M, Convey C, Lee S, Stonebraker M, Tatbul N, Zdonik S. Aurora: a new model and architecture for data stream management. VLDB J 2003, 12:120-139.
19. Balakrishnan H, Balazinska M, Carney D, Çetintemel U, Cherniack M, Convey C, Galvez E, Salz J, Stonebraker M, Tatbul N, et al. Retrospective on aurora. VLDB J 2004, 13:370-383.
20. Cherniack M, Balakrishnan H, Balazinska M, Carney D, Cetintemel U, Xing Y, Zdonik S. Scalable distributed stream processing. Proceedings of Conference on Innovative Data Systems and Research. Asilomar, CA: ACM; 2003.
21. Dean J, Ghemawat S. Mapreduce: simplified data processing on large clusters. Commun ACM 2008, 51:107-113.
22. Abadi D, Ahmad Y, Balazinska M, Cetintemel U, Cherniack M, Hwang J, Lindner W, Maskey A, Rasin A, Ryvkina E, et al. The design of the Borealis stream processing engine. Conference on Innovative Data Systems Research (CIDR). Asilomar, CA: ACM; 2005.
23. Carney D, Çetintemel U, Rasin A, Zdonik S, Cherniack M, Stonebraker M. Operator scheduling in a data stream manager. 29th International Conference on Very Large Data Bases (VLDB). Berlin, Germany: ACM; 2003, 838-849.
24. Amini L, Andrade H, Bhagwan R, Eskesen F, King R, Selo P, Park Y, Venkatramani C. SPC: a distributed, scalable platform for data mining. 4th International Workshop on Data Mining Standards, Services and Platforms, DMSSP '06. New York: ACM; 2006, 27-37.
25. Wolf J, Bansal N, Hildrum K, Parekh S, Rajan D, Wagle R, Wu KL, Fleischer L. SODA: an optimizing scheduler for large-scale stream-based distributed computer systems. 9th ACM/IFIP/USENIX International Conference on Middleware. Berlin: Springer, 2008; 306-325.
26. Gedik B, Andrade H, Wu K, Yu P, Doo M. Spade: the system s declarative stream processing engine. ACM SIGMOD International Conference on Management of Data. New York: ACM; 2008, 1123-1134.
27. Biem A, Bouillet E, Feng H, Ranganathan A, Riabov A, Verscheure O, Koutsopoulos H, Moran C. IBM InfoSphere streams for scalable, real-time, intelligent transportation services. ACM SIGMOD International Conference on Management of Data. New York: ACM; 2010, 1093-1104.
28. Isard M, Budiu M, Yu Y, Birrell A, Fetterly D. Dryad: distributed data-parallel programs from sequential building blocks. 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems (EuroSys), EuroSys '07. Lisbon, Portugal: ACM; 2007, 59-72.
29. Warneke D, Kao O. Exploiting dynamic resource allocation for efficient parallel data processing in the cloud. IEEE Trans Parallel Distrib Syst 2011, 22:985-997.
30. Bernhardt T, Vasseur A. Esper: event stream processing and correlation. O'Reilly ONJava; 2007. Available at: http://www.onjava.com/lpt/a/6955. (Accessed May 1, 2013).
31. Neumeyer L, Robbins B, Nair A, Kesari A. S4: distributed stream computing platform. 2010 IEEE International Conference on Data Mining Workshops (ICDMW). Piscataway, NJ: IEEE; 2010, 170-177.
32. Project S. Storm: distributed and fault-tolerant realtime computation. Available at: http://storm-project.net/; 2012. (Accessed May 1, 2013).
33. Stonebraker M, Çetintemel U, Zdonik S. The 8 requirements of real-time stream processing. SIGMOD Rec 2005, 34:42-47.
34. Raman V, Raman B, Hellerstein JM. Online dynamic reordering for interactive data processing. 25th International Conference on Very Large Data Bases (VLDB), VLDB '99. Edinburgh, Scotland: ACM; 1999, 709-720.
35. Avnur R, Hellerstein JM. Eddies: continuously adaptive query processing. ACM SIGMOD International Conference on Management of Data. Dallas, TX: ACM; 2000, 261-272.
36. Hull B, Bychkovsky V, Zhang Y, Chen K, Goraczko M, Miu A, Shih E, Balakrishnan H, Madden S. Cartel: a distributed mobile sensor computing system. 4th International Conference on Embedded Networked Sensor Systems. New York: ACM; 2006, 125-138.
37. Madden S, Franklin M. Fjording the stream: an architecture for queries over streaming sensor data. 18th International Conference on Data Engineering. Piscataway, NJ: IEEE; 2002, 555-566.
38. Liu M, Li M, Golovnya D, Rundensteiner E, Claypool K. Sequence pattern query processing over out-of-order event streams. 25th International Conference on Data Engineering. Shanghai, China: IEEE; 2009, 784-795.
39. Tatbul N, Çetintemel U, Zdonik S, Cherniack M, Stonebraker M. Load shedding in a data stream manager. 29th International Conference on Very Large Data Bases. Berlin, Germany: VLDB Endowment; 2003, 309-320.
40. Babcock B, Datar M, Motwani R. Load shedding for aggregation queries over data streams. 20th International Conference on Data Engineering. Piscataway, NJ: IEEE; 2004, 350-361.
41. Tatbul N, Çetintemel U, Zdonik S. Staying fit: efficient load shedding techniques for distributed stream processing. Proceedings of the 33rd International Conference on Very Large Data Bases. Vienna, Austria: VLDB Endowment; 2007, 159-170.
42. Zang C, Fan Y. Complex event processing in enterprise information systems based on RFID. Enterprise Inform Syst 2007, 1:3-23.
43. Lakshmanan G, Li Y, Strom R. Placement strategies for internet-scale data stream systems. IEEE Internet Comput 2008, 12:50-60.
44. Xing Y, Zdonik S, Hwang J. Dynamic load distribution in the Borealis stream processor. 21st International Conference on Data Engineering (ICDE). Piscataway, NJ: IEEE; 2005, 791-802.
45. Xing Y, Hwang JH, Çetintemel U, Zdonik S. Providing resiliency to load variations in distributed stream processing. 32nd International Conference on Very Large Data Bases (VLDB). Seoul, Korea: ACM; 2006, 775-786.
46. Amini L, Jain N, Sehgal A, Silber J, Verscheure O. Adaptive control of extreme-scale stream processing systems. 26th IEEE International Conference on Distributed Computing Systems (ICDCS). Piscataway, NJ: IEEE, 2006.
47. Gulisano V, Jimenez-Peris R, Patio-Martinez M, Soriente C, Valduriez P. Streamcloud: an elastic and scalable data streaming system. IEEE Trans Parallel Distrib Syst 2012, 23:2351-2365.
48. Kumar V, Cooper B, Schwan K. Distributed stream management using utility-driven self-adaptive middleware. 2nd International Conference on Autonomic Computing (ICAC). Seattle, WA: IEEE; 2005, 3-14.
49. Schneider S, Andrade H, Gedik B, Biem A, Wu KL. Elastic scaling of data parallel operators in stream processing. IEEE International Symposium on Parallel Distributed Processing (IPDPS). Rome, Italy: IEEE; 2009, 1-12.
50. Pietzuch P, Ledlie J, Shneidman J, Roussopoulos M, Welsh M, Seltzer M. Network-aware operator placement for stream-processing systems. 22nd International Conference on Data Engineering (ICDE). Los Alamitos, CA: IEEE Computer Society; 2006, 49.
51. Shah MA, Hellerstein JM, Chandrasekaran S, Franklin MJ. Flux: an adaptive partitioning operator for continuous query systems. 19th International Conference on Data Engineering (ICDE). Bangalore, India: IEEE; 2003, 25-36.
52. Hummer W, Leitner P, Satzger B, Dustdar S. Dynamic migration of processing elements for optimized query execution in event-based systems. 1st International Symposium on Secure Virtual Infrastructures (DOA-SVI'11), OnTheMove Federated Conferences (OTM'11). Crete, Greece: Springer; 2011, 451-468.
53. Kwon Y, Balazinska M, Greenberg A. Fault-tolerant stream processing using a distributed, replicated file system. Proc VLDB Endowment 2008, 1:574-585.
54. Bansal N, Bhagwan R, Jain N, Park Y, Turaga D, Venkatramani C. Towards optimal resource allocation in partial-fault tolerant applications. INFOCOM 2008. The 27th Conference on Computer Communications. Piscataway, NJ: IEEE; 2008, 1319-1327.
55. Balazinska M, Balakrishnan H, Madden SR, Stonebraker M. Fault-tolerance in the borealis distributed stream processing system. ACM Trans Database Syst 2008, 33:3:1-3:44.
56. Hwang JH, Balazinska M, Rasin A, Cetintemel U, Stonebraker M, Zdonik S. high-availability algorithms for distributed stream processing. 21st International Conference on Data Engineering (ICDE). Tokyo, Japan: IEEE, 2005.
57. Cristian F. Understanding fault-tolerant distributed systems. Commun ACM 1991, 34:56-78.
58. Satzger B. Self-healing distributed systems. Ph.D. thesis, Universität Augsburg, Germany 2008.
59. Olston C, Jiang J, Widom J. Adaptive filters for continuous queries over distributed data streams. SIGMOD International Conference on Management of Data. New York: ACM; 2003, 563-574.
60. Kleiminger W, Kalyvianaki E, Pietzuch P. Balancing load in stream processing with the cloud. 27th International Conference on Data Engineering, Workshops (ICDEW). Piscataway, NJ: IEEE; 2011, 16-21.
61. Satzger B, Hummer W, Leitner P, Dustdar S. Esc: towards an elastic stream computing platform for the cloud. 2011 IEEE Fifth International Conference on Cloud Computing. Washington, DC: IEEE; 2011, 348-355.
62. Zinn D, Hart Q, McPhillips T, Ludascher B, Simmhan Y, Giakkoupis M, Prasanna V. Towards reliable, performant workflows for streaming-applications on cloud platforms. 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid). Newport Beach, CA: IEEE; 2011, 235-244.
63. Hummer W, Raz O, Shehory O, Leitner P, Dustdar S. Testing of data-centric and event-based dynamic service compositions. Softw Test Verif Reliab, in press.
64. Banerjee P, Friedrich R, Bash C, Goldsack P, Huberman B, Manley J, Patel C, Ranganathan P, Veitch A. Everything as a service: powering the new information economy. IEEE Comput 2011, 44:36-43.
65. Shi E, Chan THH, Rieffel EG, Chow R, Song D. Privacy-preserving aggregation of time-series data (NDSS). Network and Distributed System Security Symposium. San Diego, CA: The Internet Society; 2011.
66. Hummer W, Satzger B, Leitner P, Inzinger C, Dustdar S. Distributed continuous queries over web service event streams. 7th International Conference on Next Generation Web Services Practices (NWeSP'11). Salamanca, Spain: IEEE; 2011, 176-181.
67. Loesing S, Hentschel M, Kraska T, Kossmann D. Stormy: an elastic and highly available streaming service in the cloud. Proceedings of the 2012 Joint EDBT/ICDT Workshops. Berlin, Germany: IEEE; 2012, 55-60.
68. Chen Q, Hsu M, Zeller H. Experience in Continuous analytics as a Service (CaaaS). 14th International Conference on Extending Database Technology (EDBT). New York: ACM; 2011, 509-514.
69. Brewer EA. Towards robust distributed systems. 19th ACM Symposium on Principles of Distributed Computing (PODC). New York: ACM; 2000, 19, 7-10.
70. Gilbert S, Lynch N. Brewer's conjecture and the feasibility of consistent, available, partition-tolerant Web services. ACM SIGACT News 2002, 33:51-59.
71. Vogels W. Eventually consistent. Commun ACM 2009, 52:40-44.
72. Sebepou Z, Magoutis K. Cec: Continuous eventual checkpointing for data stream processing operators. 41st IEEE/IFIP International Conference on Dependable Systems&Networks (DSN). Hong Kong, China: IEEE; 2011, 145-156.
73. Leitner P, Hummer W, Dustdar S. Cost-based optimization of service compositions. IEEE Trans Serv Comput 2013, 6:239-251.
74. Papaemmanouil O, Çetintemel U, Jannotti J. Supporting generic cost models for wide-area stream processing. IEEE International Conference on Data Engineering (ICDE). Shanghai, China: IEEE; 2009, 1084-1095.
75. Ishii A, Suzumura T. Elastic stream computing with clouds. IEEE International Conference on Cloud Computing (CLOUD). Washington, DC: IEEE; 2011, 195-202.
76. Satzger B, Hummer W, Inzinger C, Leitner P, Dustdar S. Winds of change: from vendor lock-in to the meta cloud. IEEE Internet Comput; 2013, 17:69-73.
77. Chandramouli B, Claessens J, Nath S, Santos I, Zhou W. RACE: real-time applications over cloud-edge. ACM SIGMOD International Conference on Management of Data. Scottsdale, AZ: ACM; 2012, 625-628.
78. Logothetis D, Yocum K. Wide-scale data stream management. USENIX 2008 Annual Technical Conference (ATC). Berkeley, CA: USENIX Association; 2008, 405-418.
79. Logothetis D, Yocum K. Ad-hoc data processing in the cloud. Proc VLDB Endow. 2008, 1:1472-1475.
80. Martinaitis P, Patten C, Wendelborn A. Component-based stream processing in the cloud. Workshop on Component-Based High Performance Computing. New York: ACM; 2009, 16.