Improving the predictability of distributed stream processors

Future Generation Computer Systems

Volumn 52, Issue , 2015, Pages 22-36

  Basanta Val, Pablo ^a   Fernández García, Norberto ^b   Wellings, Andy J ^c   Audsley, Neil C ^c  

^a UNIVERSIDAD CARLOS III DE MADRID   (Spain)

^b DEFENSE UNIVERSITY CENTER   (Spain)

^c UNIVERSITY OF YORK   (United Kingdom)

Author keywords

Distributed stream processing; Predictable infrastructure; Real time

Indexed keywords

ABSTRACTING; COMPUTATION THEORY; COMPUTATIONAL METHODS; DATA HANDLING; DISTRIBUTED PARAMETER CONTROL SYSTEMS;

COMPUTATIONAL CONSTRAINTS; DISTRIBUTED STREAM PROCESSING; PREDICTABLE INFRASTRUCTURE; PROCESSING INFRASTRUCTURES; REAL TIME; REAL TIME INFRASTRUCTURE; REAL TIME PERFORMANCE; REAL-TIME APPLICATION;

BIG DATA;

EID: 84930635178     PISSN: 0167739X     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.future.2015.03.023     Document Type: Article

References (51)

1. R. Buyya Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility Future Gener. Comput. Syst. 25 6 2009 599 616
2. M. Armbrust A view of cloud computing Commun. ACM 53 4 2010 50 58
3. M. García-Valls, P. Uriol-Resuela, F. Ibáñez-Vázquez, and P. Basanta-Val Low complexity reconfiguration for real-time data-intensive service-oriented applications Future Gener. Comput. Syst. 37 2014 191 200
4. Paul Zikopoulos, and Chris Eaton Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data first ed. 2011 McGraw-Hill Osborne Media
5. A. Jacobs The pathologies of big data Commun. ACM 52 8 2009 36 44
6. Y. Demchenko, Z. Zhao, P. Grosso, A. Wibisono, C. de Laat, Addressing big-data challenges for scientific data infrastructure, in: CloudCom, 2012, pp. 614-617.
7. J. Lin, and R. Dmitriy Scaling big-data mining infrastructure: the twitter experience ACM SIGKDD Explor. Newsl. 14 2 2013 6 19
8. K. Kambatla, G. Kollias, V. Kumar, and A. Grama Trends in big-data analytics J. Parallel Distrib. Comput. 74 7 2014 10.1016/j.jpdc.2014.01.003
9. G. Blelloch, Big data on small machines, in: Big Data Analytics 2013, Cambridge. May 23-24, 2013.
10. H.V. Jagadish Big data and its technical challenges Commun. ACM 57 7 2014 86 94
11. V.N. Gudivada, R. Baeza-Yates, and V.V. Raghavan Big data: Promises and problems Computer 48 3 2015 20 23
12. K. Shvachko, H. Kuang, S. Radia, R. Chansler, The Hadoop distributed file system, in: 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies, MSST, pp. 1-10.
13. Storm. Distributed and fault-tolerant real-time computation. Available (2014) on https://storm.incubator.apache.org/.
14. G. Lodi An event-based platform for collaborative threats detection and monitoring Inf. Syst. 39 2014 175 195
15. Spark. Lightning-fast cluster computing. Available (2014) on https://spark.apache.org.
16. M. Rychl, P. Skoda, P. Smrz, Scheduling decisions in stream processing on heterogeneous clusters, in: Eighth International Conference on Complex, Intelligent and Software Intensive Systems, 2014, pp. 614-619.
17. I. Gray, Y. Chan, N. Audsley, A. Wellings, Architecture-awareness for real-time big-data systems, in: Proceedings of the 21st European MPI Users' Group Meeting, pp. 151-156.
18. T. Chordia, A. Goyal, B.N. Lehmann, and G. Saar High-frequency trading J. Financ. Mark. 1386-4181 16 4 2013 637 645
19. C. Grier, K. Thomas, V. Paxson, M. Zhang, @ spam: the underground on 140 characters or less, in: Proceedings of the 17th ACM Conference on Computer and Communications Security, pp. 27-37.
20. L.T.X. Phan, Z. Zhang, B.T. Loo, and I. Lee Real-time MapReduce scheduling, in: Technical Report N. MS-CIS-10-32 2010 University of Pennsylvania
21. M.T. Higuera-Toledano, and A.J. Wellings Distributed, Embedded and Real-time Java Systems 2012 Springer 378 X
22. P. Basanta-Val, and M. Garcia-Valls A distributed real-time Java-centric architecture for industrial systems IEEE Trans. Ind. Inf. 10 1 2014 27 34
23. T. Aniello, et al. Cloud-based data stream processing, in: ACM International Conference on Distributed Event-Based Systems, DEBS'14, pp. 238-245.
24. M. Stonebraker, U. Çetintemel, and S. Zdonik The 8 requirements of real-time stream processing SIGMOD Rec. 34 4 2005 42 47
25. D.J. Abadi Aurora: a new model and architecture for data stream management VLDB J. 12 2 2003 120 139
26. A. Arasu, et al. STREAM: the stanford stream data manager, in: ACM SIGMOD International Conference on Management of Data, pp. 665-665.
27. D.J. Abadi, et al. The design of the borealis stream processing engine, in: Conference on Innovative Data Systems Research, CIDR 2005, pp. 277-289.
28. L. Amini, et al. SPC: a distributed, scalable platform for data mining, in: Data Mining Standards, Services, and Platforms, DMSSP'06, pp. 27-37.
29. M. Migliavacca, et al. SEEP: scalable and elastic event processing, in: Middleware'10. Article 4, p. 2.
30. L. Neumeyer, et al. S4: Distributed stream computing platform, in: IEEE International Conference on Data Mining, pp. 170-177.
31. C. Yixin, L. Tu, Density-based clustering for real-time stream data, in: Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.
32. J.A. Silva Data stream clustering: A survey ACM Comput. Surv. 46 1 2013 31 10.1145/2522968.2522981 Article 13
33. M. Zaharia, et al. Discretized streams: fault-tolerant streaming computation at scale, in: ACM Symposium on Operating Systems Principles, SOSP'13, pp. 423-438.
34. G. Agha Actors: A Model of Concurrent Computation in Distributed Systems 1986 MIT Press Cambridge, MA, USA
35. M. Zaharia, et al. Spark: cluster computing with working sets, in: 2nd USENIX Conference on Hot Topics in Cloud Computing, HotCloud'10, pp. 10-10.
36. L. Aniello, et al. Adaptive online scheduling in Storm, in: ACM International Conference on Distributed Event-Based Systems, DEBS'13, pp. 207-218.
37. X. Dong, Y. Wang, H. Liao, Scheduling mixed real-time and non-real-time applications in MapReduce environment, in: Parallel and Distributed Systems, ICPADS, 2011.
38. F. Teng A novel real-time scheduling algorithm and performance analysis of a MapReduce based cloud J. Supercomput. 69 2 2014 739 765
39. M. Garcia-Valls, and P. Basanta-Val Comparative analysis of two different middleware approaches for reconfiguration of distributed real-time systems J. Syst. Archit. 60 2 2014 221 233
40. L. Golab, M. Tamer Özsu, Issues in data stream management, SIGMOD Rec. 32, 2, 5-14. http://doi.acm.org/10.1145/776985.776986.
41. P. Basanta Val, and M. Garcia Valls A simple distributed garbage collector for distributed real-time Java J. Supercomput. 2014 in press. http://dx.doi.org/10.1007/s11227-014-1259-x
42. L. Sha Real-time scheduling theory: A historical perspective Real-Time Syst. 28 2-3 2004 101 155
43. J.M. López, J.L. Díaz, and D.F. García Minimum and maximum utilization bounds for multiprocessor rate monotonic scheduling IEEE Trans. Parallel Distrib. Syst. 2004 642 653
44. R.I. Davis, and A. Burns A survey of hard real-time scheduling for multiprocessor systems ACM Comput. Surv. (CSUR) 43 4 2011 35
45. P. Basanta-Val, M. García-Valls, Towards a reconfiguration service for distributed real-time Java, in: REACTION 2012 Workshops. Puerto Rico, December, 4, 2012.
46. M. Noll, Implementing real-time trending topics in Storm. Available in 2014 on http://www.michael-noll.com/blog/2013/01/18/implementing-real-time-trending-topics-in-storm/.
47. V. Chang, The business intelligence as a service in the cloud, Future Gener. Comput. Syst., 37, pp. 512-534.
48. N. Irizarry Mixing C and Java™for high performance computing. MTR130458. MITRE Technical Report 2013
49. J.C.S. dos Anjos, I. Carrera Izurieta, W. Kolberg, A.L. Tibola, L. Bezerra Arantes, and C.F.R. Geyer MRA++: Scheduling and data placement on MapReduce for heterogeneous environments Future Gener. Comput. Syst. 42 2015 22 35
50. L. Woo, K. Jin-Soo, and M. Seungryoul Large-scale incremental processing with MapReduce Future Gener. Comput. Syst. 36 2014 66 79
51. G. Lianjun, Z. Tang, and Gu. Xie The implementation of MapReduce scheduling algorithm based on priority Parallel Computational Fluid Dynamics 2014 Springer Berlin, Heidelberg