Reactive resource provisioning heuristics for dynamic dataflows on cloud infrastructure

IEEE Transactions on Cloud Computing

Volumn 3, Issue 2, 2015, Pages 105-118

  Kumbhare, Alok Gautam ^a   Simmhan, Yogesh ^b   Frincu, Marc ^a   Prasanna, Viktor K ^a  

^a University of Southern California ^*  (United States)

^b INDIAN INSTITUTE OF SCIENCE   (India)

Author keywords

cloud; Dataflows; high velocity data; resource management; runtime adaptation; scheduling; stream processing

Indexed keywords

BENCHMARKING; CLOUDS; DATA FLOW ANALYSIS; DATA STREAMS; GENETIC ALGORITHMS; SCHEDULING;

DATA FLOW; HIGH VELOCITY; RESOURCE MANAGEMENT; RUNTIME ADAPTATION; STREAM PROCESSING;

INFORMATION MANAGEMENT;

EID: 84933042168     PISSN: None     EISSN: 21687161     Source Type: Journal    
DOI: 10.1109/TCC.2015.2394316     Document Type: Article

References (51)

1. D. Laney, "3d data management: Controlling data volume, velocity, and variety, " META Group Research Note, vol. 6, 2001.
2. J. Dean and S. Ghemawat, "MapReduce: Simplified data processing on large clusters, " ACM Commun., vol. 51, no. 1, pp. 107-113, 2008.
3. M. Gatti, P. Cavalin, S. B. Neto, C. Pinhanez, C. dos Santos, D. Gribel, and A. P. Appel, "Large-scale multi-agent-based modeling and simulation of microblogging-based online social network, " in Proc. 14th Int. Workshop Multi-Agent-Based Simul., 2014, pp. 17-33.
4. I. Fronza, A. Sillitti, G. Succi, M. Terho, and J. Vlasenko, "Failure prediction based on log files using random indexing and support vector machines, " J. Syst. Softw., vol. 86, no. 1, pp. 2-11, 2013.
5. A. Biem, E. Bouillet, H. Feng, A. Ranganathan, A. Riabov, O. Verscheure, H. Koutsopoulos, and C. Moran, "IBM infosphere streams for scalable, real-time, intelligent transportation services, " in Proc. Int. Conf. Manage. Data, 2010, pp. 1093-1104.
6. Y. Simmhan, S. Aman, A. Kumbhare, R. Liu, S. Stevens, Q. Zhou, and V. Prasanna, "Cloud-based software platform for big data analytics in smart grids, " Comput. Sci. Eng., vol. 15, no. 4, pp. 38-47, Jul. 2013.
7. L. Neumeyer, B. Robbins, A. Nair, and A. Kesari, "S4: Distributed stream computing platform, " in Proc. IEEE Int. Conf. Data Min. Workshops, 2010, pp. 170-177.
8. Storm: Distributed and fault-tolerant realtime computation [Online]. Available: http://storm.apache.org/2014.
9. M. Zaharia, T. Das, H. Li, S. Shenker, and I. Stoica, "Discretized streams: an efficient and fault-tolerant model for stream processing on large clusters, " in Proc. USENIX Conf. Hot Topics Cloud Comput., 2012, p. 10.
10. B. Satzger, W. Hummer, P. Leitner, and S. Dustdar, "Esc: Towards an elastic stream computing platform for the cloud, " in Proc. IEEE Int. Conf. Cloud Comput., Jul. 2011, pp. 348-355.
11. V. Gulisano, R. Jimenez-Peris, M. Patino-Martinez, C. Soriente, and P. Valduriez, "Streamcloud: An elastic and scalable data streaming system, " IEEE Trans. Parallel Distrib. Syst., vol. 23, no. 12, pp. 2351-2365, Dec. 2012.
12. A. Iosup, N. Yigitbasi, and D. Epema, "On the performance variability of production cloud services, " in Proc. IEEE/ACM Int. Symp. Cluster, Cloud Grid Comput., 2011, pp. 104-113.
13. A. Li, X. Yang, S. Kandula, and M. Zhang, "Cloudcmp: Comparing public cloud providers, " in Proc. ACM SIGCOMM Conf. Internet Meas., 2010, pp. 1-14.
14. I. Moreno, P. Garraghan, P. Townend, and J. Xu, "Analysis, modeling and simulation of workload patterns in a large-scale utility cloud, " IEEE Trans. Cloud Comput., vol. 2, no. 2, pp. 208-221, Apr.-Jun. 2014.
15. A. Kumbhare, Y. Simmhan, and V. K. Prasanna, "Exploiting application dynamism and cloud elasticity for continuous dataflows, " in Proc. Int. Conf. High Perform. Comput., Netw., Storage Anal., 2013, p. 57.
16. A. Arasu, M. Cherniack, E. Galvez, D. Maier, A. S. Maskey, E. Ryvkina, M. Stonebraker, and R. Tibbetts, "Linear road: A stream data management benchmark, " in Proc. Int. Conf. Very Large Databases, 2004, pp. 480-491.
17. W. M. van Der Aalst, A. H. Ter Hofstede, B. Kiepuszewski, and A. P. Barros, "Workflow patterns, " in Distrib. Parallel Databases, vol. 14, no. 1, pp. 5-51, 2003.
18. M. Maheswaran, S. Ali, H. J. Siegel, D. Hensgen, and R. F. Freund, "Dynamic mapping of a class of independent tasks onto heterogeneous computing systems, " in J. Parallel Distrib. Comput., vol. 59, no. 2, pp. 107-131, 1999.
19. J. Wainer and F. de Lima Bezerra, "Constraint-based flexible workflows, " in Groupware: Design, Implementation, and Use, Berlin, Germany: Springer, 2003, pp. 151-158.
20. C. Ouyang, E. Verbeek, W. M. Van Der Aalst, S. Breutel, M. Dumas, and A. H. Ter Hofstede, "Formal semantics and analysis of control flow in WS-BPEL, " Sci. Comput. Program., vol. 67, no. 2, pp. 162-198, 2007.
21. Y. Simmhan and A. G. Kumbhare, "Floe: A continuous dataflow framework for dynamic cloud applications, " CoRR, vol. abs/1406.5977, 2014.
22. M. C. Jaeger, G. Rojec-Goldmann, and G. Muhl, "Qos aggregation for web service composition using workflow patterns, " in Proc. IEEE Int. Conf. Enterprise Distrib. Object Comput., 2004, pp. 149-159.
23. G. J. Woeginger, "Exact algorithms for np-hard problems: A survey, " in Combinatorial Optimization-Eureka, You Shrink!. Berlin, Germany: Springer, 2003, pp. 185-207.
24. T. D. Braun, H. J. Siegel, N. Beck, L. L. Boloni, M. Maheswaran, A. I. Reuther, J. P. Robertson, M. D. Theys, B. Yao, D. Hensgen, and R. F. Freund, "A comparison of eleven static heuristics for mapping a class of independent tasks onto heterogeneous distributed computing systems, " J. Parallel Distrib. Comput., vol. 61, no. 6, pp. 810-837, Jun. 2001.
25. M. E. Frincu, "Scheduling highly available applications on cloud environments, " Future Gener. Comput. Syst., vol. 32, pp. 138-153, 2014.
26. Z. Wu, X. Liu, Z. Ni, D. Yuan, and Y. Yang, " A market-oriented hierarchical scheduling strategy in cloud workflow systems, " J. Supercomput., vol. 63, no. 1, pp. 256-293, 2013.
27. F. Zamfirache, M. Frincu, and D. Zaharie, "Population-based metaheuristics for tasks scheduling in heterogeneous distributed systems, " in Proc. 7th Int. Conf. Numerical Methods Appl., 2011, vol. 6046, pp. 321-328.
28. J. H. Holland, Adaptation in Natural and Artificial Systems. Cambridge, MA, USA: MIT Press, 1992.
29. D. E. Goldberg and K. Deb, "A comparative aAnalysis of selection schemes used in genetic algorithms, " in Foundations of Genetic Algorithms, San Mateo, CA, USA: Morgan Kaufmann 1990, pp. 69-93.
30. J. Kang and S. Park, "Algorithms for the variable sized bin packing problem, " Eur. J. Oper. Res., vol. 147, no. 2, pp. 365-372, 2003.
31. R. N. Calheiros, R. Ranjan, A. Beloglazov, C. A. De Rose, and R. Buyya, "Cloudsim: A toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms, " in Softw.: Pract. Exp., vol. 41, no. 1, pp. 23-50, 2011.
32. R. Castro Fernandez, M. Migliavacca, E. Kalyvianaki, and P. Pietzuch, "Integrating scale out and fault tolerance in stream processing using operator state management, " in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2013, pp. 725-736.
33. J. Yu and R. Buyya, "A taxonomy of scientific workflow systems for grid computing, " in ACM Sigmod Rec., vol. 34, no. 3, p. 44, 2005.
34. M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica, "Spark: Cluster computing with working sets, " in Proc. USENIX Conf. Hot Topics Cloud Comput., 2010, p. 10.
35. S. Pallickara, J. Ekanayake, and G. Fox, "Granules: A lightweight, streaming runtime for cloud computing with support, for Map-Reduce, " in Proc. IEEE Int. Conf. Cluster Comput., 2009, pp. 1-10.
36. S. Babu and J. Widom, "Continuous queries over data streams, " in ACM Sigmod Rec., vol. 30, no. 3, pp. 109-120, 2001.
37. M. A. Shah, J. M. Hellerstein, S. Chandrasekaran, and M. J. Franklin, "Flux: An adaptive partitioning operator for continuous query systems, " in Proc. IEEE Int. Conf. Data Eng., 2003, pp. 25-36.
38. Y. Zhou, B. C. Ooi, K.-L. Tan, and J. Wu, "Efficient dynamic operator placement in a locally distributed continuous query system, " in On the Move to Meaningful Internet Systems: CoopIS, DOA, GADA, and ODBASE. Berlin, Germany: Springer, 2006, pp. 54-71.
39. R. Tolosana-Calasanz, J. Angel Ba~nares, C. Pham, and O. Rana, "End-to-end qos on shared clouds for highly dynamic, large-scale sensing data streams, " in Proc. IEEE/ACM Int. Symp. Cluster, Cloud Grid Comput., 2012, pp. 904-911.
40. A. Quiroz, H. Kim, M. Parashar, N. Gnanasambandam, and N. Sharma, "Towards autonomic workload provisioning for enterprise grids and clouds, " in Proc. IEEE/ACM Int. Conf. Grid Comput., Oct. 2009, pp. 50-57.
41. Y. Hu, J. Wong, G. Iszlai, and M. Litoiu, "Resource provisioning for cloud computing, " in Proc. Conf. Center Adv. Studies Collaborative Res., 2009, pp. 101-111.
42. G. J. Nutt, "The evolution towards flexible workflow systems, " Distrib. Syst. Eng., vol. 3, no. 4, p. 276, 1996.
43. D. Guinard, V. Trifa, S. Karnouskos, P. Spiess, and D. Savio, "Interacting with the SOA-based internet of things: Discovery, query, selection, and on-demand provisioning of web services, " IEEE Trans. Serv. Comput., vol. 3, no. 3, pp. 223-235, Jul.-Aug. 2010.
44. L. Wang, H. J. Siegel, V. P. Roychowdhury, and A. A. Maciejewski, "Task matching and scheduling in heterogeneous computing environments using a genetic-algorithm-based approach, " J. Parallel Distrib. Comput., vol. 47, no. 1, pp. 8-22, 1997.
45. H. Topcuoglu, S. Hariri, and M.-Y. Wu, "Performance-effective and low-complexity task scheduling for heterogeneous computing, " IEEE Trans. Parallel Distrib. Syst., vol. 13, no. 3, pp. 260-274, Mar. 2002.
46. A. Iosup, S. Ostermann, M. N. Yigitbasi, R. Prodan, T. Fahringer, and D. H. Epema, "Performance analysis of cloud computing services for many-tasks scientific computing, " in IEEE Trans. Parallel Distrib. Syst., vol. 22, no. 6, pp. 931-945, Jun. 2011.
47. K. R. Jackson, L. Ramakrishnan, K. Muriki, S. Canon, S. Cholia, J. Shalf, H. J. Wasserman, and N. J. Wright, "Performance analysis of high Perform. Comput. applications on the amazon web services cloud, " in Proc. IEEE Int. Conf. Cloud Comput. Technol. Sci., 2010, pp. 159-168.
48. Z. Ou, H. Zhuang, A. Lukyanenko, J. Nurminen, P. Hui, V. Mazalov, and A. Yla-Jaaski, "Is the same instance type created equal exploiting heterogeneity of public clouds, " IEEE Trans. Cloud Comput., vol. 1, no. 2, pp. 201-214, Jul. 2013.
49. A. Colorni, M. Dorigo, and V. Maniezzo, "Distributed optimization by ant colonies, " in Proc. Eur. Conf. Artif. Life, 1991, pp. 134-142.
50. J. Kennedy and R. Eberhart, "Particle swarm optimization, " in Proc. IEEE Int. Conf. Neural Netw., 1995, vol. 4, pp. 1942-1948.
51. N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and E. Teller, "Equation of state calculations by fast computing machines, " J. Chem. Phys., vol. 21, no. 6, pp. 1087-1092, 1953.