Towards real-time analytics in the cloud

Proceedings - 2013 IEEE 9th World Congress on Services, SERVICES 2013

Volumn , Issue , 2013, Pages 428-435

  Osman, Amr ^a   El Refaey, Mohamed ^b   Elnaggar, Ayman ^a  

^a German University in Cairo ^*  (Egypt)

^b INTEL CORPORATION   (United States)

Author keywords

Cloud Computing; Real time cloud analytics; Stream Computing; Map reduce; Distributed Machine learning; Hadoop; Big data

Indexed keywords

ANALYTIC COMPUTATIONS; BIG DATUM; DATA EXPLOSION; IT SERVICES; REAL-TIME ANALYTICS; REAL-TIME PLATFORM; STATE-OF-THE-ART REVIEWS; STREAM COMPUTING;

CLOUD COMPUTING; LEARNING SYSTEMS;

INFORMATION TECHNOLOGY;

EID: 84888022876     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/SERVICES.2013.36     Document Type: Conference Paper

References (67)

1. F. Wang, R. Lee, Q. Liu, A. Aji, X. Zhang, and J. Saltz, "Hadoop-gis: A high performance query system for analytical medical imaging with mapreduce," tech. rep., Technical report, Emory University, 2011.
2. Intel, "Bigdata101: Unstructured data analytics." [Online]. Available: http://www.intel.com/content/dam/www/public/us/en/documents/ solution-briefs/big-data-101-brief.pdf, June 2012.
3. M. Hogan, F. Liu, A. Sokol, and J. Tong, "Nist cloud computing standards roadmap," NIST Special Publication, p. 35, 2011.
4. F. Michael, C. Mike, E. Christopher, and S. Josh, "Massive data analytics and the cloud: A revolution in intelligence analysis." [Online]. Available: http://www.boozallen.com/media/file/MassiveData.pdf, 2011.
5. P. Russom, "big data analytics," TDWI Best Practices Report, 4 th Quarter 2011, 2011.
6. S. Philpott and T. B. CoE, "Advanced analytics," 2010.
7. N. R., "Get analytics right from the start." [Online]. Available: http: //www.dnm.ie/documents/whitepapers/Analyticsfromthestart.pdf, February 2010.
8. J. T., "Predictive analytics in the cloud." [Online]. Available: http://smartdatacollective.com/sites/default/docs/ SDC-3twn0H3IkKM5peAzhvQ5/Predictive%20Analytics%20Cloud% 20Position%20Paper.pdf, 2011.
9. J. Dean and S. Ghemawat, "Mapreduce: simplified data processing on large clusters," Communications of the ACM, vol. 51, no. 1, pp. 107-113, 2008.
10. S. Ghemawat, H. Gobioff, and S. Leung, "The google file system," in ACM SIGOPS Operating Systems Review, vol. 37, pp. 29-43, ACM, 2003.
11. D. Borthakur, J. Gray, J. Sarma, K. Muthukkaruppan, N. Spiegelberg, H. Kuang, K. Ranganathan, D. Molkov, A. Menon, S. Rash, et al., "Apache hadoop goes realtime at facebook," in Proceedings of the 2011 international conference on Management of data, pp. 1071-1080, ACM, 2011.
12. A. Middleton, "Hpcc systems: Introduction to hpcc (high-performance computing cluster)," 2011.
13. D. Borthakur, "The hadoop distributed file system: Architecture and design," Hadoop Project Website, vol. 11, p. 21, 2007.
14. G. ZHANG, C. LI, Y. ZHANG, C. XING, and J. YANG, "Mapreduce++: Efficient processing of mapreduce jobs in the cloud," Journal of Computational Information Systems, vol. 8, no. 14, pp. 5757-5764, 2012.
15. J. Polo, D. Carrera, Y. Becerra, J. Torres, E. Ayguadé, M. Steinder, and I. Whalley, "Performance management of mapreduce applications," 2009.
16. K. Kambatla, A. Pathak, and H. Pucha, "Towards optimizing hadoop provisioning in the cloud," in Proc. of the First Workshop on Hot Topics in Cloud Computing, 2009.
17. H. Herodotou, F. Dong, and S. Babu, "Mapreduce programming and cost-based optimization? crossing this chasm with starfish," Proceedings of the VLDB Endowment, vol. 4, no. 12, 2011.
18. H. Herodotou, H. Lim, G. Luo, N. Borisov, L. Dong, F. Cetin, and S. Babu, "Starfish: A self-tuning system for big data analytics," in Proc. of the Fifth CIDR Conf, 2011.
19. J. Polo, C. Castillo, D. Carrera, Y. Becerra, I. Whalley, M. Steinder, J. Torres, and E. Ayguadé, "Resource-aware adaptive scheduling for mapreduce clusters," Middleware2011, pp. 187-207, 2011.
20. T. Sandholm and K. Lai, "Mapreduce optimization using regulated dynamic prioritization," in Proceedings of the eleventh international joint conference on Measurement and modeling of computer systems, pp. 299-310, ACM, 2009.
21. L. Phan, Z. Zhang, Q. Zheng, B. Loo, and I. Lee, "An empirical analysis of scheduling techniques for real-time cloud-based data processing," in IEEE International Conference on Service-Oriented Computing and Applications (SOCA), 2011, pp. 1-8, IEEE, 2011.
22. D. Jiang, B. Ooi, L. Shi, and S. Wu, "The performance of mapreduce: An in-depth study," Proceedings of the VLDB Endowment, vol. 3, no. 1-2, pp. 472-483, 2010.
23. C. Gkantsidis, D. Vytiniotis, O. Hodson, D. Narayanan, and A. Rowstron, "Automatic io filtering for optimizing cloud analytics microsoft technical report msr-tr-2012-3,"
24. A. Thusoo, J. Sarma, N. Jain, Z. Shao, P. Chakka, N. Zhang, S. Antony, H. Liu, and R. Murthy, "Hive-A petabyte scale data warehouse using hadoop," in IEEE 26th International Conference on Data Engineering (ICDE), 2010, pp. 996-1005, IEEE, 2010.
25. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins, "Pig latin: a not-so-foreign language for data processing," in Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pp. 1099-1110, ACM, 2008.
26. A. Khetrapal and V. Ganesh, "Hbase and hypertable for large scale distributed storage systems," Dept. of Computer Science, Purdue University, 2006.
27. F. Chang, J. Dean, S. Ghemawat, W. Hsieh, D. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. Gruber, "Bigtable: A distributed storage system for structured data," ACM Transactions on Computer Systems (TOCS), vol. 26, no. 2, p. 4, 2008.
28. A. Lakshman and P. Malik, "Cassandra: A structured storage system on a p2p network," in Proceedings of the twenty-first annual symposium on Parallelism in algorithms and architectures, pp. 47-47, ACM, 2009.
29. J. A. Hartigan and M. A. Wong, "Algorithm as 136: A k-means clustering algorithm," Applied statistics, pp. 100-108, 1979.
30. D. E. Goldberg, "Genetic algorithms in search, optimization, and machine learning," 1989.
31. J. Ekanayake, H. Li, B. Zhang, T. Gunarathne, S. Bae, J. Qiu, and G. Fox, "Twister: a runtime for iterative mapreduce," in Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, pp. 810-818, ACM, 2010.
32. Y. Bu, B. Howe, M. Balazinska, and M. Ernst, "Haloop: Efficient iterative data processing on large clusters," Proceedings of the VLDB Endowment, vol. 3, no. 1-2, pp. 285-296, 2010.
33. T. Condie, N. Conway, P. Alvaro, J. M. Hellerstein, J. Gerth, J. Talbot, K. Elmeleegy, and R. Sears, "Online aggregation and continuous query support in mapreduce," in ACM SIGMOD, pp. 1115-1118, 2010.
34. T. Condie, N. Conway, P. Alvaro, J. M. Hellerstein, K. Elmeleegy, and R. Sears, "Mapreduce online," in Proceedings of the 7th USENIX conference on Networked systems design and implementation, pp. 21-21, 2010.
35. C. Olston, G. Chiou, L. Chitnis, F. Liu, Y. Han, M. Larsson, A. Neumann, V. B. Rao, V. Sankarasubramanian, S. Seth, et al., "Nova: continuous pig/hadoop workflows," in Proceedings of the 2011 international conference on Management of data, pp. 1081-1090, ACM, 2011.
36. M. Zaharia, M. Chowdhury, M. Franklin, S. Shenker, and I. Stoica, "Spark: cluster computing with working sets," in Proceedings of the 2nd USENIX conference on Hot topics in cloud computing, pp. 10-10, USENIX Association, 2010.
37. M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. Franklin, S. Shenker, and I. Stoica, "Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing," in Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation, 2011.
38. J. Boulon, A. Konwinski, R. Qi, A. Rabkin, E. Yang, and M. Yang, "Chukwa, a large-scale monitoring system," in Proceedings of CCA, vol. 8, 2008.
39. L. Neumeyer, B. Robbins, A. Nair, and A. Kesari, "S4: Distributed stream computing platform," in IEEE International Conference on Data Mining Workshops (ICDMW), 2010, pp. 170-177, IEEE, 2010.
40. M. Cherniack, H. Balakrishnan, M. Balazinska, D. Carney, U. Cetintemel, Y. Xing, and S. Zdonik, "Scalable distributed stream processing," in Proc. Conf. on Innovative Data Syst. Res, 2003.
41. M. Nathan, X. James, and J. Jason, "Storm: Distributed real-time computation system." [Online]. Available: http://storm-project.net/, 2012.
42. "Hstreaming." [Online]. Available: http://www.hstreaming.com/, 2011.
43. S. Melnik, A. Gubarev, J. J. Long, G. Romer, S. Shivakumar, M. Tolton, and T. Vassilakis, "Dremel: interactive analysis of web-scale datasets," Proceedings of the VLDB Endowment, vol. 3, no. 1-2, pp. 330-339, 2010.
44. M. Zaharia, T. Das, H. Li, S. Shenker, and I. Stoica, "Discretized streams: an efficient and fault-tolerant model for stream processing on large clusters," in Proceedings of the 4th USENIX conference on Hot Topics in Cloud Ccomputing, pp. 10-10, USENIX Association, 2012.
45. M. Zaharia, T. Das, H. Li, T. Hunter, S. Shenker, and I. Stoica, "Discretized streams: A fault-tolerant model for scalable stream processing," tech. rep., UC Berkeley Technical Report UCB/EECS-2012-259, 2012.
46. M. Sax, M. Castellanos, Q. Chen, and M. Hsu, "Aeolus: An optimizer for distributed intra-node-parallel streaming systems," in Demo), Will be presented at Int. Conf. on Data Engineering, 2013.
47. C. Chu, S. K. Kim, Y.-A. Lin, Y. Yu, G. Bradski, A. Y. Ng, and K. Olukotun, "Map-reduce for machine learning on multicore," Advances in neural information processing systems, vol. 19, p. 281, 2007.
48. S. Venkataraman, I. Roy, R. S. Schreiber, and A. AuYoung, "Presto: Complex and continuous analytics with distributed arrays," 2011.
49. T. H. T. D. M. Zaharia, A. Bayen, and P. Abbeel, "Large-scale online expectation maximization with spark streaming,"
50. K.-C. Lee and D. Kriegman, "Online learning of probabilistic appearance manifolds for video-based recognition and tracking," in IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005., vol. 1, pp. 852-859, IEEE, 2005.
51. M. D. Hoffman, D. M. Blei, and F. Bach, "Online learning for latent dirichlet allocation," Advances in Neural Information Processing Systems, vol. 23, pp. 856-864, 2010.
52. R. S. Sutton, S. D. Whitehead, et al., "Online learning with random representations," in Proceedings of the Tenth International Conference on Machine Learning, pp. 314-321, Citeseer, 1993.
53. S. Shalev-Shwartz, Y. Singer, and A. Y. Ng, "Online and batch learning of pseudo-metrics," in Proceedings of the twenty-first international conference on Machine learning, p. 94, ACM, 2004.
54. K. Crammer, O. Dekel, J. Keshet, S. Shalev-Shwartz, and Y. Singer, "Online passive-aggressive algorithms," The Journal of Machine Learning Research, vol. 7, pp. 551-585, 2006.
55. J. Kivinen, A. J. Smola, and R. C. Williamson, "Online learning with kernels," Signal Processing, IEEE Transactions on, vol. 52, no. 8, pp. 2165-2176, 2004.
56. S. Shalev-Shwartz, "Online learning: Theory, algorithms, and applications," 2007.
57. A. Basak, I. Brinster, and O. J. Mengshoel, "Mapreduce for bayesian network parameter learning using the em algorithm," Proc. of Big Learning: Algorithms, Systems and Tools, 2012.
58. E. B. Reed and O. J. Mengshoel, "Scaling bayesian network parameter learning with expectation maximization using mapreduce," Proc. of Big Learning: Algorithms, Systems and Tools, 2012.
59. A. G. Schwing, T. Hazan, M. Pollefeys, and R. Urtasun, "Distributed structured prediction for big data,"
60. J. Ye, J.-H. Chow, J. Chen, and Z. Zheng, "Stochastic gradient boosted distributed decision trees," in Proceedings of the 18th ACM conference on Information and knowledge management, pp. 2061-2064, ACM, 2009.
61. S. Daruru, N. M. Marin, M. Walker, and J. Ghosh, "Pervasive parallelism in data mining: dataflow solution to co-clustering large and sparse netflix data," in Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1115-1124, ACM, 2009.
62. D. Peng and F. Dabek, "Large-scale incremental processing using distributed transactions and notifications," in Proceedings of the 9th USENIX Symposium on Operating Systems Design and Implementation, 2010.
63. S. Arnuk and J. Saluzzi, "Latency arbitrage: The real power behind predatory high frequency trading," Themis Trading white paper, 2009.
64. L. D. Baskar, B. De Schutter, and H. Hellendoorn, "Model-based predictive traffic control for intelligent vehicles: Dynamic speed limits and dynamic lane allocation," in Intelligent Vehicles Symposium, 2008 IEEE, pp. 174-179, IEEE, 2008.
65. B. L. Smith and M. J. Demetsky, "Short-term traffic flow prediction models-A comparison of neural network and nonparametric regression approaches," in IEEE International Conference on Systems, Man, and Cybernetics, 1994.'Humans, Information and Technology'., 1994, vol. 2, pp. 1706-1709, IEEE, 1994.
66. M. Stonebraker, U. Çetintemel, and S. Zdonik, "The 8 requirements of real-time stream processing," ACM SIGMOD Record, vol. 34, no. 4, pp. 42-47, 2005.
67. B. Aleman-Meza, M. Nagarajan, L. Ding, A. Sheth, I. B. Arpinar, A. Joshi, and T. Finin, "Scalable semantic analytics on social networks for addressing the problem of conflict of interest detection," ACM Transactions on the Web (TWEB), vol. 2, no. 1, p. 7, 2008.