Lambda architecture for cost-effective batch and speed big data processing

Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015

Volumn , Issue , 2015, Pages 2785-2792

  Kiran, Mariam ^a   Murphy, Peter ^b   Monga, Inder ^b   Dugan, Jon ^b   Baveja, Sartaj Singh ^c  

^a UNIVERSITY OF BRADFORD   (United Kingdom)

^b Energy Sciences Network (ESnet)   (United States)

^c NETAJI SUBHAS INSTITUTE OF TECHNOLOGY   (India)

Author keywords

Amazon EC2; big data processing; lambda architecture; sensor data analysis

Indexed keywords

BATCH DATA PROCESSING; COMPUTER ARCHITECTURE; COST BENEFIT ANALYSIS; COST EFFECTIVENESS; COSTS; DATA HANDLING; INFORMATION ANALYSIS; INFORMATION MANAGEMENT; INTERACTIVE COMPUTER SYSTEMS; NETWORK ARCHITECTURE; QUERY PROCESSING; SMARTPHONES;

AMAZON EC2; ARCHITECTURE DESIGNS; DATABASE MANAGEMENT; GENERAL ARCHITECTURES; HETEROGENEOUS DEVICES; NETWORK MAINTENANCES; ONLINE DATA PROCESSING; SENSOR DATA ANALYSIS;

BIG DATA;

EID: 84963716008     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/BigData.2015.7364082     Document Type: Conference Paper

References (31)

1. D. Abadi, Data Management in the Cloud: Limitations and Opportunities, IEEE Engineering Bulletin-DEBU, vol. 32, no. 1, pp. 3-12, 2009.
2. R.M. Badia, M. Corrales, T. Dimitrakos, K. Djemame, E. Elmroth, A. Ferrer, N. Forgó, J. Guitart, F. Hernández, B. Hudzia, A. Kipp, K. Konstanteli, S. Nair, T. Sharif, C. Sheridan, J. Tordsson, T. Varvarigou, S. Wesner, W. Ziegler, C. Zsigri., Demonstration of the OPTIMIS Toolkit for Cloud Service Provisioning, Towards a Service-Based Internet LNCS vol 6994, 2011.
3. J. Brown, Migration, Integration, Challenges in Government Cloud deployments, Govtech, 2015.
4. R. Buyya, C. Yeo, S. Venugopal, J. Broberg, I. Brandic, Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility FGCS 25, 2009.
5. R. Chaiken, B. Jenkins, P.-A. Larson, B. Ramsey, D. Shakib, S. Weaver, and J. Zhou. Scope: Easy and efficient parallel processing of massive data sets. In Proc. of VLDB, 2008
6. M. Dikaiakos, G. Pallis, D. Katsaros, P. Mehra, A. Vakali, Cloud Computing: Distributed Internet Computing for IT and Scientific Research, IEEE Internet Computing, 2009.
7. e-skills UK, Jan 2013 https://www.e-skills.com/research/researchthemes/ big-data-analytics/
8. I. Foster, Y. Zhao, I. Raicu, S. Lu, Cloud Computing and Grid Computing 360-Degree Compared Proceedings of the IEEE Grid Computing Environments Workshop, pp. 1-10, 2008.
9. J. N. Hoover. Start-Ups Bring Google's Parallel Processing to Data Warehousing. InformationWeek, August 29th, 2008.
10. K. Jackson, K. Muriki, L. Ramakrishnan, K. Runge, R. Thomas, Performance and cost analysis of the Supernova factory on the Amazon AWS cloud, Scientific Programming 19, 2011.
11. T. Jin, C. Tracy, M. Veeraraghavan, Characterization of high-rate largesized flows, University of Virginia, Master's thesis, 2013, Online: http://www.cs.virginia.edu/events/colloquia/jin.html
12. Z. Liu, M. Veeraraghavan, C. Tracy, J. Tie, I. Foster, J. Dennis, J. Hick, W. Yang, On using virtual circuits for GridFTP transfers, Int. Conf. for HPC, Networking, Storage and Analysis, pp. 81:1, 2012.
13. Y. Lu, M. Wang, B. Prabhakar, F. Bonomi, ElephantTrap: A low cost device for identifying large flows, 15th Annual IEEE Symposium on High-Performance Interconnects, pp. 99-108, 2007.
14. D. Menon, Benefits and challenges in deployment of Cloud solutions for SME, InsideSAP, 2014.
15. C. Olston, B. Reed, U. Srivastava, R. Kumar, A. Tomkins. Pig latin: a notso-foreign language for data processing, SIGMOD Conference, pgs 1099-1110, 2008.
16. S. Sakr, A. Liu, D. M. Batista, M. Alomari, A Survey of Large Scale Data Management Approaches in Cloud Environments. IEEE Communication Surveys & Tutorials, vol 13, no 3, 2011.
17. US DOE Office of Science ASCR, Terabit networks for extreme-scale science workshop report. Available Online: http://science.energy.gov/, 2014.
18. S. Sarvotham, R. Riedi, R. Baraniuk, Connection-level analysis and modelling of network traffic, ACM SIGCOMM Internet Measurement Workshop, pp. 99-104, 2001.
19. P. Xiong, Y. Chi, S. Zhu, H.J. Moon, C. Pu, H. Hacigumus, Intelligent Management of Virtualized Resources for Database Systems in Cloud Environment, In ICDE, 2011.
20. N. Mitton, S. Papavassiliou, A. Puliafito, K. S Trivedi, Combining Cloud and sensors in a smart city environment, EURASIP Journal on Wireless Communications and Networking, December 2012, 2012:247
21. N. Marz, J. Warren, Big Data: Principles and best practices of scalable realtime data systems. Manning Publications, 2013.
22. R. Miana, P. Martina, J.L. Vazquez-Poletti, Provisioning data analytic workloads in a cloud, Future Generation Computer Systems 29 (2013) 1452-1458
23. C. Dobre, F. Xhafa, Intelligent services for Big Data science, Future Generation Computer Systems 37 (2014) 267-281
24. A.M. Aly, A. Sallam, B.M. Gnanasekaran, L. V. Nguyen-Dinh, W.G. Aref, M. Ouzzani, A. Ghafoor, M3: stream processing on main-memory MapReduce, in: Proceedings of the 2012 IEEE 28th International Conference on Data Engineering, ICDE '12, IEEE Computer Society, Washington, DC, USA, 2012, pp. 1253-1256.
25. T.M. Ghanem, A.K. Elmagarmid, P.-A. Larson, W.G. Aref, Supporting views in data stream management systems, ACM Transactions on Database Systems 35 (2008) 1:1-1:47
26. A. Abouzied, K. Bajda-Pawlikowski, J. Huang, D. J. Abadi, and A. Silberschatz. 2010. HadoopDB in action: building real world applications. In Proceedings of the 2010 ACM SIGMOD International Conference on Management of data. ACM, New York, USA
27. Song X-W, Dong Z-Y, Long X-Y, Li S-F, Zuo X-N, Zhu C-Z, et al. (2011) REST: A Toolkit for Resting-State Functional Magnetic Resonance Imaging Data Processing. PLoS ONE 6(9): e25031. doi:10.1371/journal.pone.0025031
28. A. Brun, Y. Liang, L. Eugene. Tools and methods for capturing Twitter data during natural disasters. First Monday, [S.l.], mar. 2012. ISSN 13960466.
29. Amazon Web Services, Lambda Architecture for Batch and Stream Processing on AWS, May 2015
30. Xin, Reynold; Rosen, Josh; Zaharia, Matei; Franklin, Michael; Shenker, Scott; Stoica, Ion, Shark: SQL and Rich Analytics at Scale, 2013
31. Y. Simmhan, V. Agarwal, S. Aman, A. Kumbhare, S. Natarajan, N. Rajguru, I. Robinson, S. Stevens, W. Yin, Q. Zhou and V. Prasanna, Adaptive Energy Forecasting and Information Diffusion for Smart Power Grids, IEEE International Scalable Computing Challenge (SCALE 2012), 2012