Log analysis in cloud computing environment with Hadoop and Spark

Proceedings of 2013 5th IEEE International Conference on Broadband Network and Multimedia Technology, IEEE IC-BNMT 2013

Volumn , Issue , 2013, Pages 273-276

  Lin, Xiuqin ^a   Wang, Peng ^a   Wu, Bin ^a  

^a BEIJING UNIVERSITY OF POSTS AND TELECOMMUNICATIONS   (China)

Author keywords

Hadoop; Hive; Log analysis; Shark; Spark

Indexed keywords

BEHAVIORAL RESEARCH; BROADBAND NETWORKS; CLOUD COMPUTING; COMPUTER SYSTEMS; DATA PROCESSING; DATA WAREHOUSES; ELECTRIC SPARKS; MULTIMEDIA SYSTEMS;

CLOUD COMPUTING ENVIRONMENTS; COMPUTING FRAMEWORKS; DISTRIBUTED FILE SYSTEMS; HADOOP; HIVE; LOG ANALYSIS; SHARK; USER BEHAVIOR ANALYSIS;

WELL LOGGING;

EID: 84903309712     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICBNMT.2013.6823956     Document Type: Conference Paper

References (15)

1. Webalizer. http://www. webalizer. com/.
2. Awstats. http://sourceforge. net/projects/awstats/
3. Hadoop. Hadoop Homepage. http://hadoop. apache. org/.
4. THUSOO A, SARMA J S, JAIN N, et al. Hive-a petabyte scale data warehouse using hadoop[C]//Data Engineering (ICDE), 2010 IEEE 26th International Conference on. 2010: 996-1005.
5. THERDPHAPIYANAK J, PIROMSOPA K. Applying Hadoop for log analysis toward distributed IDS[C]//Proceedings of the 7th International Conference on Ubiquitous Information Management and Communication. New York, NY, USA: ACM, 2013: 3:1-3:6.
6. Spark. Spark Homepage http://spark-project. org/.
7. Apache Hive. http://hive. apache. org/.
8. ZAHARIA M, CHOWDHURY M, FRANKLIN M J, et al. Spark: cluster computing with working sets[C]//Proceedings of the 2nd USENIX conference on Hot topics in cloud computing. 2010: 10-10.
9. ZAHARIA M, CHOWDHURY M, DAS T, et al. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing[C]//Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation. 2012: 2-2.
10. ENGLE C, LUPHER A, XIN R, et al. Shark: fast data analysis using coarse-grained distributed memory[C]//Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data. 2012: 689-692.
11. G. Ananthanarayanan et al. Pacman: Coordinated memory caching for parallel jobs. In NSDI, 2012.
12. XIN R, ROSEN J, ZAHARIA M, et al. Shark: SQL and rich analytics at scale [J]. arXiv preprint arXiv:1211. 6176, 2012.
13. HINDMAN B, KONWINSKI A, ZAHARIA M, et al. Mesos: A platform for fine-grained resource sharing in the data center[C]//Proceedings of the 8th USENIX conference on Networked systems design and implementation. 2011: 22-22.
14. GHODSI A, ZAHARIA M, HINDMAN B, et al. Dominant resource fairness: fair allocation of multiple resource types[C]//USENIX NSDI. 2011.
15. J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. In OSDI, 2004.