Toward scalable internet traffic measurement and analysis with hadoop

Computer Communication Review

Volumn 43, Issue 1, 2013, Pages 5-13

  Lee, Yeonhee ^a   Lee, Youngseok ^a  

^a Chungnam National University   (South Korea)

Author keywords

Analysis; Hadoop; Hive; MapReduce; NetFlow; Packet; Pcap; Traffic measurement

Indexed keywords

ANALYSIS; HADOOP; HIVE; MAP-REDUCE; NETFLOWS; PACKET; PCAP; TRAFFIC MEASUREMENTS;

BEHAVIORAL RESEARCH; DATA PROCESSING; DISTRIBUTED COMPUTER SYSTEMS; INTERNET; INTERNET PROTOCOLS;

HTTP;

EID: 84872468737     PISSN: 01464833     EISSN: 19435819     Source Type: Conference Proceeding    
DOI: 10.1145/2427036.2427038     Document Type: Review

References (24)

1. Cisco White Paper, Cicso Visual Networking Index: Forecast and Methodology, 2011-2016, May 2012.
2. K. Cho, K. Fukuda, H. Esaki, and A. Kato, Observing slow crustal movement in residential user traffic, ACM CoNEXT2008.
3. J. Dean and S. Ghemawat, MapReduce: Simplified Data Processing on Large Cluster, USENIX OSDI, 2004.
4. S. Ghemawat, H. Gobioff, and S. Leung, The Google file system, ACM SOSP, 2003.
5. Hadoop, http://hadoop.apache.org/.
6. T. White, Hadoop: the Definitive Guide, O'Reilly, 3rd ed., 2012
7. A. Thusoo, J. Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony, H. Liu, P. Wyckoff, and R. Murthy, Hive: a warehousing solution over a map-reduce framework, VLDB, August 2009.
8. Tcpdump, http://www.tcpdump.org.
9. Wireshark, http://www.wireshark.org.
10. CAIDA CoralReef Software Suite, http://www.caida.org/tools/measurement/ coralreef.
11. M. Roesch, Snort - Lightweight Intrusion Detection for Networks, USENIX LISA, 1999.
12. Bro, http://www.bro-ids.org.
13. M. Vallentin, R. Sommer, J. Lee, C. Leres, V. Paxson, and B. Tierney, The NIDS Cluster: Scalable, Stateful Network Intrusion Detection on Commodity Hardware, International Conference on Recent Advances in Intrusion Detection (RAID), 2007.
14. A. Finamore, M. Mellia, M. Meo, M. M. Munafo, and D. Rossi, Live traffic monitoring with tstat: Capabilities and experiences, 8th International Conference on Wired/Wireless Internet Communication, 2010.
15. Cisco NetFlow, http://www.cisco.com/web/go/netflow.
16. M. Fullmer and S. Romig, The OSU Flow-tools Package and Cisco NetFlow Logs, USENIX LISA, 2000.
17. D. Plonka, FlowScan: a Network Traffic Flow Reporting and Visualizing Tool, USENIX Conference on System Administration, 2000.
18. QoSient, LLC, argus: newtork audit record generation and utilization system, http://www.qosient.com/argus/.
19. Arbor Networks, http://www.arbornetworks.com.
20. Y. Lee, W. Kang, and Y. Lee, A Hadoop-based Packet Trace Processing Tool, International Workshop on Traffic Monitoring and Analysis (TMA 2011), April 2011.
21. RIPE Hadoop PCAP, https://labs.ripe.net/Members/wnagele/large-scalepcap- data-analysis-using-apache-hadoop, Nov. 2011.
22. F. Fusco and L. Deri, High Speed Network Traffic Analysis with Commodity Multi-core Systems, ACM IMC 2010, Nov. 2010.
23. Y. Lee and Y. Lee, Detecting DDoS Attacks with Hadoop, ACM CoNEXT Student Workshop, Dec. 2011.
24. CNU Project on traffic analysis in Hadoop, https://sites.google.com/a/ networks.cnu.ac.kr/dnlab/research/hadoop.