Scalable analysis of network measurements with Hadoop and Pig

Proceedings of the 2012 IEEE Network Operations and Management Symposium, NOMS 2012

Volumn , Issue , 2012, Pages 1254-1259

  Samak, Taghrid ^a   Gunter, Daniel ^a   Hendrix, Valerie ^a  

^a LAWRENCE BERKELEY NATIONAL LABORATORY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CROSS-CUTTING; DATA SETS; DISTRIBUTED MONITORING; LARGE-SCALE ANALYSIS; NETWORK MANAGEMENT SERVICE; NETWORK MEASUREMENT; OFF-LINE ANALYSIS; PERFSONAR; REAL MEASUREMENTS; SCALABILITY ISSUE; SCALABLE ANALYSIS;

NETWORK PERFORMANCE; VISUALIZATION;

NETWORK MANAGEMENT;

EID: 84864188519     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/NOMS.2012.6212060     Document Type: Conference Paper

References (17)

1. perfSONAR. http://www.perfsonar.net/.
2. pS-BUOY. http://psps.perfsonar.net/psb/index.html.
3. R. www.r-project.org.
4. Apache avro. http://avro.apache.org/.
5. Apache hadoop. http://hadoop.apache.org/.
6. Apache pig. http://pig.apache.org/.
7. Chuck Cranor, Theodore Johnson, Oliver Spataschek, and Vladislav Shkapenyuk. Gigascope: a stream database for network applications. In Proceedings of the 2003 ACM SIGMOD international conference on Management of data, SIGMOD '03, pages 647-651, New York, NY, USA, 2003. ACM.
8. Jeffrey Dean and Sanjay Ghemawat. Mapreduce: simplified data processing on large clusters. In Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation - Volume 6, pages 10-10, Berkeley, CA, USA, 2004. USENIX Association.
9. Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung. The google file system. In Proceedings of the nineteenth ACM symposium on Operating systems principles, SOSP '03, pages 29-43, New York, NY, USA, 2003. ACM.
10. Y. Lee, W. Kang, and Y. Lee. A hadoop-based packet trace processing tool. Third International Workshop on Traffic Monitoring and Analysis, pages 51-63, 2011.
11. Youngseok Lee, Wonchul Kang, and Hyeongu Son. An internet traffic analysis method with mapreduce. In Network Operations and Management Symposium Workshops (NOMS Wksps), 2010 IEEE/IFIP, pages 357 -361, april 2010.
12. Brian Tierney, Jeff Boote, Eric Boyd, Aaron Brown, Maxim Grigoriev, Joe Metzger, Martin Swany, Matt Zekauskas, Yee-Ting Li, and Jason Zurawski. Instantiating a global network measurement framework. Technical report, Lawrence Berkeley National Laboratory, January 2009.
13. Tom White. Hadoop: The Definitive Guide. O'Reilly Media, original edition, June 2009.
14. Tomasz Wiktor Wlodarczyk, Chunming Rong, Csongor Nyulas, and Mark A. Musen. An efficient approach to intelligent real-time monitoring using ontologies and hadoop. In HPCS'10, pages 209-215, 2010.
15. Wei Xu, Ling Huang, Armando Fox, David Patterson, and Michael I. Jordan. Mining console logs for large-scale system problem detection. In 3rd Workshop on Tackling System Problems with Machine Learning Techniques (SysML'08), 2008.
16. Wei Xu, Ling Huang, Armando Fox, David Patterson, and Michael I. Jordan. Detecting large-scale system problems by mining console logs. In Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles, SOSP '09, pages 117-132, New York, NY, USA, 2009. ACM.
17. J. Zurawski, M. Swany, and D. Gunter. A scalable framework for representation and exchange of network measurements. In TRIDENTCOM, 2006.