Similarity as a central approach to flow-based anomaly detection

International Journal of Network Management

Volumn 24, Issue 4, 2014, Pages 318-336

  Drašar, Martin ^a   Vizváry, Martin ^a   Vykopal, Jan ^a  

^a MASARYK UNIVERSITY   (Czech Republic)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER APPLICATIONS; COMPUTER NETWORKS;

ACADEMIC CONFERENCES; ANOMALY DETECTION; ANOMALY DETECTION METHODS; DETECTION METHODS; FALSE POSITIVE RATES; LARGE-SIZE NETWORKS; NETWORK ANOMALIES; STATE OF THE ART;

DATA PROCESSING;

EID: 84904187835     PISSN: 10557148     EISSN: 10991190     Source Type: Journal    
DOI: 10.1002/nem.1867     Document Type: Article

References (94)

1. Steinberger J, Schehlmann L, Abt S, Baier H,. Anomaly detection and mitigation at Internet scale: a survey. In Emerging Management Mechanisms for the Future Internet, Doyen G, Waldburger M, Čeleda P, Sperotto A, Stiller B, (eds.), Lecture Notes in Computer Science, vol. 7943. Springer: Berlin, 2013; 49-60.
2. Sperotto A, Schaffrath G, Sadre R, Morariu C, Pras A, Stiller B,. An overview of IP flow-based intrusion detection, IEEE Communications Surveys and Tutorials 2010; 12 (3): 343-356.
3. Li B, Springer J, Bebis G, Gunes MH,. A survey of network flow applications, Journal of Network and Computer Applications 2013; 36 (2): 567-581.
4. Bhuyan M, Bhattacharyya D, Kalita J,. Network anomaly detection: methods, systems and tools, IEEE Communications Surveys and Tutorials 2014; 16: 303-336.
5. Kruegel C, Valeur F, Vigna G,. Intrusion Detection and Correlation: Challenges and Solutions, Advances in Information Security, vol. 14, Springer: Berlin, 2005.
6. Vigna G,. Network intrusion detection: dead or alive? In Proceedings of the 26th Annual Computer Security Applications Conference: ACSAC'10, ACM: New York, 2010; 117-126.
7. Hofstede R, Drago I, Sperotto A, Sadre R, Pras A,. Measurement artifacts in NetFlow data. In Proceedings of the 14th International Conference on Passive and Active Measurement, Lecture Notes in Computer Science, vol. 7799. Springer: Berlin, 2013; 1-10.
8. Brauckhoff D, Tellenbach B, Wagner A, May M, Lakhina A,. Impact of packet sampling on anomaly detection metrics, In Proceedings of the 6th ACM SIGCOMM Conference on Internet measurement: IMC'06. ACM: New York, 2006; 159-164. (Pubitemid 47165598)
9. Mai J, Chuah CN, Sridharan A, Ye T, Zang H,. Is sampled data sufficient for anomaly detection? In Proceedings of the 6th ACM SIGCOMM Conference on Internet Measurement: IMC'06. ACM: New York, 2006; 165-176. (Pubitemid 47165599)
10. Qiao P, Huang Yf, Zeng Pf,. Reduction of traffic sampling impact on anomaly detection, In Proceedings of the 7th International Conference on Computer Science Education, Melbourne, Australia, 2012; 438-443.
11. Deri L, Luconi Trombacchi L, Martinelli M, Vannozzi D,. A distributed DNS traffic monitoring system, In Proceedings of the 8th International Conference for Wireless Communications and Mobile Computing, Limassol, Cyprus, 2012; 30-35.
12. Velan P, Jirsík T, Čeleda P,. Design and evaluation of HTTP protocol parsers for IPFIX measurement. In Advances in Communication Networking, Lecture Notes in Computer Science, vol. 8115. Springer: Berlin, 2013; 136-147.
13. Elich M, Velan P, Jirsík T, Čeleda P,. An Investigation Into Teredo and 6to4 transition mechanisms: traffic analysis, In IEEE 38th Conference on Local Computer Networks, Sydney, Australia, 2013; 1046-1052.
14. Miller KW, Voas J, Hurlburt GF,. BYOD: Security and privacy considerations, IT Professional 2012; 14 (5): 53-55.
15. Atzori L, Iera A, Morabito G,. The Internet of Things: a survey, Computer Networks 2010; 54 (15): 2787-2805.
16. Trammell B, Boschi E,. An introduction to IP flow information export (IPFIX), IEEE Communications Magazine 2011; 49 (4): 89-95.
17. Claise B, Trammell B, Aitken P,. Specification of the IP Flow Information Export (IPFIX) Protocol for the exchange of flow information RFC 7011 (Internet Standard), September 2013. Available: http://www.ietf.org/rfc/rfc7011.txt [31 October 2013].
18. Igure V, Williams R,. Taxonomies of attacks and vulnerabilities in computer systems, IEEE Communications Surveys Tutorials 2008; 10 (1): 6-19.
19. Hansman S, Hunt R,. A taxonomy of network and computer attacks, Computers and Security 2005; 24 (1): 31-43. (Pubitemid 40415398)
20. Zezula P, Amato G, Dohnal V, Batko M,. Similarity Search: The Metric Space Approach, Springer: New York, 2005.
21. Cha SH,. Comprehensive survey on distance/similarity measures between probability density functions, International Journal of Mathematical Models and Methods in Applied Sciences 2007; 1 (4): 300-307.
22. Choi SS, Cha SH, Tappert C,. A Survey of binary similarity and distance measures, Journal on Systemics, Cybernetics and Informatics 2010; 8 (1): 43-48.
23. Lesot MJ, Rifqi M, Benhadda H,. Similarity measures for binary and numerical data: a survey, International Journal of Knowledge Engineering and Soft Data Paradigms 2009; 1 (1): 63-84.
24. Boriah S, Chandola V, Kumar V,. Similarity measures for categorical data: a comparative evaluation, In Proceedings of the 8th SIAM International Conference on Data Mining, Atlanta, Georgia, USA, 2008; 243-254.
25. Dübendorfer T, Plattner B,. Host behaviour based early detection of worm outbreaks in Internet backbones, In Proceedings of the 14th IEEE International Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprise, Linköping, Sweden, 2005; 166-171. (Pubitemid 46090869)
26. Vykopal J, Plesník T, Minařík P,. Network-based dictionary attack detection, In Proceedings of the International Conference on Future Networks, Bangkok, Thailand, 2009; 23-27.
27. Krmíček V, Plesník T,. Detecting botnets with NetFlow, 2011. Available: http://www.cert.org/flocon/2011/proceedings.html [31 October 2013].
28. Galtsev AA, Sukhov AM,. Network attack detection at flow level. In Smart Spaces and Next Generation Wired/Wireless Networking, Balandin S, Koucheryavy Y, Hu H, (eds.), Lecture Notes in Computer Science, vol. 6869. Springer: Berlin, 2011; 326-334.
29. Lakhina A, Crovella M, Diot C,. Diagnosing network-wide traffic anomalies, SIGCOMM Computer Communication Review 2004; 34 (4): 219-230. (Pubitemid 40954882)
30. D'Alconzo A, Coluccia A, Ricciato F, Romirer-Maierhofer P,. A distribution-based approach to anomaly detection and application to 3G mobile traffic, In Proceedings of IEEE Global Telecommunications Conference, Honolulu, Hawaii, USA, 2009; 1-8.
31. Mazzola C, Tragesser T,. Security Incident discovery and correlation on.gov networks, In CERT FloCon Workshop: Salt Lake City, UT, 2011. Available: http://www.cert.org/flocon/2011/presentations/Mazzola-Security.pdf [31 October 2013].
32. Drašar M,. Protocol-independent detection of dictionary attacks. In Advances in Communication Networking, Bauschert T, (ed.), Lecture Notes in Computer Science, vol. 8115. Springer: Berlin, 2013; 304-309.
33. Goldfarb J,. Identifying anomalous network traffic through the use of client port distribution, In CERT FloCon Workshop: Vancouver, WA, 2006. Available: http://www.cert.org/flocon/2006/presentations/clientport-dist1205.pdf [31 October 2013].
34. François J, Wang S, State R, Engel T,. BotTrack: tracking botnets using NetFlow and PageRank. In IFIP NETWORKING 2011, Domingo-Pascual J, Manzoni P, Palazzo S, Pont A, Scoglio C, (eds.), Lecture Notes in Computer Science, vol. 6640. Springer: Berlin, 2011; 1-14.
35. François J, Wang S, Bronzi W, State R, Engel T,. BotCloud: detecting botnets using MapReduce, In Proceedings of the IEEE International Workshop on Information Forensics and Security, Iguacu Falls, Brazil, 2011; 1-6.
36. Dewaele G, Himura Y, Borgnat P, Fukuda K, Abry P, Michel O, Fontugne R, Cho K, Esaki H,. Unsupervised host behavior classification from connection patterns, International Journal of Network Management 2010; 20 (5): 317-337.
37. Gao Y, Li Z, Chen Y,. A DoS resilient flow-level intrusion detection approach for high-speed networks, In Proceedings of the 26th IEEE International Conference on Distributed Computing Systems, 2006; 39-46.
38. Salem O, Vaton S, Gravey A,. A scalable, efficient and informative approach for anomaly- based intrusion detection systems: theory and practice, International Journal of Network Management 2010; 20 (5): 271-293.
39. Yan R, Shao C,. Hierarchical method for anomaly detection and attack identification in high-speed network, Information Technology Journal 2012; 11 (9): 1243-1250.
40. Ishibashi K, Kondoh T, Harada S, Mori T, Kawahara R, Asano S,. Detecting anomalies in interhosts communication graph, In CERT FloCon Workshop: Scottsdale, AZ, 2009. Available: http://www.cert.org/flocon/2009/presentations/ Ishibashi-GraphAnomalies.pdf [31 October 2013].
41. Yurcik W,. Visflowconnect-ip: a link-based visualization of NetFlows for security monitoring, VizSEC '06 Proceedings of the 3rd International Workshop on Visualization for Computer Security 2006.
42. Tegeler F, Fu X, Vigna G, Kruegel C,. BotFinder: finding bots in network traffic without deep packet inspection, In Proceedings of the 8th International Conference on Emerging Networking Experiments and Technologies: CoNEXT'12. ACM: New York, 2012; 349-360.
43. Jung J, Paxson V, Berger AW, Balakrishnan H,. Fast portscan detection using sequential hypothesis testing, In Proceedings of the IEEE Symposium on Security and Privacy, Berkeley, CA, USA, 2004; 211-225.
44. Nagaonkar V, McHugh J,. Revisiting the threshold random walk scan detector, In CERT FloCon Workshop: Savannah, GA, 2008. Available: http://www.cert.org/flocon/2008/presentations/flocon08-mchugh-vagi.pdf [31 October 2013].
45. Xu K, Zhang ZL, Bhattacharyya S,. Profiling Internet backbone traffic: behavior models and applications, In Proceedings of the 2005 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications: SIGCOMM'05. ACM: New York, 2005; 169-180. (Pubitemid 46323502)
46. The Cyber Systems and Technology Group. DARPA intrusion detection data sets. Available: http://www.ll.mit.edu/mission/communications/cyber/CSTcorpora/ ideval/data/ [31 October 2013].
47. The MAWI Working Group. MAWI Working Group traffic archive. Available: http://mawi.wide.ad.jp/mawi/ [31 October 2013].
48. KDD Cup 1999 data. Available: http://kdd.ics.uci.edu/databases/kddcup99/ kddcup99.html [31 October 2013].
49. Nehinbe JO,. A critical evaluation of datasets for investigating IDSs and IPSs researches, In Proceedings of the 10th IEEE International Conference on Cybernetic Intelligent Systems, London, UK, 2011; 92-97.
50. McHugh J,. Testing intrusion detection systems: a critique of the 1998 and 1999 DARPA intrusion detection system evaluations as performed by Lincoln Laboratory, ACM Transactions on Information and System Security; 3 (4): 262-294.
51. Mahoney MV, Chan PK,. An analysis of the 1999 DARPA/Lincoln Laboratory evaluation data for network anomaly detection. In Recent Advances in Intrusion Detection, Vigna G, Kruegel C, Jonsson E, (eds.), Lecture Notes in Computer Science, vol. 2820, 2003; 220-237. (Pubitemid 137633167)
52. Tavallaee M, Bagheri E, Lu W, Ghorbani AA,. A detailed analysis of the KDD CUP 99 data set, In Proceedings of the 2009 IEEE Symposium on Computational Intelligence in Security and Defense Applications, Ottawa, ON, Canada, 2009; 1-6.
53. Tavallaee M, Bagheri E, Lu W, Ghorbani AA,. NSL-KDD data set. Available: http://nsl.cs.unb.ca/NSL-KDD/ [31 October 2013].
54. Gogoi P, Bhuyan MH, Bhattacharyya DK, Kalita JK,. Packet and flow based network intrusion dataset. In Contemporary Computing, Parashar M, Kaushik D, Zomaya A, (eds.), Vol. 306. Springer: Berlin, 2012; 322-334.
55. Fontugne R, Borgnat P, Abry P, Fukuda K,. Mawilab: combining diverse anomaly detectors for automated anomaly labeling and performance benchmarking, In Proceedings of the 6th International Conference: Co-NEXT'10. ACM: New York, 2010.
56. Tavallaee M, Stakhanova N, Ghorbani AA,. Toward credible evaluation of anomaly-based intrusion- detection methods, IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews 2010; 40 (5): 516-524.
57. Sperotto A, Sadre R, van Vliet F, Pras A,. A labeled data set for flow-based intrusion detection. In Proceedings of the 9th IEEE International Workshop on IP Operations and Management, Venice, Lecture Notes in Computer Science, vol. 5843. Springer: Berlin, 2009; 39-50.
58. Sangster B, O'Connor TJ, Cook T, Fanelli R, Dean E, Morrell C, Conti G,. Toward instrumenting network warfare competitions to generate labeled datasets, In Proceedings of the 2nd Workshop on Cyber Security Experimentation and Test, Montreal, Canada, 2009.
59. Ringberg H, Roughan M, Rexford J,. The need for simulation in evaluating anomaly detectors, SIGCOMM Computer Communication Review 2008; 38 (1): 55-59.
60. Bartoš V, Žádík M,. Framework for comparison of network anomaly detection algorithms. Technical Report, Faculty of Information Technology BUT, 2012. Available: http://www.fit.vutbr.cz/research/ view-pub.php?id=10070 [31 October 2013].
61. Axelsson S,. The base-rate fallacy and the difficulty of intrusion detection, ACM Transactions on Information and System Security 2000; 3 (3): 186-205.
62. Vykopal J,. A flow-level taxonomy and prevalence of brute force attacks. In Advances in Computing and Communications, Abraham A, Mauri JL, Buford JF, Suzuki J, Thampi SM, (eds.), Communications in Computer and Information Science, vol. 191. Springer: Berlin, 2011; 666-675.
63. Rehák M, Čeleda P, Pěchouček M, Novotný J,. CAMNEP: multistage collective network behavior analysis system with hardware accelerated NetFlow probes, In CERT FloCon Workshop: Scottsdale, AZ, 2009. Available: http://www.cert.org/flocon/2009/presentations/ Rehak-Camnep.pdf [31 October 2013].
64. Mahimkar A, Lall A, Wang J, Xu J, Yates J, Zhao Q,. SYNERGY: detecting and diagnosing correlated network anomalies. AT&T Labs Research, 2009. Available: http://www.research.att.com/export/sites/att-labs/techdocs/TD-7KEJWS. pdf [31 October 2013].
65. Wang Y, Zhang X,. Complex event processing over distributed probabilistic event streams, In Proceedings of the 9th International Conference on Fuzzy Systems and Knowledge Discovery, Chongqing, Sichuan, China, 2012; 1489-1493.
66. McHugh J,. Considerations for scan detection using flow data, In CERT FloCon Workshop: Albuquerque, NM, 2013. Available: http://www.cert.org/flocon/ 2013/presentations/mchugh-john-scan-detection.pdf [31 October 2013].
67. Kim MS, Kong HJ, Hong SC, Chung SH, Hong JW,. A flow-based method for abnormal network traffic detection, In Proceedings of the IEEE/IFIP Network Operations and Management Symposium: Application Session, Vol. 1, Seoul, Korea, 2004; 599-612.
68. Krmíček V, Vykopal J, Krejčí R,. Netflow based system for NAT detection, In Proceedings of the 5th International Student Workshop on Emerging Networking Experiments and Technologies: Co-Next Student Workshop'09. ACM: New York, 2009; 23-24.
69. Zhang H,. Study on the TOPN abnormal detection based on the NetFlow data set, Computer and Information Science 2009; 2 (3): 103-108.
70. Chan YTF, Shoniregun CA, Akmayeva GA,. A NetFlow based Internet-worm detecting system in large network, In Proceedings of the 3rd International Conference on Digital Information Management, London, UK, 2008; 581-586.
71. Lu W, Ghorbani AA,. Network anomaly detection based on wavelet analysis, EURASIP Journal on Advances in Signal Processing 2009; 4: 1-16.
72. Hellemons L, Hendriks L, Hofstede R, Sperotto A, Sadre R, Pras A,. SSHCure: a flow-based SSH intrusion detection system. In Dependable Networks and Services, Sadre R, Novotný J, Čeleda P, Waldburger M, Stiller B, (eds.), Lecture Notes in Computer Science, vol. 7279. Springer: Berlin, 2012; 86-97.
73. McNutt J,. Analysis of the US-CERT DAC, In CERT FloCon Workshop, Pittsburgh, PA, 2004. Available: http://www.cert.org/flocon/2004/proceedings/ flocon2004-analysis-mcnutt.pdf [31 October 2013].
74. Kondoh T, Ishibashi K,. Identifying anomalous traffic using Delta Traffic, In CERT FloCon Workshop: Savannah, GA, 2008. Available: http://www.cert.org/flocon/2008/presentations/kondoh-flocon2008.pdf [31 October 2013].
75. Hashim F, Munasinghe KS, Jamalipour A,. Biologically inspired anomaly detection and security control frameworks for complex heterogeneous networks, IEEE Transactions on Network and Service Management; 7 (4): 268-281.
76. Casas P, Mazel J, Owezarski P,. Unsupervised network intrusion detection systems: detecting the unknown without knowledge, Computer Communications 2012; 35 (7): 772-783.
77. Karagiannis T, Papagiannaki K, Faloutsos M,. BLINC: multilevel traffic classification in the dark, In Proceedings of the 2005 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications: SIGCOMM'05. ACM: New York, 2005; 229-240. (Pubitemid 46323507)
78. Livadas C, Walsh R, Lapsley D, Strayer WT,. Using machine learning techniques to identify botnet traffic, In Proceedings of the 31st IEEE Conference on Local Computer Networks, Tampa, Florida, USA, 2006; 967-974.
79. McLeod R, Nagaonkar V,. Anomaly detection through blind flow analysis inside a local network, In CERT FloCon Workshop: Vancouver, WA, 2006. Available: http://www.cert.org/flocon/2006/presentations/anomaly-detect2006.pdf [31 October 2013].
80. Kind A, Stoecklin MP, Dimitropoulos X,. Histogram-based traffic anomaly detection, IEEE Transactions on Network and Service Management 2009; 6 (2): 110-121.
81. Kent A, Fisk M, Gavrilov E,. Network host classification using statistical analysis of flow data, In CERT FloCon Workshop: New Orleans, LA, 2010. Available: http://www.cert.org/flocon/2010/presentations/Kent- NetworkHostClassification.pdf [31 October 2013].
82. McHugh J,. Locality based analysis of network flows, In CERT FloCon Workshop: Pittsburgh, PA, 2004. Available: http://www.cert.org/flocon/2004/ proceedings/flocon2004-analysis-mchugh.pdf [31 October 2013].
83. Lakhina A, Crovella M, Diot C,. Mining anomalies using traffic feature distributions, SIGCOMM Computer Communication Review 2005; 35 (4): 217-228. (Pubitemid 46323506)
84. Li Z, Gao Y, Chen Y,. Towards a high-speed router-based anomaly/intrusion detection system, In Proceedings of the Special Interest Group on Data Communication (SIGCOMM): Philadelphia, PA, 2005. Available: http://conferences. sigcomm.org/sigcomm/2005/poster-121.pdf [20 May 2014].
85. McNutt J, De Shon M,. Correlations between quiescent ports in network flows, In CERT FloCon Workshop: Pittsburgh, PA, 2005. Available: http://www.cert.org/flocon/2005/presentations/McNutt-Correlation-FloCon2005.pdf [31 October 2013].
86. Wagner A, Plattner B,. Proceedings of the 14th IEEE International Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprise, 2005; 172-177.
87. Stoecklin MP, Boudec JY, Kind A,. A two-layered anomaly detection technique based on multi-modal flow behavior models. In Passive and Active Network Measurement, Claypool M, Uhlig S, (eds.), Lecture Notes in Computer Science, vol. 4979. Springer: Berlin, 2008; 212-221. (Pubitemid 351702294)
88. Zseby T, Hirsch T,. A flexible DDoS detection system using IPFIX, In CERT FloCon Workshop: Savannah, GA, 2008. Available: http://www.cert.org/flocon/ 2008/presentations/zseby-Flocon2008.pdf [31 October 2013].
89. Frias-Martinez V, Sherrick J, Stolfo SJ, Keromytis AD,. A network access control mechanism based on behavior profiles, In Proceedings of the Conference on Computer Security Applications, Honolulu, Hawaii, 2009; 3-12.
90. Yin K, Nianqing T,. Study on the risk detection about network security based on grey theory, In Proceedings of the International Forum on Information Technology and Applications, Vol. 1, 2009; 411-413.
91. Janies J,. Protographs: graph-based approach to NetFlow analysis, In CERT FloCon Workshop: Salt Lake City, UT, 2011. Available: http://www.cert.org/ flocon/2011/presentations/Janies-Protographs.pdf [31 October 2013].
92. Yin Kx, Zhu Jq,. A novel DoS detection mechanism, In Proceedings of the International Conference on Mechatronic Science, Electric Engineering and Computer, Jilin, China, 2011; 296-298.
93. Sawaya Y, Kubota A, Miyake Y,. Detection of attackers in services using anomalous host behavior based on traffic flow statistics, In Proceedings of the 11th International Symposium on Applications and the Internet (SAINT): Munich, Germany, 2011; 353-359.
94. Zseby T,. Entropy in IP Darkspace data, In CERT FloCon Workshop: Austin, TX, 2012. Available: http://www.cert.org/flocon/2012/presentations/zseby- entropy-in-IP-darkspace-data.pdf [31 October 2013].