An entropy-based network anomaly detection method

Entropy

Volumn 17, Issue 4, 2015, Pages 2367-2408

  Bereziński, Przemysław ^a   Jasiul, Bartosz ^a   Szpyrka, Marcin ^b  

^a MILITARY COMMUNICATION INSTITUTE   (Poland)

^b AGH UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Poland)

Author keywords

Anomaly detection; Entropy; Malware detection

Indexed keywords

EID: 84930363113     PISSN: None     EISSN: 10994300     Source Type: Journal    
DOI: 10.3390/e17042367     Document Type: Article

References (140)

1. Denning, D.E. An intrusion-detection model. IEEE Trans. Softw. Eng. 1987, 13, 222-232.
2. Li, Z.; Das, A.; Zhou, J. USAID: Unifying Signature-Based and Anomaly-Based Intrusion Detection. In Advances in Knowledge Discovery and Data Minin, Ho, T., Cheung, D., Liu, H., Eds.; Volume 3518, Lecture Notes in Computer Science; Springer: Berlin/Heidelberg, Germany, 2005; pp 702-712.
3. Cheng, T.H.; Lin, Y.D.; Lai, Y.C.; Lin, P.C. Evasion Techniques: Sneaking through Your Intrusion Detection/Prevention Systems. IEEE Commun. Surv. Tutor. 2012, 14, 1011-1020.
4. Jasiul, B.; Śliwa, J.; Gleba, K.; Szpyrka, M. Identification of malware activities with rules. In Proceedings of the 2014 Federated Conference on Computer Science and Information Systems (FedCSIS), Warsaw, Poland, 7-10 September 2014; Ganzha, M., Maciaszek, L., Paprzycki, M., Eds.; pp. 101-110.
5. Gascon, H.; Orfila, A.; Blasco, J. Analysis of update delays in signature-based network intrusion detection systems. Comput. Secur. 2011, 30, 613-624.
6. Eimann, R. Network Event Detection with Entropy Measures. Ph.D. Thesis, University of Auckland, Auckland, New Zealand, 2008.
7. Wagner, A.; Plattner, B. Entropy Based Worm and Anomaly Detection in Fast IP Networks. In Proceedings of the 14th IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprise (WETICE'05, Linköping University, Linköping, Sweden, 13-15 June 2005; pp 172-177.
8. Nychis, G.; Sekar, V.; Andersen, D.G.; Kim, H.; Zhang, H. An Empirical Evaluation of Entropy-based Traffic Anomaly Detection. In Proceedings of the 8th ACM SIGCOMM Conference on Internet Measurement (IMC '08), Vouliagmeni, Greece, 20-22 October 2008 ; pp. 151-156.
9. Tellenbach, B. Detection, Classification and Visualization of Anomalies using Generalized Entropy Metric. Ph.D. Thesis, ETH, Zürich, Switzerland, 2012; Ph.D. Dissertation Nr. 20929.
10. Xiang, Y.; Li, K.; Zhou, W. Low-Rate DDoS Attacks Detection and Traceback by Using New Information Metrics. IEEE Trans. Inf. Forensics Secur. 2011, 6, 426-437.
11. Kopylova, Y.; Buell, D.A.; Huang, C.T.; Janies, J. Mutual information applied to anomaly detection. J. Commun. Netw. 2008, 10, 89-97.
12. HP-The Bot Threat. Available online: http://www.bitpipe.com/detail/RES/1384218191_706. html (accessed on 16 April 2015).
13. Sophos-Security Threat Report 2014 Smarter, Shadier, Stealthier Malware. Available online: https://cccure.training/m/articles/view/Sophos-Security-Threat-Report-2014 (accessed on 16 April 2015).
14. Scanlon, M.; Kechadi, M.T. The Case for a Collaborative Universal Peer-to-Peer Botnet Investigation Framework. In Proceedings of the 9th International Conference on Cyber Warfare and Security (ICCWS 2014, Purdue University, West Lafayette, IN, USA, 24-25 March 2014; pp 287-293.
15. Tellenbach, B.; Burkhart, M.; Sornette, D.; Maillart, T. Beyond Shannon: Characterizing Internet Traffic with Generalized Entropy Metrics. In Proceedings of the 10th International Conference on Passive and Active Network Measurement (PAM'09, Seoul, Korea, 1-3 April 2009; pp 239-248.
16. NfSen-Netflow Sensor. Available online: http://nfsen.sourceforge.net (accessed on 16 April 2015).
17. Barford, P.; Kline, J.; Plonka, D.; Ron, A. A Signal Analysis of Network Traffic Anomalies. In Proceedings of the 2nd ACM SIGCOMM Workshop on Internet Measurement (IMW '02, Marseille, France, 6-8 November 2002; pp 71-82.
18. Kim, M.S.; Kong, H.J.; Hong, S.C.; Chung, S.H.; Hong, J. A flow-based method for abnormal network traffic detection. Presented at IEEE/IFIP Network Operations and Management Symposium (NOMS 2004), Seoul, Korea, 19-23 April 2004; pp. 599-612.
19. NtopNg-High-Speed Web-based Traffic Analysis and Flow Collection. Available online:http://www.ntop.org (accessed on 16 April 2015).
20. Witten, I.H.; Frank, E.; Hall, M.A. Data Mining: Practical Machine Learning Tools and Techniques, 3rd ed.; Morgan Kaufmann Publishers Inc.: San Francisco, CA, USA, 2011.
21. Bhattacharyya, D.K.; Kalita, J.K. Network Anomaly Detection: A Machine Learning Perspective; Chapman & Hall/CRC: Boca Raton, FL, USA, 2013.
22. Aggarwal, C. Outlier Analysis; Springer: New York, NY, USA, 2013.
23. Hastie, T.; Tibshirani, R.; Friedman, J. The Elements of Statistical Learning: Data Mining, Inference and Prediction, 2 ed.; Springer: New York, NY, USA, 2009.
24. Chandola, V.; Banerjee, A.; Kumar, V. Anomaly Detection: A Survey. ACM Comput. Surv. 2009, 41, 15:1-15:58.
25. Hodge, V.; Austin, J. A Survey of Outlier Detection Methodologies. Artif. Intell. Rev. 2004, 22, 85-126.
26. Estevez-Tapiador, J.M.; Garcia-Teodoro, P.; Diaz-Verdejo, J.E. Anomaly Detection Methods in Wired Networks: A Survey and Taxonomy. Comput. Commun. 2004, 27, 1569-1584.
27. Patcha, A.; Park, J.M. An Overview of Anomaly Detection Techniques: Existing Solutions and Latest Technological Trends. Comput. Netw. 2007, 51, 3448-3470.
28. Callegari, C. Statistical approaches for network anomaly detection. In Proceedings of the 4th International Conference on Internet Monitoring and Protection (ICIMP, Venice/Mestre, Italy, 24-28 May 2009.
29. Callado, A.; Kamienski, C.; Szabo, G.; Gero, B.; Kelner, J.; Fernandes, S.; Sadok, D. A Survey on Internet Traffic Identification. IEEE Commun. Surv. Tutor. 2009, 11, 37-52.
30. Garcia-Teodoro, P.; Diaz-Verdejo, J.; Macia-Fernandez, G.; Vazquez, E. Anomaly-based network intrusion detection: Techniques, systems and challenges. Comput. Secur. 2009, 28, 18-28.
31. Bhuyan, M.; Bhattacharyya, D.; Kalita, J. Network anomaly detection: methods, systems and tools. IEEE Commun. Surv. Tutor. 2013, 16, 1-34.
32. Sperotto, A.; Schaffrath, G.; Sadre, R.; Morariu, C.; Pras, A.; Stiller, B. An Overview of IP Flow-Based Intrusion Detection. IEEE Commun. Surv. Tutor. 2010, 12, 343-356.
33. Huang, L.; Nguyen, X.; Garofalakis, M.; Jordan, M.; Joseph, A.D.; Taft, N. In-Network PCA and Anomaly Detection; Technical Report UCB/EECS-2007-10; EECS Department, University of California: Berkeley, CA, USA, 2007.
34. Shyu, M.-L.; Chen, S.-C.; Sarinnapakorn, K.; Chang, L. A novel anomaly detection scheme based on principal component classifier. In Proceedings of IEEE Foundations and New Directions of Data Mining Worksho, in conjunction with the Third IEEE International Conference on Data Mining (ICDM'03), Melbourne, FL, USA, 19-22 November 2003; pp 171-179.
35. Lee, Y.J.; Yeh, Y.R.; Wang, Y.C.F. Anomaly Detection via Online Oversampling Principal Component Analysis. IEEE Trans. Knowl. Data Eng. 2013, 25, 1460-1470.
36. Lu, W.; Ghorbani, A.A. Network Anomaly Detection Based on Wavelet Analysis. EURASIP J. Adv. Sig. Proc. 2009, 2009, doi:10.1155/2009/837601.
37. Lu, W.; Tavallaee, M.; Ghorbani, A.A. Detecting Network Anomalies Using Different Wavelet Basis Functions. In Proceedings of Sixth Annual Conference on Communication Networks and Services Research (CNSR 2008, Halifax, Nova Scotia, Canada, 5-8 May 2008; pp 149-156.
38. Limthong, K.; Watanapongse, P.; Kensuke, F. A wavelet-based anomaly detection for outbound network traffic. Presented at 8th Asia-Pacific Symposium on Information and Telecommunication Technologies (APSITT), Kuching, Sarawak, Malaysia, 15-18 June 2010; pp. 1-6.
39. Ye, N.; Zhang, Y.; Borror, C.M. Robustness of the Markov-chain model for cyber-attack detection. IEEE Trans. Reliab. 2004, 53, 116-123.
40. Sha, W.; Zhu, Y.; Huang, T.; Qiu, M.; Zhu, Y.; Zhang, Q. A Multi-order Markov Chain Based Scheme for Anomaly Detection. In Proceedings of IEEE 37th Annual Computer Software and Applications Conferen, COMPSAC Workshops 2013, Kyoto, Japan, 22-26 July 2013; pp 83-88.
41. Syarif, I.; Prugel-Bennett, A.; Wills, G. Unsupervised Clustering Approach for Network Anomaly Detection. In Networked Digital Technologie, Volume 293, Communications in Computer and Information Science; Springer: Berlin/Heidelberg, Germany, 2012; pp 135-145.
42. Riad, A.; Elhenawy, I.; Hassan, A.; Awadallah, N. Visualize Network Anomaly Detection By Using K-Means Clustering Algorithm. Int. J. Comput. Netw. Commun. 2013, 5, doi:10.5121/ijcnc.2013.5514.
43. Bazan, J.; Szpyrka, M.; Szczur, A.; Dydo, L.; Wojtowicz, H. Classifiers for Behavioral Patterns Identification Induced from Huge Temporal Data. Fundam. Inform. 2015, in press.
44. Kind, A.; Stoecklin, M.P.; Dimitropoulos, X. Histogram-based Traffic Anomaly Detection. IEEE Trans. Netw. Serv. Manag. 2009, 6, 110-121.
45. Soule, A.; Salamatia, K.; Taft, N.; Emilion, R.; Papagiannaki, K. Flow Classification by Histograms: Or How to Go on Safari in the Internet. In Proceedings of the Joint International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS-Performance 2004, Columbia University, New York, NY, USA, 12-16 June 2004; pp 49-60.
46. Stoecklin, M.P.; Le Boudec, J.Y.; Kind, A. A Two-layered Anomaly Detection Technique Based on Multi-modal Flow Behavior Models. In Proceedings of the 9th International Conference on Passive and Active Network Measurement (PAM'08, Cleveland, OH, USA, 29-30 April 2008; pp 212-221.
47. Brauckhoff, D.; Dimitropoulos, X.;Wagner, A.; Salamatian, K. Anomaly Extraction in Backbone Networks Using Association Rules. In Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement Conference (IMC '09, Chicago, IL, USA, 4-6 November 2009; pp 28-34.
48. Iglesias, F.; Zseby, T. Entropy-Based Characterization of Internet Background Radiation. Entropy 2014, 17, 74-101.
49. Harrington, D.; Presuhn, R.; Wijnen, B. An Architecture for Describing Simple Network Management Protocol (SNMP) Management Frameworks. Available online: http://www.ietf. org/rfc/rfc3411.txt (accessed on 16 April 2015).
50. Claise, B. Cisco Systems NetFlow Services Export Version 9. Available online: http://tools.ietf. org/html/rfc3954 (accessed on 16 April 2015).
51. Kambourakis, G.; Kolias, C.; Gritzalis, S.; Park, J.H. DoS attacks exploiting signaling in {UMTS} and {IMS}. Comput. Commun. 2011, 34, 226-235.
52. Choi, K.; Chen, X.; Li, S.; Kim, M.; Chae, K.; Na, J. Intrusion Detection of NSM Based DoS Attacks Using Data Mining in Smart Grid. Energies 2012, 5, 4091-4109.
53. Liu, Y.; Xiong, N.; Park, J.; Yang, C.; Xu, K. Fair incentive mechanism with pyramidal structure for peer-to-peer networks. IET Commun. 2010, 4, 1-12.
54. Lee, D.C.; Park, B.; Kim, K.E.; Lee, J.J. Fast Traffic Anomalies Detection Using SNMP MIB Correlation Analysis. In Proceedings of the 11th International Conference on Advanced Communication Technology (ICACT'09), Phoenix Park, Korea, 15-18 February 2009; Volume 1, pp. 166-170.
55. Casas, P.; Fillatre, L.; Vaton, S.; Nikiforov, I. Volume Anomaly Detection in Data Networks:An Optimal Detection Algorithm vs. the PCA Approach. In Traffic Management and Traffic Engineering for the Future Interne, Valadas, R., Salvador, P., Eds.; Volume 5464, Lecture Notes in Computer Science; Springer: Berlin/Heidelberg, Germany, 2009; pp 96-113.
56. Plixer Scrutinizer-Incydent Response System. Available online: http://www.plixer.com (accessed on 16 April 2015).
57. Peassler PRTG-Network Monitor. Available online: http://www.paessler.com (accessed on 16 April 2015).
58. Solarwinds Network Traffic Analyzer. Available online: http://www.solarwinds.com (accessed on 16 April 2015).
59. Invea-Tech FlowMon. Available online: https://www.invea.com (accessed on 16 April 2015).
60. AKMA Labs FlowMatrix. Available online: http://www.akmalabs.com (accessed on 16 April 2015).
61. Jingle, I.; Rajsingh, E. ColShield: An effective and collaborative protection shield for the detection and prevention of collaborative flooding of DDoS attacks in wireless mesh networks. Human-centric Comput. Inf. Sci. 2014, 4, doi: 10.1186/s13673-014-0008-8.
62. Zhou, W.; Jia, W.; Wen, S.; Xiang, Y.; Zhou, W. Detection and defense of application-layer {DDoS} attacks in backbone web traffic. Future Gener. Comput. Syst. 2014, 38, 36-46.
63. Brauckhoff, D.; Tellenbach, B.;Wagner, A.; May, M.; Lakhina, A. Impact of Packet Sampling on Anomaly Detection Metrics. In Proceedings of the 6th ACM SIGCOMM Conference on Internet Measurement (IMC '06, Rio de Janeiro, Brazil, 25-27 October 2006; pp 159-164.
64. Lakhina, A.; Crovella, M.; Diot, C. Mining Anomalies Using Traffic Feature Distributions. In Proceedings of the 2005 Conference on Application, Technologies, Architectures, and Protocols for Computer Communications (SIGCOMM'05), Philadelphia, PA, USA, 22-26 August 2005; pp 217-228.
65. Shannon, C. A Mathematical Theory of Communication. Bell Syst. Tech. J. 1948, 27, 379-423.
66. Baez, J.C.; Fritz, T.; Leinster, T. A Characterization of Entropy in Terms of Information Loss. Entropy 2011, 13, 1945-1957.
67. Lee, W.; Xiang, D. Information-theoretic measures for anomaly detection. In Proceedings of 2001 IEEE Symposium on Security and Privac, Oakland, CA, USA, 14-16 May 2001; pp 130-143.
68. Grünwald, P.; Vitányi, P. Kolmogorov Complexity and Information Theory. With an Interpretation in Terms of Questions and Answers. J. Logic Lang. Inf. 2003, 12, 497-529.
69. Teixeira, A.; Matos, A.; Souto, A.; Antunes, L. Entropy Measures vs. Kolmogorov Complexity. Entropy 2011, 13, 595-611.
70. Ranjan, S.; Shah, S.; Nucci, A.; Munafo, M.; Cruz, R.; Muthukrishnan, S. DoWitcher: Effective Worm Detection and Containment in the Internet Core. In Proceedings of 26th IEEE International Conference on Computer Communications (INFOCOM 2007, Anchorage, AL, USA, 6-12 May 2007; pp 2541-2545.
71. Gu, Y.; McCallum, A.; Towsley, D. Detecting Anomalies in Network Traffic Using Maximum Entropy Estimation. In Proceedings of the 5th ACM SIGCOMM Conference on Internet Measurement (IMC '05, Berkeley, CA, USA, 19-21 October 2005; pp 32-32.
72. Speidel, U.; Eimann, R.; Brownlee, N. Detecting network events via T-entropy. In Proceedings of 6th International Conference on Informatio, Communications Signal Processing, Singapore, Singapore, 10-13 December 2007 , 2007; pp 1-5.
73. Eimann, R.; Speidel, U.; Brownlee, J. A T-entropy Analysis of the Slammer Worm Outbreak. In Proceedings of Asia-Pacific Network Operations and Management Symposiu, Okinawa, Japan, 27-30 September 2005; pp 434-445.
74. Titchener, M.R.; Nicolescu, R.; Staiger, L.; Gulliver, T.A.; Speidel, U. Deterministic Complexity and Entropy. Fundam. Inform. 2005, 64, 443-461.
75. Pawelec, J.; Bereziński, P.; Piotrowski, R.; Chamela, W. Entropy Measures For Internet Traffic Anomaly Detection. In Proceedings of 16th International Conference on Computer Systems Aided Science, Industry and Transport (TransComp), hlcity, country, date 2012; pp 309-318.
76. Tsallis, C. Possible generalization of Boltzmann-Gibbs statistics. J. Stat. Phys. 1988, 52, 479-487.
77. Tsallis, C. The Nonadditive Entropy Sq and Its Applications in Physics and Elsewhere: Some Remarks. Entropy 2011, 13, 1765-1804.
78. Prehl, J.; Essex, C.; Hoffmann, K.H. Tsallis Relative Entropy and Anomalous Diffusion. Entropy 2012, 14, 701-716.
79. Renyi, A. Probability Theory; Enlarged version of Wahrscheinlichkeitsrechnung, Valoszinusegszamitas and Calcul des probabilites. English translation by Laszlo Vekerdi; North-Holland: Amsterdam, The Netherlands, 1970.
80. Csiszár, I. Axiomatic Characterizations of Information Measures. Entropy 2008, 10, 261-273.
81. Ziviani, A.; Gomes, A.; Monsores, M.; Rodrigues, P. Network anomaly detection using nonextensive entropy. IEEE Commun. Lett. 2007, 11, 1034-1036.
82. Shafiq, M.Z.; Khayam, S.A.; Farooq, M. Improving Accuracy of Immune-inspired Malware Detectors by Using Intelligent Features. In Proceedings of the 10th Annual Conference on Genetic and Evolutionary Computation (GECCO '08, Atlanta, GA, USA, 12-16 July 2008; pp 119-126.
83. Lima, C.F.L.; de Assis, F.M.; de Souza, C.P. A Comparative Study of Use of Shannon, Rényi and Tsallis Entropy for Attribute Selecting in Network Intrusion Detection. In Proceedings of the 13th Intl Conf. on Intelligent Data Engineering and Automated Learning (IDEAL'12, Natal, Brazil, 29-31 August 2012; pp 492-501.
84. Tellenbach, B.; Burkhart, M.; Schatzmann, D.; Gugelmann, D.; Sornette, D. Accurate Network Anomaly Classification with Generalized Entropy Metrics. Comput. Netw. 2011, 55, 3485-3502.
85. Zhang, J.; Chen, X.; Xiang, Y.; Zhou, W.; Wu, J. Robust Network Traffic Classification. IEEE/ACM Trans. Netw. 2014, PP, 1-1.
86. Clausius, R.; Hirst, T. The Mechanical Theory of Heat: With its applications to the steam-engine and to the physical properties of bodies. J. van Voorst: London, UK, 1867.
87. Karmeshu, J. Entropy Measures, Maximum Entropy Principle and Emerging Applications; Springer: New York, NY, USA, 2003.
88. Harremoes, P.; Topsoe, F. Maximum Entropy Fundamentals. Entropy 2001, 3, 191-226.
89. Kullback, S. Information Theory and Statistics; Wiley: New York, NY, USA, 1959.
90. Cover, T.; Thomas, J. Elements of Information Theory; Wiley: Hoboken, NJ, USA, 2006.
91. Maszczyk, T.; Duch, W. Comparison of Shannon, Renyi and Tsallis Entropy Used in Decision Trees. In Artificial Intelligence and Soft Computing-ICAISC 200, Rutkowski, L., Tadeusiewicz, R., Zadeh, L., Zurada, J., Eds.; Volume 5097, Lecture Notes in Computer Science; Springer: Berlin/Heidelberg, Germany, 2008; pp 643-651.
92. Marco, M. A step beyond Tsallis and Rényi entropies. Phys. Lett. A 2005, 338, 217-224.
93. Wȩdrowska, E. Miary entropii i dywergencji w analizie struktur; Wydawnictwo Uniwersytetu Warminsko-Mazurskiego: Olsztyn, Poland, 2012.
94. Softflowd-Flow-based Network Traffic Analyser. Available online: http://code.google.com/p/ softflowd/(accessed on 16 April 2015).
95. Gigamon-SPAN Port Or TAP? White Paper. Available online: https://www.netdescribe.com/ downloads/span_port_or_tap_web.pdf (accessed on 16 April 2015).
96. Trammell, B.; Wagner, A.; Claise, B. Flow Aggregation for the IP Flow Information Export (IPFIX) Protocol. Available online: http://tools.ietf.org/html/rfc7015 (accessed on 16 April 2015).
97. Reimann, C.; Filzmoser, P.; Garrett, R.G. Background and threshold: critical comparison of methods of determination. Sci. Total Environ. 2005, 346, 1-16.
98. Szpyrka, M.; Jasiul, B.; Wrona, K.; Dziedzic, F. Telecommunications Networks Risk Assessment with Bayesian Networks. In Computer Information Systems and Industrial Managemen, Saeed, K., Chaki, R., Cortesi, A.,Wierzchoń, S., Eds.; Volume 8104, Lecture Notes in Computer Science; Springer: Berlin/Heidelberg, Germany, 2013; pp 277-288.
99. Hall, M.; Frank, E.; Holmes, G.; Pfahringer, B.; Reutemann, P.; Witten, I. The WEKA Data Mining Software: An Update. SIGKDD Explor. Newslett. 2009, 11, 10-18.
100. Jasiul, B.; Szpyrka, M.; Śliwa, J. Detection and Modeling of Cyber Attacks with Petri Nets. Entropy 2014, 16, 6602-6623.
101. Jasiul, B.; Szpyrka, M.; Śliwa, J. Malware Behavior Modeling with Colored Petri Nets. In Computer Information Systems and Industrial Managemen, Saeed, K., Snasel, V., Eds.; Volume 8838, Lecture Notes in Computer Science; Springer: Berlin/Heidelberg, Germany, 2014; pp 667-679.
102. Jasiul, B.; Szpyrka, M.; Śliwa, J. Formal Specification of Malware Models in the Form of Colored Petri Nets. In Computer Science and its Application, Park, J.J.J.H., Stojmenovic, I., Jeong, H.Y., Yi, G., Eds.; Volume 330, Lecture Notes in Electrical Engineering; Springer: Berlin/Heidelberg, Germany, 2015; pp 475-482.
103. ACM Sigcomm Internet Traffic Archive. Available online: http://www.sigcomm.org/ITA (accessed on 16 April 2015).
104. Lawrence Berkeley National Laboratory/International Computer Science Institute Enterprise Tracing. Available online: http://www.icir.org/enterprise-tracing/ (accessed on 16 April 2015).
105. SimpleWeb. Available online: http://www.simpleweb.org/wiki/Traces (accessed on 16 April 2015).
106. Center for Applied Internet Data Analysis (CAIDA). Available online: http://www.caida.org/ data/overview (accessed on 16 April 2015).
107. Cluster of European Projects aimed at Monitoring and Measurement (MoMe). Available online:http://www.ist-mome.org/database/MeasurementData (accessed on 16 April 2015).
108. Waikato Internet Traffic Storage (WITS). Available online: http://wand.net.nz/wits (accessed on 16 April 2015).
109. UMass Trace Repository (UMass). Available online: http://traces.cs.umass.edu (accessed on 16 April 2015).
110. Verizon Data Breach Investigations Report. Available online: http://www.verizonenterprise.com/ DBIR/2014/(accessed on 16 April 2015).
111. Symantec Internet Security Threat Report. Available online: http://www.symantec.com/security_ response/publications/threatreport.jsp (accessed on 16 April 2015).
112. CERT Poland Raport. Availableonline:http://www.cert.pl/PDF/Report_CP_2013.pdf (accessed on 16 April 2015).
113. Saad, S.; Traore, I.; Ghorbani, A.A.; Sayed, B.; Zhao, D.; Lu, W.; Felix, J.; Hakimian, P. Detecting P2P botnets through network behavior analysis and machine learning. In Proceedings of 2011 Ninth Annual International Conference on Privac, Security and Trust (PST), Montreal, QC, Canada, 19-21 July 2011; pp 174-180.
114. García, S.; Grill, M.; Stiborek, J.; Zunino, A. An Empirical Comparison of Botnet Detection Methods. Comput. Secur. 2014, 45, 100-123.
115. Sperotto, A.; Sadre, R.; Vliet, F.; Pras, A. A Labeled Data Set for Flow-Based Intrusion Detection. In Proceedings of the 9th IEEE International Workshop on IP Operations and Management (IPOM '09), Venice, Italy, 29-30 October 2009, Springer: Berlin/Heidelberg, Germany, 2009; pp 39-50.
116. Bereziński, P.; Pawelec, J.; Małowidzki, M.; Piotrowski, R. Entropy-Based Internet Traffic Anomaly Detection: A Case Study. In Proceedings of the Ninth International Conference on Dependability and Complex Systems DepCoS-RELCOMEX, Brunów, Poland, 30 June - 4 July 201, Zamojski,W., Mazurkiewicz, J., Sugier, J.,Walkowiak, T., Kacprzyk, J., Eds.; Volume 286, Advances in Intelligent Systems and Computing; Springer: Cham, Switzerland, 2014; pp 47-58.
117. Haines, J.; Lippmann, R.; Fried, D.; Zissman, M.; Tran, E.; Boswell, S. 1999 DARPA Intrusion Detection Evaluation: Design and Procedures; Technical Report 1062; MIT Lincoln Laboratory: Lexington, MA, USA, 2001; Available online:https://www.ll.mit.edu/mission/communications/cyber/CSTcorpora/files/TR-1062.pdf, (accessed on 16 April 2015).
118. The Third International Knowledge Discovery and Data Mining Tools (KDD) Cup 1999 Data. Available online: http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html (accessed on 16 April 2015).
119. Tavallaee, M.; Bagheri, E.; Lu, W.; Ghorbani, A.A. A Detailed Analysis of the KDD CUP 99 Data Set. In Proceedings of the 2nd IEEE Intl Conference on Computational Intelligence for Security and Defense Application, Ottawa, ON, Canada, 8-10 July 2009; pp 53-58.
120. McHugh, J. Testing Intrusion Detection Systems: A Critique of the 1998 and 1999 DARPA Intrusion Detection System Evaluations As Performed by Lincoln Laboratory. ACM Trans. Inf. Syst. Secur. 2000, 3, 262-294.
121. Mahoney, M.V.; Chan, P.K. An Analysis of the 1999 DARPA/Lincoln Laboratory Evaluation Data for Network Anomaly Detection. In Recent Advances in Intrusion Detectio, Vigna, G.; Kruegel, C., Jonsson, E., Eds.; Volume 2820, Lecture Notes in Computer Science; Springer:Berlin/Heidelberg, Germany, 2003; pp 220-237.
122. Thomas, C.; Sharma, V.; Balakrishnan, N. Usefulness of DARPA dataset for intrusion detection system evaluation. SPIE Proc. 2008, doi:10.1117/12.777341.
123. Brauckhoff, D.; Wagner, A.; May, M. FLAME: A Flow-level Anomaly Modeling Engine. In Proceedings of the Conference on Cyber Security Experimentation and Test (CSET'08), San Jose, CA, USA, 28 July 2008; pp 1-6.
124. Brauckhoff, D. Network traffic anomaly detection and evaluatio. Ph.D. Thesis, ETH Zürich, Switzerland, 2010; PhD Dissertation Nr. 18835.
125. Shiravi, A.; Shiravi, H.; Tavallaee, M.; Ghorbani, A.A. Toward Developing a Systematic Approach to Generate Benchmark Datasets for Intrusion Detection. Comput. Secur. 2012, 31, 357-374.
126. Bereziński, P.; Szpyrka, M.; Jasiul, B.; Mazur, M. Network Anomaly Detection Using Parameterized Entropy. In Computer Information Systems and Industrial Managemen, Saeed, K., Snasel, V., Eds.; Volume 8838, Lecture Notes in Computer Science; Springer:Berlin/Heidelberg, Germany, 2014; pp 465-478.
127. Tomer, B. Morto Post Mortem: Dissecting a Worm; Available online: http://blog.imperva.com/2011/09/morto-post-mortem-a-worm-deep-dive.html (accessed on 16 April 2015).
128. Damon, E.; Dale, J.; Laron, E.; Mache, J.; Land, N.; Weiss, R. Hands-on Denial of Service Lab Exercises Using SlowLoris and RUDY. In Proceedings of the 2012 Information Security Curriculum Development Conference, (InfoSecCD '12, Kennesaw, GA, USA, 12-13 October 2012; pp 21-29.
129. Bencsáth, B.; Pék, G.; Buttyán, L.; Félegyházi, M. The Cousins of Stuxnet: Duqu, Flame, and Gauss. Future Internet 2012, 4, 971-1003.
130. Denning, D.E. Stuxnet: What Has Changed? Future Internet 2012, 4, 672-687.
131. Kührer, M.; Hupperich, T.; Rossow, C.; Holz, T. Exit from Hell? Reducing the Impact of Amplification DDoS Attacks. In Proceedings of the 23rd USENIX Security Symposiu, San Diego, CA, USA, 20-22 August 2014.
132. Hauke, J.; Kossowski, T. Comparison of Values of Pearson's and Spearman's Correlation Coefficients on the Same Sets of Data. Quaest. Geogr. 2011, 30, 87-93.
133. Davis, J.; Goadrich, M. The Relationship Between Precision-Recall and ROC Curves. In Proceedings of the 23rd International Conference on Machine Learning, (ICML'06, Pittsburgh, PA, USA, 25-29 June 2006; pp 233-240.
134. Wu, Y.; Cai, S.; Yang, S.; Zheng, F.; Xiang, N. Classification of Knee Joint Vibration Signals Using Bivariate Feature Distribution Estimation and Maximal Posterior Probability Decision Criterion. Entropy 2013, 15, 1375-1387.
135. Rifkin, R. MIT-Multiclass Classification. Available online: http://www.mit.edu/~9.520/ spring09/Classes/multiclass.pdf (accessed on 16 April 2015).
136. Sumner, M.; Frank, E.; Hall, M. Speeding up Logistic Model Tree Induction. In Proceedings of 9th European Conference on Principles and Practice of Knowledge Discovery in Database, Porto, Portugal, 3-7 October 2005; pp 675-683.
137. Seber, G.; Lee, A. Linear Regression Analysis; Wiley Series in Probability and Statistics; Wiley:Hoboken, NJ, USA, 2012.
138. Landwehr, N.; Hall, M.; Frank, E. Logistic Model Trees. Mach. Learn. 2005, 59, 161-205.
139. Madjarov, G.; Kocev, D.; Gjorgjevikj, D.; Deroski, S. An Extensive Experimental Comparison of Methods for Multi-label Learning. Pattern Recogn. 2012, 45, 3084-3104.
140. MEKA: A Multi-label Extension to WEKA. Available online: http://meka.sourceforge.net/ (accessed on 16 April 2015).