Exploring discrepancies in findings obtained with the KDD Cup '99 data set

Intelligent Data Analysis

Volumn 15, Issue 2, 2011, Pages 251-276

  Engen, Vegard ^a   Vincent, Jonathan ^a   Phalp, Keith ^a  

^a BOURNEMOUTH UNIVERSITY   (United Kingdom)

Author keywords

intrusion detection; KDD Cup '99 data set; Machine learning; methodology

Indexed keywords

DATA SETS; EMPIRICAL INVESTIGATION; INTRUSION DETECTION SYSTEMS; KDD CUP '99 DATA SET; MACHINE-LEARNING; METHODOLOGY; ON-MACHINES; UNDERLYING CAUSE;

LEARNING SYSTEMS; RESEARCH;

INTRUSION DETECTION;

EID: 79953303626     PISSN: 1088467X     EISSN: 15714128     Source Type: Journal    
DOI: 10.3233/IDA-2010-0466     Document Type: Article

References (90)

1. M. AlSubaie andM. Zulkernine, Efficacy of HiddenMarkovModels Over Neural Networks in Anomaly Intrusion Detection. In 30th Annual International Computer Software and Applications Conference (COMPSAC'06), 2006, pages 325- 332. (Pubitemid 46661664)
2. J. Albus, A new approach to manipulator control: The cerebellar model articulation controller (cmac), Transactions ASME, Journal Dynamic Systems, Measurement, and Control 97 (1975), 220-227.
3. N. Ben Amor, S. Benferhat and Z. Elouedi, Naive Bayes vs Decision Trees in Intrusion Detection Systems. In SAC '04: Proceedings of the 2004 ACM symposium on Applied computing, New York, NY, USA, 2004, pages 420-424. ACM.
4. S. Benferhat and K. Tabia, On the combination of naive bayes and decision trees for intrusion detection. In CIMCA '05: Proceedings of the International Conference on Computational Intelligence for Modelling, Control and Automation and International Conference on Intelligent Agents, Web Technologies and Internet Commerce Vol1 (CIMCAIAWTIC' 06), Washington, DC, USA, 2005, pages 211-216. IEEE Computer Society.
5. A. Bosin, N. Dessì and B. Pes, Intelligent Bayesian Classifiers in Network Intrusion Detection. In Proceedings of the 18th international conference on Innovations in Applied Artificial Intelligence, London, UK, 2005, pages 445-447. Springer Verlag.
6. Y. Bouzida, Principal Component Analysis for Intrusion Detection and Supervised Learning for New Attack Detection, PhD thesis, Graduate Faculty of Ecole Nationale Supérieure des Télécommunications de Bretagne, 2006.
7. Y. Bouzida and F. Cuppens, Detecting Known and Novel Network Intrusions. In IFIP/SEC 2006, 21st IFIP TC11 International Information Security Conference, 2006.
8. Y. Bouzida and F. Cuppens, Neural networks vs. decision trees for intrusion detection. In IEEE / IST Workshop on Monitoring, Attack Detection and Mitigation (MonAM), 2006.
9. T. Brugger, Kdd cup '99 dataset (network intrusion) considered harmful. KDnuggets, n18(item4), September 2007. Available online: [http://www.kdnuggets. com/news/2007/n18/4i.html] [Accessed November 23 2007].
10. T. Brugger and J. Chow, An assessment of the DARPA IDS Evaluation Dataset using Snort, Technical Report CSE20071, University of California, Department of Computer Science, 2007.
11. J. Burez and D. van den Poel, Handling Class Imbalance in Customer Churn Prediction, Expert Systems with Applications 36(3) (April 2009), 4626-4636.
12. J. Cannady, Next Generation Intrusion Detection: Autonomous Reinforcement Learning of Network Attacks. In Proceedings of the 23rd National Information Systems Secuity Conference, 2000.
13. N.V. Chawla, C4.5 and Imbalanced Data sets: Investigating the effect of sampling method, probabilistic estimate, and decision tree structure. In Workshop on Learning from Imbalanced Datasets II, ICML, 2003.
14. S. Chebrolu, A. Abraham and J.P. Thomas, Feature Deduction and Ensemble Design of Intrusion Detection Systems, Computers and Security 24(4) (June 2005), 295-307. (Pubitemid 40752313)
15. P.Cheeseman and J. Stutz,BayesianClassification (AutoClass): Theory andResults. In Advances inKnowledgeDiscovery and Data Mining, 1996, pages 153-180.
16. G. Cohen, M. Hilario, H. Sax, S. Hugonnet and A. Geissbuhler, Learning from imbalanced data in surveillance of nosocomial infection, Artificial Intelligence in Medicine 37(1) (2006), 7-18.
17. D. Dasgupta, editor, Artificial Immune Systems and Their Applications, Springer, 1998.
18. L.N. De Castro and J. Timmis, Artificial Immune Systems: A New Computational Intelligence Approach, Springer, 2002.
19. H. Debar, An Introduction to Intrusion Detection Systems. In An Introduction to Intrusion Detection Systems, 2000.
20. H. Debar, M. Dacier and A. Wespi, Towards a Taxonomy of Intrusion Detection Systems Computer Networks 31(9) (1999), 805-822.
21. D.E. Denning, An intrusiondetection model, IEEE Transactions on Software Engineering 13(2) (1987), 222-232.
22. O. Depren, M. Topallar, E. Anarim and M.K. Ciliz, An Intelligent Intrusion Detection System for Anomaly and Misuse Detection in Computer Networks, Expert systems with Applications 29 (2005), 713-722. (Pubitemid 41394445)
23. M. Dorigo and T. Stutzle, Ant Colony Optimization, The MIT Press, July 2004.
24. C. Elkan, Results of the KDD'99 Classifier Learning Contest, 1999. Available online: [http://wwwcse. ucsd.edu/ ̃elkan/clresults.html].
25. C. Elkan, Results of the KDD'99 classifier learning, SIGKDD Explor Newsl 1(2) (2000), 63-64.
26. D. Endler, Intrusion Detection: Applying Machine Learning to Solaris AuditData. In Proceedings of the AnnualComputer Security Applications conference, 1998, pages 267-279.
27. V. Engen, J. Vincent and K. Phalp, Enhancing Network Based Intrusion Detection for Imbalanced Data. International Journal of Knowledge Based Intelligent Engineering Systems (KES) 12(5,6) (December 2008), 357-367.
28. E. Eskin, A. Arnold,M. Prerau, L. Portnoy and S. Stolfo, A Geometric Framework for Unsupervised Anomaly Detection: Detecting Intrusions in Unlabeled Data. In Data Mining for Security Applications, Kluwer, 2002.
29. F. Gharibian and A.A. Ghorbani, Comparative Study of Supervised Machine Learning Techniques for Intrusion Detection. In CNSR '07: Proceedings of the Fifth Annual Conference on Communication Networks and Services Research, Washington, DC, USA, 2007, pages 350-358. IEEE Computer Society.
30. G. Giachinto, R. Perdisci, M. Del Rio and F. Roli, Intrusion detection in computer networks by a modular ensemble of oneclass classifiers, Information Fusion 9 (2008), 69-82. (Pubitemid 47589059)
31. D. Gollmann, Computer Security, Wiley, 2nd edition, 2006.
32. S. Haykin, Neural Networks: A Comprehensive Foundation, Prentice Hall, 2 edition, 1998.
33. Y.M. Huang, C.M. Hung and H.C. Jiau, Evaluation of neural networks and data mining methods on a credit assessment task for class imbalance problem, Nonlinear Analysis: Real World Applications 7(4) (2006),720-747.
34. K. Ilgun, USTAT: A Real Time Intrusion Detection System for UNIX. In SP '93: Proceedings of the 1993 IEEE Symposium on Security and Privacy, Washington, DC, USA, 1993, page 16. IEEE Computer Society.
35. K. Ilgun, R.A. Kemmerer and P.A. Porras, State Transition Analysis: A Rule Based Intrusion Detection Approach, Software Engineering 21(3) (1995), 181-199.
36. N. Japkowicz, Learning from Imbalanced Data Sets: A Comparison of Various Strategies. In Proceedings of Learning from Imbalanced Data Sets, Papers from the AAAI Workshop, Technical Report WS0005, 2000, pages 10-15.
37. N. Japkowicz, The Class Imbalance Problem: Significance and Strategies. In Proceedings of the 2000 International Conference on Artificial Intelligence (ICAI' 2000), volume 1, 2000, pages 111-117.
38. T. Jo and N. Japkowicz, Class Imbalances versus Small Disjuncts, SIGKDD Explorations 6(1) (2004), 40-49.
39. O. Kachirski and R. Guba, Effective Intrusion Detection Using Multiple Sensors in Wireless Ad Hoc Networks. In Proceedings of the 36th Annual Hawaii International Conference on System Sciences, 2003.
40. M. Kearns, A Bound on the Error of Cross Validation Using the Approximation and Estimation Rates, with Consequences for the TrainingTest Split. In Advances in Neural Information Processing Systems 8, The MIT Press, 1996, pages 183- 189.
41. R.A. Kemmerer and G. Vigna, Intrusion detection: A brief history and overview, Security &Privacy 35(4) (2002), 27-30.
42. K. Kendall, A Database of Computer Attacks for the Evaluation of Intrusion Detection Systems, Master's thesis, Massachusetts Institute of Technology, 1999.
43. K. Kermanidis, M. Maragoudakis, N. Fakotakis and G. Kokkinakis, Learning Greek verb complements: Addressing the class imbalance. In COLING '04: Proceedings of the 20th international conference on Computational Linguistics, page 1065, Morristown, NJ, USA, 2004. Association for Computational Linguistics.
44. G.H. Kim and P.J. Bentley, Towards Artificial Immune Systems for Network Intrusion Detection: An Investigation of Dynamic Clonal Selection. In Proceedings of the Congress on Evolutionary Computation, 2002.
45. L. Kobyliński and A. Przepiórkowski, Definition Extraction with Balanced Random Forests. In Advances in Natural Language Processing: Proceedings of the 6th International Conference on Natural Language Processing, Springer Verlag, 2008, pages 237-247.
46. R. Kohavi, A Study of Cross Validation and Bootstrap for Accuracy Estimation and Model Selection. In International Joint Conference on Artificial Intelligence (IJCAI), 1995.
47. A. Kolcz, A. Chowdhury and J. Alspector, Data duplication: An imbalance problem? In ICML'2003 Workshop on Learning from Imbalanced Data Sets II, 2003.
48. C. Kruegel, F. Valeur and G. Vigna, Intrusion Detection and Correlation: Challenges and Solutions, Springer Verlag Telos, 2004.
49. Y. Le Cun, L. Bottou, G.B. Orr and K.R. Mueller, Efficient Back Prop. In Neural Networks: Tricks of the Trade, this book is an outgrowth of a 1996 NIPS workshop, London, UK, 1998, pages 9-50. Springer Verlag.
50. W. Lee and S.J. Stolfo, A Framework for Constructing Features and Models for Intrusion Detection Systems, ACM Transactions on Information and System Security 3(4) (2000), 227-261.
51. W. Lee and D. Xiang, Information Theoretic Measures for Anomaly Detection. In IEEE Symposium on Security and Privacy, Oakland, CA, May 2001.
52. K. Leung and C. Leckie, Unsupervised anomaly detection in network intrusion detection using clusters. In ACSC '05: Proceedings of the 28th Australasian conference on Computer Science, Darlinghurst, Australia, Australia, 2005, pages 333-342. Australian Computer Society, Inc.
53. L.M. Lewis, A Case Based Reasoning Approach to the Resolution of Faults in Communication Networks. In Proceedings of the IFIP TC6/WG6.6 Third International Symposium on Integrated Network Management with participation of the IEEE Communications Society CNOM and with support from the Institute for Educational Services, NorthHolland, 1993, pages 671-682.
54. U. Lindqvist and P.A. Porras, Detecting Computer and Network Misuse Through the Production Based Expert System Toolset (PBEST). In IEEE Symposium on Security and Privacy, 1999, pages 146-161.
55. U. Lindqvist and P.A. Porras, eXpertBSM: A Host Based Intrusion Detection Solution for Sun Solaris. In Proceedings of the 17th Annual computer Security Applications Conference, New Orleans, USA, Dec 2001, pages 240-251.
56. R. Lippmann, D. Fried, I.Graf, J.Haines, K.Kendall,D.McClung, D.Weber, S.Webster,D.Wyschogrod, R.Cunningham and M. Zissman, Evaluating Intrusion Detection Systems: The 1998 DARPA Offline Intrusion Detection Evaluation. In Proceedings of the DARPA Information Survivability Conference and Exposition 2, IEEE Computer Society Press, 2000, pages 12-26.
57. R. Lippmann, J.W. Haines, D.J. Fried, J. Korba and K. Das, The 1999 darpa offline intrusion detection evaluation, Comput Networks 34(4) (2000), 579-595.
58. M.V. Mahoney and P.K. Chan, An analysis of the 1999 darpa/lincoln laboratory evaluation data for network anomaly detection. In Proc 6th Intl Symp on Recent Advances in Intrusion Detection (RAID 2003) 2820, 2003, pages 220-237. (Pubitemid 137633167)
59. M.A. Mazurowski, P.A. Habas, J.M. Zurada, J.Y. Lo, J.A. Baker and G.D. Tourassi. Training neural network classifiers for medical decision making: The effects of imbalanced datasets on classification performance, Neural Networks 21(2-3) (2008), 427-436.
60. J. McHugh, Testing intrusion detection systems: A critique of the 1998 and 1999 darpa intrusion detection system evaluations as performed by lincoln laboratory, ACM Transactions on Information and System Security 3(4) (2000), 262-294.
61. L.Mena and J.A.Gonzalez,Machine learning for imbalanced datasets: Application inmedical diagnostic. In Proceedings of the 19th International FLAIRS Conference, 2006.
62. E. Michailidis, S.K. Katsikas and E. Georgopoulos, Intrusion Detection Using Evolutionary Neural Networks. In PCI '08: Proceedings of the 2008 Panhellenic Conference on Informatics, Washington, DC, USA, 2008, pages 8-12. IEEE Computer Society.
63. D. Michie, D.J. Spiegelhalter and C.C. Taylor, editors. Machine learning, neural and statistical classification. Ellis Horwood, 1994. Available online: [http://www.amsta.leeds.ac.uk/ charles/statlog/].
64. MIT Lincoln Labratory, DARPA Intrusion Detection Evaluation Data Sets, 2000. Available online: [http://www.ll.mit. edu/IST/ideval/data/data index.html].
65. T.M. Mitchell, Machine Learning, McGraw-Hill, 1997.
66. S. Mukkamala and A.H. Sung, Identifying Key Features for Intrusion Detection Using Neural Networks. In ICCC '02: Proceedings of the 15th international conference on Computer communication, Washington, DC, USA, 2002, pages 1132-1138. International Council for Computer Communication.
67. S. Mukkamala and A.H. Sung, Artificial Intelligent Techniques for Intrusion Detection. In Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, 2003.
68. Z.S. Pan, S.C. Chen, G.B Hu and D.Q. Zhang, Hybrid Neural Network and C4.5 for Misuse Detection. In Machine Learning and Cybernetics, Xi'an, 2003, pages 2463-2467.
69. M. Panda and M.R. Patra, Network intrusion detection using naive bayes, IJCSNS International Journal of Computer Science and Network Security 7(12) (2007), 258-263.
70. M. Panda and M.R. Patra, Ensemble of classifiers for detecting network intrusion. In ICAC3 '09: Proceedings of the International Conference on Advances in Computing, Communication and Control, New York, NY, USA, 2009, pages 510-515. ACM.
71. S. Peddabachigari,A.Abraham, C.Grosan and J.Thomas,Modeling Intrusion Detection Systems Using Hybrid Intelligent Systems, Journal of Network and Computer Applications 30(1) (January 2007), 114-132. (Pubitemid 44666486)
72. L. Portnoy, E. Eskin and S. Stolfo, Intrusion Detection With Unlabeled Data Using Clustering. In Proceedings of the ACM Workshop on Data Mining Applied to Security, 2001.
73. J.R. Quinlan, C4.5: programs for machine learning, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1993.
74. L.R. Rabiner, A tutorial on hidden markov models and selected applications in speech recognition, Proceedings of the IEEE 77(2) (1989), 257-286.
75. V. Ramos and A. Abraham, ANTIDS: Self Organized Antbased Clustering Model for Intrusion Detection System, 2004.
76. M. Sabhnani and G. Serpen, Application of Machine Learning Algorithms to KDD Intrusion Detection Dataset within Misuse Detection Context. In Proceedings of the International Conference on Machine Learning, Models, Technologies and Applications (MLMTA 2003) 1 2003, pages 209-215.
77. M. Sabhnani and G. Serpen, Why machine learning algorithms fail in misuse detection on kdd intrusion detection data set, Intelligent Data Analysis 8(4) (2004), 403-415.
78. E. Stamatatos, Author identification: Using text sampling to handle the class imbalance problem, Information Processing and Management 44(2) (2008), 790-799. (Pubitemid 351288480)
79. A.H. Sung and S. Mukkamala, Identifying Important Features for Intrusion Detection Using Support Vector Machines and Neural Networks. In SAINT '03: Proceedings of the 2003 Symposium on Applications and the Internet, Washington, DC, USA, 2003, pages 209-216. IEEE Computer Society.
80. A. Tajbakhsh, M. Rahmati and A. Mirzaei, Intrusion detection using fuzzy association rules, Appl Soft Comput 9(2) (2009), 462-469.
81. The UCI KDD Archive, Kdd cup '99 data set, 1999. Available online: [http://kdd.ics.uci.edu/databases/kddcup99/ kddcup99.html].
82. G. Vigna and R.A. Kemmerer, NetSTAT: A Network Based Intrusion Detection Approach. In Proceedings of the 14th Annual Computer Security Application Conference, 1998, pages 25-34.
83. W. Wang, X. Guan and X. Zhang, Processing of massive audit data streams for realtime anomaly intrusion detection, Comput Commun 31(1) (2008), 58-72. (Pubitemid 350251566)
84. Weka 3, Data Mining Software in Java, 2008. http://www.cs.waikato.ac.nz/ ml/weka/.
85. I.H. Witten and E. Frank, Data Mining: Practical machine Learning Tools and Techniques. Morgan Kaufmann, 2nd edition, 2005.
86. C. Wright, F. Monrose and G.M. Masson, HMM Profiles for Network Traffic Classification. In VizSEC/DMSEC '04: Proceedings of the 2004 ACM workshop on Visualization and data mining for computer security, New York, NY, USA, 2004, pages 9-15. ACM. (Pubitemid 40821168)
87. C. Xiang, P.C. Yong and L.S. Meng, Design of multiplelevel hybrid classifier for intrusion detection system using Bayesian clustering and decision trees, Pattern Recogn Lett 29(7) (2008), 918-924.
88. Y. Xie, X. Li, E.W.T. Ngai and W. Ying, Customer Churn Prediction using Improved Balanced Random Forests, Expert Systems with Applications 36(3) (2009), 5445-5449.
89. X. Yang, Y. Zheng, M. Siddique and G. Beddoe, Learning from imbalanced data: A comparative study for colon CAD. In Society of PhotoOptical Instrumentation Engineers (SPIE) Conference Series, volume 6915 of Presented at the Society of PhotoOptical Instrumentation Engineers (SPIE) Conference, April 2008.
90. A. Zainal, M.A. Maarof, S.M. Shamsuddin and A. Abraham, Ensemble of One Class Classifiers for Network Intrusion Detection System. In Proceedings of the 4th International Conference on Information Assurance and Security, IEEE, 2008, pages 180-185.