Outside the closed world: On using machine learning for network intrusion detection

Proceedings - IEEE Symposium on Security and Privacy

Volumn , Issue , 2010, Pages 305-316

  Sommer, Robin ^a   Paxson, Vern ^b  

^a LAWRENCE BERKELEY NATIONAL LABORATORY   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Anomaly detection; Intrusion detection; Machine learning; Network security

Indexed keywords

ACADEMIC RESEARCH; ANOMALY DETECTION; IN-NETWORK; INTRUSION DETECTION APPROACHES; MACHINE LEARNING COMMUNITIES; MACHINE-LEARNING; NETWORK INTRUSION DETECTION; OTHER APPLICATIONS;

LEARNING SYSTEMS; NETWORK SECURITY; RESEARCH;

INTRUSION DETECTION;

EID: 77955209381     PISSN: 10816011     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/SP.2010.25     Document Type: Conference Paper

References (66)

1. C. Ko, M. Ruschitzka, and K. Levitt, "Execution Monitoring of Security-Critical Programs in Distributed Systems: A Specification-based Approach," in Proc. IEEE Symposium on Security and Privacy, 1997.
2. D. R. Ellis, J. G. Aiken, K. S. Attwood, and S. D. Tenaglia, "A Behavioral Approach to Worm Detection," in Proc. ACM CCS WORM Workshop, 2004.
3. G. Linden, B. Smith, and J. York, "Amazon.com Recommendations: Item-to-Item Collaborative Filtering," IEEE Internet Computing, vol. 7, no. 1, pp. 76-80, 2003.
4. J. Bennett, S. Lanning, and N. Netflix, "The Netflix Prize," in Proc. KDD Cup and Workshop, 2007.
5. L. Vincent, "Google Book Search: Document Understanding on a Massive Scale," 2007.
6. R. Smith, "An Overview of the Tesseract OCR Engine," in Proc. International Conference on Document Analysis and Recognition, 2007.
7. F. J. Och and H. Ney, "The Alignment Template Approach to Statistical Machine Translation," Comput. Linguist, vol. 30, no. 4, pp. 411-449, 2004.
8. P. Graham, "A Plan for Spam," in Hackers & Painters. O'Reilly, 2004.
9. D. E. Denning, "An Intrusion-Detection Model," IEEE Transactions on Software Engineering, vol. 13, no. 2, pp. 222-232, 1987.
10. H. S. Javitz and A. Valdes, "The NIDES Statistical Component: Description and Justification," SRI International, Tech. Rep., 1993.
11. W. Lee and D. Xiang, "Information-Theoretic Measures for Anomaly Detection," in Proc. IEEE Symposium on Security and Privacy, 2001.
12. Z. Zhang, J. Li, C. Manikopoulos, J. Jorgenson, and J. Ucles, "HIDE: a Hierarchical Network Intrusion Detection System Using Statistical Preprocessing and Neural Network Classification," in Proc. IEEE Workshop on Information Assurance and Security, 2001.
13. W. Hu, Y. Liao, and V. R. Vemuri, "Robust Anomaly Detection Using Support Vector Machines," in Proc. International Conference on Machine Learning, 2003.
14. C. Sinclair, L. Pierce, and S. Matzner, "An Application of Machine Learning to Network Intrusion Detection," in Proc. Computer Security Applications Conference, 1999.
15. S. A. Hofmeyr, "An Immunological Model of Distributed Detection and its Application to Computer Security," Ph.D. dissertation, University of New Mexico, 1999.
16. V. Chandola, A. Banerjee, and V. Kumar, "Anomaly Detection: A Survey," University of Minnesota, Tech. Rep., 2007.
17. R. J. Bolton and D. J. Hand, "Statistical Fraud Detection: A Review," Statistical Science, vol. 17, no. 3, 2002.
18. C. Gates and C. Taylor, "Challenging the Anomaly Detection Paradigm: A Provocative Discussion," in Proc. Workshop on New Security Paradigms, 2007.
19. "Peakflow SP," http://www.arbornetworks.com/en/ peakflow-sp.html.
20. "StealthWatch," http://www.lancope.com/products/.
21. I. H. Witten and E. Frank, Data Mining: Practical Machine Learning Tools and Techniques (2nd edition). Morgan Kaufmann, 2005.
22. R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification (2nd edition). Wiley Interscience, 2001.
23. S. Axelsson, "The Base-Rate Fallacy and Its Implications for the Difficulty of Intrusion Detection," in Proc. ACM Conference on Computer and Communications Security, 1999.
24. "Make Data Useful," Greg Linden, Data Mining Seminar, Stanford University, 2006. http://glinden.blogspot.com/2006/ 12/slides-from-my-talk-at- stanford.htm%1.
25. C. Allauzen, M. Riley, J. Schalkwyk, W. Skut, and M. Mohri, "OpenFst: A General and Efficient Weighted Finite-state Transducer Library," in Proc. International Conference on Implementation and Application of Automata, 2007.
26. R. Sommer, "Viable Network Intrusion Detection in High-Performance Environments," Ph.D. dissertation, TU München, 2005.
27. C. V. Wright, L. Ballard, F. Monrose, and G. M. Masson, "Language Identification of Encrypted VoIP traffic: Alejandra y Roberto or Alice and Bob?" in Proc. USENIX Security Symposium, 2007.
28. T.-F. Yen, X. Huang, F. Monrose, and M. K. Reiter, "Browser Fingerprinting from Coarse Traffic Summaries: Techniques and Implications," in Proc. Conference on Detection of Intrusions and Malware & Vulnerability Assessment, 2009.
29. A. Narayanan and V. Shmatikov, "Robust De-anonymization of Large Sparse Datasets," in IEEE Symposium on Security and Privacy, 2008.
30. A. Kumar, V. Paxson, and N. Weaver, "Exploiting Underlying Structure for Detailed Reconstruction of an Internet-scale Event," in ACM SIGCOMM Internet Measurement Conference, 2005.
31. Jim Mellander, Lawrence Berkeley National Laboratory, via personal communication, 2009.
32. W. Willinger, M. S. Taqqu, R. Sherman, and D. V. Wilson, "Self-Similarity Through High-Variability: Statistical Analysis of Ethernet LAN Traffic at the Source Level," IEEE/ACM Transactions on Networking, vol. 5, no. 1, 1997.
33. A. Feldmann, A. C. Gilbert, and W. Willinger, "Data Networks As Cascades: Investigating the Multifractal Nature of Internet WAN Traffic," in Proc. ACM SIGCOMM, 1998.
34. V. Paxson, "Bro: A System for Detecting Network Intruders in Real-Time," Computer Networks, vol. 31, no. 23-24, pp. 2435-2463, 1999.
35. P. Gill, M. Arlitt, Z. Li, and A. Mahanti, "YouTube Traffic Characterization: A View From the Edge," in Proc. ACM SIGCOMM Internet Measurement Conference, 2008.
36. A. Nazir, S. Raza, and C-N. Chuah, "Unveiling Facebook: A Measurement Study of Social Network Based Applications," in Proc. SIGCOMM Internet Measurement Conference, 2008.
37. A. Haley, P. Norvig, and F. Pereira, "The Unreasonable Effectiveness of Data," IEEE Intelligent Systems, March/April 2009.
38. D. S. Anderson, C. Fleizach, S. Savage, and G. M. Voelker, "Spamscatter: Characterizing Internet Scam Hosting Infrastructure," in Proc. USENIX Security Symposium, 2007.
39. G. V. Cormack and T. R. Lynam, "Online Supervised Spam Filter Evaluation," ACM Transactions on Information Systems, vol. 25, no. 3, 2007.
40. J. van Beusekom, F. Shafait, and T. M. Breuel, "Automated OCR Ground Truth Generation," in Proc. DAS 2008, Sep 2008.
41. R. Lippmann, R. K. Cunningham, D. J. Fried, I. Graf, K. R. Kendall, S. E. Webster, and M. A. Zissman, "Results of the 1998 DARPA Offline Intrusion Detection Evaluation," in Proc. Recent Advances in Intrusion Detection, 1999.
42. R. Lippmann, J. W. Haines, D. J. Fried, J. Korba, and K. Das, "The 1999 DARPA Off-line Intrusion Detection Evaluation," Computer Networks, vol. 34, no. 4, pp. 579-595, October 2000.
43. "KDD Cup Data," http://kdd.ics.uci.edu/databases/kddcup99/ kddcup99.html.
44. J. McHugh, "Testing Intrusion detection systems: A critique of the 1998 and 1999 DARPA intrusion detection system evaluations as performed by Lincoln Laboratories," ACM Transactions on Information and System Security, vol. 3, no. 4, pp. 262-294, November 2000.
45. M. V. Mahoney and P. K. Chan, "An Analysis of the 1999 DARPA/Lincoln Laboratory Evaluation Data for Network Anomaly Detection," in Proc. Recent Advances in Intrusion Detection, 2003.
46. "ClarkNet-HTTP," http://ita.ee.lbl.gov/html/contrib/ ClarkNet-HTTP.html.
47. Martin Arlitt, via personal communication, 2008.
48. S. Floyd and V. Paxson, "Difficulties in Simulating the Internet," IEEE/ACM Transactions on Networking, vol. 9, no. 4, 2001.
49. "tcpdpriv," http://ita.ee.lbl.gov/html/contrib/tcpdpriv.html.
50. J. Xu, J. Fan, M. Ammar, and S. Moon, "On the Design and Performance of Prefix-Preserving IP Traffic Trace Anonymization," in Proc. ACM SIGCOMM Internet Measurement Workshop, Nov. 2001.
51. R. Pang, M. Allman, V. Paxson, and J. Lee, "The Devil and Packet Trace Anonymization," in Computer Communication Review, 2006.
52. "The Internet Traffic Archive (ITA)," http://ita.ee.lbl.gov.
53. "PREDICT," http://www.predict.org.
54. S. E. Coull, C. V. Wright, F. Monrose, M. P. Collins, and M. K. Reiter, "Playing Devil's Advocate: Inferring Sensitive Information from Anonymized Network Traces," in Proc. Network and Distributed Security Symposium, 2007.
55. K. S. Killourhy and R. A. Maxion, "Toward Realistic and Artifact-Free Insider-Threat Data," Proc. Computer Security Applications Conference, 2007.
56. K. M. Tan and R. A. Maxion, ""Why 6?" Defining the Operational Limits of Stide, an Anomaly-Based Intrusion Detector," in Proc. IEEE Symposium on Security and Privacy, 2002.
57. T. H. Ptacek and T. N. Newsham, "Insertion, Evasion, and Denial of Service: Eluding Network Intrusion Detection," Secure Networks, Inc., Tech. Rep., January 1998.
58. P. Fogla and W. Lee, "Evading Network Anomaly Detection Systems: Formal Reasoning and Practical Techniques," in Proc. ACM Conference on Computer and Communications Security, 2006.
59. M. Barreno, B. Nelson, R. Sears, A. D. Joseph, and J. D. Tygar, "Can Machine Learning Be Secure?" in Proc. ACM Symposium on Information, Computer and Communications Security, 2006.
60. C. Kruegel and G. Vigna, "Anomaly Detection of Webbased Attacks," in Proc. ACM Conference on Computer and Communications Security, 2003.
61. G. Gu, P. Porras, V. Yegneswaran, M. Fong, and W. Lee, "BotHunter: Detecting Malware Infection Through IDSDriven Dialog Correlation," in Proc. USENIX Security Symposium, August 2007.
62. K. G. Anagnostakis, S. Sidiroglou, P. Akritidis, K. Xinidis, E. Markatos, and A. D. Keromytis, "Detecting Targeted Attacks Using Shadow Honeypots," in Proc. USENIX Security Symposium, 2005.
63. N. Provos and T. Holz, Virtual Honeypots: From Botnet Tracking to Intrusion Detection. Addison Wesley, 2007.
64. P. Mittal, V. Paxson, R. Sommer, and M. Winterrowd, "Securing Mediated Trace Access Using Black-box Permutation Analysis," in Proc. ACM Workshop on Hot Topics in Networks, 2009.
65. V. Paxson, "Strategies for Sound Internet Measurement," in ACM SIGCOMM Internet Measurement Conference, Oct. 2004.
66. N. Fraser, "Neural Network Follies," http://neil.fraser.name/ writing/tank, 1998.