Testing Intrusion Detection Systems: A Critique of the 1998 and 1999 DARPA Intrusion Detection System Evaluations as Performed by Lincoln Laboratory

ACM Transactions on Information and System Security

Volumn 3, Issue 4, 2000, Pages 262-294

  Mchugh, John ^a  

^a CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

Computer security; intrusion detection; receiver operating curves (ROC); Security; software evaluation

Indexed keywords

EID: 85019691440     PISSN: 10949224     EISSN: 15577406     Source Type: Journal    
DOI: 10.1145/382912.382923     Document Type: Article

References (30)

1. Alessandri, D. 2000. Using rule-based activity descriptions to evaluate intrusion-detection systems. In RAID2000, H. Debar, L. Me, and S. F. Wu, Eds. Springer-Verlag, New York, NY, 183-196
2. Allen, J., Christie, A., Fithen, W., Mchugh, J., Pickel, J., and Stoner, E. 2000. State of the practice of intrusion detection technologies. CMU/SEI-99-TR-028,CMU/SEI. Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA
3. Axelsson, S. 1999. The base-rate fallacy and its implications for the difficulty of intrusion detection. In Proceedings of the 6th ACM Conference on Computer and Communications Security. 1-7
4. Axelsson, S. 2000. Intrusion-detection systems: A taxonomy and survey. 99-115 (March)
5. Axelsson, S. 2000. A preliminary attempt to apply detection and estimation theory to intrusion detection. 00--4 (March)
6. Bellovin, S. M. 1993. Packets found on an internet. SIGCOMM Comput. Commun. Rev. 23, 3 (July), 26-31
7. Das, K. 2000. Attack development for intrusion detection. Master's Thesis. Massachusetts Institute of Technology, Cambridge, MA
8. Debar, H., Dacier, M., Wespi, A., and Lampart, S. 1998. An experimentation workbench for intrusion detection systems. Res. Rep. RZ 2998 (#93044) (Sept.). Research Division, IBM, New York, NY
9. Dharanipragada, S. and Roukos, S. 1998. A fast vocabulary independent algorithm for spotting words in speech. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (May). 233-236
10. Durst, R., Champion, T., Witten, B., Miller, E., and Spagnuolo, L. 1999. Testing and evaluating computer intrusion detection systems. Commun. ACM 42, 7, 53-61
11. Egan, J. P. 1975. Signal Detection Theory and ROC Analysis. Academic Press, Inc., Orlando, FL
12. Einbinder, H. 1964. The Myth of the Britannica. Grove Press, New York, NY
13. Huff, D. 1954. How to Lie with Statistics. W. W. Norton & Co., Inc., New York, NY
14. James, D. A. and Young, S. J. 1994. A fast lattice-based approach to vocabulary independent wordspotting. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing 2. 337-380
15. Jeanrenaud, P., Siu, M., Rohlicek, J. R., Meteer, M., and Gish, H. 1994. Spotting events in continuous speech. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing 2. 381-384
16. Junkawitsch, J. and HÖge, H. 1998. Keyword verification considering the correlation of succeeding feature vectors. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (May). 221-224
17. Kendall, K. 1999. A database of computer attacks for the evaluation of intrusion detection systems. Master's Thesis. Massachusetts Institute of Technology, Cambridge, MA
18. Klatt, D. H. 1977. Review of the ARPA speech understanding project. J. Acoust. Soc. Amer. 62, 1345-1366
19. Korba, J. 2000. Windows NT attacks for the evaluation of intrusion detection systems. Master's Thesis. Massachusetts Institute of Technology, Cambridge, MA
20. Lippmann, R., Haines, J. W., Fried, D. J., Korba, J., and Das, K. 2000. The 1999 DARPA off-line intrusion detection evaluation. In RAID2000, H. Debar, L. Me, and S. F. WU, EDS. SPRINGER-VERLAG, NEW YORK, NY, 162-182
21. Lippmann, R. P., Chang, E. I., and Jankowski, C. R. 1994. Wordspotter training using figure-of-merit back propagation. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing 2. 385-388
22. Lippmann, R. P., Fried, D., Graf, I., Haines, J., Kendall, K., Mcclung, D., Webber, D., Webster, S., Wyschograd, D., Cunninghan, R., and Zissman, M. 2000. Evaluating intrusion detection systems: The 1998 DARPA off-line intrusion detection evaluation. In Proceedings of the on DARPA Information Survivability Conference and Exposition (DISCEX '00, Hilton Head, South Carolina, Jan. 25-27). IEEE Computer Society Press, Los Alamitos, CA, 12-26
23. Maxion, R. A. and Tan, K. M. C. 2000. Benchmarking anomaly-based detection systems. In Proceedings of International Conference on Dependable Systems and Networks (June). 623-630
24. Paxson, V. 1999. Bro: A system for detecting network intruders in real-time. Comput. Netw. J. 23-24 (Dec.), 2435-2463
25. Puketza, N., Chung, M., Olsson, R. A., and Mukherjee, B. 1997. A software platform for testing intrusion detection systems. IEEE Software 14, 5 (Sept.), 43-51
26. Puketza, N. J., Zhang, K., Chung, M., Mukherjee, B., and Olsson, R. A. 1996. A methodology for testing intrusion detection systems. IEEE Trans. Softw. Eng. 22, 10, 719-729
27. Swets, J. A. 1988. Measuring the accuracy of diagnostic systems. Science 24, 48, 1285-1293
28. Swets, J. A. and Pickett, R. M. 1982. Evaluation of Diagnostic Systems. Academic Press, Inc., New York, NY
29. Weber, D. 1998. A taxonomy of computer intrusions. Master's Thesis. Massachusetts Institute of Technology, Cambridge, MA
30. Whiting-O'Keefe, Q. E., Henke, C., and Simborg, D. W. 1984. Choosing the correct unit of analysis in medical care experiments. Med. Care 22, 12 (Dec.), 1101-1114