On the infeasibility of modeling polymorphic shellcode : Re-thinking the role of learning in intrusion detection systems

Machine Learning

Volumn 81, Issue 2, 2010, Pages 179-205

  Song, Yingbo ^a   Locasto, Michael E ^b   Stavrou, Angelos ^b   Keromytis, Angelos D ^a   Stolfo, Salvatore J ^a  

^a Department of Earth and Environmental Sciences   (United States)

^b Krasnow Institute for Advanced Study   (United States)

Author keywords

Blending; Metrics; Polymorphism; Shellcode

Indexed keywords

CURRENT TRENDS; DUAL PROBLEM; IDS SENSORS; INTRUSION DETECTION SYSTEMS; MALICIOUS CODES; METRICS; QUANTITATIVE ANALYSIS; SECURITY SENSORS; SHELLCODE; STATISTICAL MODELS;

BLENDING; COMPUTER CRIME; CRYPTOGRAPHY; POLYMORPHISM; SENSORS;

INTRUSION DETECTION;

EID: 78049530989     PISSN: 08856125     EISSN: 15730565     Source Type: Journal    
DOI: 10.1007/s10994-009-5143-5     Document Type: Article

References (57)

1. Abadi, M., Budiu, M., Erlingsson, U., & Ligatti, J. (2005). Control-flow integrity: principles, implementations, and applications. In Proceedings of the ACM conference on computer and communications security (CCS).
2. AlephOne (2001). Smashing the stack for fun and profit. Phrack, 7(49-14).
3. Anagnostakis, K. G., Sidiroglou, S., Akritidis, P., Xinidis, K., Markatos, E., & Keromytis, A. D. (2005). Detecting targeted attacks using shadow honeypots. In Proceedings of the 14th USENIX security symposium.
4. Bania, P. (2009). Tapion polymorphic engine. http://pb.specialised.info/ all/tapion/.
5. Baratloo, A., Singh, N., & Tsai, T. (2000). Transparent run-time defense against stack smashing attacks. In Proceedings of the USENIX annual technical conference.
6. Barrantes, E. G., Ackley, D. H., Forrest, S., Palmer, T. S., Stefanovic, D., & Zovi, D. D. (2003). Randomized instruction set emulation to distrupt binary code injection attacks. In Proceedings of the 10th ACM conference on computer and communications security (CCS).
7. Bhatkar, S., DuVarney, D. C., & Sekar, R. (2003). Address obfuscation: an efficient approach to combat a broad range of memory error exploits. In Proceedings of the 12th USENIX security symposium (pp. 105-120).
8. Biondi, P. (2006). Shellforge project. http://www.secdev.org/projects/ shellforge/.
9. Brumley, D., Newsome, J., Song, D., Wang, H., & Jha, S. (2006). Towards automatic generation of vulnerability-based signatures. In Proceedings of the IEEE symposium on security and privacy.
10. CERT (2001). Code red I/II worm. http://www.cert.org/advisories/CA-2001- 19.html.
11. Chinchani, R., & Berg, E. V. D. (2005). A fast static analysis approach to detect exploit code inside network flows. In Proceedings of the 8th international symposium on recent advances in intrusion detection (RAID) (pp. 284-304).
12. Costa, M., Crowcroft, J., Castro, M., & Rowstron, A. (2005). Vigilante: end-to-end containment of Internet worms. In Proceedings of the symposium on systems and operating systems principles (SOSP).
13. Cowan, C., Pu, C., Maier, D., Hinton, H., Walpole, J., Bakke, P., Beattie, S., Grier, A., Wagle, P., & Zhang, Q. (1998). Stackguard: automatic adaptive detection and prevention of buffer-overflow attacks. In Proceedings of the USENIX security symposium.
14. Crandall, J. R., Su, Z., Wu, S. F., & Chong, F. T. (2005a). On deriving unknown vulnerabilities from zero-day polymorphic and metamorphic worm exploits. In Proceedings of the 12th ACM conference on computer and communications security (CCS).
15. Crandall, J. R., Wu, S. F., & Chong, F. T. (2005b). Experiences using minos as a tool for capturing and analyzing novel worms for unknown vulnerabilities. In Detection of intrusions and malware and vulnerability assessment (DIMVA).
16. Cui, W., Peinado, M., Wang, H. J., & Locasto, M. E. (2007). ShieldGen: automated data patch generation for unknown vulnerabilities with informed probing. In Proceedings of the IEEE symposium on security and privacy.
17. Detristan, T., Ulenspiegel, T., Malcom, Y., & von Underduk, M. S. (2003). Polymorphic shellcode engine using spectrum analysis. Phrack, 11(61-9).
18. Etoh, J. (2000). GCC extension for protecting applications from stack-smashing attacks. http://www.trl.ibm.com/projects/security/ssp.
19. Fogla, P., & Lee, W. (2006). Evading network anomaly detection systems: formal reasoning and practical techniques. In Proceedings of the 13th ACM conference on computer and communications security (CCS) (pp. 59-68). http://doi.acm.org/10.1145/1180405.1180414.
20. Fogla, P., Sharif, M., Perdisci, R., Kolesnikov, O., & Lee, W. (2006). Polymorphic blending attacks. In Proceedings of the USENIX security conference.
21. Foster, J. C., Osipov, V., Bhalla, N., & Heinen, N. (2005). Buffer overflow attacks: detect, exploit, prevent. Syngress.
22. Joshi, A., King, S. T., Dunlap, G. W., & Chen, P. M. (2005). Detecting past and present intrusions through vulnerability-specific predicates. In Proceedings of the symposium on systems and operating systems principles (SOSP).
23. K2 (2003). ADMmutate documentation. http://www.ktwo.ca/ADMmutate-0.8.4. tar.gz.
24. Kc, G. S., Keromytis, A. D., & Prevelakis, V. (2003). Countering code-injection attacks with instruction-set randomization. In Proceedings of the 10th ACM conference on computer and communications security (CCS) (pp. 272-280).
25. Kim, H. A., & Karp, B. (2004). Autograph: toward automated, distributed worm signature detection. In Proceedings of the USENIX security conference.
26. Kiriansky, V., Bruening, D., & Amarasinghe, S. (2002). Secure execution via program shepherding. In Proceedings of the 11th USENIX security symposium.
27. Kolesnikov, A., & Lee, W. (2006). Advanced polymorphic worms: evading IDS by blending in with normal traffic. In Proceedings of the USENIX security conference.
28. Kruegel, C., & Vigna, G. (2003). Anomaly detection of web-based attacks. In Proceedings of the 10th ACM conference on computer and communications security (CCS).
29. Krugel, C., Kirda, E., Mutz, D., Robertson, W., & Vigna, G. (2005). Polymorphic worm detection using structural information of executables. In Proceedings of the 8th international symposium on recent advances in intrusion detection (RAID) (pp. 207-226).
30. Liang, Z., & Sekar, R. (2005). Fast and automated generation of attack signatures: a basis for building self-protecting servers. In Proceedings of the 12th ACM conference on computer and communications security (CCS).
31. Locasto, M. E., Wang, K., Keromytis, A. D., & Stolfo, S. J. (2005). FLIPS: hybrid adaptive intrusion prevention. In Proceedings of the 8th international symposium on recent advances in intrusion detection (RAID) (pp. 82-101).
32. Metasploit Development Team (2006). Metasploit project. http://www.metasploit.com.
33. Nethercote, N., & Seward, J. (2003). Valgrind: a program supervision framework. In Electronic notes in theoretical computer science (Vol. 89).
34. Newsome, J., & Song, D. (2005). Dynamic taint analysis for automatic detection, analysis, and signature generation of exploits on commodity software. In Proceedings of the 12th symposium on network and distributed system security (NDSS).
35. Newsome, J., Karp, B., & Song, D. (2005). Polygraph: automatically generating signatures for polymorphic worms. In Proceedings of the IEEE symposium on security and privacy.
36. Obscou (2003). Building IA32 'Unicode-Proof' shellcodes. Phrack, 11(61-11).
37. Panda Labs (2007). MPack uncovered. http://pandalabs.pandasecurity.com/.
38. Polychronakis, M., Anagnostakis, K. G., & Markatos, E. P. (2006). Network-level polymorhpic shellcode detection using emulation. In Detection of intrusions and malware and vulnerability assessment (DIMVA).
39. Rix (2001). Writing IA-32 alphanumeric shellcodes. Phrack, 11(57-15).
40. Russell, S., & Norvig, P. (2002). Artificial intelligence: a modern approach. New York: Prentice Hall.
41. SANS (2004a). IISMedia Exploit. http://www.sans.org/newsletters/cva/vol2- 21.php.
42. SANS (2004b). Santy worm. http://isc.sans.org/diary.html?date=2004-12-21.
43. SANS (2004c). Webdav exploit. http://www.sans.org/resources/malwarefaq/ webdav-exploit.php.
44. Siddharth, S. (2005). Evading NIDS. http://www.securityfocus.com/infocus/ 1852.
45. Sidiroglou, S., Giovanidis, G., & Keromytis, A. D. (2005). A dynamic mechanism for recovering from buffer overflow attacks. In Proceedings of the 8th information security conference (ISC) (pp. 1-15).
46. Singh, S., Estan, C., Varghese, G., & Savage, S. (2004). Automated worm fingerprinting. In Proceedings of symposium on operating systems design and implementation (OSDI).
47. Snort Development Team (2009). Snort project. http://www.snort.org/.
48. Song, Y., Locasto, M. E., Stavrou, A., Keromytis, A. D., & Stolfo, S. J. (2007). On the infeasibility of modeling polymorphic shellcode. In Proceedings of the ACM conference on computer and communications security (CCS).
49. D. Spinellis 2003 Reliable identification of bounded-length viruses is NP-complete IEEE Transactions on Information Theory 49 1 280 284 10.1109/TIT.2002.806137 1966706 1063.68045
50. Tcpdump (2009). http://www.tcpdump.org.
51. Toth, T., & Kruegel, C. (2002). Accurate buffer overflow detection via abstract payload execution. In Proceedings of the 5th international symposium on recent advances in intrusion detection (RAID) (pp. 274-291).
52. Wang, K., & Stolfo, S. J. (2004). Anomalous payload-based network intrusion detection. In Proceedings of the 7th international symposium on recent advances in intrusion detection (RAID) (pp. 203-222).
53. Wang, H. J., Guo, C., Simon, D. R., & Zugenmaier, A. (2004). Shield: vulnerability-driven network filters for preventing known vulnerability exploits. In Proceedings of the ACM SIGCOMM conference (pp. 193-204).
54. Wang, K., Cretu, G., & Stolfo, S. J. (2005). Anomalous payload-based worm detection and signature generation. In Proceedings of the 8th international symposium on recent advances in intrusion detection (RAID) (pp. 227-246).
55. Wang, K., Parekh, J. J., & Stolfo, S. J. (2006a). Anagram: a content anomaly detector resistant to mimicry attack. In Proceedings of the 9th international symposium on recent advances in intrusion detection (RAID).
56. Wang, X., Pan, C. C., Liu, P., & Zhu, S. (2006b). SigFree: a signature-free buffer overflow attack blocker. In Proceedings of the 15th USENIX security symposium (pp. 225-240).
57. Yegneswaran, V., Giffin, J. T., Barford, P., & Jha, S. (2005). An architecture for generating semantics-aware signatures. In Proceedings of the 14th USENIX security symposium.