Recognizing malicious software behaviors with tree automata inference

Formal Methods in System Design

Volumn 41, Issue 1, 2012, Pages 107-128

  Babić, Domagoj ^a   Reynaud, Daniel ^a   Song, Dawn ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Behavioral malware detection; Tree automata inference

Indexed keywords

DATAFLOW; DEPENDENCY GRAPHS; MALICIOUS SOFTWARE; MALWARE DETECTION; MALWARES; SYSTEM CALLS; TREE AUTOMATA;

COMPUTER AIDED ANALYSIS; COMPUTER CRIME; FORESTRY;

AUTOMATA THEORY;

ALGORITHMS; COMPUTER PROGRAMS; FORESTRY;

EID: 84863215860     PISSN: 09259856     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10703-012-0149-1     Document Type: Conference Paper

References (37)

1. Aho AV, Sethi R, Ullman JD (1986) Compilers: principles, techniques, and tools. Addison-Wesley Longman, Boston
2. Babić D (2008) Exploiting structure for scalable software verification. PhD thesis, University of British, Columbia, Vancouver, Canada
3. Babić D, Reynaud D, Song D (2011) Malware analysis with tree automata inference. In: CAV'11: proceedings of the 23rd int conference on computer aided verification. Lecture notes in computer science, vol 6806. Springer, Berlin, pp 116-131
4. G Bonfante M Kaczmarek J-Y Marion 2009 Architecture of a morphological malware detector J Comput Virol 5 263 270 10.1007/s11416-008-0102-4
5. Brumley D, Hartwig C, Zhenkai Liang JN, Song D, Yin H (2008) Automatically identifying trigger-based behavior in malware. In: Botnet detection countering the largest security threat. Advances in information security, vol 36. Springer, Berlin, pp 65-88
6. Chow J, Pfaff B, Garfinkel T, Christopher K, Rosenblum M (2004) Understanding data lifetime via whole system simulation. In: Proc of 13th USENIX security symposium
7. Christodorescu M, Jha S (2004) Testing malware detectors. In: ISSTA'04: proc of the 2004 ACM SIGSOFT int symp on software testing and analysis. ACM Press, New York, pp 34-44
8. Christodorescu M, Jha S, Seshia SA, Song D, Bryant RE (2005) Semantics-aware malware detection. In: SP'05: proc of the 2005 IEEE symp. on security and privacy. IEEE Computer Society Press, Los Alamitos, pp 32-46
9. Christodorescu M, Jha S, Kruegel C (2007) Mining specifications of malicious behavior. In: Proc of the 6th joint meeting of the European software engineering conf and the ACM SIGSOFT symp on the foundations of software engineering. ACM Press, New York, pp 5-14
10. Comon H, Dauchet M, Gilleron R, Löding C, Jacquemard F, Lugiez D, Tison S, Tommasi M (2007) Tree automata techniques and applications. http://tata.gforge.inria.fr/
11. Cormen TH, Leiserson CE, Rivest RL, Stein C (2001) Introduction to algorithms, 2nd edn. The MIT Press, Cambridge
12. Crandall J, Chong F (2005) Minos: control data attack prevention orthogonal to memory model. In: Proc of the 37th int symp on microarchitecture. IEEE Press, New York, pp 221-232
13. Forrest S, Hofmeyr SA, Somayaji A, Longstaff TA (1996) A sense of self for Unix processes. In: Proc of the 1996 IEEE symp on security and privacy. IEEE Computer Society Press, Los Alamitos, pp 120-129
14. Fredrikson M, Jha S, Christodorescu M, Sailer R, Yan X (2010) Synthesizing near-optimal malware specifications from suspicious behaviors. In: Proc of the 2010 IEEE symposium on security and privacy. IEEE Computer Society Press, Los Alamitos, pp 45-60
15. García P (1993) Learning k-testable tree sets from positive data. Technical report, Dept. Syst. Inform. Comput., Univ. Politecnica Valencia, Valencia, Spain
16. P García E Vidal 1990 Inference of k-testable languages in the strict sense and application to syntactic pattern recognition IEEE Trans Pattern Anal Mach Intell 12 920 925 10.1109/34.57687
17. Godefroid P, Klarlund N, Sen K (2005) DART: directed automated random testing. In: PLDI'05: proc of the ACM SIGPLAN conf on prog lang design and implementation. ACM Press, New York, pp 213-223
18. EM Gold 1978 Complexity of automaton identification from given data Inf Control 37 3 302 320 495194 0376.68041 10.1016/S0019-9958(78)90562-4
19. Holzer A, Kinder J, Veith H (2007) Using verification technology to specify and detect malware. In: Lecture notes in computer science, vol 4739. Springer, Berlin, pp 497-504
20. Kang MG, McCamant S, Poosankam P, Song D (2011) DTA++: dynamic taint analysis with targeted control-flow propagation. In: Proceedings of the 18th annual network and distributed system security symposium, San Diego, CA
21. Khoussainov B, Nerode A (2001) Automata theory and its applications. Birkhauser, Basel
22. Kinder J, Katzenbeisser S, Schallhart C, Veith H (2005) Detecting malicious code by model checking. In: Julisch K, Krügel C (eds) GI SIG SIDAR conference on detection of intrusions and malware and vulnerability assessment. Lecture notes in computer science, vol 3548. Springer, Berlin, pp 174-187
23. JC King 1976 Symbolic execution and program testing Commun ACM 19 7 385 394 0329.68018 10.1145/360248.360252
24. Knuutila T (1993) Inference of k-testable tree languages. In: Bunke H (ed) Advances in structural and syntactic pattern recognition: proc of the int workshop. World Scientific, Singapore, pp 109-120
25. Kolbitsch C, Milani P, Kruegel C, Kirda E, Zhou X, Wang X (2009) Effective and efficient malware detection at the end host. In: The 18th USENIX security symposium
26. D López JM Sempere P García 2004 Inference of reversible tree languages IEEE Trans Syst Man Cybern, Part B, Cybern 34 4 1658 1665 10.1109/TSMCB.2004.827190
27. Luk C, Cohn R, Muth R, Patil H, Klauser A, Lowney G, Wallace S, Reddi V, Hazelwood K (2005) Pin: building customized program analysis tools with dynamic instrumentation. In: PLDI'05: proc of the 2005 ACM SIGPLAN conf on prog lang design and impl. ACM Press, New York, pp 190-200
28. Martignoni L, Paleari R (2010) The libwst library (a Part of WUSSTrace)
29. Matrosov A, Rodionov E, Harley D, Malcho J (2010) Stuxnet under the microscope. Technical report, Eset
30. Moser A, Kruegel C, Kirda E (2007) Exploring multiple execution paths for malware analysis. In: SP'07: proc of the 2007 IEEE symposium on security and privacy, Washington, DC, USA. IEEE Computer Society Press, Los Alamitos, pp 231-245
31. Newsome J, Song D (2005) Dynamic taint analysis: automatic detection, analysis, and signature generation of exploit attacks on commodity software. In: Proc of the network and distributed systems security symposium
32. G Suh J Lee D Zhang S Devadas 2004 Secure program execution via dynamic information flow tracking Oper Syst Rev 38 5 85 96 10.1145/1037949.1024404
33. Symantec (2010) Symantec global internet security threat report: trends for 2009, vol xv. Technical report, Symantec, April
34. Wagner D, Dean D (2001) Intrusion detection via static analysis. In: Proc of the 2001 IEEE symposium on security and privacy. IEEE Computer Society Press, Los Alamitos, p 156
35. Wagner D, Soto P (2002) Mimicry attacks on host-based intrusion detection systems. In: CCS'02: proc of the 9th ACM conf on comp and comm security. ACM Press, New York, pp 255-264
36. You I, Yim K (2010) Malware obfuscation techniques: a brief survey. In: Int conf on broadband, wireless computing, communication and applications, pp 297-300
37. Y Zalcstein 1972 Locally testable languages J Comput Syst Sci 6 2 151 167 307538 0242.68038 10.1016/S0022-0000(72)80020-5