April 1, 2011
Wenkee Lee
Georgia Institute of Technology
Title: How Bad Guys Use Theory Against Us
Abstract:
Have you ever wondered about the poor state of cyber security, where the bad guys always seem to have the upper hand? Well, it turns out that that they have theory on their side. That is, knowingly or unknowingly, the bad guys have evolved to make detection and analysis of security threats difficult, even from a theoretical point of view.
This talk will discuss challenges caused by and practical solutions to dismaying theoretical results that put security researchers at a distinct disadvantage. Not surprisingly, many of the problems facing traditional host-based defenses (i.e., antivirus software), such as the need to reverse arbitrary transformations that make the malicious portions of program code appear as seemingly benign data, end in undecidability when held to the light of computability theory. I will present a formal model for malicious software's (malware's) use of "unpack-execution", along with a generic algorithm that reverses such transformations by focusing on the results of unpack-execution instead of the specific obfuscation mechanism.
Even when we look to next-generation dynamic (behavioral) analysis tools, the principle of non-interference makes the fulfillment of formal malware analyzer transparency requirements difficult (e.g., maintaining identical instruction execution semantics) or impossible (e.g., preserving an identical notion of time) to completely satisfy. Yet, implementations that ignore these problems become vulnerable to detection attacks -- categorical abstractions of detection techniques employed by modern malicious software. However, we show that through clever application of features in modern hardware, we can construct malware analyzers that both optimally meet formal transparency requirements and are useful to the security practitioner.
