Native Software Security Hardening in the Real World : Compatibility, Modularity, Expressiveness, and Performance

This dissertation presents a series of new technologies that significantly bridge the gap between theory and practice of software hijacking defenses based on control-flow integrity (CFI) and in-lined reference monitors (IRMs). CFI has emerged over the past 15 years as one of the strongest known defenses against code-reuse attacks, which are among the top threats to modern software ecosystems. Such attacks wrest control of critical software systems away from lawful users into the hands of adversaries by reusing or repurposing legitimate code blocks for malicious purposes. CFI offers provably strong protections against code-reuse attacks by confining vulnerable software to a strict security policy that constrains its flow of control to paths chosen in advance by developers and legitimate users. Research over the past decade has increased the power and performance of CFI defenses; however, effectively applying many of the strongest CFI algorithms to large, production-level software products have remained difficult and challenging. To expose the root causes of these difficulties, this dissertation presents a new evaluation methodology and microbenchmarking suite, ConFIRM, that is designed to measure applicability, compatibility, and performance characteristics relevant to CFI algorithm evaluation. It provides a set of 20 tests of various CFI-relevant code features and coding idioms (e.g., event-driven callbacks and exceptions), which are widely found in commodity COTS software products and constitute the greatest barriers to more widespread CFI adoption. To overcome a significant class of fundamental challenges identified by ConFIRM, the dissertation then presents object flow integrity (OFI), which is the first source-agnostic CFI system that augments CFI protections with secure, first-class support for binary object exchange across inter-module trust boundaries. A prototype implementation for Microsoft Component Object Model (COM) demonstrates that OFI scales to component-based, eventdriven consumer software with low overheads of under 1%. The approach is demonstrated in practice through an interface-driven approach that is the first to secure full COTS, GUIdriven Windows products with CFI without needing the application source code. Finally, the IRM technology underlying CFI is shown to be effective in web domains for enforcing safety policies by injecting runtime security guards into binary web scripts. In particular, a method of detecting and interrupting unauthorized, browser-based cryptomining is proposed, based on semantic signature-matching. The approach addresses a new wave of cryptojacking attacks, including XSS-assisted, web gadget-exploiting, counterfeit mining. Evaluation shows that the approach is more robust than current static code analysis defenses, which are susceptible to code obfuscation attacks.