Source-Free, Component-Driven Software Security Hardening

Hardening COTS binary software products (e.g., via control-flow integrity (CFI) and/or software fault isolation (SFI) defenses) is extremely complex in contexts where the surrounding software environment includes closed-source, immutable, and possibly obfuscated binary components, such as system libraries, OS kernels, and virtualization layers. It is demonstrated that many code hardening algorithms, when applied only to the user-level software products in such environments, leave open critical vulnerabilities that arise from mismatches between the application-agnostic security policies enforced by the system modules versus the application-specific policies enforced at the application layer. Similar challenges also exist in web environments, which typically involve components of cross-language web scripts. This dissertation proposes the first Control Flow Integrity system to successfully harden multiple, large (millions of lines) binary Windows COTS software without sources. It implements a prototype for Microsoft COM (largest production component-based architecture in the world) with low overhead. Experiences developing and refining this approach for Microsoft Windows environments are reported and discussed. To evaluate and compare various CFI/SFI protections, the dissertation also introduces ConFIRM, a new evaluation methodology and benchmarking suite aimed at better assessing compatibility, applicability, and relevance of control-flow integrity (CFI) protections for preserving the intended semantics of real-world software while protecting it from abuse via hijacking. Reevaluation of CFI/SFI solutions using ConFIRM reveals that there remain significant unsolved challenges in securing many large classes of software products with CFI/SFI, including software for market-dominant OSes (e.g., Windows) and code employing certain ubiquitous coding idioms (e.g., event-driven callbacks and delay-loaded components). In addition, A method of detecting and interrupting unauthorized, browser-based cryptomining is proposed, based on semantic signature-matching. The approach addresses a new wave of cryptojacking attacks, including XSS-assisted, web gadget-exploiting counterfeit mining. Evaluation shows that the approach is more robust than current static code analysis defenses, which are susceptible to code obfuscation attacks. An implementation based on in-lined reference monitoring offers a browser-agnostic deployment strategy that is applicable to average end-user systems without specialized hardware or operating systems.