Automated Binary Software Attack Surface Reduction

dc.contributor.ORCID0000-0002-8576-4856 (Ghaffarinia, M)
dc.contributor.advisorHamlen, Kevin W
dc.creatorGhaffarinia, Masoud
dc.date.accessioned2020-09-09T14:26:15Z
dc.date.available2020-09-09T14:26:15Z
dc.date.created2020-05
dc.date.issued2020-05
dc.date.submittedMay 2020
dc.date.updated2020-09-09T14:26:17Z
dc.description.abstractAutomated removal of potentially exploitable, abusable, or unwanted code features from binary software is potentially valuable in scenarios where security-sensitive organizations wish to employ general-purpose, closed-source, commercial software in specialized computing contexts where some of the software’s functionalities are unneeded. This dissertation proposes binary control-flow trimming, a new method of automatically reducing the attack surfaces of binary software, affording code consumers the power to remove features that are unwanted or unused in a particular deployment context. The approach targets stripped binary native code with no source-derived metadata or symbols, can remove semantic features irrespective of whether they were intended and/or known to code developers, and anticipates consumers who can demonstrate desired features (e.g., via unit testing), but who may not know the existence of specific unwanted features, and who lack any formal specifications of the code’s semantics. Through a combination of runtime tracing, machine learning, in-lined reference monitoring, and contextual control-flow integrity enforcement, it is demonstrated that automated code feature removal is nevertheless feasible under these constraints, even for complex programs such as compilers and servers. The approach additionally accommodates consumers whose demonstration of desired features is incomplete; a tunable entropy-based metric detects coverage lapses and conservatively preserves unexercised but probably desired flows. A prototype implementation for Intel x86-64 exhibits low runtime overhead for trimmed binaries (about 1.87%), and case studies show that consumer-side control-flow trimming can successfully eliminate zero-day vulnerabilities. Binary control-flow trimming relies foundationally upon control-flow integrity (CFI) enforcement, which has become a mainstay of protecting certain classes of software from code-reuse attacks. Using CFI to enforce the highly complex, context-sensitive security policies needed for feature removal requires a detailed analysis of CFI’s compatibility with large, binary software products. However, prior analyses of CFI in the literature have primarily focused on assessing CFI’s security weaknesses and performance characteristics, not its ability to preserve intended program functionalities (semantic transparency) of large classes of diverse, mainstream software products. This is in part because although there exist many performance and security benchmarking suites, there remains no standard regimen for assessing compatibility. Researchers must often therefore resort to anecdotal assessments, consisting of tests on homogeneous software collections with limited variety (e.g., GNU Coreutils), or on CPU benchmarks (e.g., SPEC) whose limited code features are not representative of large, mainstream software products. To fill this void, this dissertation presents ConFIRM (CONtrol-Flow Integrity Relevance Metrics), a new evaluation methodology and microbenchmarking suite for assessing compatibility, applicability, and relevance of CFI protections for preserving the intended semantics of software while protecting it from abuse. Reevaluation of CFI solutions using ConFIRM reveals that there remain significant unsolved challenges in securing many large classes of software products with CFI, including software for market-dominant OSes (e.g., Windows) and code employing certain ubiquitous coding idioms (e.g., event-driven callbacks and exceptions). An estimated 47% of CFI-relevant code features with high compatibility impact remain incompletely supported by existing CFI algorithms, or receive weakened controls that leave prevalent threats unaddressed (e.g., return-oriented programming attacks). Discussion of these open problems highlights issues that future research must address to bridge these important gaps between CFI theory and practice.
dc.format.mimetypeapplication/pdf
dc.identifier.urihttps://hdl.handle.net/10735.1/8861
dc.language.isoen
dc.rights©2020 Masoud Ghaffarinia. All rights reserved.
dc.subjectMachine learning
dc.subjectComputer networks -- Security measures -- Software
dc.subjectComputer programming
dc.titleAutomated Binary Software Attack Surface Reduction
dc.typeDissertation
dc.type.materialtext
thesis.degree.departmentComputer Science
thesis.degree.grantorThe University of Texas at Dallas
thesis.degree.levelDoctoral
thesis.degree.namePHD

Files

Original bundle

Now showing 1 - 1 of 1
Loading...
Thumbnail Image
Name:
ETD-5608-011D-261923.28.pdf
Size:
903.79 KB
Format:
Adobe Portable Document Format
Description:
Dissertation

License bundle

Now showing 1 - 2 of 2
No Thumbnail Available
Name:
LICENSE.txt
Size:
1.85 KB
Format:
Plain Text
Description:
No Thumbnail Available
Name:
PROQUEST_LICENSE.txt
Size:
5.85 KB
Format:
Plain Text
Description: