Over the past year, HardenedBSD has been hard at work in integrating the Cross-DSO CFI implementation in llvm. We have reached a point where we can release an early (pre-alpha) public Call For Testing (CFT) of this work.
For reasons which will be described below, we recommend this CFT be used by those using root-on-ZFS with boot environments. We recommend testing in a dedicated boot environment.
This initial round of testing is best suited for development server installations. Production servers and desktops/laptops are not advised for testing at this time. We're looking for feedback on what works and doesn't work.
Introduction
Control Flow Integrity, or CFI, is an exploit mitigation that aims to make it harder for an attacker to hijack the control flow of an executable image. llvm's CFI implementation provides forward-edge protection, meaning it protects call sites and non-return code branches. llvm includes basic and incomplete backward-edge protection via SafeStack.
CFI in llvm consists of two flavors:
1. Non-Cross-DSO CFI
2. Cross-DSO CFI
For over a year now, HardenedBSD has adopted non-Cross-DSO CFI in 12-CURRENT/amd64. Support for non-Cross-DSO CFI was added for 12-CURRENT/arm64 on 01 July 2018. Non-Cross-DSO CFI applies CFI to the applications themselves, but not on the shared objects they depend on. Cross-DSO CFI applies CFI to both applications and shared objects, enforcing CFI across shared object boundaries.
When an application or shared object is compiled, its source files typically get compiled first to intermediate object files. Enabling Cross-DSO CFI requires compiling and linking both static and shared libraries with Link Time Optimization (LTO). When LTO is enabled, these object files are no longer ELF object files, but rather LLVM IR bitcode object files.
Linking applications that have been compiled with LTO generally only requires ld.lld as the linker. Linking libraries that have been compiled with LTO requires switching certain compiler toolchain components to ones that understand LLVM IR bitcode. To prepare for Cross-DSO CFI, we switched ar, ranlib, nm, and objdump to their respective llvm compiler toolchain components. This gives us the ability to use LTO across-the-board for the HardenedBSD userland, with a few exceptions.
Note that because Cross-DSO CFI requires storing metadata regarding the shared library boundaries at runtime, Cross-DSO CFI requires ASLR and PaX NOEXEC at a minimum to be effective. If an attacker knows the address of the metadata pages, the attacker can first perform data-only attacks for later code execution/code reuse attacks. Similarily, if an attacker is able to mark non-executable, yet writable, pages as executable while still obeying CFI, (example: JIT compiled code) the attacker can still gain perform execution/code reuse attacks.
Known Issues And Limitations
There are a few known issues. Before we dive into the testing procedure, I would like to talk a bit about known regressions. Note that this list of known issues essentially also constitutes a "work-in-progress" and every known issue will be fixed prior to the
official launch of Cross-DSO CFI.
It seems llvm does not like statically compiling applications with LTO that have a mixture of C and C++ code. /sbin/devd is one of these applications. As such, when Cross-DSO CFI is enabled, devd is compiled as a Position-Independent Executable (PIE). Doing this breaks UFS systems where /usr is on a separate partition. We are currently looking into solving this issue to allow devd to be statically compiled again.
NO_SHARED is now unset in the tools build stage (aka, bootstrap-tools, cross-tools). This is related to the static compilation issue above. Unsetting NO_SHARED for to tools build stage is only a band-aid until we can resolve static compliation with LTO.
One goal of our Cross-DSO CFI integration work is to be able to support the cfi-icall scheme when dlopen(3) and dlsym(3)/dlfunc(3) is used. This means the runtime linker (RTLD), must be enhanced to know and care about the CFI runtime. This enhancement is not currently implemented, but is planned.
When Cross-DSO CFI is enabled, SafeStack is disabled. This is because compiling with Cross-DSO CFI brings in a second copy of the sanitizer runtime, violating the One Definition Rule (ODR). Resolving this issue should be straightforward: Unify the sanitizer runtime into a single common library that both Cross-DSO CFI and SafeStack can link against.
As of 07 Jun 2018, libpmc and friends are receiving a lot of code churn in upstream FreeBSD. The jevents application in lib/libpmc/pmu-events is used as a build tool to generate code. Enabling Cross-DSO CFI disables building PMC-related tools (libpmc and friends) due to the jevents application segfaulting during the build process.
When the installed world has Cross-DSO CFI enabled, performing a buildworld with Cross-DSO CFI disabled fails. This is somewhat related to the static compilation issue described above.
Linking with Cross-DSO CFI can cause lld to use an extremely large amount of memory. For each parallel build job, budget around 15GB of memory for the linker.
Due to the issues discussed above, this CFT is applicable to users who either use ZFS or where /usr is contained within the root filesystem.
Procedure For Testing
Use git to clone locally the HardenedBSD Playground repo. The instructions below assume using /usr/src as the location for the source tree. It also blows away your existing source tree, if it exists. If you want to keep your existing source tree, feel free to
modify the steps below to your liking.
Due to the complexity of building Cross-DSO CFI, the buildworld step must be completed twice: once without Cross-DSO CFI and a second time with. The non-Cross-DSO CFI world must be installed prior to performing the second build. As noted above, we will work to ensure this doesn't need to happen later.
These instructions make the following assumptions:
1. The root filesystem is on ZFS, with the proper layout for ZFS Boot Environments.
2. The beadm package is installed.
3. The existing installation is running 12-CURRENT on amd64.
# cd /usr
# rm -rf src
# git clone https://github.com/HardenedBSD/hardenedBSD-playground.git \
src
# cd src
# git checkout -b hardened/current/cross-dso-cfi \
origin/hardened/current/cross-dso-cfi
# make -sj$(sysctl -n hw.ncpu) buildworld buildkernel
# beadm create cfi-01
# beadm mount cfi-01 /tmp/newbe
# make -s installworld installkernel DESTDIR=/tmp/newbe
# mergemaster -iFUD /tmp/newbe
# beadm umount cfi-01
# beadm activate cfi-01
# shutdown -r now
Binary Updates
We will provide binary updates in base for the hardened/current/cross-dso-cfi feature branch on amd64 until this work gets merged into hardened/current/master. Take a look at Appendix A for a sample hbsd-update.conf configuration file for the Cross-DSO CFI work.
Future Work
We're not done, yet! There's still plenty of work to do. Of upmost importance is fixing static compilation with LTO enabled. Without it, statically-linked applications will crash. devd can go back to being a statically-linked application and users with /usr on a separate non-ZFS filesystem will be able to take advantage of Cross-DSO CFI.
Secondly, we need to re-integrate SafeStack, giving us backward-edge protections once again.
Third up is integration with the RTLD. Without it, we still need to disable the cfi-icall scheme for applications that make use of dlopen(3)+dlsym(3)/dlfunc(3).
Given that we're in uncharted territory, we will likely find other issues. We will keep the community updated and informed. Once all issues have been resolved, we will work on integration with ports.
We need to ensure buildworld works with the various CFI (MK_CFI and MK_CROSS_DSO_CFI) options toggled, regardless of installed world state.
Given the tremendous memory requirements, HardenedBSD may not be able to apply Cross-DSO CFI across the entire package repository. The current amd64 package building system is a dual Xeon system with eight cores per CPU (sixteen cores total), 192GB RAM, and 64GB swap. Without Cross-DSO CFI, building the package repo for amd64 takes around 82 hours to complete using all sixteen cores. With Cross-DSO CFI enabled, the package build server eventually runs out of swap using only eight of the sixteen cores.
Though we plan to support Cross-DSO CFI in arm64, amd64 will be the primary development platform until the major issues are worked out. The sanitizer framework needs to be updated to take FreeBSD/HardenedBSD into account on arm64. As of 14 July 2018, the llvm sanitizer framework does not support FreeBSD/arm64. With time, we plan to change that.
HardenedBSD may very well be the first UNIX-like operating system with full Cross-DSO CFI integration across its entire base operating system userland.
Appendix A - hbsd-update.conf
# hbsd-update.conf
# Configuration settings for hbsd-update.
# This file is read in through a /bin/sh shell and uses that syntax.
# dnsrec:
# DNS TXT record to use when looking up the version info for the
# latest update.
#
# This record name seems redundant, but it provides the following
# information:
# 1) architecture
# 2) branch (hardened/current/master) in reverse form
# 3) repo (hardenedbsd)
dnsrec="$(uname -m).cross-dso-cfi.current.hardened.hardenedbsd-playground.updates.hardenedbsd.org"
# kernel:
# Which kernel to install.
# By default, this is intelligently detected by parsing `uname -v`
# output.
#kernel="HARDENEDBSD"
# capath:
# Location of the trusted root certificate store.
capath="/usr/share/keys/hbsd-update/trusted"
# branch:
# Which branch/tag we are pointing to. This option is only used in
# this file for the baseurl option below.
branch="hardened/current/cross-dso-cfi"
# baseurl:
# Where to get the update from.
baseurl="http://updates.hardenedbsd.org/pub/HardenedBSD-playground/updates/${branch}/$(uname -m)"
# dnssec:
# Use DNSSEC for validating the DNS TXT record. Default: yes
#dnssec="no"
