HardenedBSD February 2024 Status Report

I spent most of February getting 15-CURRENT working again. FreeBSD introduced a new library, libsys, which is where the userland side of performing syscalls is implemented. There's an intricate dance between libsys, libc, libthr, and the CSU. I spent some time learning about that dance, and I still feel like there's more to learn.

HardenedBSD 15-CURRENT is mostly fixed. Prior to the libsys change, we built libc with Link-Time Optimizations (LTO). Building libc with LTO was part of the problem, though not the only issue. Once all the issues are resolved, I will re-enable LTO for libc.

FreeBSD also introduced a new pam_xdg(8) PAM module. This module had a few vulnerabilities, which are fixed in HardenedBSD. The two NULL deref bugs are fixed in FreeBSD now, too. The filesystem race condition and recursion limit issues are somewhat mitigated in HardenedBSD, but not completely.

HardenedBSD now has two VisionFive StarFive2 64-bit RISCV SBCs. I spent a little bit of time toying around with them. The kernel boots to the mountroot prompt. I've been wanting to learn hardware hacking, including writing drivers, so these little SBCs might be great for that.

In ports:

  1. the u-boot ports are fixed
  2. dns/unbound was updated to 1.19.1
  3. the net/vnstat port was fixed
  4. graphics/mupdf is now built as a PIE with SafeStack enabled
  5. The secadm ports were updated

Looking forward into March: I'm hoping to close two gaps of knowledge: the dance mentioned above and I'd like to return to jemalloc hardening. I plan to also do some infrastructure maintenance--routine updates.

HardenedBSD January 2024 Status Report

Happy new year! January saw a few changes in the src repository:

  1. Update some internal netlink(4) functions to use an unsigned integer for buffer size.
  2. The NETLINK option is now effectively required by our upstream FreeBSD. I would encourage the community to study its code for potential issues.
  3. The sysctl knobs vm.objects and vm.swap_objects are now only made available to privileged unjailed processes.
  4. A new sysctl node is introduced that disables new USB device connections: hardening.pax.prohibit_new_usb. Possible values to set it to:
    • 0: disabled
    • 1: enabled
    • 2: enabled without possibility to disable without incurring a reboot

    It would be cool to see a new option: 3, enforce a USB device allowlist. I would like to delegate that to anyone who wants to volunteer to do that work. :-)

In ports:

  1. An unneeded patch for dns/c-ares was remooved now that the port has been updated by upstream
  2. Fix build of devel/ivykis
  3. Bump default llvm ports version to 17

In the infrastructure:

As detailed in this announcement, HardenedBSD will slow its OS/update build cadence from bi-weekly to monthly.

The issue plaguing git clone https://... of src or ports has been resolved! Major thanks to h3artbl33d for helping resolve this issue.

Change in Build Cadence

Building packages takes quite a while, especially for 14-STABLE. The last package build for that branch took 371 hours, around 15-ish days. We currently build installer images and base OS updates every 14 days. This means that the 14-STABLE package builds will always be at least one build behind.

So, we need to slow the cadence of our installer image and base OS update builds. Instead of building installer images and base OS updates bi-weekly, we will move to monthly builds.

This will allow us to, at the start of every month, perform a clean package build. After that initial clean package build completes and is synced, we can then perform incremental package builds for the remainder of the month.

There are still a few potential issues with this. Sometimes, a port gets updated that 20,000+ other ports depend on, effectively meaning that we're doing a clean build even though an incremental build was requested.

If an incremental package build is unable to complete by the time the 1st day of the next month comes around, we will cancel that build in favor of starting the clean build on the 1st.

Slowing the frequency to monthly for the installers and base OS updates also opens up new opportunities for the community to explore. For example, we would welcome generation of .torrent files. Who knows, maybe we could help perform the initial seeding. :-)

These changes will take place on 01 Feb 2024.

HardenedBSD December 2023 Status Report

Happy new year! In December, the focus was on finishing the move to our new home. Updates were applied across the entire infrastructure.

In src, we reverted commit 8cf35a2cbe0270666845a5f2634cfc50c90696f1, which had originally set the default umask to 027. This is a bit too strict to work in the real world as a default for HardenedBSD, so we chose to revert back to 022.

In ports, textproc/jq was updated to 1.7.1. CFI and SafeStack were enabled for textproc/jq. ports-mgmt/poudriere-hbsd was updated to 3.4.

The secadm project was brought up-to-date to account for recent VFS-related changes in FreeBSD. The ports entry was updated accordingly.

While not an official HardenedBSD project, libhijack has given inspiration to new hardening techniques. libhijack is a post-exploitation tool that can be used to inject arbitrary code and hook dynamically-loaded functions. It's a tool that I created with origins back to the early 2000's. libhijack can now inject shared objects over the ptrace boundary anonymously. For those curious, libhijack can be found here: https://github.com/SoldierX/libhijack

HardenedBSD November 2023 Status Report

The focus in November was navigating and closing the purchase of a new home. I worked a little bit on the change to jemalloc that will optionally return NULL for zero-sized allocation requests. I realized that I lack a lot of knowledge on how jemalloc works and I need to fill those gaps fully before landing this feature.

I worked a bit on Cross-DSO CFI, including working in the ports tree. I started toying around a little with applying CFI (via LLVM kCFI) to select kernel modules. Some work needs to happen in the kernel ELF linker to support newer relocation types ld.lld emits when linking with -fsanitize=kcfi.

In src:

  1. The vfs.zfs.bclone_enabled sysctl tunable is set to 0 by default. We hope to re-enable it after a long soak time in FreeBSD.
  2. Generation of OpenSSH RSA host keys is disabled by default.

In ports:

  1. Loic F fixed the lang/gcc11 and lang/gcc12 ports
  2. Shawn Webb fixed the databases/postgresql*-server ports
  3. Shawn Webb fixed dns/void-zones-tools
  4. Shawn Webb added a new port: security/evilginx2

IMPORTANT INFRASTRUCTURE NOTE:

I'm hoping to move the HardenedBSD development/build infrastructure over to the new house as soon as this weekend. Electrical work still needs to happen, but I might have a workaround available until a proper solution is in place.

If the move does not happen this weekend (02-03 Dec 2023), then it will definitely happen the following weekend.

Other projects:

While writing this very status report, I've kicked off another build of hbsdfw. This includes the latest ZFS changes from upstream and more OpenSSL fixes. I'm hoping to have it tested and uploaded this weekend.

Once I have the jemalloc feature sorted out, I plan to resume work on libhijack. I've been letting my brain think about what needs to happen next over the past few months. Implementing an RTLD over the PTrace boundary is a bit more difficult than one might think. :-)

I've also started writing a little HardenedBSD testing framework. This will help us identify and resolve regressions (like the PaX NOEXEC regression I still have yet to fully resolve.)

If you would like to help with HardenedBSD development, but aren't sure where to start, there's this nifty issue board that shows all the bugs, features, and other work we would like help with: https://git.hardenedbsd.org/groups/hardenedbsd/-/boards/11

HardenedBSD October 2023 Status Report

This status is going to be pretty short. The focus in October has been on the home purchase.

News regarding the home purchase: if things go according to plan, we will take possession of the property towards the end of November 2023 (this month).

In ports:

  1. Loic F fixed archivers/rpm4
  2. Loic F fixed graphics/sane-backends
  3. Shawn Webb fixed lang/perl5.36
  4. Shawn Webb updated ports-mgmt/pkg to 1.20.8
  5. Shawn Webb "fixed" audio/pulseaudio by ripping out SIMD support. A more proper fix is needed. If the community can help us re-gain SIMD support for this port, that would be very much appreciated.

I worked on hbsdfw a little bit, figured out some of the issues in building the base OS packages. Now I need to figure out why some ports fail to build.

For November, I'm working on a new malloc.conf(5) option: bool nullonzero, default: false. This malloc.conf(5) option will cause malloc(3)-and-friends to return NULL when a zero size is passed in. This would prevent a class of integer overflow vulnerabilities that can lead to buffer overflows.

For example, in the common OpenSolaris CTF code (used in DTrace), we find the following function:

    char *
    xstrndup(char *str, size_t len)
    {
        char *newstr;
        
        /*
         * HardenedBSD note:
         *
         * If len happens to be SIZE_MAX, the size passed to malloc is zero.
         */
        if ((newstr = malloc(len + 1)) == NULL)
            memory_bailout();
    
        /*
         *HardenedBSD note:
         *
         * malloc(0) happily handed us a non-NULL pointer, sized zero. We now
         * copy SIZE_MAX bytes from str to newstr.
         */
        (void) strncpy(newstr, str, len);
        newstr[len] = '\0';
    
        return (newstr);
    }

HardenedBSD would prefer developers get used to providing reasonableness checks regarding size calculations. This xstrndup function could include this defensive coding technique of applying reasonable checks like so:

    char *
    xstrndup(char *str, size_t len)
    {
        char *newstr;
    
        if (str == NULL || len + 1 == 0) {
            return (NULL);
        }
    
        if (len == 0 && str[0] != '\0') {
            return (NULL);
        }
    
        /*
         * If len is still 0, that means we're duplicating an empty string.
         * That's fine, since strdup("") is a thing.
         */
        
        if ((newstr = malloc(len + 1)) == NULL)
            memory_bailout();
    
        (void) strncpy(newstr, str, len);
        newstr[len] = '\0';
    
        return (newstr);
    }

These new reasonable checks ensure that sleepy, caffeine-deprived developers have a harder time making understandable mistakes. We want to encourage safe and defensive programming techniques, especially before we launch this feature. I will fix the issues that come up in the base OS. When it lands, this feature will be defaulted to disabled, but the reasonableness checks and fixes will remain in place regardless.

So that's a preview of what's going on. I hope to land this before the end of the month. Perhaps mid-2024 we'll have it enabled by default.

HardenedBSD September 2023 Status Report

The HardenedBSD 14-STABLE build infrastructure is back online. A new package build is running. I apologize for the outage, and I appreciate the patience.

My wife and I are investigating some potential opportunities to purchase a home and plant our roots in Colorado. There is a chance we might significantly accelerate our plans at purchasing a home, moving the date from around June 2024 to even potentially November or December 2023. Should things go the way I'm thinking they might, the downtime for the HardenedBSD infrastructure would be limited to a single weekend, perhaps even a single Saturday.

We would like to ask for more public mirrors. Please reach out to netops@hardenedbsd.org if you would like to mirror our installation media and OS update artifacts. This may be especially useful in case we find unexpected dead trees in the metaphorical forest of purchasing a new-to-us home.

In HardenedBSD's src repo:

  1. A conditional in the virtual memory subsystem pertaining to our PaX NOEXEC-inspired strict W^X implementation. I suspect there may be one or two more conditionals to double-check.
  2. The output provided by the `newvers.sh` build script should be more correct.

In the ports tree:

  1. First-time submitter Shion Yorigami provided a fix for lang/gcc-aux.
  2. Shion Yorigami provided a fix for security/py-cryptography.
  3. Shawn Webb patched ports-mgmt/poudriere-hbsd to take into account the hardening of the vfs.lookup_cap_dotdot and vfs.lookup_cap_dotdot_nonlocal sysctl nodes.
  4. ports-mgmt/pkg is now built with PIE and LTO.
  5. devel/boost-libs now builds.
  6. math/symengine now builds.
  7. The default version of llvm in ports was bumped from 15 to 16. Because we build base system libraries with LTO, the default minimum ports llvm version needs to match the base llvm version.

Additional infrastructure info: the rsync service was moved to a new VM to account for the additional 14-STABLE build artifacts. I hope to deploy the Tor Onion Service endpoints for the 14-STABLE build infrastructure this week.

I also worked a bit on hbsdfw, forward-porting the changes to its HardenedBSD 14-STABLE feature branch. I'm still hoping to get a new build out soon-ish, but it is indeed taking longer than I originally anticipated.

We still have a number of ports that are broken, graphics/sane-backends being broken prevents editors/libreoffice from building. I'm hoping we can get some help from the community in fixing broken ports. I really appreciate those who contribute, no matter the form of contribution--code patches, advocacy, funding, documentation, etc. It's all equally important and very much appreciated.

HardenedBSD August 2023 Status Report

August was an interesting month. We updated the core networking components (firewall, switch, cabling) to support 2.5Gbps internally. This will enable us to grow to handle additional internal network load. The NICs in the servers still need to be updated to 2.5Gbps, but that will come with time.

FreeBSD created the stable/14 branch. Shortly afterward, I followed along and created the hardened/14-stable/master branch for HardenedBSD. We do not have the build infrastructure in place for 14-stable, but will soon. I need to order new SSD drives to increase capacity on one of the build servers. Afterwards, I'll build the necessary VMs. I suspect it will take another month or so for the hardened/14-stable/master build VMs to be brought up.

Speaking of the next month or so, I also plan to move hbsdfw's src tree to 14-stable in September.

In January, I plan to merge the hardened/current/cross-dso-cfi branch into hardened/current/master. Between now and then, I need to determine how best to get DTrace working again with Cross-DSO CFI. Please let me know if you require use of DTrace in hardened/current/master. Letting me know usage patterns will help me best determine my own priorities.

So, let's get into the src changes:

  1. hbsd-update was taught how to regenerate the default HTTPS root trust store.
  2. unbound-host(1) build in hardened/current/master was fixed.
  3. Two sysctl knobs were hardened:
    • vfs.lookup_cap_dotdot (old default: 1, new: 0)
    • vfs.lookup_cap_dotdot_nonlocal (old default: 1, new: 0)
  4. Memory permission transition code was debugged and some fixes committed. There might be one more change needed to fully address this.

There was only two changes of note in the ports tree: Update ports-mgmt/pkg to 1.20.6 and a HardenedBSD-specific change to ftp/curl.. After getting my research network back online, it appeared that pkg could not resolve .onion addresses anymore. I knew that the pkg 1.19.x line could. FreeBSD switched pkg from libfetch to libcurl with the 1.20.x line. In March of this year, libcurl introduced code rejecting (with no possibility of override) resolutions of .onion addresses. This means that on my research device (running HardenedBSD), I could no longer update packages.

I reverted the prohibition on Tor Onion Services within both the libcurl embedded in the pkg source code, updating the ports-mgmt/pkg and ftp/curl ports.

Now that the prohibition on resolving .onion addresses has been removed from pkg and libcurl, I was able to verify direct access into our infrastructure through our Tor Onion Services.

So, if you have configured Tor as a transparent proxy, you can continue using curl/libcurl like normal on HardenedBSD. As a project, we believe everyone should have equal access to resources. Placing non-optional arbitrary restrictions in yet another monocultured ubiquitous project harms more than it helps. We encourage application developers to implement toggles should they be deemed appropriate. Firefox provides a good example here by providing an easy toggle (the "network.dns.blockDotOnion" option in `about:config`).

The LAYLO mirror in the Netherlands stood up a Tor Onion Service endpoint for their mirror: http://zqsjg25lnx7zratmne3dhbcqt5paehitom3qp2rjmwttuy7gzbzqwayd.onion/pu...

We are grateful to the community for their continued support of HardenedBSD. Your contributions, no matter the form in which they come, are noticed and greatly appreciated.

HardenedBSD July 2023 Status Report

There was only one notable change in the src repo in July 2023. But first, a little background info:

A long time ago, I started on a project that makes anonymous remote code injection and PLT/GOT redireciton techniques over the ptrace boundary easy in one little consumable API. The end-goal of the tool is to support injection of shared objects in a completely anonymous manner, and to be able to hijack PLT/GOT entries to point to their counterpart in the newly-injected shared object. I started on this work well over a decade ago (it has roots back to ideas I had in 2003.) The project is aptly named libhijack[0].

One common technique is to rely on shared memory-backed file descriptors, writing the shared object to the shmfd, lseek(shmfd,0,seek_set), then fdlopen(shmfd). This causes the RTLD to load the shared object from anonymous memory. In fact, I've published a PoC[1] that shows this very technique.

I'm currently working on providing shmfd-based anonymous shared object injection support in libhijack, so it can be performed over the ptrace boundary with a simple API call. Imagine something like this on a webserver running nginx:


pid_t pid = get_nginx_pid();
const char *path_to_shared_object = "/lib/libpcap.so.8";
InjectSharedObject(pid, path_to_shared_object);

This would cause the target process to create a new shmfd, write the contents of libpcap.so.8 to it, seek to the beginning, then load it via fdlopen.

I have this mostly implemented, but I'm running into some ptrace oddities.

Where this ties into HardenedBSD's src is: while writing this code, I kept thinking to myself, how would I defend against this kind of technique? Results from fstat(2) aren't helpful as there's no way to detect that we're looking at an anonymous shared memory-backed file descriptor.

However, I noticed that calling fstatfs(2) on the shmfd causes undocumented behavior: fstatfs(2) will return EINVAL. The underlying code for shared memory file descriptors doesn't implement a handler for fstatfs(2), causing the syscall to return EINVAL.

I then added code to the RTLD to check the return value of a call to fstatfs(2) on the file descriptor passed in when RTLD hardening is enabled. If fstatfs(2) fails and sets errno to EINVAL, we prohibit loading the object.

This would force an attacker to fully implement what I call a "remote RTLD": an out-of-process RTLD that loads objects over a boundary (the boundary in this particular case being ptrace.) The attacker would have to force the application to call mmap (which libhijack supports), inject into those new mapping (which libhijack supports), but then perform all the RTLD logic and fixups over the boundary (which libhijack does not support.)

My hope is that one day, libhijack gains that last little bit. That last little bit is the most complex bit. That's why I'm going the shmfd route first: to prove the concept and to flush out a "rough draft" of what's in my head.

HardenedBSD's PaX NOEXEC-inspired strict W^X implementation is effective over the ptrace boundary, further frustrating the concept of a remote RTLD.

[0]: https://github.com/SoldierX/libhijack
[1]: https://git.hardenedbsd.org/shawn.webb/random-code/-/tree/main/memdlopen

Pages

Subscribe to HardenedBSD RSS