HardenedBSD November 2023 Status Report

The focus in November was navigating and closing the purchase of a new home. I worked a little bit on the change to jemalloc that will optionally return NULL for zero-sized allocation requests. I realized that I lack a lot of knowledge on how jemalloc works and I need to fill those gaps fully before landing this feature.

I worked a bit on Cross-DSO CFI, including working in the ports tree. I started toying around a little with applying CFI (via LLVM kCFI) to select kernel modules. Some work needs to happen in the kernel ELF linker to support newer relocation types ld.lld emits when linking with -fsanitize=kcfi.

In src:

  1. The vfs.zfs.bclone_enabled sysctl tunable is set to 0 by default. We hope to re-enable it after a long soak time in FreeBSD.
  2. Generation of OpenSSH RSA host keys is disabled by default.

In ports:

  1. Loic F fixed the lang/gcc11 and lang/gcc12 ports
  2. Shawn Webb fixed the databases/postgresql*-server ports
  3. Shawn Webb fixed dns/void-zones-tools
  4. Shawn Webb added a new port: security/evilginx2


I'm hoping to move the HardenedBSD development/build infrastructure over to the new house as soon as this weekend. Electrical work still needs to happen, but I might have a workaround available until a proper solution is in place.

If the move does not happen this weekend (02-03 Dec 2023), then it will definitely happen the following weekend.

Other projects:

While writing this very status report, I've kicked off another build of hbsdfw. This includes the latest ZFS changes from upstream and more OpenSSL fixes. I'm hoping to have it tested and uploaded this weekend.

Once I have the jemalloc feature sorted out, I plan to resume work on libhijack. I've been letting my brain think about what needs to happen next over the past few months. Implementing an RTLD over the PTrace boundary is a bit more difficult than one might think. :-)

I've also started writing a little HardenedBSD testing framework. This will help us identify and resolve regressions (like the PaX NOEXEC regression I still have yet to fully resolve.)

If you would like to help with HardenedBSD development, but aren't sure where to start, there's this nifty issue board that shows all the bugs, features, and other work we would like help with: https://git.hardenedbsd.org/groups/hardenedbsd/-/boards/11

HardenedBSD October 2023 Status Report

This status is going to be pretty short. The focus in October has been on the home purchase.

News regarding the home purchase: if things go according to plan, we will take possession of the property towards the end of November 2023 (this month).

In ports:

  1. Loic F fixed archivers/rpm4
  2. Loic F fixed graphics/sane-backends
  3. Shawn Webb fixed lang/perl5.36
  4. Shawn Webb updated ports-mgmt/pkg to 1.20.8
  5. Shawn Webb "fixed" audio/pulseaudio by ripping out SIMD support. A more proper fix is needed. If the community can help us re-gain SIMD support for this port, that would be very much appreciated.

I worked on hbsdfw a little bit, figured out some of the issues in building the base OS packages. Now I need to figure out why some ports fail to build.

For November, I'm working on a new malloc.conf(5) option: bool nullonzero, default: false. This malloc.conf(5) option will cause malloc(3)-and-friends to return NULL when a zero size is passed in. This would prevent a class of integer overflow vulnerabilities that can lead to buffer overflows.

For example, in the common OpenSolaris CTF code (used in DTrace), we find the following function:

    char *
    xstrndup(char *str, size_t len)
        char *newstr;
         * HardenedBSD note:
         * If len happens to be SIZE_MAX, the size passed to malloc is zero.
        if ((newstr = malloc(len + 1)) == NULL)
         *HardenedBSD note:
         * malloc(0) happily handed us a non-NULL pointer, sized zero. We now
         * copy SIZE_MAX bytes from str to newstr.
        (void) strncpy(newstr, str, len);
        newstr[len] = '\0';
        return (newstr);

HardenedBSD would prefer developers get used to providing reasonableness checks regarding size calculations. This xstrndup function could include this defensive coding technique of applying reasonable checks like so:

    char *
    xstrndup(char *str, size_t len)
        char *newstr;
        if (str == NULL || len + 1 == 0) {
            return (NULL);
        if (len == 0 && str[0] != '\0') {
            return (NULL);
         * If len is still 0, that means we're duplicating an empty string.
         * That's fine, since strdup("") is a thing.
        if ((newstr = malloc(len + 1)) == NULL)
        (void) strncpy(newstr, str, len);
        newstr[len] = '\0';
        return (newstr);

These new reasonable checks ensure that sleepy, caffeine-deprived developers have a harder time making understandable mistakes. We want to encourage safe and defensive programming techniques, especially before we launch this feature. I will fix the issues that come up in the base OS. When it lands, this feature will be defaulted to disabled, but the reasonableness checks and fixes will remain in place regardless.

So that's a preview of what's going on. I hope to land this before the end of the month. Perhaps mid-2024 we'll have it enabled by default.

HardenedBSD September 2023 Status Report

The HardenedBSD 14-STABLE build infrastructure is back online. A new package build is running. I apologize for the outage, and I appreciate the patience.

My wife and I are investigating some potential opportunities to purchase a home and plant our roots in Colorado. There is a chance we might significantly accelerate our plans at purchasing a home, moving the date from around June 2024 to even potentially November or December 2023. Should things go the way I'm thinking they might, the downtime for the HardenedBSD infrastructure would be limited to a single weekend, perhaps even a single Saturday.

We would like to ask for more public mirrors. Please reach out to netops@hardenedbsd.org if you would like to mirror our installation media and OS update artifacts. This may be especially useful in case we find unexpected dead trees in the metaphorical forest of purchasing a new-to-us home.

In HardenedBSD's src repo:

  1. A conditional in the virtual memory subsystem pertaining to our PaX NOEXEC-inspired strict W^X implementation. I suspect there may be one or two more conditionals to double-check.
  2. The output provided by the `newvers.sh` build script should be more correct.

In the ports tree:

  1. First-time submitter Shion Yorigami provided a fix for lang/gcc-aux.
  2. Shion Yorigami provided a fix for security/py-cryptography.
  3. Shawn Webb patched ports-mgmt/poudriere-hbsd to take into account the hardening of the vfs.lookup_cap_dotdot and vfs.lookup_cap_dotdot_nonlocal sysctl nodes.
  4. ports-mgmt/pkg is now built with PIE and LTO.
  5. devel/boost-libs now builds.
  6. math/symengine now builds.
  7. The default version of llvm in ports was bumped from 15 to 16. Because we build base system libraries with LTO, the default minimum ports llvm version needs to match the base llvm version.

Additional infrastructure info: the rsync service was moved to a new VM to account for the additional 14-STABLE build artifacts. I hope to deploy the Tor Onion Service endpoints for the 14-STABLE build infrastructure this week.

I also worked a bit on hbsdfw, forward-porting the changes to its HardenedBSD 14-STABLE feature branch. I'm still hoping to get a new build out soon-ish, but it is indeed taking longer than I originally anticipated.

We still have a number of ports that are broken, graphics/sane-backends being broken prevents editors/libreoffice from building. I'm hoping we can get some help from the community in fixing broken ports. I really appreciate those who contribute, no matter the form of contribution--code patches, advocacy, funding, documentation, etc. It's all equally important and very much appreciated.

HardenedBSD August 2023 Status Report

August was an interesting month. We updated the core networking components (firewall, switch, cabling) to support 2.5Gbps internally. This will enable us to grow to handle additional internal network load. The NICs in the servers still need to be updated to 2.5Gbps, but that will come with time.

FreeBSD created the stable/14 branch. Shortly afterward, I followed along and created the hardened/14-stable/master branch for HardenedBSD. We do not have the build infrastructure in place for 14-stable, but will soon. I need to order new SSD drives to increase capacity on one of the build servers. Afterwards, I'll build the necessary VMs. I suspect it will take another month or so for the hardened/14-stable/master build VMs to be brought up.

Speaking of the next month or so, I also plan to move hbsdfw's src tree to 14-stable in September.

In January, I plan to merge the hardened/current/cross-dso-cfi branch into hardened/current/master. Between now and then, I need to determine how best to get DTrace working again with Cross-DSO CFI. Please let me know if you require use of DTrace in hardened/current/master. Letting me know usage patterns will help me best determine my own priorities.

So, let's get into the src changes:

  1. hbsd-update was taught how to regenerate the default HTTPS root trust store.
  2. unbound-host(1) build in hardened/current/master was fixed.
  3. Two sysctl knobs were hardened:
    • vfs.lookup_cap_dotdot (old default: 1, new: 0)
    • vfs.lookup_cap_dotdot_nonlocal (old default: 1, new: 0)
  4. Memory permission transition code was debugged and some fixes committed. There might be one more change needed to fully address this.

There was only two changes of note in the ports tree: Update ports-mgmt/pkg to 1.20.6 and a HardenedBSD-specific change to ftp/curl.. After getting my research network back online, it appeared that pkg could not resolve .onion addresses anymore. I knew that the pkg 1.19.x line could. FreeBSD switched pkg from libfetch to libcurl with the 1.20.x line. In March of this year, libcurl introduced code rejecting (with no possibility of override) resolutions of .onion addresses. This means that on my research device (running HardenedBSD), I could no longer update packages.

I reverted the prohibition on Tor Onion Services within both the libcurl embedded in the pkg source code, updating the ports-mgmt/pkg and ftp/curl ports.

Now that the prohibition on resolving .onion addresses has been removed from pkg and libcurl, I was able to verify direct access into our infrastructure through our Tor Onion Services.

So, if you have configured Tor as a transparent proxy, you can continue using curl/libcurl like normal on HardenedBSD. As a project, we believe everyone should have equal access to resources. Placing non-optional arbitrary restrictions in yet another monocultured ubiquitous project harms more than it helps. We encourage application developers to implement toggles should they be deemed appropriate. Firefox provides a good example here by providing an easy toggle (the "network.dns.blockDotOnion" option in `about:config`).

The LAYLO mirror in the Netherlands stood up a Tor Onion Service endpoint for their mirror: http://zqsjg25lnx7zratmne3dhbcqt5paehitom3qp2rjmwttuy7gzbzqwayd.onion/pu...

We are grateful to the community for their continued support of HardenedBSD. Your contributions, no matter the form in which they come, are noticed and greatly appreciated.

HardenedBSD July 2023 Status Report

There was only one notable change in the src repo in July 2023. But first, a little background info:

A long time ago, I started on a project that makes anonymous remote code injection and PLT/GOT redireciton techniques over the ptrace boundary easy in one little consumable API. The end-goal of the tool is to support injection of shared objects in a completely anonymous manner, and to be able to hijack PLT/GOT entries to point to their counterpart in the newly-injected shared object. I started on this work well over a decade ago (it has roots back to ideas I had in 2003.) The project is aptly named libhijack[0].

One common technique is to rely on shared memory-backed file descriptors, writing the shared object to the shmfd, lseek(shmfd,0,seek_set), then fdlopen(shmfd). This causes the RTLD to load the shared object from anonymous memory. In fact, I've published a PoC[1] that shows this very technique.

I'm currently working on providing shmfd-based anonymous shared object injection support in libhijack, so it can be performed over the ptrace boundary with a simple API call. Imagine something like this on a webserver running nginx:

pid_t pid = get_nginx_pid();
const char *path_to_shared_object = "/lib/libpcap.so.8";
InjectSharedObject(pid, path_to_shared_object);

This would cause the target process to create a new shmfd, write the contents of libpcap.so.8 to it, seek to the beginning, then load it via fdlopen.

I have this mostly implemented, but I'm running into some ptrace oddities.

Where this ties into HardenedBSD's src is: while writing this code, I kept thinking to myself, how would I defend against this kind of technique? Results from fstat(2) aren't helpful as there's no way to detect that we're looking at an anonymous shared memory-backed file descriptor.

However, I noticed that calling fstatfs(2) on the shmfd causes undocumented behavior: fstatfs(2) will return EINVAL. The underlying code for shared memory file descriptors doesn't implement a handler for fstatfs(2), causing the syscall to return EINVAL.

I then added code to the RTLD to check the return value of a call to fstatfs(2) on the file descriptor passed in when RTLD hardening is enabled. If fstatfs(2) fails and sets errno to EINVAL, we prohibit loading the object.

This would force an attacker to fully implement what I call a "remote RTLD": an out-of-process RTLD that loads objects over a boundary (the boundary in this particular case being ptrace.) The attacker would have to force the application to call mmap (which libhijack supports), inject into those new mapping (which libhijack supports), but then perform all the RTLD logic and fixups over the boundary (which libhijack does not support.)

My hope is that one day, libhijack gains that last little bit. That last little bit is the most complex bit. That's why I'm going the shmfd route first: to prove the concept and to flush out a "rough draft" of what's in my head.

HardenedBSD's PaX NOEXEC-inspired strict W^X implementation is effective over the ptrace boundary, further frustrating the concept of a remote RTLD.

[0]: https://github.com/SoldierX/libhijack
[1]: https://git.hardenedbsd.org/shawn.webb/random-code/-/tree/main/memdlopen

HardenedBSD June 2023 Status Report

I'm a few days late to June's status report. But I have extremely good news. The development and build infrastructure is back online! All of our equipment made it safe and sound to Colorado. I'm currently being serenaded by the hum of the infrastructure in my home office.

If you cannot reach any bit of our public infrastructure, please let me know. There's a chance we may bring more servers online, depending on power availability and stability.

In addition to moving across the country, here's what happened in the src tree:

  1. Process tracing a process that has entered capabilities mode (aka, it successfully called `cap_enter(2)`) is now prohibited by default. A new per-jail sysctl tunable, `hardening.prohibit_ptrace_capsicum`, has been added. System administrators can also use `hbsdcontrol(8)` to toggle the restriction on a per-application basis.
  2. Some llvm work. FreeBSD imported llvm 16 into base. We needed to fix a few things regarding llvm16.

In ports:

  1. First-time patch submitter, vujo, disabled RELRO for editors/vscode.

HardenedBSD May 2023 Status Report

May 2023 has been a very busy month behind-the-scenes here at HardenedBSD. I want to start off, though, by talking about infrastructure downtime as it's the most important take-away from this status report.

On 03 June 2023, in preparation for our move to Colorado, I will take down the following servers (hosted out of my home):

  1. GitLab ( https://git.hardenedbsd.org/ )
  2. The main installers mirror ( https://installers.hardenedbsd.org/pub )
  3. The package build servers
  4. The OS and update build servers
  5. The Tor Onion Service nodes

The last pre-move builds will start tomorrow on 01 June 2023. They should complete by 02 June 2023. The builds will be accessible via our two mirrors. The link to the list of up-to-date HardenedBSD mirrors is: https://hardenedbsd.org/content/mirrors

We're grateful for those who provide mirrors. You ensure that HardenedBSD's installers (and hopefully one day: packages) are available regardless of our own infrastructure. The project, and I'm sure the entire community, thanks you.

At the time of writing this status report, we have a tentative temporary hosting solution with limited bandwidth capabilities. We're evaluating multiple hosting possibilities, with this tentative temporary hosting solution being only a backup in case those other options don't pan out. We do not have an ETA for bringing the infrastructure back online.

Now, let's get on with the fun stuff! The HardenedBSD Foundation is now fully registered in Colorado! We're moving the headquarters from Maryland to Colorado while still maintaining a presence my Maryland. Keeping a Maryland presence allows us to continue serving local communities. For those curious, our Colorado registration can be found at: https://www.coloradosos.gov/ccsa/ViewReports.do?ceId=325482

We will soon update the US federal side, the completion of which finishes the move to Colorado. The HardenedBSD Foundation Board of Directors has been a great help in supporting the move.

And now, in the src tree:

  1. Shawn fixed NULL pointer derefs in the following portions of the kernel:
    • mrsas(4)
    • powerpc code
    • NVIDIA Tegra CPU Frequency code <- integer overflow, too
    • iw_cxgbe
    • drm2 <- integer overflow, too
    • hdspe-pcm
    • linsysfs(5)
    • beri's virtio support <- integer overflow, too
    • firewire(4)
    • qnnxe(4)
    • axgbe pci code <- integer overflows, too
    • mmc(4)
  2. On 14-CURRENT (and soon in hbsdfw), SHM hardening places restricitons on what can be done with the shared memory subsystem (see `shm_open(2)`.) This feature is launching with one technique (with more planned):
    • Use of `shm_open(2)/__sys_shm_open2` system calls is prohibited when:
      1. The `hardening.harden_shm` sysctl tunable is enabled;
      2. The process has not opted out of the feature;
      3. The process has entered capability mode (aka, Capsicum mode)
  3. FreeBSD introduced an old vulnerability, the infamous (argc == 0) CVE, but this time in the linuxulator. Shawn ensured that HardenedBSD remains invulnerable.
  4. Shawn fixed the 14-CURRENT builds by fixing a FreeBSD commit for building ELF Toolchain's `strings(1)` utility. By default, we use llvm's.

And in ports:

  1. MrUnix disabled the JIT and re-enabled MPROTECT for www/ungoogled-chromium

In last month's status report, I mentioned that I decided to punish myself by running HardenedBSD 14-CURRENT/amd64 with Cross-DSO CFI enabled for base userland on my primary laptop, even during this move to Colorado. Shawn is happy to report that two months in, and he's still rocking (and keeping up-to-date with) Cross-DSO CFI! :-)

As a reminder, we mirror our src and ports repos on GitHub. They will remain available (and updated as best as I can). Links below.

GitHub src repo: https://github.com/HardenedBSD/hardenedBSD
GitHub ports repo: https://github.com/hardenedBSD/ports
Installation media mirrors: https://hardenedbsd.org/content/mirrors
Documentation https://github.com/HardenedBSD/gitlab-wiki/blob/master/Home.md

HardenedBSD April 2023 Status Report

April was busy from an administrative perspective, with me working to get the Foundation and the Project ready to move to Colorado. We have around 90% of what we need to file as a not-for-profit, tax-exempt charitable organization in Colorado. We're hoping to file by the end of 05 May 2023.

Once the Colorado organization is live, we'll switch the federal side to point the headquarters to the new, yet-to-be-determined Colorado address. I believe once that's done, we should be fully re-headquartered.

On 02 June 2023, I plan to do our last pre-move code sync. Remember, we provide read-only mirrors on GitHub (listed below) for our base and ports repositories.

On 03 June 2023, I plan to take the build infrastructure down for the move. We do not have an ETA for bringing it back up, but bringing up the infrastructure will be of highest priority. I'll be unpacking and powering on equipment before I unpack the kitchen. ;-)

Package repos will remain online even during the move. However, we will need to rely on our mirrors (link to the mirrors page below) to provide installation media. We are grateful for those who provide public mirrors.

Please reach out to us (core@hardenedbsd.org or netops@hardenedbsd.org) to get set up as a public mirror if you're interested. The sooner we can get new mirrors launched, the better poised we (the community) are for the move.

Let's get to the changes! In src:

  1. The installer will no longer ask to install the kernel debug distset. We do not support downloading the distsets at install time.
  2. FreeBSD kept trying various methods to enable Netlink support in the kernel. Given our concerns about Netlink's current code quality, we kept trying to follow in disabling Netlink by default.

In ports:

  1. MrUnix disabled the JIT in www/chromium and www/iridium, which switches the default javascript engine to one that's PaX MPROTECT-safe.
  2. MrUnix fixed multimedia/obs-studio
  3. MrUnix disabled PaX MPROTECT for www/node18
  4. MrUnix disabled PaX MPROTECT for www/node19
  5. MrUnix disabled PaX PAGEEXEC for devel/valgrind and devel/valgrind-devel

I have decided to punish myself by running HardenedBSD 14-CURRENT/amd64 with Cross-DSO CFI enabled for base on my primary laptop. My goal here is to see if I can effectuate the move to Colorado while running with Cross-DSO CFI. The first problem I experienced was i3wm, which would crash upon launching any command. Interestintly, xfce4 mostly works. The xfce4-panel application crashes, but the rest of xfce4 seems to work just fine and dandy.

We are grateful for the past, present, and future contributions from the community. There are many ways to contribute to the project. You don't have to be a security expert or even know how to program! We appreciate contributions in any form they come in, like advocacy, monetary donations, documentation, bug reports, etc. Thank you for making this project possible!

GitHub src repo: https://github.com/HardenedBSD/hardenedBSD
GitHub ports repo: https://github.com/hardenedBSD/ports
Installation media mirrors: https://hardenedbsd.org/content/mirrors

HardenedBSD March 2023 Status Report

I missed February's status report, so March's will contain both months. Let's start off with a bit of personal news, though. My wife, our two dogs, and I are officially moving to Colorado! Our move-out date is June 10th.

Additionally, my wife and I have set up a GoFundMe. Please note that donations to the GoFundMe are NOT eligible for tax deductions, but donations directly to The HardenedBSD Foundation are.

I plan to purchase a number of pieces of equipment to help with the move to make sure that the equipment arrives intact. HardenedBSD could still definitely use your continued support, for which I and the rest of the community are grateful.

This move obviously will impact the project. I've spent most of March working on getting the project and the HardenedBSD Foundation ready administratively for the move. There's only three things left to do, and the Foundation will have officially moved headquarters from Maryland to Colorado. We will be leaving a skeleton organziation in Maryland in case we want to do Maryland-specific things. For example, there could be some future Maryland-specific fundraiser or service opportunity.

I plan to keep the infrastructure online as long as I can. I suspect the infrastructure will be taken offline on around 09 June 2023, when we start loading the truck. However, there's a chance that the infrastructure will be taken down up to a week in advance. Since we don't know where we're going to land, I cannot provide an ETA as to when the infrastructure would be back online.

We plan to rent only for up to one year at most. We want to explore Colorado a bit to find out where we want to plant our roots. So there will be another infrastructure-disrupting move on the inside of a year from June. Hopefully that move will be to the home in which we retire in two-ish decades. I'm going to make having full self-hosting capabilities a priority in our "forever" home. So, now, let's get on to the src and ports updates since the end of January 2023.

In src:

  1. tarfs is marked as insecure.
  2. tarfs has been opted into -ftrivial-var-auto-init=zero.
  3. geli has been opted into -ftrivial-var-auto-init=zero.
  4. FreeBSD merged llvm 15 into base, so some fixups were needed on our side.
  5. pf has been opted into -ftrivial-var-auto-init=zero.
  6. pfsync has been opted into -ftrivial-var-auto-init=zero.
  7. pflog has been opted into -ftrivial-var-auto-init=zero.
  8. Default net.inet6.icmp6.nodeinfo to 0. This prevents host information disclosures via ICMP6.
  9. Added some safty nets in the new netlink code
  10. Dislabed netlink multiple times in multiple ways, chasing FreeBSD's continued attempt at enabling netlink by default.
  11. cfi-icall has been disabled for various NFS-related applications.
  12. The TTY pushback vulnerability has been mitigated by hardening the TIOCSTI ioctl. Attempts to use the TIOCSTI ioctl node will fail with EPERM. A new per-jail sysctl tunable (`hardening.harden_tty`) has been added, defaulted to 1.
  13. The default packet TTL is randomized at boot-time, with a minimum of 32 hops. This feature is controlled by the new `HBSD_RESIST_FINGERPRINTING` kernel option, which is enabled by default. This feature can help with resisting fingerprinting attacks and preventing information leaks.
  14. 0x1eef fixed a few typos in libhbsdcontrol.3.

SPECIAL NOTE ON ITEM 13: This has the potential for network disruption. I probably need to adjust the minimum hops as 32 might be too limiting. Please reach out to me directly if you are unable to get to resources that you used to get to. The sysctl node being set is `net.inet.ip.ttl`, and that can be adjusted anytime post-boot.

In ports:

  1. DTRACE is disabled for www/node18.
  2. CFI is disabled for security/sudo.
  3. java/openjdk11 was fixed.
  4. MrUNIX disabled PaX MPROTECT for bin/kdeconnect-app
  5. MrUNIX disabled PaX MPROTECT for multimedia/mpv
  6. MrUNIX disabled PaX MPROTECT for www/librewolf
  7. MrUNIX disabled PaX MPROTECT and PaX PAGEEXEC for www/otter-browser
  8. MrUNIX disabled PaX MPROTECT for www/tor-browser
  9. MrUNIX enabled CFI and SafeStack for x11-wm/i3-gaps

HardenedBSD Firewall (hbsdfw)

hbsdfw is a fork of OPNsense based on HardenedBSD 13-STABLE.

A new build of hbsdfw is out. We still don't provide official in-place updatesupport, so your update process should be as follows:

1. Backup your existing config
2. Reinstall with the new image
3. Restore the config

The default username and password is:

Username: root
Password: hbsdfw

HardenedBSD Infrastructure

We now regularly build 13-STABLE and 14-CURRENT, on the first and fourteenth day of each month. The 13-STABLE and 14-CURRENT package build jobs are still kicked off manually, but usually follow on the same day or the day after the installation media build.

We're looking for more mirrors for the installation media. We're grateful for those who currently run public mirrors.


Subscribe to HardenedBSD RSS