HardenedBSD September 2023 Status Report

The HardenedBSD 14-STABLE build infrastructure is back online. A new package build is running. I apologize for the outage, and I appreciate the patience.

My wife and I are investigating some potential opportunities to purchase a home and plant our roots in Colorado. There is a chance we might significantly accelerate our plans at purchasing a home, moving the date from around June 2024 to even potentially November or December 2023. Should things go the way I'm thinking they might, the downtime for the HardenedBSD infrastructure would be limited to a single weekend, perhaps even a single Saturday.

We would like to ask for more public mirrors. Please reach out to netops@hardenedbsd.org if you would like to mirror our installation media and OS update artifacts. This may be especially useful in case we find unexpected dead trees in the metaphorical forest of purchasing a new-to-us home.

In HardenedBSD's src repo:

  1. A conditional in the virtual memory subsystem pertaining to our PaX NOEXEC-inspired strict W^X implementation. I suspect there may be one or two more conditionals to double-check.
  2. The output provided by the `newvers.sh` build script should be more correct.

In the ports tree:

  1. First-time submitter Shion Yorigami provided a fix for lang/gcc-aux.
  2. Shion Yorigami provided a fix for security/py-cryptography.
  3. Shawn Webb patched ports-mgmt/poudriere-hbsd to take into account the hardening of the vfs.lookup_cap_dotdot and vfs.lookup_cap_dotdot_nonlocal sysctl nodes.
  4. ports-mgmt/pkg is now built with PIE and LTO.
  5. devel/boost-libs now builds.
  6. math/symengine now builds.
  7. The default version of llvm in ports was bumped from 15 to 16. Because we build base system libraries with LTO, the default minimum ports llvm version needs to match the base llvm version.

Additional infrastructure info: the rsync service was moved to a new VM to account for the additional 14-STABLE build artifacts. I hope to deploy the Tor Onion Service endpoints for the 14-STABLE build infrastructure this week.

I also worked a bit on hbsdfw, forward-porting the changes to its HardenedBSD 14-STABLE feature branch. I'm still hoping to get a new build out soon-ish, but it is indeed taking longer than I originally anticipated.

We still have a number of ports that are broken, graphics/sane-backends being broken prevents editors/libreoffice from building. I'm hoping we can get some help from the community in fixing broken ports. I really appreciate those who contribute, no matter the form of contribution--code patches, advocacy, funding, documentation, etc. It's all equally important and very much appreciated.

HardenedBSD August 2023 Status Report

August was an interesting month. We updated the core networking components (firewall, switch, cabling) to support 2.5Gbps internally. This will enable us to grow to handle additional internal network load. The NICs in the servers still need to be updated to 2.5Gbps, but that will come with time.

FreeBSD created the stable/14 branch. Shortly afterward, I followed along and created the hardened/14-stable/master branch for HardenedBSD. We do not have the build infrastructure in place for 14-stable, but will soon. I need to order new SSD drives to increase capacity on one of the build servers. Afterwards, I'll build the necessary VMs. I suspect it will take another month or so for the hardened/14-stable/master build VMs to be brought up.

Speaking of the next month or so, I also plan to move hbsdfw's src tree to 14-stable in September.

In January, I plan to merge the hardened/current/cross-dso-cfi branch into hardened/current/master. Between now and then, I need to determine how best to get DTrace working again with Cross-DSO CFI. Please let me know if you require use of DTrace in hardened/current/master. Letting me know usage patterns will help me best determine my own priorities.

So, let's get into the src changes:

  1. hbsd-update was taught how to regenerate the default HTTPS root trust store.
  2. unbound-host(1) build in hardened/current/master was fixed.
  3. Two sysctl knobs were hardened:
    • vfs.lookup_cap_dotdot (old default: 1, new: 0)
    • vfs.lookup_cap_dotdot_nonlocal (old default: 1, new: 0)
  4. Memory permission transition code was debugged and some fixes committed. There might be one more change needed to fully address this.

There was only two changes of note in the ports tree: Update ports-mgmt/pkg to 1.20.6 and a HardenedBSD-specific change to ftp/curl.. After getting my research network back online, it appeared that pkg could not resolve .onion addresses anymore. I knew that the pkg 1.19.x line could. FreeBSD switched pkg from libfetch to libcurl with the 1.20.x line. In March of this year, libcurl introduced code rejecting (with no possibility of override) resolutions of .onion addresses. This means that on my research device (running HardenedBSD), I could no longer update packages.

I reverted the prohibition on Tor Onion Services within both the libcurl embedded in the pkg source code, updating the ports-mgmt/pkg and ftp/curl ports.

Now that the prohibition on resolving .onion addresses has been removed from pkg and libcurl, I was able to verify direct access into our infrastructure through our Tor Onion Services.

So, if you have configured Tor as a transparent proxy, you can continue using curl/libcurl like normal on HardenedBSD. As a project, we believe everyone should have equal access to resources. Placing non-optional arbitrary restrictions in yet another monocultured ubiquitous project harms more than it helps. We encourage application developers to implement toggles should they be deemed appropriate. Firefox provides a good example here by providing an easy toggle (the "network.dns.blockDotOnion" option in `about:config`).

The LAYLO mirror in the Netherlands stood up a Tor Onion Service endpoint for their mirror: http://zqsjg25lnx7zratmne3dhbcqt5paehitom3qp2rjmwttuy7gzbzqwayd.onion/pu...

We are grateful to the community for their continued support of HardenedBSD. Your contributions, no matter the form in which they come, are noticed and greatly appreciated.

HardenedBSD July 2023 Status Report

There was only one notable change in the src repo in July 2023. But first, a little background info:

A long time ago, I started on a project that makes anonymous remote code injection and PLT/GOT redireciton techniques over the ptrace boundary easy in one little consumable API. The end-goal of the tool is to support injection of shared objects in a completely anonymous manner, and to be able to hijack PLT/GOT entries to point to their counterpart in the newly-injected shared object. I started on this work well over a decade ago (it has roots back to ideas I had in 2003.) The project is aptly named libhijack[0].

One common technique is to rely on shared memory-backed file descriptors, writing the shared object to the shmfd, lseek(shmfd,0,seek_set), then fdlopen(shmfd). This causes the RTLD to load the shared object from anonymous memory. In fact, I've published a PoC[1] that shows this very technique.

I'm currently working on providing shmfd-based anonymous shared object injection support in libhijack, so it can be performed over the ptrace boundary with a simple API call. Imagine something like this on a webserver running nginx:


pid_t pid = get_nginx_pid();
const char *path_to_shared_object = "/lib/libpcap.so.8";
InjectSharedObject(pid, path_to_shared_object);

This would cause the target process to create a new shmfd, write the contents of libpcap.so.8 to it, seek to the beginning, then load it via fdlopen.

I have this mostly implemented, but I'm running into some ptrace oddities.

Where this ties into HardenedBSD's src is: while writing this code, I kept thinking to myself, how would I defend against this kind of technique? Results from fstat(2) aren't helpful as there's no way to detect that we're looking at an anonymous shared memory-backed file descriptor.

However, I noticed that calling fstatfs(2) on the shmfd causes undocumented behavior: fstatfs(2) will return EINVAL. The underlying code for shared memory file descriptors doesn't implement a handler for fstatfs(2), causing the syscall to return EINVAL.

I then added code to the RTLD to check the return value of a call to fstatfs(2) on the file descriptor passed in when RTLD hardening is enabled. If fstatfs(2) fails and sets errno to EINVAL, we prohibit loading the object.

This would force an attacker to fully implement what I call a "remote RTLD": an out-of-process RTLD that loads objects over a boundary (the boundary in this particular case being ptrace.) The attacker would have to force the application to call mmap (which libhijack supports), inject into those new mapping (which libhijack supports), but then perform all the RTLD logic and fixups over the boundary (which libhijack does not support.)

My hope is that one day, libhijack gains that last little bit. That last little bit is the most complex bit. That's why I'm going the shmfd route first: to prove the concept and to flush out a "rough draft" of what's in my head.

HardenedBSD's PaX NOEXEC-inspired strict W^X implementation is effective over the ptrace boundary, further frustrating the concept of a remote RTLD.

[0]: https://github.com/SoldierX/libhijack
[1]: https://git.hardenedbsd.org/shawn.webb/random-code/-/tree/main/memdlopen

HardenedBSD June 2023 Status Report

I'm a few days late to June's status report. But I have extremely good news. The development and build infrastructure is back online! All of our equipment made it safe and sound to Colorado. I'm currently being serenaded by the hum of the infrastructure in my home office.

If you cannot reach any bit of our public infrastructure, please let me know. There's a chance we may bring more servers online, depending on power availability and stability.

In addition to moving across the country, here's what happened in the src tree:

  1. Process tracing a process that has entered capabilities mode (aka, it successfully called `cap_enter(2)`) is now prohibited by default. A new per-jail sysctl tunable, `hardening.prohibit_ptrace_capsicum`, has been added. System administrators can also use `hbsdcontrol(8)` to toggle the restriction on a per-application basis.
  2. Some llvm work. FreeBSD imported llvm 16 into base. We needed to fix a few things regarding llvm16.

In ports:

  1. First-time patch submitter, vujo, disabled RELRO for editors/vscode.

HardenedBSD May 2023 Status Report

May 2023 has been a very busy month behind-the-scenes here at HardenedBSD. I want to start off, though, by talking about infrastructure downtime as it's the most important take-away from this status report.

On 03 June 2023, in preparation for our move to Colorado, I will take down the following servers (hosted out of my home):

  1. GitLab ( https://git.hardenedbsd.org/ )
  2. The main installers mirror ( https://installers.hardenedbsd.org/pub )
  3. The package build servers
  4. The OS and update build servers
  5. The Tor Onion Service nodes

The last pre-move builds will start tomorrow on 01 June 2023. They should complete by 02 June 2023. The builds will be accessible via our two mirrors. The link to the list of up-to-date HardenedBSD mirrors is: https://hardenedbsd.org/content/mirrors

We're grateful for those who provide mirrors. You ensure that HardenedBSD's installers (and hopefully one day: packages) are available regardless of our own infrastructure. The project, and I'm sure the entire community, thanks you.

At the time of writing this status report, we have a tentative temporary hosting solution with limited bandwidth capabilities. We're evaluating multiple hosting possibilities, with this tentative temporary hosting solution being only a backup in case those other options don't pan out. We do not have an ETA for bringing the infrastructure back online.

Now, let's get on with the fun stuff! The HardenedBSD Foundation is now fully registered in Colorado! We're moving the headquarters from Maryland to Colorado while still maintaining a presence my Maryland. Keeping a Maryland presence allows us to continue serving local communities. For those curious, our Colorado registration can be found at: https://www.coloradosos.gov/ccsa/ViewReports.do?ceId=325482

We will soon update the US federal side, the completion of which finishes the move to Colorado. The HardenedBSD Foundation Board of Directors has been a great help in supporting the move.

And now, in the src tree:

  1. Shawn fixed NULL pointer derefs in the following portions of the kernel:
    • mrsas(4)
    • powerpc code
    • NVIDIA Tegra CPU Frequency code <- integer overflow, too
    • iw_cxgbe
    • drm2 <- integer overflow, too
    • hdspe-pcm
    • linsysfs(5)
    • beri's virtio support <- integer overflow, too
    • firewire(4)
    • qnnxe(4)
    • axgbe pci code <- integer overflows, too
    • mmc(4)
  2. On 14-CURRENT (and soon in hbsdfw), SHM hardening places restricitons on what can be done with the shared memory subsystem (see `shm_open(2)`.) This feature is launching with one technique (with more planned):
    • Use of `shm_open(2)/__sys_shm_open2` system calls is prohibited when:
      1. The `hardening.harden_shm` sysctl tunable is enabled;
      2. The process has not opted out of the feature;
      3. The process has entered capability mode (aka, Capsicum mode)
  3. FreeBSD introduced an old vulnerability, the infamous (argc == 0) CVE, but this time in the linuxulator. Shawn ensured that HardenedBSD remains invulnerable.
  4. Shawn fixed the 14-CURRENT builds by fixing a FreeBSD commit for building ELF Toolchain's `strings(1)` utility. By default, we use llvm's.

And in ports:

  1. MrUnix disabled the JIT and re-enabled MPROTECT for www/ungoogled-chromium

In last month's status report, I mentioned that I decided to punish myself by running HardenedBSD 14-CURRENT/amd64 with Cross-DSO CFI enabled for base userland on my primary laptop, even during this move to Colorado. Shawn is happy to report that two months in, and he's still rocking (and keeping up-to-date with) Cross-DSO CFI! :-)

As a reminder, we mirror our src and ports repos on GitHub. They will remain available (and updated as best as I can). Links below.

GitHub src repo: https://github.com/HardenedBSD/hardenedBSD
GitHub ports repo: https://github.com/hardenedBSD/ports
Installation media mirrors: https://hardenedbsd.org/content/mirrors
Documentation https://github.com/HardenedBSD/gitlab-wiki/blob/master/Home.md

HardenedBSD April 2023 Status Report

April was busy from an administrative perspective, with me working to get the Foundation and the Project ready to move to Colorado. We have around 90% of what we need to file as a not-for-profit, tax-exempt charitable organization in Colorado. We're hoping to file by the end of 05 May 2023.

Once the Colorado organization is live, we'll switch the federal side to point the headquarters to the new, yet-to-be-determined Colorado address. I believe once that's done, we should be fully re-headquartered.

On 02 June 2023, I plan to do our last pre-move code sync. Remember, we provide read-only mirrors on GitHub (listed below) for our base and ports repositories.

On 03 June 2023, I plan to take the build infrastructure down for the move. We do not have an ETA for bringing it back up, but bringing up the infrastructure will be of highest priority. I'll be unpacking and powering on equipment before I unpack the kitchen. ;-)

Package repos will remain online even during the move. However, we will need to rely on our mirrors (link to the mirrors page below) to provide installation media. We are grateful for those who provide public mirrors.

Please reach out to us (core@hardenedbsd.org or netops@hardenedbsd.org) to get set up as a public mirror if you're interested. The sooner we can get new mirrors launched, the better poised we (the community) are for the move.

Let's get to the changes! In src:

  1. The installer will no longer ask to install the kernel debug distset. We do not support downloading the distsets at install time.
  2. FreeBSD kept trying various methods to enable Netlink support in the kernel. Given our concerns about Netlink's current code quality, we kept trying to follow in disabling Netlink by default.

In ports:

  1. MrUnix disabled the JIT in www/chromium and www/iridium, which switches the default javascript engine to one that's PaX MPROTECT-safe.
  2. MrUnix fixed multimedia/obs-studio
  3. MrUnix disabled PaX MPROTECT for www/node18
  4. MrUnix disabled PaX MPROTECT for www/node19
  5. MrUnix disabled PaX PAGEEXEC for devel/valgrind and devel/valgrind-devel

I have decided to punish myself by running HardenedBSD 14-CURRENT/amd64 with Cross-DSO CFI enabled for base on my primary laptop. My goal here is to see if I can effectuate the move to Colorado while running with Cross-DSO CFI. The first problem I experienced was i3wm, which would crash upon launching any command. Interestintly, xfce4 mostly works. The xfce4-panel application crashes, but the rest of xfce4 seems to work just fine and dandy.

We are grateful for the past, present, and future contributions from the community. There are many ways to contribute to the project. You don't have to be a security expert or even know how to program! We appreciate contributions in any form they come in, like advocacy, monetary donations, documentation, bug reports, etc. Thank you for making this project possible!

GitHub src repo: https://github.com/HardenedBSD/hardenedBSD
GitHub ports repo: https://github.com/hardenedBSD/ports
Installation media mirrors: https://hardenedbsd.org/content/mirrors

HardenedBSD March 2023 Status Report

I missed February's status report, so March's will contain both months. Let's start off with a bit of personal news, though. My wife, our two dogs, and I are officially moving to Colorado! Our move-out date is June 10th.

Additionally, my wife and I have set up a GoFundMe. Please note that donations to the GoFundMe are NOT eligible for tax deductions, but donations directly to The HardenedBSD Foundation are.

I plan to purchase a number of pieces of equipment to help with the move to make sure that the equipment arrives intact. HardenedBSD could still definitely use your continued support, for which I and the rest of the community are grateful.

This move obviously will impact the project. I've spent most of March working on getting the project and the HardenedBSD Foundation ready administratively for the move. There's only three things left to do, and the Foundation will have officially moved headquarters from Maryland to Colorado. We will be leaving a skeleton organziation in Maryland in case we want to do Maryland-specific things. For example, there could be some future Maryland-specific fundraiser or service opportunity.

I plan to keep the infrastructure online as long as I can. I suspect the infrastructure will be taken offline on around 09 June 2023, when we start loading the truck. However, there's a chance that the infrastructure will be taken down up to a week in advance. Since we don't know where we're going to land, I cannot provide an ETA as to when the infrastructure would be back online.

We plan to rent only for up to one year at most. We want to explore Colorado a bit to find out where we want to plant our roots. So there will be another infrastructure-disrupting move on the inside of a year from June. Hopefully that move will be to the home in which we retire in two-ish decades. I'm going to make having full self-hosting capabilities a priority in our "forever" home. So, now, let's get on to the src and ports updates since the end of January 2023.

In src:

  1. tarfs is marked as insecure.
  2. tarfs has been opted into -ftrivial-var-auto-init=zero.
  3. geli has been opted into -ftrivial-var-auto-init=zero.
  4. FreeBSD merged llvm 15 into base, so some fixups were needed on our side.
  5. pf has been opted into -ftrivial-var-auto-init=zero.
  6. pfsync has been opted into -ftrivial-var-auto-init=zero.
  7. pflog has been opted into -ftrivial-var-auto-init=zero.
  8. Default net.inet6.icmp6.nodeinfo to 0. This prevents host information disclosures via ICMP6.
  9. Added some safty nets in the new netlink code
  10. Dislabed netlink multiple times in multiple ways, chasing FreeBSD's continued attempt at enabling netlink by default.
  11. cfi-icall has been disabled for various NFS-related applications.
  12. The TTY pushback vulnerability has been mitigated by hardening the TIOCSTI ioctl. Attempts to use the TIOCSTI ioctl node will fail with EPERM. A new per-jail sysctl tunable (`hardening.harden_tty`) has been added, defaulted to 1.
  13. The default packet TTL is randomized at boot-time, with a minimum of 32 hops. This feature is controlled by the new `HBSD_RESIST_FINGERPRINTING` kernel option, which is enabled by default. This feature can help with resisting fingerprinting attacks and preventing information leaks.
  14. 0x1eef fixed a few typos in libhbsdcontrol.3.

SPECIAL NOTE ON ITEM 13: This has the potential for network disruption. I probably need to adjust the minimum hops as 32 might be too limiting. Please reach out to me directly if you are unable to get to resources that you used to get to. The sysctl node being set is `net.inet.ip.ttl`, and that can be adjusted anytime post-boot.

In ports:

  1. DTRACE is disabled for www/node18.
  2. CFI is disabled for security/sudo.
  3. java/openjdk11 was fixed.
  4. MrUNIX disabled PaX MPROTECT for bin/kdeconnect-app
  5. MrUNIX disabled PaX MPROTECT for multimedia/mpv
  6. MrUNIX disabled PaX MPROTECT for www/librewolf
  7. MrUNIX disabled PaX MPROTECT and PaX PAGEEXEC for www/otter-browser
  8. MrUNIX disabled PaX MPROTECT for www/tor-browser
  9. MrUNIX enabled CFI and SafeStack for x11-wm/i3-gaps

HardenedBSD Firewall (hbsdfw)

hbsdfw is a fork of OPNsense based on HardenedBSD 13-STABLE.

A new build of hbsdfw is out. We still don't provide official in-place updatesupport, so your update process should be as follows:

1. Backup your existing config
2. Reinstall with the new image
3. Restore the config

The default username and password is:

Username: root
Password: hbsdfw

HardenedBSD Infrastructure

We now regularly build 13-STABLE and 14-CURRENT, on the first and fourteenth day of each month. The 13-STABLE and 14-CURRENT package build jobs are still kicked off manually, but usually follow on the same day or the day after the installation media build.

We're looking for more mirrors for the installation media. We're grateful for those who currently run public mirrors.

HardenedBSD January 2023 Status Report

It has been a number of months since the last status report. I got really sick in November/December and am now recovering from COVID. Fun! Regardless, there have been a ton of enhancements and goings on in the project.

In src:

  1. devd(8) is now compiled as a PIE default
  2. FreeBSD commit 140ceb5d956bb8795a77c23d3fd5ef047b0f3c68 introduced the ability to easily force a remote process to perform a syscal (`ptrace(PT_SC_REMOTE)`). This is restricted by the `hardening.prohibit_ptrace_syscall` sysctl node by default. While an attacker can force the remote process to execute syscalls regardless, I think exposing an API to make this easier is not attractive.
  3. The netlink(4) module is marked as insecure. FreeBSD recently landed support for basic netlink socket support. However, the implementation quality seems to be sub-par and has me worried about security implications. FreeBSD has enabled netlink by default in the GENERIC kernel config (from which we inherit). I've disabled it default in the HardenedBSD kernel configs.
  4. I fixed a number of potential NULL deref vulnerabilities in netlink(4).
  5. I fixed a bug in the insecure kmod loading enforcement code.
  6. I fixed a number of code consitency issues in netlink(4).
  7. I introduced a new feature: malloc(9) hardening. A new kernel option, PAX_HARDEN_KMALLOC, enables all malloc(9) hardening features by default. The only feature implemented currently is zeroing allocations on creation and free. To enable allocation zeroing without having to compile a custom kernel, set the `hardening.kmalloc_zero` sysctl tunable node to 1. It is disabled by default due to obvious performance impacts.
  8. Kernel modules can now opt-in to -ftrivial-var-auto-init=zero. Kernel modules currently opted in:
    • virtio-net
    • netlink
    • zfs
    • linuxkpi
    • tmpfs
  9. uuidgen(1) now defaults to generating UUIDv4 identifiers. A new option, -R, provides backwards compatibility for generating UUIDv1 identifiers.

Ports hasn't seen much change, just regular maintenance. So not much to report there.

The infrastructure is slowly coming back online. I'm now hosting GitLab and one build server at home. I'm rewriting the build scripts from scratch, tying together the `hbsd-update` artifacts and the installation media together. I'm about 99% done with that. Once that reaches 100% (hopefully by the end of this week), I will reach out to those who showed desire to host public mirrors of our builds.

I suspect my electricity bills are going to go up by around $250USD/month. Once I'm done with the infrastructure work, I will sit down and switch to the administrative side of the house.

Here are my goals for February 2023:

  1. Write a 2023 project roadmap with firm donation goals, coordinating this with the HardenedBSD Foundation Board of Directors.
  2. Send out donation receipts for 2022 US-based donors who donated $250USD or more for tax deduction purposes.
  3. Determine if/how to provide better cooling to our unfinished basement to prepare for summer. In the summer, our basement can get up to 90F without any servers or computer equipment. Obviously, that presents issues for continued hosting out of our basement.
  4. Documentation! Keep making more progress on updating our wiki with relevant documentation.

Upcoming HardenedBSD Infrastructure Changes

On Sunday, 27 Nov 2022, we will be temporarily taking down the HardenedBSD development and build infrastructure. We are working to find a new home for it and will be working those details out over the next few months.

We provide a read-only mirror of src and ports at GitHub:

src: https://github.com/HardenedBSD/hardenedBSD
ports: https://github.com/HardenedBSD/ports

Here are the short-term physical infrastructure changes:

1. GitLab will (hopefully) be brought back up within one week.
2. The Tor Onion Service endpoints will be brought back up at around the same time.
3. The nightly build server will be taken offline for an extended period of time until we find a new home for it.
4. The binary update server will be taken offline and brought back up at around the same time as the nightly build server.
5. The package building servers will be taken offline for an even more extended period of time. We encourage our community to learn how to use Poudriere to build their own packages.

Here are the short-term infrastructure policy changes:

1. Once the nightly build server is back online, we will no longer produce nightly builds. We will likely build twice per month on a bi-weekly schedule.
2. We will do the same for the binary update server. We will do our best to pair the hbsd-update builds with the bi-weekly builds to keep our users more in-sync with what is installed versus the latest update artifact.

If you know of a hosting provider within a two to four hour drive of Baltimore, Maryland, USA that would host our infrastructure for free with at least one public IPv4 address, please reach out. As a reminder, all donations--whether that's funding, equipment, services, or otherwise--are eligible for tax deductions in the US.

In full, our infrastructure takes around 24U, but we can get around with 16U. The physical infrastructure changes outlined above would be around 5U worth of equipment.

We appreciate the continued help and support from the community. HardenedBSD's continued success is enabled by the many generous ways in which the community contributes to the project.

Please reach out to me (shawn.webb@hardenedbsd.org and/or foundation@hardenedbsd.org) if you have any questions, comments, or concerns.

Pages

Subscribe to HardenedBSD RSS