HardenedBSD April 2021 Status Report

In April, HardenedBSD added a new developer named Loic! Loic is primarily focused on cleaning up old cruft in the HardenedBSD project and fixing bugs. It's my hope that, with time and increasing familiarity, he takes on more complex projects. Please join me in a warm welcome.

Just yesterday (30 Apr 2021), I set up the new 12-STABLE/amd64 nightly build server. After months of being offline, we are now building 12-STABLE every night again. I also updated all the servers/VMs/jails in our infrastructure.

I taught hbsd-update how to force IPv4 or IPv6. We now support the typical "-4" and "-6" options that most network applications support.

secadm is having issues right now. We're investigating the right fix. So, if you use secadm for toggling exploit mitigations, you'll probably want to use hbsd-control for now.

FreeBSD's ports migration to git and its impact on HardenedBSD

FreeBSD completed their ports migration from subversion to git. Prior to the official switch, we used the read-only mirror FreeBSD had at GitHub[1]. The new repo is at [2]. A cursory glance at the new repo will show that the commit hashes changed. This presents an issue with HardenedBSD's ports tree in our merge-based workflow.

I'm going to archive our old ports repo[3] and create a new repo[4]. Due to the nature of our changes and how far back our history goes, creating a new repo is necessary. Attempting to do a `git merge --allow-unrelated-histories` works, but brings gitlab to its knees, eventually failing.

For projects downstream of HardenedBSD using the same kind of merge-based workflow, you will need to effectively do the same thing we did. Here's the process I took:

  1. Match the last commit of the old repo (e7ad26d92beff76bbead7b6b675ad5c551e86fa9) with its corresponding commit in FreeBSD's old ports repo (4010f7bbc03638d71781ce091bf40a0907fa12fe).
  2. Clone FreeBSD's new repo to a temporary location
  3. Generate a diff between the old repo and the new: git diff 4010f7bbc03638d71781ce091bf40a0907fa12fe e7ad26d92beff76bbead7b6b675ad5c551e86fa9
  4. Create the hardenedbsd/main branch
  5. Apply the diff created in step 3
  6. Commit and push

Instead of committing directly to the master branch, we created a new branch "hardenedbsd/main". Upstream's branch in our repo is "freebsd/main".

I apologize for the breakage downstream to us. I made a "best effort" attempt at trying to prevent the breakage, but the ports repo is just too large to do it.

Going forward, please use the new repo at [4]. No merge requests and bug reports for the old repo at [3] will be accepted.

I plan to restore the every-six-hour autosync script this weekend. While there's still dust to settle, I plan to do the syncs myself. The dust should be settled this weekend.

[1]: https://github.com/freebsd/freebsd-ports/
[2]: https://cgit.freebsd.org/ports/
[3]: https://git.hardenedbsd.org/hardenedbsd/hardenedbsd-ports/
[4]: https://git.hardenedbsd.org/hardenedbsd/ports

HardenedBSD March 2021 Status Report

This month, I worked on finding and fixing the regression that caused kernel panics on our package builders. I think I found the issue: I made it so that the HARDENEDBSD amd64 kernel just included GENERIC so that we follow FreeBSD's toggling of features. Doing so added QUEUE_MACRO_DEBUG_TRASH to our kernel config. That option is the likely culprit. If the next package build (with the option removed) completes, I will commit the change that removes QUEUE_MACRO_DEBUG_TRASH from the HARDENEDBSD amd64 kernel.

I still have one new server to set up. I plan to use it for our 12-STABLE builds. I enabled the 14-CURRENT/arm64 nightly builds and we've now completed two production package builds.

I'm giving a virtual presentation on 07 Apr 2021 I'm giving titled "HardenedBSD 2021 State of the Hardened Union." It details the work we've been doing since the last HardenedBSD State of the Union.

As part of that presentation, I'd like to highlight areas in which HardenedBSD is used. If you or your employer uses HardenedBSD and would like me to add a slide about it, please reach out to me.

In April, I plan to focus on the ports tree. I'm going to audit all the ports that fail to build and determine if I can easily get them to build. A large number of ports ignore our setting -fPIC and -fPIE compiler flags and subsequently fail to build.

Jason Donenfeld of the Wireguard project is looking for a maintainer/developer for the Wireguard FreeBSD kernel module. If you are familiar with the networking kernel code and would like to help, please reach out to me. I'll get you in touch with Jason. I'm hoping that the HardenedBSD community can fill a gap where the FreeBSD community failed: developing a robust in-kernel Wireguard implementation properly blessed by the Wireguard project. I would be happy to dedicate some HardenedBSD infrastructure resources to help support this effort. Those resources include, but are not limited to: a repo on our self-hosted git server and a VM for nightly builds.

HardenedBSD February 2021 Status Report

This month was a busy one, especially for our infrastructure. We purchased and received three refurbished Dell servers and one Cavium ThunderX server. Two of the three Dell servers were deployed. One is being used for our GitLab migration[1][2]. The second one is being set up as the HardenedBSD 14-CURRENT nightly build and package build server.

With the ThunderX server, we now have a production-capable arm64 server. I'm hoping that we can promote arm64 as a teir 1 architecture by the end of March. Though the server is deployed, I still need to copy the build scripts over and perform is final configuration.

FreeBSD recently landed a W^X implementation (congrats!), but it interferes with our PaX-inspired implementation. In addition to infrastructure work, I've been looking into this. I'm only able to reproduce when building packages--when there's a heavy load on the system. I'm working on some candiate patches, but they're not ready for commit.

The new package build server can be reached at [3]. The Tor onion service address is [4].

The next non-hackery item on my list is to reach out to 2020 donors that donated $250 USD or more to HardenedBSD. I need to give them their tax statement in case they itemize their taxes. I also need to reach out to recent donors to see if they want their name listed on our donations page.

I appreciate the help and support from the community. It's because of you that this project is possible. Your donations enable me to serve you.

[1]: https://git.hardenedbsd.org/
[2]: https://groups.google.com/a/hardenedbsd.org/g/users/c/wZgihY3PR8E/m/gVbZ...
[3]: http://ci-08.md.hardenedbsd.org/
[4]: http://t3xwovqp3hijxp2wskcg2ot5jqpwjd63uaeodsp3ltduhvpxb47aegad.onion/

HardenedBSD December 2020 Status Report

Happy New Year! On this the last day of 2020, I submit December's status report.

FreeBSD recently switched its source (/usr/src) repo from subversion to git. This switch had a major impact on HardenedBSD. This month was spent almost entirely on that impact. I had planned to work on Cross-DSO CFI this month, but I wasn't able to find any time for that.

I spent some time with my Foundation administrative hat on. I sent emails to recent donors, asking them whether they wanted to be listed on the donor's page. If you have donated and haven't received an email from me, please reach out.

This month alone, the community blew through our 2020 donation goal. We received around $13,000 USD in December 2020. We are immensely thankful for those who contribute. Monetary donations go straight to improving the project. In January, we will purchase a new server to replace our existing self-hosted Gitea server.

It has been an honor to serve you, the HardenedBSD Community, in 2020. I'm grateful for the continued opportunity to serve in 2021. Our Roadmap page hasn't been updated in three years now. In January, I will be updating the roadmap.


HardenedBSD November 2020 Status Report

Hey all,

We're getting close to the end of November. My wife and I have plans this weekend, so I thought I'd take the time to write November's status report today.

On amd64, with commit fd35ed2087409c7ae83ed6cf8ec041d47432a758, I added the HARDENEDBSD-NODEBUG kernel. Note that this disables INVARIANTS, which has a lot of security side-benefits due to assertion checking. Disabling INVARIANTS also means enabling PAX_INSECURE_MODE.

I've been working on the ports tree, updating more ports that need extra CFLAGS/LDFLAGS and/or need ports-based hardening features disabled (like PIE, RELRO+BIND_NOW).

I've worked on merging in some code and documentation contributions. We had our first vendor code contribution this month. I still need to review and merge in their patch for the hardening-check port.

Our self-hosted git server is still experiencing issues, but is still getting more stable as time goes on. We've noticed malicious bot scanning activity over our Tor Onion Service endpoints that put a lot of pressure on Gitea, to the point of crashing Gitea. Happens every other week or so.

There's some work that I failed to highlight in October:

  1. Per-thread stacks. We apply both a new delta to mmap(MAP_STACK) and libthr's _thr_stack_alloc(). The delta is is disabled when the hardening.pax.aslr.status syctl node/tunable is set to 1 or 0, or if ASLR is disabled for the process. I need to determine a way to disable the additional randomization added in libthr so that ports like sbcl work again. sbcl requires a deterministic memory layout.
  2. I hardeninged the debug.dump_modinfo flag. Only privileged, unjailed accounts can now access debug.dump_modinfo.

I took a brief look at Cross-DSO CFI again, merging in the latest changes from hardened/current/master into the Cross-DSO CFI feature branch. It seems that FreeBSD's switch to ZFS-on-Linux as the new basis for the upstream repo, the ZFS utilities (/sbin/zpool, /sbin/zfs) and perhaps libraries violate Cross-DSO CFI in some form or fashion. I need to set up a UFS VM and test ZFS regressions there, because it's not possible to boot a userland with Cross-DSO CFI and root-on-ZFS anymore.

In early December, I plan to catch up on HardenedBSD administrative tasks, like reaching out to October and November's donors to see if they want to be listed on the donor's page. (We default to keeping donations anonymous given the security and privacy aspects HardenedBSD promots.)

Cross-DSO CFI will start becoming my main focus again as we go into 2021. Cross-DSO CFI is a hard beast to wrangle when you have to maintain millions of lines of code in the base OS _and_ 33,000+ ports/packages built on top of that. It's a huge effort, and I'm happy to be the guinea pig for llvm's implementation of CFI in a complete OS ecosystem (base OS + third-party programs/packages.)


HardenedBSD October 2020 Status Report

A bit of progress has been made in October. I've started working on randomizing per-thread stacks. I've added a new delta to our ASLR implementation. That delta is applied on all mmap(MAP_STACK) calls. That causes a different delta to be applied than the regular mmap delta, and we're able to introduce a lot more entropy than the delta applied to regular mmap calls. This is only phase one. Phase two will likely be implemented in libthr, to apply a per-thread delta at pthread_create time, thus completing the per-thread stack randomization implementation.

I've fixed a number of ports that had issues with PIE being set. More ports should compile now, as some of these ports that I fixed were dependencies of others.

The infrastructure is becoming more and more stable. I've been tweaking Gitea's config to prevent it from consuming more memory than it can handle on the system. The system Gitea is deployed on has a lot of memory, but its CPU is a bit on the slow side. That slow CPU means Gitea has a tough time crawling through its in-app caches. Disabling caching made Gitea perform better for our particular use caase.

I created the HardenedBSD 12.2-RELEASE branch. I plan to provide base updates for that branch. Due to the way FreeBSD designed the package manager, we'll not be able to provide packages for the 12.2-RELEASE branch since we put the priority on 12-STABLE.

Having the 12.2-RELEASE branch, though, allows projects downstream of us to have a release branch from which to build upon.


HardenedBSD August 2020 Status Report and Call for Donations

This last month has largely been a quiet one. I've restarted work on porting five-year-old work from the Code Pointer Integrity (CPI) project into HardenedBSD. Chiefly, I've started forward-porting the libc and rtld bits from the CPI project and now need to look at llvm compiler/linker enhancements. We need to be able to apply SafeStack to shared objects, not just application binaries. This forward-porting work I'm doing is to support that effort.

The infrastructure has settled and is now churning normally and happily. We're still working out bandwidth issues. We hope to have a new fiber line ran by the end of September.

As part of this status report, I'm issuing a formal call for donations. I'm aiming for $4,000.00 USD for a newer self-hosted Gitea server. I hope to purchase the new server before the end of 2020.

Last year, I migrated us away from GitHub as the source-of-truth for HardenedBSD's source code and ports tree. The server hosting the source is a rather ancient one (a > 10yo Dell R410) with insufficient CPU and RAM. I'm formally calling for donations to go towards a newer server to host our code.

I wanted to self-host our source for a couple reasons:

1. A single source-of-truth under our control that we can monitor and guarantee the security of.
2. Provide unique access to the HardenedBSD ecosystem. We provide Tor Onion Services for those who need it. We plan to deploy other mixnets, anonymization services, and privacy-related tech for reaching HardenedBSD's infrastructure in unique ways.

We use Gitea as our code sharing platform. It has a similar look and feel to GitHub and provides bug reporting, wiki, and pull request features.

Given that HardenedBSD is downstream from FreeBSD, which shares a history greater than twenty five years, this places a huge burden on Gitea. It's having a hard time on our hand-me-down Dell R410 system and desperately needs an upgrade.

We appreciate every type of contribution--whether it be financial, code, documentation, advocacy, or otherwise. HardenedBSD would not exist without the continued help and support from the community.


HardenedBSD July 2020 Status Report

The filesystem extended attribute support work is now completed. As time progresses, more ports will be marked with toggles. I've started documenting which ports are marked with exploit mitigations toggled.

On to a different subject: hosting network stability. Since the migration of the build infrastructure to BlackhawkNests's hosting facility, we've had a number of network-related issues, especially with regards to network stability. The stability issues have hopefully been worked out as of this morning.

We're also in talks with Verizon FiOS and will hopefully switch to FiOS once installation is completed. The switch will also drastically increase available bandwidth.

I'm also working on migrating our gitea instance from using sqlite as the database backend to mysql. Migrating to mysql should drastically increase speed and stability of our self-hosted git service. I will keep you, the HardenedBSD community, updated with my progress.

DEF CON and Jeff Moss have donated funds for a new development laptop for me. I received it last week and am migrating to it. Once the self-hosted git service work is completed, I plan to start completing our SafeStack integration such that SafeStack can be applied to shared objects, opening the door to applying SafeStack to our entire ecosystem (base world + ports).


Deep Integration of Filesystem Extended Attribute Support

I've been working on integrating filesystem extended attribute support in tmpfs, libarchive, and pkg(8).

Other operating systems tag ELF objects with various flags. We in HardenedBSD prefer not to use such a heavy-handed approach. Making use of filesystem extended attributes enables out-of-band (OOB) management of security flags. HardenedBSD makes use of extended attributes to toggle exploit mitigations on a per-binary basis. Using an OOB method provides flexibility along with an easy avenue for future growth.

I've made changes to libarchive in the base OS and have submitted a patch upstream. The patch takes a best-effort approach to restoring system-level extended attributes. Setting system-level extended attributes is a privileged operation. If an archive entry contains a systeam-level extended attribute and the extraction process is not privileged, setting the extended attribute will fail. The failure will be ignored and the extraction process will continue as normal. (The same holds true today without the patch.)

Extended attribute support in tmpfs is a bare minimum, with the ability to add and list, but not remove extended attributes. Anyone desiring to provide complete extended attribute support is welcome to provide a patch.

Finally, HardenedBSD forked FreeBSD's package manager, aptly named pkg. The package manager must be aware of filesystem extended attributes. pkg can now include any filesystem extended attribute and is not limited to HardenedBSD's use. I will make some attempt at upstreaming the changes to pkg after the changes to libarchive have been upstreamed.

The future is bright for filesystem extended attributes. One could imagine a future in which pkg stores the hash of files as extended attributes, and the kernel checks the hash against the stored attribute. The sky is the limit.

I am now integrating exploit mitigation toggling into the ports tree such that HardenedBSD ships packages with exploit mitigations toggled for those misbehaving applications (like firefox, java, nodejs, etc.)


Subscribe to HardenedBSD RSS