Hey all,

We're getting close to the end of November. My wife and I have plans this weekend, so I thought I'd take the time to write November's status report today.

On amd64, with commit fd35ed2087409c7ae83ed6cf8ec041d47432a758, I added the HARDENEDBSD-NODEBUG kernel. Note that this disables INVARIANTS, which has a lot of security side-benefits due to assertion checking. Disabling INVARIANTS also means enabling PAX_INSECURE_MODE.

I've been working on the ports tree, updating more ports that need extra CFLAGS/LDFLAGS and/or need ports-based hardening features disabled (like PIE, RELRO+BIND_NOW).

I've worked on merging in some code and documentation contributions. We had our first vendor code contribution this month. I still need to review and merge in their patch for the hardening-check port.

Our self-hosted git server is still experiencing issues, but is still getting more stable as time goes on. We've noticed malicious bot scanning activity over our Tor Onion Service endpoints that put a lot of pressure on Gitea, to the point of crashing Gitea. Happens every other week or so.

There's some work that I failed to highlight in October:

1. Per-thread stacks. We apply both a new delta to mmap(MAP_STACK) and libthr's _thr_stack_alloc(). The delta is is disabled when the hardening.pax.aslr.status syctl node/tunable is set to 1 or 0, or if ASLR is disabled for the process. I need to determine a way to disable the additional randomization added in libthr so that ports like sbcl work again. sbcl requires a deterministic memory layout.
2. I hardeninged the debug.dump_modinfo flag. Only privileged, unjailed accounts can now access debug.dump_modinfo.

I took a brief look at Cross-DSO CFI again, merging in the latest changes from hardened/current/master into the Cross-DSO CFI feature branch. It seems that FreeBSD's switch to ZFS-on-Linux as the new basis for the upstream repo, the ZFS utilities (/sbin/zpool, /sbin/zfs) and perhaps libraries violate Cross-DSO CFI in some form or fashion. I need to set up a UFS VM and test ZFS regressions there, because it's not possible to boot a userland with Cross-DSO CFI and root-on-ZFS anymore.

In early December, I plan to catch up on HardenedBSD administrative tasks, like reaching out to October and November's donors to see if they want to be listed on the donor's page. (We default to keeping donations anonymous given the security and privacy aspects HardenedBSD promots.)

Cross-DSO CFI will start becoming my main focus again as we go into 2021. Cross-DSO CFI is a hard beast to wrangle when you have to maintain millions of lines of code in the base OS _and_ 33,000+ ports/packages built on top of that. It's a huge effort, and I'm happy to be the guinea pig for llvm's implementation of CFI in a complete OS ecosystem (base OS + third-party programs/packages.)

This last month has largely been a quiet one. I've restarted work on porting five-year-old work from the Code Pointer Integrity (CPI) project into HardenedBSD. Chiefly, I've started forward-porting the libc and rtld bits from the CPI project and now need to look at llvm compiler/linker enhancements. We need to be able to apply SafeStack to shared objects, not just application binaries. This forward-porting work I'm doing is to support that effort.

The infrastructure has settled and is now churning normally and happily. We're still working out bandwidth issues. We hope to have a new fiber line ran by the end of September.

As part of this status report, I'm issuing a formal call for donations. I'm aiming for $4,000.00 USD for a newer self-hosted Gitea server. I hope to purchase the new server before the end of 2020.

Last year, I migrated us away from GitHub as the source-of-truth for HardenedBSD's source code and ports tree. The server hosting the source is a rather ancient one (a > 10yo Dell R410) with insufficient CPU and RAM. I'm formally calling for donations to go towards a newer server to host our code.

I wanted to self-host our source for a couple reasons:

1. A single source-of-truth under our control that we can monitor and guarantee the security of.
2. Provide unique access to the HardenedBSD ecosystem. We provide Tor Onion Services for those who need it. We plan to deploy other mixnets, anonymization services, and privacy-related tech for reaching HardenedBSD's infrastructure in unique ways.

We use Gitea as our code sharing platform. It has a similar look and feel to GitHub and provides bug reporting, wiki, and pull request features.

Given that HardenedBSD is downstream from FreeBSD, which shares a history greater than twenty five years, this places a huge burden on Gitea. It's having a hard time on our hand-me-down Dell R410 system and desperately needs an upgrade.

We appreciate every type of contribution--whether it be financial, code, documentation, advocacy, or otherwise. HardenedBSD would not exist without the continued help and support from the community.

I've been working on integrating filesystem extended attribute support in tmpfs, libarchive, and pkg(8).

Other operating systems tag ELF objects with various flags. We in HardenedBSD prefer not to use such a heavy-handed approach. Making use of filesystem extended attributes enables out-of-band (OOB) management of security flags. HardenedBSD makes use of extended attributes to toggle exploit mitigations on a per-binary basis. Using an OOB method provides flexibility along with an easy avenue for future growth.

I've made changes to libarchive in the base OS and have submitted a patch upstream. The patch takes a best-effort approach to restoring system-level extended attributes. Setting system-level extended attributes is a privileged operation. If an archive entry contains a systeam-level extended attribute and the extraction process is not privileged, setting the extended attribute will fail. The failure will be ignored and the extraction process will continue as normal. (The same holds true today without the patch.)

Extended attribute support in tmpfs is a bare minimum, with the ability to add and list, but not remove extended attributes. Anyone desiring to provide complete extended attribute support is welcome to provide a patch.

Finally, HardenedBSD forked FreeBSD's package manager, aptly named pkg. The package manager must be aware of filesystem extended attributes. pkg can now include any filesystem extended attribute and is not limited to HardenedBSD's use. I will make some attempt at upstreaming the changes to pkg after the changes to libarchive have been upstreamed.

The future is bright for filesystem extended attributes. One could imagine a future in which pkg stores the hash of files as extended attributes, and the kernel checks the hash against the stored attribute. The sky is the limit.

I am now integrating exploit mitigation toggling into the ports tree such that HardenedBSD ships packages with exploit mitigations toggled for those misbehaving applications (like firefox, java, nodejs, etc.)

Hey HardenedBSD Community,

It has been a while since I've written a status report, and now is definitely time to do so. Over the past few months, I've put my focus on infrastructure stability and merge conflict resolution. The work on exploit mitigations is still somewhat on pause, though I've made slight progress on Cross-DSO CFI.

Our build infrastructure has been hosted at my current employer for a few years now. I'm so grateful for G2, Inc (now Huntington Ingalls Industries) for their support and help in ensuring the continued success of the project.

After over five years of service at my current employer, I've tendered my letter of resignation. The people I've met, the projects I've worked on, and the culture and virtues instilled in me made me fall in love with G2.

I've decided to take a new employment opportunity. BlackhawkNest will host the HardenedBSD build infrastructure with room to grow. I've architected the infrastructure such that the migration should be mostly plug-n-play, only needing to change a few IP addresses.

I plan to shut down the infrastructure in preparation for the migration on 02 May 2020, which is one week in advance of my start date. Builds will resume once the infrastructure has been deployed at the new facility. Note that published builds and package repos will still be accessible. Only the build infrastructure, which is separate from the infrastructure serving the builds and package repos, will be down. I do not currently have a date for when the infrastructure will be back online, but I suspect around two to three weeks from 02 May 2020.

I'm excited for this new opportunity, and especially for BlackhawkNest for agreeing to host the build infrastructure. I'm positive that the relationship between HardenedBSD and BlackhawkNest will be symbiotic.

Thank you so much for your help for and support of the project.

In the last status report, we stood up our own git server. Since then, we've migrated our entire infrastructure to point to our self-hosted git as the source-of-truth repo.

Over the past month, we purchased and deployed the new 13-CURRENT/amd64 package building server. We published our first 13-CURRENT/amd64 production package build using that server. We then rebuilt the old package building server to act as the 12-STABLE/amd64 package building server. This post signifies a very important milestone: we have now fully recovered from last year's death of our infrastructure. Our 12-STABLE/amd64 repo, previously out-of-date by many months, is now fully up-to-date!

We now have four build servers in total:

1. ci-01.md.hardenedbsd.org: nightly build server for 13-CURRENT/amd64 and 13-CURRENT/arm64.
2. ci-02.md.hardenedbsd.org: nightly build server for 12-STABLE/amd64.
3. ci-03.md.hardenedbsd.org: Package building server for 13-CURRENT/amd64.
4. ci-04.md.hardenedbsd.org: Package building server for 12-STABLE/amd64.

From here, we have two major improvements to make:

1. Deploy Kerberos + LDAP across our infrastructure. Not only do we have those four servers, but we have others along with a number of jails. Unifying authentication would drastically simplify management.
2. Set up various Tor Onion Service v3 endpoints for the various parts of our infrastructure. Distribute those Onion Service hostnames to the various stake holders (there will be a notion of public endpoints versus private).

HardenedBSD is in a very unique position to provide innovative solutions to at-risk and underprivileged populations. As such, we are making human rights endeavors a defining area of focus. Our infrastructure will integrate various privacy and anonymity enhancing technologies and techniques to protect lives. Our operating system's security posture will increase, especially with our focus on exploit mitigations.

Navigating the intersection between human rights and information security directly impacts lives. HardenedBSD's 2020 mission and focus is to deliver an entire hardened ecosystem that is unfriendly towards those who would oppress or censor their people. This includes a subtle shift in priorities to match this new mission and focus. While we implement exploit mitigations and further harden the ecosystem, we will seek out opportunities to contribute a tangible and unique impact on human rights issues. Providing Tor Onion Services for our core infrastructure is the first step in likely many to come towards securely helping those in need.

We are grateful for the opportunity to serve. Let us welcome 2020 with a rebuilt infrastructure and a renewed purpose!