HardenedBSD August 2020 Status Report and Call for Donations

This last month has largely been a quiet one. I've restarted work on porting five-year-old work from the Code Pointer Integrity (CPI) project into HardenedBSD. Chiefly, I've started forward-porting the libc and rtld bits from the CPI project and now need to look at llvm compiler/linker enhancements. We need to be able to apply SafeStack to shared objects, not just application binaries. This forward-porting work I'm doing is to support that effort.

The infrastructure has settled and is now churning normally and happily. We're still working out bandwidth issues. We hope to have a new fiber line ran by the end of September.

As part of this status report, I'm issuing a formal call for donations. I'm aiming for $4,000.00 USD for a newer self-hosted Gitea server. I hope to purchase the new server before the end of 2020.

Last year, I migrated us away from GitHub as the source-of-truth for HardenedBSD's source code and ports tree. The server hosting the source is a rather ancient one (a > 10yo Dell R410) with insufficient CPU and RAM. I'm formally calling for donations to go towards a newer server to host our code.

I wanted to self-host our source for a couple reasons:

1. A single source-of-truth under our control that we can monitor and guarantee the security of.
2. Provide unique access to the HardenedBSD ecosystem. We provide Tor Onion Services for those who need it. We plan to deploy other mixnets, anonymization services, and privacy-related tech for reaching HardenedBSD's infrastructure in unique ways.

We use Gitea as our code sharing platform. It has a similar look and feel to GitHub and provides bug reporting, wiki, and pull request features.

Given that HardenedBSD is downstream from FreeBSD, which shares a history greater than twenty five years, this places a huge burden on Gitea. It's having a hard time on our hand-me-down Dell R410 system and desperately needs an upgrade.

We appreciate every type of contribution--whether it be financial, code, documentation, advocacy, or otherwise. HardenedBSD would not exist without the continued help and support from the community.


HardenedBSD July 2020 Status Report

The filesystem extended attribute support work is now completed. As time progresses, more ports will be marked with toggles. I've started documenting which ports are marked with exploit mitigations toggled.

On to a different subject: hosting network stability. Since the migration of the build infrastructure to BlackhawkNests's hosting facility, we've had a number of network-related issues, especially with regards to network stability. The stability issues have hopefully been worked out as of this morning.

We're also in talks with Verizon FiOS and will hopefully switch to FiOS once installation is completed. The switch will also drastically increase available bandwidth.

I'm also working on migrating our gitea instance from using sqlite as the database backend to mysql. Migrating to mysql should drastically increase speed and stability of our self-hosted git service. I will keep you, the HardenedBSD community, updated with my progress.

DEF CON and Jeff Moss have donated funds for a new development laptop for me. I received it last week and am migrating to it. Once the self-hosted git service work is completed, I plan to start completing our SafeStack integration such that SafeStack can be applied to shared objects, opening the door to applying SafeStack to our entire ecosystem (base world + ports).


Deep Integration of Filesystem Extended Attribute Support

I've been working on integrating filesystem extended attribute support in tmpfs, libarchive, and pkg(8).

Other operating systems tag ELF objects with various flags. We in HardenedBSD prefer not to use such a heavy-handed approach. Making use of filesystem extended attributes enables out-of-band (OOB) management of security flags. HardenedBSD makes use of extended attributes to toggle exploit mitigations on a per-binary basis. Using an OOB method provides flexibility along with an easy avenue for future growth.

I've made changes to libarchive in the base OS and have submitted a patch upstream. The patch takes a best-effort approach to restoring system-level extended attributes. Setting system-level extended attributes is a privileged operation. If an archive entry contains a systeam-level extended attribute and the extraction process is not privileged, setting the extended attribute will fail. The failure will be ignored and the extraction process will continue as normal. (The same holds true today without the patch.)

Extended attribute support in tmpfs is a bare minimum, with the ability to add and list, but not remove extended attributes. Anyone desiring to provide complete extended attribute support is welcome to provide a patch.

Finally, HardenedBSD forked FreeBSD's package manager, aptly named pkg. The package manager must be aware of filesystem extended attributes. pkg can now include any filesystem extended attribute and is not limited to HardenedBSD's use. I will make some attempt at upstreaming the changes to pkg after the changes to libarchive have been upstreamed.

The future is bright for filesystem extended attributes. One could imagine a future in which pkg stores the hash of files as extended attributes, and the kernel checks the hash against the stored attribute. The sky is the limit.

I am now integrating exploit mitigation toggling into the ports tree such that HardenedBSD ships packages with exploit mitigations toggled for those misbehaving applications (like firefox, java, nodejs, etc.)

HardenedBSD June 2020 Status Report

Now that HardenedBSD's infrastructure has found its new home, it's time to ramp up development again. We're working out kinks with regards to bandwidth and hope to increase bandwidth to our infrastructure on the inside of two months.

I've started working on adding filesystem extended attributes support to tmpfs. Once support is added, we should be able to integrate with ports/packages such that our users will no longer need to worry about toggling exploit mitigations--they'll already come pre-toggled for misbehaving applications.

I suspect this work will take a few months to complete. I've never done filesystem development, so I'm treading new waters. Once filesystem extended attribute support is added, I plan to integrate exploit mitigation toggling in the ports tree.

When all is said and done, I'm thinking around six months time frame. Granted, I have health issues, so there's no guarantees. I'll keep everyone updated.

My next goal will be integration of SafeStack into the RTLD. This is needed in order to apply SafeStack to both shared libraries and applications. This integration work relates directly with Cross-DSO CFI support, since Cross-DSO CFI requires the same/similar types of integrations.

I'm interviewing a few people to add to the HardenedBSD Board of Directors. We've added Jordan Boland to the team. He will help maintain the infrastructure. I plan to get with him once the bandwidth issues have been resolved.

I've included an intro to Jordan below. We will have more exciting news to share soon with regards to the Board of Directors.

I'm very excited to be getting more involved with HardenedBSD and to have an opportunity to serve on the Board.

I'm a lifelong tinkerer and open-source enthusiast. I was introduced to Linux in middle school and was fascinated with it until I was introduced to FreeBSD while in college. I ran FreeBSD on my personal machines for almost a decade until I was introduced to HardenedBSD, which quickly took over as my OS of choice.

My degree includes a specialization in network administration, and although I love that field I've worked in too many small IT shops to avoid becoming a generalist, and these days I do nothing related to it in my professional life. I've worked in higher education,
healthcare, telecommunications, and (to the complete surprise of my 17-year-old self) have somehow arrived at Microsoft, where I am a support engineer in the research division.

I'm not a programmer, I'm a person who occasionally has a problem that requires writing some code. On that journey I've dabbled in C, C++, C#, Java, Python, Perl, Powershell, and BASH/Bourne Shell (extensively). I really admire those that can write kernel code
and have such a deep understanding of the hardware and what is happening "under the hood", and I'd love to have that kind of proficiency someday. In the meantime, my best contribution to this project will likely be infrastructure-related. Deployment of Kerberos
and LDAP comes to mind, and perhaps digging around inside Gitea to understand why it gives 5xx errors to us. Let me know if you have any questions I didn't cover here. I'm excited to get to work with all of you!


HardenedBSD April 2020 Status Report

Hey HardenedBSD Community,

It has been a while since I've written a status report, and now is definitely time to do so. Over the past few months, I've put my focus on infrastructure stability and merge conflict resolution. The work on exploit mitigations is still somewhat on pause, though I've made slight progress on Cross-DSO CFI.

Our build infrastructure has been hosted at my current employer for a few years now. I'm so grateful for G2, Inc (now Huntington Ingalls Industries) for their support and help in ensuring the continued success of the project.

After over five years of service at my current employer, I've tendered my letter of resignation. The people I've met, the projects I've worked on, and the culture and virtues instilled in me made me fall in love with G2.

I've decided to take a new employment opportunity. BlackhawkNest will host the HardenedBSD build infrastructure with room to grow. I've architected the infrastructure such that the migration should be mostly plug-n-play, only needing to change a few IP addresses.

I plan to shut down the infrastructure in preparation for the migration on 02 May 2020, which is one week in advance of my start date. Builds will resume once the infrastructure has been deployed at the new facility. Note that published builds and package repos will still be accessible. Only the build infrastructure, which is separate from the infrastructure serving the builds and package repos, will be down. I do not currently have a date for when the infrastructure will be back online, but I suspect around two to three weeks from 02 May 2020.

I'm excited for this new opportunity, and especially for BlackhawkNest for agreeing to host the build infrastructure. I'm positive that the relationship between HardenedBSD and BlackhawkNest will be symbiotic.

Thank you so much for your help for and support of the project.


HardenedBSD Tor Onion Service v3 Nodes

I've been working today on deploying Tor Onion Service v3 nodes across our build infrastructure. I'm happy to announce that the public portion of this is now completed. Below you will find various onion service hostnames and their match to our infrastructure.

hardenedbsd.org: lkiw4tmbudbr43hbyhm636sarn73vuow77czzohdbqdpjuq3vdzvenyd.onion
ci-01.nyi.hardenedbsd.org: qspcqclhifj3tcpojsbwoxgwanlo2wakti2ia4wozxjcldkxmw2yj3yd.onion
ci-03.md.hardenedbsd.org: eqvnohly4tjrkpwatdhgptftabpesofirnhz5kq7jzn4zd6ernpvnpqd.onion
ci-04.md.hardenedbsd.org: rfqabq2w65nhdkukeqwf27r7h5xfh53h3uns6n74feeyl7s5fbjxczqd.onion
git-01.md.hardenedbsd.org: dacxzjk3kq5mmepbdd3ai2ifynlzxsnpl2cnkfhridqfywihrfftapid.onion

A GPG-signed version of this post is here: https://groups.google.com/a/hardenedbsd.org/d/msg/users/hmEL0qAE3J8/mLjs...


The Idealistic Future of HardenedBSD

In the last status report, we stood up our own git server. Since then, we've migrated our entire infrastructure to point to our self-hosted git as the source-of-truth repo.

Over the past month, we purchased and deployed the new 13-CURRENT/amd64 package building server. We published our first 13-CURRENT/amd64 production package build using that server. We then rebuilt the old package building server to act as the 12-STABLE/amd64 package building server. This post signifies a very important milestone: we have now fully recovered from last year's death of our infrastructure. Our 12-STABLE/amd64 repo, previously out-of-date by many months, is now fully up-to-date!

We now have four build servers in total:

  1. ci-01.md.hardenedbsd.org: nightly build server for 13-CURRENT/amd64 and 13-CURRENT/arm64.
  2. ci-02.md.hardenedbsd.org: nightly build server for 12-STABLE/amd64.
  3. ci-03.md.hardenedbsd.org: Package building server for 13-CURRENT/amd64.
  4. ci-04.md.hardenedbsd.org: Package building server for 12-STABLE/amd64.

From here, we have two major improvements to make:

  1. Deploy Kerberos + LDAP across our infrastructure. Not only do we have those four servers, but we have others along with a number of jails. Unifying authentication would drastically simplify management.
  2. Set up various Tor Onion Service v3 endpoints for the various parts of our infrastructure. Distribute those Onion Service hostnames to the various stake holders (there will be a notion of public endpoints versus private).

HardenedBSD is in a very unique position to provide innovative solutions to at-risk and underprivileged populations. As such, we are making human rights endeavors a defining area of focus. Our infrastructure will integrate various privacy and anonymity enhancing technologies and techniques to protect lives. Our operating system's security posture will increase, especially with our focus on exploit mitigations.

Navigating the intersection between human rights and information security directly impacts lives. HardenedBSD's 2020 mission and focus is to deliver an entire hardened ecosystem that is unfriendly towards those who would oppress or censor their people. This includes a subtle shift in priorities to match this new mission and focus. While we implement exploit mitigations and further harden the ecosystem, we will seek out opportunities to contribute a tangible and unique impact on human rights issues. Providing Tor Onion Services for our core infrastructure is the first step in likely many to come towards securely helping those in need.

We are grateful for the opportunity to serve. Let us welcome 2020 with a rebuilt infrastructure and a renewed purpose!


Happy Holidays From HardenedBSD

We at HardenedBSD would like to wish the community a happy end to 2019 and a joyful beginning to 2020.

Just today, we finished putting all the pieces in places to migrate away from GitHub. HardenedBSD's build infrastructure is now fully self-hosted. We plan to make the repos on GitHub read-only by the end of January.

We will still maintain a presence on GitHub. The repos will still live on at GitHub. However, they will be a read-only mirror of our self-hosted source-of-truth repository at git-01.md.hardenedbsd.org.

Going forward, please file bug reports and pull requests at our Gitea server: https://git-01.md.hardenedbsd.org/

We will likely make a more appealing subdomain (perhaps git.hardenedbsd.org) later. We'll keep everyone updated if/when we do.

We will update our site accordingly soon. Happy holidays and we hope the community enjoys this little end-of-year gift. :-)

December 2019 Infrastructure Status

I thought I'd take a moment to update the community on where we stand on the infrastructure.

Our infrastructure received its first community contribution over the last week with this completed and deployed pull request: https://github.com/HardenedBSD/build/pull/4

Earlier today, I deployed LetsEncrypt on ci-01.nyi.hardenedbsd.org, our primary mirror.

The last piece of the puzzle is to set up rsync once again such that our mirrors can re-enable syncing with us. I'll probably tackle this in January of 2020, taking a small break from this little bit, especially during the holidays.

If you love infrastructure work and want to contribute, take a look at these open issues: https://github.com/HardenedBSD/build/issues

As always, if you have an itch to scratch, don't wait for me to feel the same itch. Submit a patch to proactively help me scratch your itch. :)

My next major focus will be on package builds.

HardenedBSD Status Report

We at HardenedBSD have a lot of news to share. On 05 Nov 2019, Oliver Pinter resigned amicably from the project. All of us at HardenedBSD owe Oliver our gratitude and appreciation. This humble project, named by Oliver, was born out of his thesis work and the collaboration with Shawn Webb. Oliver created the HardenedBSD repo on GitHub in April 2013. The HardenedBSD Foundation was formed five years later to carry on this great work.

As I rebuild the HardenedBSD build infrastructure, I will be performing the following user-facing changes:

1. The hardenedBSD-STABLE.git repo will be archived off. HardenedBSD will still utilize the hardenedBSD-Playground.git repo for collaboration with third parties and extremely experimental code.
2. We are slowly transitioning to being fully self-hosted. It is my goal to complete the transition on or before 31 Dec 2019. This means we will stop using GitHub altogether.
3. Downgrading 11-STABLE to community support. Given all that's on my plate, I can only maintain 13-CURRENT and 12-STABLE right now. Therefore, if the community wants 11-STABLE support, the community will need to provide it.
4. git commits performed by our infrastructure will be signed by our dev key. Think: our auto-sync scripts that run every six hours.

Now for random bits of other news:

I am currently working on getting the sync scripts running on the new infrastructure. I'm not too far off, but it will likely take around another week to re-enable the auto-sync.

Our amd64 package builder is experiencing stability issues. Due to some upstream network changes, some packages are failing to sync. Our package repos for 13-CURRENT and 12-STABLE are woefully out-of-date. I'm actively working on this as time permits. I have no ETA for updated repos.

Ben La Monica from The HardenedBSD Foundation is looking into LDAP/Kerberos integration for our infrastructure. We're looking to unify everything in order to enable finer-grained control of our infrastructure along with easier centralized management.

The new build scripts are coming along very nicely. One last change I need to make is to skip the build if no commit happened between the last build and the freshly started one. With commit https://github.com/HardenedBSD/build/commit/7aa3f2f3617db85727ac679ddc62..., the build scripts now track the revision of the source tree. This can be used to check whether there have been any updates since the last successful build.

By the end of November, I hope to turn the build scripts into a port/package. It is my goal to be able to `pkg install` our entire infrastructure.

Given the complete rebuild of our infrastructure, we will no longer use the domain installer.hardenedbsd.org. Our primary mirror is now ci-01.nyi.hardenedbsd.org. I will update our website to reflect the changes.

To our mirror operators: due to the complete rebuild of our infrastructure, I have not yet re-enabled rsync on our primary mirror. I will be taking a different approach to authentication than before. I will provide example steps to convert your existing configuration to the new one.

I'm excruciatingly behind with the administrative side of HardenedBSD. If you have donated and I have not reached out to you, please forgive my tardiness. Know that you're not forgotten and I will get to you soon. HardenedBSD, and especially me, appreciate every contribution, no matter the form it comes in (code, money, advocacy, etc.)

Lastly, I'd like to thank everyone for their patience. I know this downtime has been extensive. I'm grateful to have the opportunity to serve the community in my spare time. Thank you for providing me the opportunity to serve you.


Subscribe to HardenedBSD RSS