HardenedBSD August 2022 Status Report

It's that time of the month for the HardenedBSD status report! My own status is pretty darn simple: Little time, no hacks. I hope to be back in the swing of things by the beginning of November. Life is keeping me busy. So I'm ever more grateful for the continued contributions by the HardenedBSD community.

However, Loic and MrUnix fixed a number of issues in both the source and ports repos.

In src:

  1. Loic fixed an issue MrUnix reported about a missing PaX ASLR macro when building a kernel with COMPAT_FREEBSD32 enabled.
  2. Loic updated bsdinstall with a few changes, updating which sysctl nodes to set.
  3. I pulled in a change from OpenBSD that randomizes how often the chacha20-based arc4random(3) reseeds itself.
  4. HardenedBSD user "apache2" enabled multi-console booting by default, enabling use of the serial console by default.

In ports:

  1. Loic disabled PIE for java/eclipse
  2. I disabled SafeStack for x11-servers/xorg-server
  3. Loic added a new port: hardenedbsd/kernel-nodebug
  4. Loic disabled PIE for sysutils/grub2-efi
  5. Loic disabled PIE for net-im/profanity
  6. Loic disabled PIE for astr/xephem
  7. Loic disabled PIE for lang/zig-devel
  8. Loic fixed sysutils/pefs-kmod
  9. Loic fixed textproc/sxml
  10. Loic disabled PIE for sysutils/fluent-bit
  11. Loic disabled PIE for mat/4ti2
  12. Loic disabled PIE for mat/mprime
  13. Loic disabled DTRACE for lang/erlang-runtime25
  14. Loic disabled the PDF option in comms/fl_moxgen
  15. Loic fixed mail/bogofilter
  16. Loic fixed lang/gcc13-devel
  17. Shawn disable variable auto-init for security/tor
  18. MrUNIX disabled the JIT for net-im/signal-desktop
  19. MrUNIX disabled MPROTECT and PAGEEXEC for games/veloren
  20. MrUNIX fixed the build of lang/mono5.10, lang/mono5.20, and lang/mono6.8

For hbsdfw:

hbsdfw, aka the HardenedBSD Firewall, has a new build for this month. As usual, the process for updating is:

  1. Backup your config
  2. Reinstall with the new build
  3. Restore your config

The default username and password have been changed:

Username: root
Password: hbsdfw

You can find the new build at [0].

[0]: https://hardenedbsd.org/~shawn/hbsdfw/hbsdfw_installer_vga_13.1-20220824...

SHA256 (hbsdfw_installer_vga_13.1-20220824-140520.iso.xz) = 0656808643dfaf2ba640c561686da5f861969dadd3ebb9185abfa7c640a6af44

HardenedBSD July 2022 Status Report

This month was a crazy month for me (Shawn Webb). My wife and I adopted a new puppy, so life has been a bit on the exciting side. I'm hoping to get back into the swing of things in the next month or two.

With that said, let's get right into it.

In src:

  1. TPE and RTLD hardening were merged into 13-STABLE. I had posted a HEADS UP email on the users@ mailing list[0]. If you build your own ports/packages, please take note. RTLD hardening can cause issues when building ports/packages.

In ports:

  1. Loic fixed misc/rump
  2. Loic fixed sysutils/bareos18-server
  3. Loic disabled PaX MPROTECT and PAGEEXEC for lang/python39
  4. Loic fixed math/libpgmath
  5. Loic fixed building openjdk8 and openjdk11 for 14-CURRENT
  6. Loic fixed graphics/scrot
  7. Loic fixed devel/objecthash
  8. Loic fixed lang/perl5.36
  9. Loic fixed GCC 12 and 13-devel
  10. Loic fixed net/waypipe
  11. Loic fixed devel/vxlog
  12. Loic fixed www/vdr-plugin-live
  13. Loic fixed comms/telldus-core
  14. Loic fixed graphics/enblend
  15. Shawn enabled MTP support by default for multimedia/vlc
  16. Loic disabled PIE for net/ndpi
  17. Ibrahim Kaikaa (Mr.UNIX) disabled PaX SEGVGUARD for memcheck-amd64-freebsd in devel/valgrind-devel and devel/valgrind
  18. Ibrahim Kaikaa disabled PaX MPROTECT for net-im/signal-desktop
  19. Ibrahim Kaikaa fixed lang/gcc11

For hbsdfw (the HardenedBSD 13-STABLE fork of OPNsense):

Today (30 Jul 2022), I published a new build[1]. It migrates us to PHP 8.0 and Python 3.9. It appears that the PHP 8.0 Radius extension (php80-pecl-radius) has issues, so I removed the package from the build. So if you're testing hbsdfw out and rely on Radius authentication, you'll want to skip this build.

I haven't had the time to fully bring up the infrastructure needed for in-place updates for hbsdfw, so the normal process of backing up the running config, reinstalling with the new build, and restoring the config is needed for this build and at least the following next few builds.

Please test the build out and let me know how it goes for you. Any message, whether it's "works fine for me" or "hey, we got a problem" helps me determine follow-up tasks for this fork.

The default username is "root" and the password is "dynfi". (The reason for the password being "dynfi" is because we use a forked version of the dynfi build scripts, which pull in the default dynfi opnsense config.)

SHA256 (hbsdfw_installer_vga_13.1-20220729-224841.iso.xz) =
99876a3ba436a274564f4ce51f83b71f901559d8e49926a18c438b483e3d288c

[0]: https://groups.google.com/a/hardenedbsd.org/g/users/c/u6HcO415_OE/m/8g2N...
[1]: https://hardenedbsd.org/~shawn/hbsdfw/hbsdfw_installer_vga_13.1-20220729...

HardenedBSD June 2022 Status Report

June saw some cool security enhancements to HardenedBSD. So let's kick off our usual list:

In src, 14-CURRENT:

  1. The HardenedBSD amd64 kernel configs have been unified to be based off of HARDENEDBSD-CORE.
  2. OpenSSH's ssh-sk-helper program violates the cfi-icall scheme. Until I get time to dive in (or if someone beats me to it), I've disabled the cfi-icall scheme for that program. Users can now use the integrated FIDO2/U2F key support in OpenSSH.
  3. Our Trusted Path Execution feature from secadm now exists in base. There are some differences, which I will document in our wiki soon. TPE violations are logged. One major thing left to do is integrate with mmap(fd, PROT_EXEC). This would also prevent a PaX NOEXEC bypass by virtue of creating a file with an executable payload, mapping it in memory, and executing it.
  4. The RTLD has been significantly hardened. This has the potential to cause issues, especially when building ports/packages. A new sysctl node (hardening.harden_rtld) has been added and is defaulted to 1 (enabled).

I plan to MFC all of the above to 13-STABLE soon. If you build your own packages or ports, please take special note of item four above. Here's a few more details on how we've hardened the RTLD (when hardening.harden_rtld is set to 1):

  1. LD_PRELOAD is fully prohibited.
  2. Set dangerous_ld_env, which isn't used much in the RTLD, but could be used more in the future.
  3. Sensitive LD_* environment variables are scrubbed.
  4. Using the RTLD to execute applications is prohibited.
  5. Tracing of loaded objects is prohibited. This change in particular breaks ldd(1), which is used by a lot of ports during the build process. This is what can cause the most headaches.

In ports:

  1. SafeStack and CFI are disabled if PKGNAMESUFFIX ends with -static.
  2. PaX PAGEEXEC is disabled for sysutils/syslog-ng
  3. New port added: sysutils/pc-sysinstall
  4. SMB support was added to multimedia/ffmpeg
  5. PaX MPROTECT is disabled for emulators/wine
  6. PaX MPROTECT is disabled for emulators/wine-proton
  7. PaX MPROTECT is disabled for net-im/nheko
  8. PaX MPROTECT is disabled for net-im/quaternion
  9. PaX MPROTECT is disabled for www/node16

Other projects:

  1. Work is now officially underway to provide the HardenedBSD community with a HardenedBSD 13-STABLE based fork of OPNsense. We're really close to providing a proof-of-concept build--likely before the end of July 2022. We will provide periodic (montly? bi-weekly?) updates. If you'd like to follow along, the repos are here.
  2. The old 12-STABLE package building server will be used to perform periodic automated builds of Loic's LiveCD project, making it an official HardenedBSD project. This server will also build the HardenedBSD+OPNsense builds. Time frame for completing this will likely be in August 2022.

HardenedBSD May 2022 Status Report

In May 2022, HardenedBSD saw a few changes.

In src:

  1. chroot(2) is now prohibited when a directory file descriptor is opened.
  2. The HARDENEDBSD-NODEBUG kernel configuration was updated to remove a few more debugging-related options.
  3. Loic merged a lot of updates to 13-STABLE, especially regarding hbsd-update.

In ports:

  1. LTO is now disabled for the firefox port.
  2. The virtualbox ports were fixed by Loic.
  3. net/opennx port is fixed.
  4. Loic fixed devel/mingw32-gcc
  5. Loic fixed devel/bmake

Other projects:

Recent changes in FreeBSD caused breakages with secadm. Shawn fixed secadm by complying to those ABI/API-breaking changes made by FreeBSD.

HardenedBSD April 2022 Status Report

In src:

  1. Shawn introduced the notion of an "insecure/untrustworthy" kernel module. Certain kernel modules, like this linux syscall translation layer commonly called the "linuxulator", may create interesting attack vectrors. Some modules are old and likely contain vulnerabilities (old: smbfs, vulnerable: fusefs.) By default, HardenedBSD prevents loading these kernel modules post-boot (eg, via rc.conf(5)'s `kld_list`). The list of kernel modules currently tagged as "insecure" is below at the end of this status report.
  2. Loic hardened the default sshd_config. Please reference commit b7961aade549f05f62d65b0906db495b9423c940 for more information. The changes that might carry the most impact are:
    • MaxSessions 5
    • AllowTcpForwarding no
    • AllowAgentForwarding no

In ports:

  1. Shawn fixed the harfbuzz bug that plagued devel/doxygen (via pango). Though the errant code was indeed in pango, the harfbuzz project did not do a thorough job at ensuring the sanity of arguments passed in to one of its provided APIs (a NULL dereference bug in harfbuzz, manifest by errant code in pango.)
  2. Loic fixed a compiler error in the wine ports.
  3. Loic fixed the virtualbox-ose-* ports.

Other projects or items of note:

  1. The HardenedBSD Foundation's Ben Welch has been working on a new static site for us, migrating us away from Drupal. There's a few things to wrap up, but I suspect on the inside of three months, the HardenedBSD website will look quite a bit different from what it looks like today.
  2. I (Shawn) am quite far behind on the administrative side of the HardenedBSD project. I need to do the financials and other administrative things. I apologize for the delays on the various administrative tasks.

Kernel modules currently marked as insecure:

  1. smbfs
  2. accf_http
  3. accf_dns
  4. linux_common
  5. linux/linux64
  6. lindebugfs (NOTE: this impacts drm-*-kmod KMS drivers)
  7. fusefs

As of this writing, HardenedBSD 14-CURRENT (both amd64 and arm64) users can overwrite these insecure markings by using hbsdcontrol:


# hbsdcontrol pax disable insecure_kmod /path/to/kernel/module

This is especially useful for drm-*-kmod users. I plan to MFC the hbsdcontrol integration commit mid-to-late next week (so somewhere between 05 May and 08 May 2022) after more thorough testing on my HardenedBSD laptops.

Please note that April 2022 concludes official support for the 12-STABLE branch. Effective 01 May 2022, support for the 12-STABLE branch must come from the wider HardenedBSD community. On 31 Dec 2022, the package repo and all build artifacts pertaining to 12-STABLE will be fully removed.

HardenedBSD March 2022 Status Report

We made some cool progress in March. Though I, Shawn, am writing this report, I'll refer to myself in the third person for the rest of the report. :-)

In src:

  1. Shawn reverted a potential LPE race condition in ZFS introduced by OpenZFS. Only 14-CURRENT was affected.
  2. Shawn merged in zlib's CVE-2018-25032 fix.
  3. Shawn enabled -ftrivial-auto-var-init=zero in base clang by default. This now means we don't need to pass in any compiler flags to support this feature. All ports that rely on the system compiler will now automatically benefit.
  4. FreeBSD introduced a change that enables dumpon to use the underlying non-encrypted swap device when encrypted swap is used. Shawn reverted this change since users who choose to encrypt their swap encrypt it for a reason--HardenedBSD should proactively protect users by respecting their encryption preferences.
  5. Loic found and fixed an issue with the linuxulator in HardenedBSD, with the default stack permissions.
  6. Coming up soon: sponsored work by BlackhawkNest, Inc that provides support in base for (more) easily building HardenedBSD 13-STABLE based versions of OPNsense. This is in the hopes to provide the wider community with the ability to produce their own builds. Some of this work has landed in a special feature branch.

In ports:

  1. In tandem with src change #3, Shawn modified the ports tree to rely on the system compiler's application of -ftrivial-auto-var-init=zero. There's no need to apply that feature via CFLAGS injection.
  2. Loic removed `stackautoinit:off` USE_HARDENING flag from a very large number of ports. This was a huge lift and his work on this is very much appreciated. He and Shawn worked a lot on this.
  3. Ibrahim Kaikaa (Mr.UNIX) has helped fix a number of ports. We still have a number of outstanding merge requests that I need to verify.

Infrastructure:

  1. The HardenedBSD GitLab server had a drive failure. I had already planned to rebuild the pool from a bunch of older 1TB spinning rust drives to a bunch of 2TB SSDs. The drive failure accellerated the pool rebuild, which completed successfully.
  2. After a large number of months of downtime, our arm64 package building server has come back online! We're now building 14-CURRENT/arm64 packages.
  3. After src change #3 landed, all of the build infrastructure servers were updated.

Cool projects:

  1. Loic released an unofficial livecd of HardenedBSD that boots into XFCE[0]. I've started the discussion with him to convert that from from an "unofficial" project to an "official" one. :-)

Special notes:

  1. As a reminder, support for HardenedBSD 12-STABLE will be delegated to the community. As such, binary updates and package builds will cease. The hardened/12-stable/master branch will no longer be auto-synced.
  2. Please remember to let us know if you have any thoughts to share on whether HardenedBSD should support the linuxulator by 15 Apr 2022.
  3. Please remember to let us know if you have any thoughts to share on the proposed changes to the default sshd configuration[1].

[0]: https://groups.google.com/a/hardenedbsd.org/g/users/c/QUTUJfm30Dg/m/0VNK...
[1]: https://git.hardenedbsd.org/hardenedbsd/HardenedBSD/-/merge_requests/60

HardenedBSD January 2022 Status Report

The first month of the year started strong. I'm going to get right into it.

In base:

  1. FreeBSD landed their Virtual Dynamic Shared Object (VDSO) late last year. I brought those changes in to 14-CURRENT (and subsequently 13-STABLE). I ensured that the VDSO's userland mapping was still randomized. As it stands today, the VDSO is active on amd64, but not on arm64. I tested to make sure that both scenarios still work by testing on the development ThunderX1.
  2. In the back port of the VDSO work to 13-STABLE, I introduced a single-character typo. I chased that down, squashing the bug.
  3. FreeBSD's pkgbase work has advanced quite a bit. Our pkgbase integration didn't follow those advancements, so I belatedly fixed that. This was needed for work sponsored by BlackhawkNest, Inc.

  4. Loic made it so that the kernel's config file isn't embedded in the kernel.
  5. I made it so that we apply a separate delta for the RTLD upon image activation time. Previously, we would apply the same delta we use for regular mmap calls. Re-using the mmap delta placed the RTLD right next to the other dependent shared objects. As the RTLD contains high-value targets, applying a separate delta helps protect those targets.
  6. I identified a problem with the OpenSSH import in base whereby the in-app sftpd service would crash when a Windows client attempted to connect. I resolved that issue by fixing a kernel check on the process control KPI.
  7. FreeBSD introduced a new `security.bsd.allow_ptrace` sysctl node and its corresponding integration code. When PAX_HARDENING is set in the kernel (it is set by default), the sysctl node will default to 0.
  8. CVE-2021-4034 came out. I took inspiration from OpenBSD's general mitigation of checking that (argc == 0) in the kernel before allowing execve(2) to complete. The implementation of the argc check differs between FreeBSD and HardenedBSD. FreeBSD's implementation occurs much sooner in the code path, whereas my implementation occurs after the sysentvec-specific sv_copyout_strings function pointer is called. Should there be a weird sv_copyout_strings implementation, our argc check will be more resilient than the FreeBSD check.
  9. The old Berkely DataBase (bdb) implementation requires downstream consumers to reference function pointers in a contextual structure returned (and filled out) by dbopen(3). These function pointers point to uninstrumented functions in libc. Thus, when the cfi-icall scheme is applied, applications attempting to call those function pointers will be killed with SIGILL by CFI.
  10. I spent around twenty hours this month on Cross-DSO CFI, the longest amount of time spent in a while. I've identified some chicken-and-egg scenarios, especially revolving around the dlopen dance in libc and the rtld. The Cross-DSO CFI runtime intercepts calls to dlopen in order to add the shared object's memory to the CFI allow list.

In ports:

  1. Loic hardened the multimedia/ffmpeg port
  2. I fixed the misc/seabios port.
  3. Loic removed an unneeded patch in the security/osiris port.
  4. I disabled the DTRACE option for a few ports. DTrace support is broken when using a more complete llvm compiler toolchain. I need to file a bug report upstream in llvm to address this, but haven't due to lack of time.
  5. I fixed the openjdk ports.
  6. Loic fixed the devel/aarch64-none-elf-gcc ports.
  7. I disabled PaX MPROTECT for the mongodb ports.

Infrastructure work:

All of the HardenedBSD build infrastructure is graciously hosted by my employer, BlackhawkNest, Inc. Due to COVID-19, we're still working remotely 99.9% of the time, myself included. The 27th of this month was the first time in around three months since I had been in the office.

With my focus being more on the development side, I've not taken the time to set up a proper network and system monitoring solution. Some servers are experiencing hardware failures (mostly just dead drives).

Starting 02 Feb 2022, I plan to go up to the office at least twice per week. I've ordered replacement drives and will install them as soon as they arrive. The server hit hardest is the 12-STABLE nightly build server, which is at the point where I will need to rebuild it with new drives, performing a fresh reinstall and reconfiguration.

I plan to divert some of my development time towards infrastructure maintenance in February, setting up that infrastructure monitoring solution. I'd like to get us to the point where we can be anticipatory with our infrastructure's needs rather than reactionary.

Conclusion:

HardenedBSD has had a very busy and productive January. We've made several substantial improvements. This project would not be possible without the generous and greatly appreciated contributions of the community. The HardenedBSD team and I are grateful for the opportunity to serve you.

HardenedBSD December 2021 Status Report

It has been a busy December! I worked on 14-CURRENT/arm64 support. HardenedBSD now builds (nearly) all of world (both libraries and applications) with Link-Time Optimization (LTO). We now have two ThunderX1 systems. Over the past few months, FreeBSD introduced one or more commits that is causing the ThunderX1 system we use for package builds to fail to boot. Oddly, the other ThunderX1 server (used for arm64 development, rather than package builds) boots just fine. I've been working on tracking down which commit(s) are the culprit, but doing so takes time.

FreeBSD also landed a proper VDSO implementation. However, the implementation lacks ASLR support. Due to scheduling issues, I've reverted the VDSO-related commits until I have a solid weekend to hack on it, applying our PaX-inspired ASLR implementation to it. I hope to have that time mid-January or early February.

I narrowed down a few more issues in ports related to our switch to a more complete LLVM compiler toolchain. There are still a large number of ports to fix, which stands as a testament that the development community relies heavily on a GNU-based toolchain. Ideally, projects shouldn't care what toolchain is being used.

Loic landed ClonOS support in the HardenedBSD ports tree. He also helped address more LLVM toolchain fallout. I have a number of merge requests to review from him. Keep up the good work, Loic!

On Sunday (26 Dec 2021), I plan to work on HardenedBSD financials. I'm a bit late in sending out the typical "would you like to be listed on our donor's page" emails. I hope to also work on a 2022 project roadmap.

HardenedBSD has had a lot of help in 2021. The community's contributions have directly improved HardenedBSD. We received a number of server donations, which enables us to build packages quicker and more reliably. We were able to expand our arm64 support. All donations have gone to support either hardware or the few monthly expenses we have. I am grateful for any contribution, no matter the form it comes in--whether that's advocacy, patch submissions, monetary donations, hardware donations, etc. Your generosity enables the success of this project.

HardenedBSD November 2021 Status Report

November saw a number of improvements to HardenedBSD. Loic fixed a bunch of old cruft in base. Among the changes from Loic:

  1. Remove Oliver Pinter's old kernel config
  2. Clean up line breaks
  3. Fix the motd generation code to use HardenedBSD's motd template
  4. Bug fixes in hbsd-update
  5. Use HTTPS with hbsd-update (possible now that FreeBSD distributs trusted CA root certificates.)

I need to MFC a bunch of his work to 13-STABLE and 12-STABLE where applicable.

The HardenedBSD Foundation's Ben La Monica has been stellar at keeping our self-hosted GitLab up-to-date and making sure that runs smoothly.

FreeBSD updated llvm in 14-CURRENT base to llvm 13. I've been working on addressing the fallout from that. Note that though there is fallout, it's the good kind: the llvm compiler toolchain is progressing and finding buggy code. The problem comes when you build 30,000+ packages. ;-)

Speaking of building packages, the 14-CURRENT/amd64 package build server experienced a catastrophic failure. Just today (30 Nov 2021), I went into the datacenter to rebuild the server. 14-CURRENT/amd64 packages will lag behind for a little bit while I transfer backed up config files and the like and kick off a new build.

The 14-CURRENT/arm64 package builder is also in a paused state. I'm working on bisecting one or more commits from FreeBSD that trigger a kernel panic on the ThunderX1.

We also purchased and received another ThunderX1. This new TX1 will be used for development purposes (for example: porting SafeStack to HardenedBSD/arm64). The TX1 referenced in the paragraph above is solely for package builds. The git bisect is being performed on this second TX1. I've yet to find the offending commit(s), but hope to by the end of this coming weekend.

To better facilitate expansion and development efforts, I have installed a new 25U rack at home, which is where the second TX1 currently lives. My employer (BlackhawkNest, Inc, who graciously hosts the HardenedBSD build infrastructure) recently installed a third rack. We have a few servers to deploy into it, which will likely happen mid-December.

I'd like to take a moment to thank the wider HardenedBSD community. Your help and support is not only crucial to the project, but immensely appreciated. Contributions come in all forms, some of which are advocacy, patch submissions, monetary donations, and community support. Every contribution, no matter the form, helps the project grow. Especially as we enter the last month of the year, we are incredibly grateful for your continued support.

If you have an itch to scratch, please do! We review all patches for the project that come our way, regardless of whether they're security-related or not.

As a reminder, for those who create new accounts on our self-hosted GitLab, please email netops{AT}hardenedbsd{DOT}org for account activation.

Call For Testing: Removing and hardening sensitive files

I've merged into a feature branch (hardened/current/sensitive) a merge request from Loic that hardens file and directory permissions for a handful of files/directories.

I plan to let the feature branch soak for around two months, giving the HardenedBSD community time to test the changes prior to them landing. Of course, should issues arise, we'll take care of them.

I've enabled binary updates for that feature branch and I've configured the auto-sync application to sync the branch along with all the other branches every six hours.

The feature branch will share the same package repo as hardened/current/master (aka, HardenedBSD 14-CURRENT). If you track 14-CURRENT, please help test the hardened/current/sensitive branch.

Attached to this email are the two hbsd-update config files: one for normal hbsd-update users, and another for Tor users.

Pages

Subscribe to HardenedBSD RSS