For the longest time I used the built-in VNC support to connect to machines because the vm console command would use /usr/bin/cu which would inevitably trap my console and no amount of ~>~>~><~D~>D<S~>D<~><D~<L would help me exit.

Somewhere along the line tmux support was added to vm-bhyve and now vm console <name> simply opens up a new tmux window!

I host everything under /vm on the machine, so in /vm/.config/system.conf:

console="tmux"

This seems like a simple thing to be excited about, and it is, but it makes VMs wildly more accessible and useful for me.

I don’t have a very clear idea what caused the baseline increase in fan speed, but the updates I mentioned previously did include adding a PCI-e card to an otherwise PCI-e card-less machine. I would guess that something in the IPMI/BMC resulted in the power profiles changing when the riser cards in the chassis started to utilize power, indicating new peripherals that would benefit from increased airflow.

Another homelab operator on Reddit found the magic bytes to poke into the IPMI with ipmitool that would manipulate the fan control for similar systems, in essence:

# ipmitool raw 0x3a 0x07 {fan_id} {speed} {fan_override}

My first attempt on this FreeBSD machine kicked back an error:

Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory

Oops! I need the kernel module, which can be loaded at runtime with kldload ipmi, or at boot time with impi_load="YES".

In my system’s case, the two fan IDs and settings that I ended up configuring were:

# ipmitool raw 0x3a 0x07 0x01 0x20 0x01
# ipmitool raw 0x3a 0x07 0x02 0x20 0x01

That was enough to knock the speed down to a nice quiet whirr rather than a medium-loud whoosh! Should you find yourself in a similar position, I recommend following nixCraft’s article to monitor the CPU temperatures as you fiddle with the appropriate balance of noise to cooling power!

The FreeBSD machine doesn’t have any use for the PCI-e device but I might be able to make use of the device from a bhyve virtual machine. bhyve supports pci passthrough but I have not actually used it in my many years of FreeBSD virtualization.

Executing vm passthru lists out all the devices and whether they’re ready for use, and suffice it to say even after my first crack at things, my PCI device was not ready!

The device must be “hidden” from the host in order for guests to acquire/use the device which means a pptdevs entry in /boot/loader.conf. The configuration syntax has an antique feel to it, so I re-read my loader.conf numerous times, tried a couple reboots, and then went off searching the internet.

I spotted a one-off comment on a semi-related GitHub issue stating:

Make sure vmm_load="YES" is in the loader.conf

I have been running bhyve for a while, with vm_enable="YES" and assorted settings in /etc/rc.conf…but I didn’t have anything associated in the loader.conf! For those unfamiliar, loader.conf is like your grub.conf which has the kernel configuration and early boot time parameters, whereas rc.conf is utilized at a much later stage of the boot process.

I posit that my pptdevs entry in loader.conf was not working because I was relying on rc.conf to dynamically load the necessary bhyve kernel modules, but by that point in the boot process it is just too late to configure PCI passthrough devices!

A deep breath and one more reboot, and tada! The machine now is passing a PCI-e card up to a Linux virtual machine which sees it as a native device!

There’s no telling what kind of mischief this will lead to!

The thing about “server grade hardware” is that it never really is. The consumer grade stuff is junk, the small-medium business grade stuff is junk. It’s all junk. Servers tend to have more redundancies to compensate, but at the end of the day: junk.

Anyways, the seller had trouble getting the server to recognize more than one stick of RAM. I carefully unseated and reseated the RAM… and it just worked!

Onto the next challenge, sometimes the machine wouldn’t reboot. Somehow a cold boot, versus a “reboot” didn’t work properly. I found a critical problem in the IMM which could have resulted in me bricking the board and upgraded the out-of-date firmware, dreading any potential triggering of the bug.

I then fought against the RAID card, which really means I fought against the Serial-Attached-SCSI disks. Did you know that Serial-Attached-SCSI has a cable that’s functionally compatible with SATA? Did you also know that SAS disks are obscenely expensive, while SATA SSDs are plentiful? The difference it turns out is largely in software and many SAS supporting devices also support SATA!

I wanted to boot directly into FreeBSD via ZFS, with a nice array of SATA SSDs , but try as I might I could not get FreeBSD to boot consistently. Inevitiably mfi(4) errors would appear and the system would become unusable. There were so many goblins in this machine, I continued searching forums, StackOverflow, and random mailing list posts. Until I finally met Mr. Sas.

From the mrsas(4) man page:

Using /boot/device.hints (as mentioned below), the user can provide a preference for the mrsas driver to detect a MR-Fusion card instead of the mfi(4) driver.

The mfi(4) driver and my hardware simply would not cooperate, so I updated /boot/loader.conf with:

hw.mfi.mrsas_enable="1"

I crossed my fingers, blew the petals off a dandelion, clutched a lucky charm and rebooted the machine..

…

…

As the machine POSTed and booted into FreeBSD I quietly waited for something else to go wrong.

But it didn’t.

…

How about that? It just worked.

Mr. Sas saved the day!

I have probably spent $1000 on upgrades for the machine, which was retired from some guy’s garage for $100. In turn I’ve gotten far more benefit from this goofy steel slap resting at the bottom of the rack.

The machine has since had its disk capacity maxed out, it’s been upgraded to 136GB of RAM. It has become the central workhorse of my homelab: storing backups, hosting a dozen jails, and even a few VMs!

None of which would have been possible without the friendly help of mrsas(4).

watermelon# service jail start gitea
Starting jails: cannot start jail  "gitea":
ifconfig: interface epair14 already exists
jail: gitea: ifconfig epair14 create up: failed
.
watermelon# service jail stop gitea
Stopping jails:.

What perplexed me about this issue is that I would run ifconfig epair14a after the failure to start the jail, and the interface would be there. “Surely this must be a FreeBSD bug!”

The “eureka!” happened earlier today, not while I was sleeping, but rather while I was solving other problems. “I bet there’s something fishy in the configuration, I should just rewrite it” I thought to myself. Most esoteric bugs are not bugs with the compiler, libraries, or operating systems. Usually they’re the user doing something slightly stupid and not realizing it.

My jail configuration (/etc/jail.conf) resembled the following:

gitea {
        $id = "14";
        $ip_addr = "10.10.10.${id}";

        vnet.interface = "epair${id}b";

        exec.prestart = "ifconfig epair${id} create up";
        exec.prestart += "ifconfig epair${id}a up descr vnet-${name}";
        exec.prestart += "ifconfig $public_bridge addm epair${id}a up";

        exec.start = "/sbin/ifconfig epair${id}b ${ip_addr}";
        exec.start += "/sbin/route add default ${public_gw}";
        exec.start += "/bin/sh /etc/rc";

        exec.prestop = "ifconfig epair${id}b -vnet ${name}";
        exec.poststop = "ifconfig ${public_bridge} deletem epair${id}a";
        exec.poststop += "ifconfig epair${id}a destroy";
}

Looking at the block and comparing it to other functional jails, I saw something missing: a vnet; declaration:

--- jail.conf   2023-11-12 20:09:03.028010000 -0800
+++ /etc/jail.conf      2023-11-12 19:59:02.867271000 -0800
@@ -230,6 +230,7 @@
        $id = "14";
        $ip_addr = "10.10.10.${id}";

+       vnet;
        vnet.interface = "epair${id}b";

        exec.prestart = "ifconfig epair${id} create up";

Sometimes you have to just walk away from a problem for a bit, but yeesh was that a silly one!

This FreeBSD machine uses UEFI and boots directly to ZFS. Imagine my surprise that the operating system had complaints about my boot partitions…after it had already booted. This machine had recently been rebuilt with new disks after I discovered that the previous disks I had been sold were “SNR” (Shingled Magnetic Recording), which have such abhorrent performance that it’s a wonder they’re even marketed at all. Suffice it to say, disk issues on this machine terrify me. I doni’t want to deal with another rebuild!

The boot process failed half-way through, which means that FreeBSD drops you into a single-user mode in the console. With that I could poke around a little bit:

• zfs list showed all data sets I expected
• zpool status showed that each disk in the pool was healthy.
• zpool scrub for good measure to make sure the pool was legitimately healthy.
• gpart showed that the partitions on all the disks were in tact as well.
• fsck reported errors on the EFI partitions for three of the four disks.

For whatever reason, the efi partitions were all hosed in the same way on 3/4th of the disks: Invalid signature in boot block.

I am still not entirely sure how this corruption occurred but getting the machine back online to do more disk diagnostics was a key step forward. Fortunately with one valid efi partition, I was able to dd its contents onto every other disk, since they’re all supposed to be identical anyways:

dd if=/dev/ada0p1 of=/dev/ada1p1 bs=4M

After a round of copying bytes around, I was able to reboot and everything came up perfectly fine!

Since there are no other indications of disk failure or problems, I may never know what originally caused the corruption. The consensus on IRC however is that building a foundational part of the boot process on an unreliable filesystem was perhaps a bad idea.

I have attempted to consolidate the home lab before, but not with this level of success. There was the time I tried deploying a single-node Kubernetes cluster, which worked for a time until some upgrade caused the software-defined networking to break in a way I wasn’t interested in debugging in my free time. Following that I tried to go “old school” and started spinning up libvirt-based virtual machines which worked well for a long time. The major downside of that approach is that I simply wasn’t able to get much density because of the significant overhead for each virtual machine. At some point you run out of memory to commit to each VM.

Converting virtual machines to FreeBSD Jails took a few hours over a weekend, and since then the quantity of what is running has jumped dramatically. Originally I was running:

• Nextcloud (Apache)
• MySQl
• Gopher server
• Tor relay
• Gitea

I now also run:

• Elasticsearch
• Graylog
• Icinga
• Icecast
• MongoDB
• Nginx
• Peertube
• PostgreSQL, which replaced MySQL for Nextcloud and is now running for multiple services.
• Redis
• Plus some personal web apps

To accomplish this, I use ZFS and Jails heavily. All the jails run on a striped and mirrored ZFS disk array which allows for better performance and redundancy than my previous incantation. These services are low-utilization which allows them to cooperate effectively within this machine’s 4 cores and 16GB of RAM.

Benefits

The higher utilization of this single machine, which is always-on, has a better power consumption profile than running multiple machines. More power efficiency wasn’t my original motivation however, I was much more interested in some of the system administration benefits of bringing these services “under one roof.”

Backups are much easier than before. I’m using some ZFS scripts to perform automatic snapshots on daily/weekly/monthly basis. Separately from those, I regularly push backups off-site and that’s trivial because at the “host” machine level, not the jail level, I can access all the filesystems at once.

Security is much easier as well since Jails are obviously providing a level of isolation. On top of that, I am using the vnet functionality in FreeBSD to provide additional network segmentation, something I never did with physical or virtual machines.

Management is pretty straightforward with all these services running in Jails. I typically do all my administration from the host rather than the jail level. As such I can pretty easily shuffle configuration files around. Unfortunately to date I haven’t found a good configuration management tool, including my own tinkering that provides a Jails-aware set of tools for more automation.

Downsides

The most notable downside to this approach is that a hardware failure could take me offline for a bit. These services are all dependent on a single power supply (PSU) and CPU. As such a failure of either would require me to keep all these services offline until that part could be replaced. For a home lab setup, I’m not expecting to support a specific SLA, so I’m quite comfortable with this risk.

I can imagine some security arguments that could be made, but frankly I think this Jails approach is much better secured than my previous home lab setups.

At present this machine has just under 10GB of RAM in use and a load average that floats between 1.0 and 2.0. Despite all the services that are running, it still uses less than my workstation with a few browsers open. To deploy your own “home lab” set up in this way you absolutely do not need a high powered machine. I would argue that a ZFS-based disk array is likely more important, after all most home lab tasks, that aren’t video-encoding, tend to be more disk I/O heavy rather than memory/CPU heavy.

This path isn’t quite entry-level simple, but if you have some systems administration experience, I think FreeBSD with ZFS and Jails is worth considering for your home or office lab.

One of the first blog posts I read was this post by klara systems, which despite being painfully devoid of diagrams, does a fairly good job of introducing the vnet feature set. In short, vnet allows jails to each have their own isolated networking stack including the ability to assign virtual or hardware interfaces to the jail. I cannot overstate how much of a super-power this is.

Consider an example where you want to do some testing of network security tooling whether your firewall rules, intrusion detection, etc. You can set up a bridge device to act as a “virtual switch” along with epair(4) devices to act as “virtual patch cables” to create a completely virtualized network as diagrammed below:

+---------------------+
| Host                |
|                     |
| +-----------------+ |
| | ixl0 (ethernet) | |  +-------------+
| +-----------------+ |  | snort-jail  |
|                     |  |             |
| +---------+     +------> epair1b     |
| | bridge0 |     |   |  +-------------+
| |         |     |   |
| | epair1a +-----+   |  +-----------------+
| | epair2a +------+  |  | some other jail |
| |         |      |  |  |                 |
| +---------+      +-----> epair2b         |
+---------------------+  +-----------------+

In the setup described above, epair1a and epair2a would not have IP addresses:

• bridge0: could be set up with 10.0.100.1
• epair1b: would then be assigned 10.0.100.2
• epair2b: would also take 10.0.100.3

If it was desired to have snort-jail be able to route out to the network connected on the host’s ixl0 interface, then the host would need to be configured as a gateway and snort-jail would the treat 10.0.100.1 as its default router.

Contrary to my experiences with Docker-based networking this is natively part of the FreeBSD network stack, rather than iptables-style smoke and mirrors. As a result, I have found this set up not only to be incredibly reliable but also very nestable. It’s totally feasible to nest these virtualized network setups to create completely isolated test networks or all manners of different network topologies for development or production workloads.

On top of that, because you can assign hardware interfaces to Jails, it is also possible to take another NIC (e.g. ixl1) and assign it into a Jail such that the Jail could manage its own network, pf, and gateway configuration, completely separate from the jail host. Thereby allowing a fairly strong segmentation of physical networks by way of the FreeBSD vnet jails infrastructure.

So cool!

For my own purposes I have been using an approach which is much better documented in this page for isolating network services. For example, I run a Tor relay inside of a Jail with an isolated network, and then use pf on the jail host to restrict that network from ever reaching out onto my LAN. It can only talk to the public internet, no other virtual or physical networks. From a security standpoint this should mean even a compromise of the Tor daemon would limit the blast-radius to that jail alone.

Since there are few practical limits on how many bridges and epair devices I can create, I have taken the path of creating multiple independent networks for Jails which don’t need to talk to each other or the same set of shared resources. This gives me some level of comfort when trying out new pieces of software or sharing services with other users that have varying trust profiles.

Overall I am immensely pleased with what has been possible between FreeBSD, Jails, and pf to create rich topologies of software-defined networks for my production and development needs. If you are curious to learn more, this blog post and this page are definitely worth reading!

With Nginx 1.15.9 a feature I have seen referred to as “dynamic certificates” was released. Originally the ssl_certificate and ssl_certificate_key were loaded when Nginx started. This meant that you could not refer to any of the Nginx variables when creating the setting. With dynamic certificates, the resolution of the ssl_certificate directive is done later by the worker(s) process(es). It’s a very handy feature!

server {
  listen 443 ssl http2;
  listen [::]:443 ssl http2;
  server_name dotdotvote.com www.dotdotvote.com dotdot.vote;
  ##
  # Certificates
  ssl_certificate     /usr/local/etc/letsencrypt/live/$ssl_server_name/fullchain.pem;
  ssl_certificate_key /usr/local/etc/letsencrypt/live/$ssl_server_name/privkey.pem;

  # ... snip ...
  location / {
    # ... snip ...
    proxy_pass http://dotdotvote;
  }
}

In the example above, I’m using the $ssl_server_name variable which will correspond to the server name requested in the SNI part of the TLS payload. This ensures that the right hostname’s certificate is utilized.

My first attempt was with $server_name, which I recommend you avoid using $server_name since that will not be computed per request. For example, in the block below the $server_name variable will always be dotdotvote.com and requests served on www.dotdotvote.com will use the incorrect certificate.:

server {
    server_name dotdotvote.com www.dotdotvote.com;

    ssl_certificate '/some/path/$server_name/fullchain.pem';
}

When I was originally setting this up, I also stumbled into some “Permission denied” errors from Nginx. With the static certificate declaration, the main Nginx process would load the file. That process would run as root before dropping privileges to the www user for the Nginx worker(s). To address this I needed to go change filesystem ownership in order for the www user to properly read the certificate files.

In retrospect, this feature seems relatively simple to use if you have a good understanding of the Nginx process and permissions model. Suffice it to say, I wouldn’t have figured this out without a bit of help from the folks in #nginx. With the change in place my Nginx configurations are now much more succinct and readable!

2021-02-14 update with more tips below

Note: if you’re already familiar with how great FreeBSD jails are, you can skip ahead

Without trying to start a flamewar, I think FreeBSD jails are basically what Linux containers aspired to be when they grew up. In short:

The jail mechanism is an implementation of FreeBSD’s OS-level virtualisation that allows system administrators to partition a FreeBSD-derived computer system into several independent mini-systems called jails, all sharing the same kernel, with very little overhead

From Wikipedia

Since FreeBSD 12, FreeBSD jails can “natively” do something which I haven’t seen done seriously in other container or virtualization stacks: provide full network isolation. VNET allows attaching an entire virtualized network stack to a jail, which allows you to delegate an actual hardware interface to a jail, after which the host no longer sees the interface at all!

VNET is the name of a technique to virtualize the network stack. The basic idea is to change global resources most notably variables into per network stack resources and have functions, sysctls, eventhandlers, etc. access and handle them in the context of the correct instance. Each (virtual) network stack is attached to a prison, with vnet0 being the unrestricted default network stack of the base system.

From vnet(9)

Using this capability you can set up entirely software-defined virtualized networks inside of FreeBSD jails for everything from network software testing (e.g. VPNs), to pre-flighting firewall changes in a simulated environment. If this sounds compelling to you, I recommend bouncing over to this blog post to learn more.

FreeBSD has also supported ZFS natively, including as the boot / partition for a number of major releases. It should therefore be no surprise that jails integrate very well with ZFS. In my setup each jail is its own file system in the main ZFS pool (zroot), which allows me to snapshot each jail independently for backup purposes.

“Excuse me, I believe I was promised some pkg(1) related content?”

Yes yes, getting to that. Since FreeBSD jails are not well appreciated by the broader systems engineering world, I wanted to take a moment to highlight some of what we’ve been missing out the past couple years while we’ve been screwing around with container orchestration engines.

pkg(1) is a binary package management tool akin to apt or yum. On modern FreeBSD installations compiling ports is no longer required, which is an incredibly welcome change to FreeBSD in recent years. Like its Linux relatives, pkg retrieves package metadata and tarballs from a remote repository by default, from the “offline jail” perspective this is a problem.

Jails however, are just running off slices of my host filesystem, e.g. /jails/postgresql which allows me to commands from the host in that file tree, for example:

freebsd# ls /jails/postgressql
.cshrc          bin             entropy         libexec         net             root            tmp
.profile        boot            etc             media           proc            sbin            usr
COPYRIGHT       dev             lib             mnt             rescue          sys             var
freebsd#

You should note that this looks surprisingly identical to what a default base install of FreeBSD, and that’s because it is!

Installing packages

The pkg(1) utility has a very useful flag -c which I can use for these jails:

-c <chroot path>, --chroot <chroot path>
    pkg will chroot in the <chroot path> environment.

From the jail host’s perspective, I can poke around the jail’s filesystem with ease, which makes installing packages trivial from the host’s perspective. For example:

freebsd# pkg -c /jails/postgresql install postgresql13-server

The above command will install the PostgreSQL v13 package and all its dependencies within the jail’s filesystem path:

# postgres --version
postgres (PostgreSQL) 13.1
#

“Try this one weird trick and your jails will never have to know anything about packages ever again!”

Caveats:

• My host distribution is identical to the distribution installed in the jail, e.g. 12.2-RELEASE. If there were drift I’m sure there would be problems with using pkg(1) in this way.
• This approach can feel goofy with configuration management, since the jail host is where all the configuration and package management happens.

Nonetheless, I was incredibly excited to discover this feature which made setting up and managing my heavily isolated vnet-based jails much easier!

2021-02-14 update: Here’s a tip passed along from a reader:

you might be interested in a tip for your jail post.

pkg -o ABI=... \
--chroot /path/to/jail \
--config /some/package/repo/FreeBSD.conf \
install -y ...

where ABI could be FreeBSD:14:aarch64 (FreeBSD current on armv8/aarch64) or a more prosaic FreeBSD:12:amd64 (amd64 architecture, FreeBSD 12).

the config file is similar to /etc/pkg/FreeBSD.conf and is acquired by pkg(8) before dropping into the chroot.

NB I usually need to mount_devfs inside the jail before & after pkg, as it now requires /dev/null - see #1763

This allows us to install packages into a directory that is not for the same FreeBSD version (and possibly even architecture!), such as a FreeBSD 13.0 armv8 nfs mount, running on a FreeBSD 14.0-current amd64 server.

There’s also pkg –rootdir … which is similar to chroot, but IIRC handles pre/post/user/group scripts differently.

In the future I hope to write more about the home lab FreeBSD set up that I have been working on over the past couple weeks. FreeBSD 12.x is by far the most exciting FreeBSD release I have used since the transition to full SMP in the 4.x -> 6.x timeframe. If you haven’t looked at FreeBSD lately, I highly recommend giving it a spin!