Whenever a system comes offline I try to always make multiple updates at once. When I recently received some new mounting hardware for one of my big FreeBSD servers, I decided to plonk an idle PCI-e device into the chassis while it was out of the rack.

The FreeBSD machine doesn’t have any use for the PCI-e device but I might be able to make use of the device from a bhyve virtual machine. bhyve supports pci passthrough but I have not actually used it in my many years of FreeBSD virtualization.

Executing vm passthru lists out all the devices and whether they’re ready for use, and suffice it to say even after my first crack at things, my PCI device was not ready!

The device must be “hidden” from the host in order for guests to acquire/use the device which means a pptdevs entry in /boot/loader.conf. The configuration syntax has an antique feel to it, so I re-read my loader.conf numerous times, tried a couple reboots, and then went off searching the internet.

I spotted a one-off comment on a semi-related GitHub issue stating:

Make sure vmm_load="YES" is in the loader.conf

I have been running bhyve for a while, with vm_enable="YES" and assorted settings in /etc/rc.conf…but I didn’t have anything associated in the loader.conf! For those unfamiliar, loader.conf is like your grub.conf which has the kernel configuration and early boot time parameters, whereas rc.conf is utilized at a much later stage of the boot process.

I posit that my pptdevs entry in loader.conf was not working because I was relying on rc.conf to dynamically load the necessary bhyve kernel modules, but by that point in the boot process it is just too late to configure PCI passthrough devices!

A deep breath and one more reboot, and tada! The machine now is passing a PCI-e card up to a Linux virtual machine which sees it as a native device!

There’s no telling what kind of mischief this will lead to!