Sailing towards frowntown with minikube --vm-driver=none
Kubernetes is so hot right now. So hot. Not to brag, but the Jenkins has been using Kubernetes for a couple years, in production even! While Kubernetes is certainly worth the hype, so hot, I have traditionally done all of my development with Kubernetes resources using a personal Azure Kubernetes Service instance rather than running minikube locally. Recently I took some time to do some hobby-hacking, which included some Kubernetes work, and ended up revisiting using minikube on my local machine. It went…okay.
minikube tries to utilize a local hypervisor like xhyve (macOS),
VirtualBox, or KVM (Linux). There is also a promising option:
--vm-driver=none which utilizes the local
dockerd already running on a
Linux system. For those like me, running Linux as their primary operating
none option appears to be a nice compact way to run a
single-local-node of Kubernetes. Unfortunately, the theory of running minikube
with no VM is a bit nicer than the reality of it.
When configuring minikube for the first time, a number of certificates are
generated and stored for accessing the Kubernetes cluster with
I did not notice right away was that these certificates embed my IP
wlan0) address into the certificate and subsequently generated
apiVersion: v1 clusters: - cluster: certificate-authority: /home/tyler/.minikube/ca.crt server: https://192.168.1.59:8443 name: minikube
The reason I didn’t notice this right away was because it
only proved to be a problem once I joined a different network, with a wildly
different subnet, and all of a sudden
kubectl was no longer able to access my
minikube does provide an
update-context command which claims to “Verify the
IP address of the running cluster in kubeconfig.” For me this updated the
~/.kube/config but didn’t re-generate or update the certificates in the
cluster itself, so while I could re-connect, the certificates would no longer
authorize against the environment.
Sadly, I still haven’t figured out how to resolve this.
When minikube starts without a hypervisor, it installs a local
service on your host machine, which is important to know for later.
Right now it seems that
minikube start is the only command aware of
minikube stop keeps resulting in errors related
docker-machine, and as luck would have it also results in none of the
Kubernetes containers terminating, nor the
kubelet service stopping.
Of course, if you wish to actually terminate minikube, you will need to execute
kubelet stop and then ensure the k8s containers are removed from the output in
Aside from a bunch of other containers running, there are a few side effects
from running minikube. I found that the dashboard pod failed to load or connect
to some of the other pods in the cluster, and as a result infinitely spammed
/var/log/syslog with errors along those lines. Over a couple days, my
uncompressed syslog bumped up from a few hundred kilobytes to a hundred or so
The most notable side effect of having minikube running locally however, was
that my Docker images kept disappearing! It turned out to be due to one of the
kubelet service’s features: garbage collecting Docker images which are not
being used after some time delay. Let’s say you ran
docker build in your
application directory, but then before running a test you went to grab a cup of
coffee. By the time you would return, depending on your brew time, executing
docker run with that container would fail because the container no longer
While this behavior is really useful when Kubernetes is the only thing talking
to the Docker daemon, it’s pretty stinking annoying when you’re using
for other work.
To stop this, at least temporarily, stop the
kubelet service with:
service kubelet stop when you’re not exclusively using
minikube. As noted by
those who maintain the minikube repo (https://github.com/kubernetes/minikube/blob/master/docs/vmdriver-none.md)
Kubernetes is a really wonderful technology, so hot right now. And I wish the
local development experience were a bit nicer, especially for those of us who
already have a nice Docker environment already up and running. Unfortunately, I
minikube is a bit too untested, and perhaps a bit too cavalier, with a
“stock” Linux system. For now, it’s probably best to keep in a virtual machine,
despite the networking and performance headaches that come along with that.
At least with a virtual machine, if something goes wrong, it’s easy to delete.
Nuke it from orbit, the only way to be sure.