JKS? jfc. Adding a root certificate

TLS certificates have the largest “complexity/importance” scores imaginable. Everything about them is error prone and seemingly over-engineered from top to bottom, yet they are one of the most important pieces of security and authentication in our software architectures. From an engineering management standpoint, I am finding myself adopting the rule of: estimates for any project involving certificates should be multiplied tenfold. If the project involves the Java Virtual Machine (JVM) and the Java Key Store (JKS), multiply by another ten I suppose. For my own future convenience, in this blog post I would like to outline how to add a root certificate to a Java Key Store in Red Hat-derived environments.

Like many corporate environments, we have our own internal Certificate Authorities (CAs) which all derive their chain of trust from our internal root certificate. Accessing internal services requires that the operating system has that root certificate, or when accessing those internal services from anything running atop the JVM, the default JKS must have the root certificate.

If you search around the web for how to add root certificates, you might find the update-ca-certificates command, whose CentOS/RHEl manpage has the following:

The directory /etc/pki/ca-trust/extracted/java/ contains a CA
certificate bundle in the java keystore file format. Distrust information
cannot be represented in this file format, and distrusted certificates are
missing from these files. File cacerts contains CA certificates trusted for TLS
server authentication.

You might assume, as I did, that this means the update-ca-certificates tool is going to create files that the JVM picks up properly and your default JKS will have the root certificate in place.

This is false. At least in the environments which I have tested this.

Digging further I found this blog post and used the following command to import the root certificate into JKS after installing it on the system at large:

keytool -importcert -alias startssl -keystore $JAVA_HOME/jre/lib/security/cacerts -storepass changeit -file ca.der

Using the SSLPoke tool referenced in this Atlassian knowledgebase article I was then finally able to access the same internal services from native utilities (e.g. curl) and from the Java-based services which I was working with at the time.

In my situation, the fact that all of this was happening within Docker containers further complicated the debugging: multiple by another 2-5 on that engineering estimate.

Certificates are too important to be this painful.