Vegatron 🌈✨

Checking Certs

Even among microservices it’s prudent to use HTTPS to communicate. It builds more trust in the system.

Unhelpful error

But when the certificates fail in some way the error is quite unhelpful.

There’s no information about which certificate failed, or why it failed. The stacktrace and exception message do not help to diagnose the problem:

sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:148)
    ...
 	at io.ktor.network.tls.TLSClientHandshake.handleCertificatesAndKeys(TLSClientHandshake.kt:257)
    ...

Certificate issues

We hit this exact problem recently, when an upstream system had a certificate issue.

The error only happened once we sent a request. Requests only happen when a consumer is involved, so the consumer hit the error before we did. Our system reported healthy the whole time, even though the fault was ours to fix.

Check proactively

Instead of waiting for a consumer to hit an error, I built a simple certificate check to run on startup. We didn’t want to block the pod startup, so this doesn’t throw an error.

package startup.diagnostics

import io.ktor.util.logging.KtorSimpleLogger
import java.net.URI
import java.security.KeyStore
import java.security.SecureRandom
import java.security.cert.X509Certificate
import javax.net.ssl.SSLContext
import javax.net.ssl.SSLSocket
import javax.net.ssl.TrustManagerFactory
import javax.net.ssl.X509TrustManager

/**
 * Connects to a serviceUrl and logs the certificate chain the server presents,
 * plus whether the JVM's default trust manager accepts it. Wraps the platform
 * default X509TrustManager so it logs the chain (subject/issuer/validity) even
 * when validation fails with SunCertPathBuilderException, since that exception
 * alone doesn't say which cert/host caused it.
 *
 * - https://docs.oracle.com/en/java/javase/21/docs/api/java.base/javax/net/ssl/X509TrustManager.html
 * - The JVM flag -Djavax.net.debug=ssl:handshake:verbose:trustmanager is extremely verbose. Don't use it.
 */
object CertDiagnostics {
    private val logger = KtorSimpleLogger("CertDiagnostics")

    private fun defaultTrustManager(): X509TrustManager {
        val factory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm())
        factory.init(null as KeyStore?)
        return factory.trustManagers.filterIsInstance<X509TrustManager>().first()
    }

    private fun loggingTrustManager(
        delegate: X509TrustManager,
        host: String
    ) = object : X509TrustManager {
        override fun getAcceptedIssuers(): Array<X509Certificate> = delegate.acceptedIssuers

        override fun checkClientTrusted(
            chain: Array<out X509Certificate>,
            authType: String
        ) = delegate.checkClientTrusted(chain, authType)

        override fun checkServerTrusted(
            chain: Array<out X509Certificate>,
            authType: String
        ) {
            chain.forEachIndexed { index, cert ->
                logger.info(
                    "tls-diagnostics host=$host cert[$index] subject=${cert.subjectX500Principal} " +
                        "issuer=${cert.issuerX500Principal} notBefore=${cert.notBefore} " +
                        "notAfter=${cert.notAfter} serial=${cert.serialNumber}"
                )
            }
            try {
                delegate.checkServerTrusted(chain, authType)
                logger.info("tls-diagnostics host=$host chain TRUSTED (${chain.size} certs)")
            } catch (e: Exception) {
                logger.error("tls-diagnostics host=$host chain NOT TRUSTED: ${e.message}", e)
                throw e
            }
        }
    }
    
    /**
     * Diagnostic connection; should never throw.
     * Call at service startup for the upstream system so a broken chain shows up in logs immediately.
     */
    fun logCertificateChain(uri: URI) {
        if (uri.scheme != "https") {
            logger.info("tls-diagnostics skipping non-https url=$uri")
            return
        }
        val host = uri.host
        val port = if (uri.port != -1) uri.port else 443

        try {
            val trustManager = loggingTrustManager(defaultTrustManager(), host)
            val sslContext = SSLContext.getInstance("TLS")
            sslContext.init(null, arrayOf(trustManager), SecureRandom())

            // .use closes the socket for us, even if connect/handshake throws
            sslContext.socketFactory.createSocket().use { socket ->
                socket as SSLSocket
                socket.connect(java.net.InetSocketAddress(host, port), 5000)
                socket.soTimeout = 5000
                socket.startHandshake()
                logger.info("tls-diagnostics host=$host port=$port handshake succeeded")
            }
        } catch (e: Exception) {
            logger.error("tls-diagnostics host=$host port=$port handshake FAILED: ${e.message}", e)
        }
    }
}

The idea

The code wraps the TrustManager as a delegate object. Connecting to the upstream port triggers the TLS handshake, which starts the certificate check. We iterate over the certs to log them. This allows us to pinpoint which certificate is causing problems.

The Java option -Djavax.net.debug=ssl:handshake:verbose:trustmanager is verbose. It prints output for all the connections that your system creates.

Going further

We considered a more robust check, but skipped it. We didn’t have the time or resources. 1

Health check

We could fold this into our health check, so it alerts us as soon as an upstream system redeploys with a broken chain.

  1. This is one of many political problems with microservices. It wasn’t ours to solve: we just had to find who was responsible. ↩︎

#certs #kotlin #tls