javax.net.ssl.SSLHandshakeException: (certificate_unknown) using custom trust store and self-signed certificates

Background + Setup:

I have created a Scala+AKKA-http app that is a client and a server. For explanation, I will refer to one "server+client" as instance.

Each instance generates a self-signed certificate and keystore using:

      val genKeyCommand = s"openssl genrsa -out $SSL_KEY_PATH 4096"      Seq("bash", "-c", genKeyCommand) !!      val sslCommand = s"openssl req -new -x509 -sha256 -days 36500 -addext 'subjectAltName = IP:$SERVER_IP'" +        s"-subj '/C=UK/ST=London/L=London/O=Dis/CN=hydra-${OS.getOS.toString.toLowerCase}-test.com' -key $SSL_KEY_PATH " +        s"-out $SSL_CERTIFICATE_PATH"      Seq("bash", "-c", sslCommand) !!      // load the key into keystore and create      val keyStoreCommand = s"openssl pkcs12 -export -out $KEY_STORE_PATH -in $SSL_CERTIFICATE_PATH -inkey $SSL_KEY_PATH " +        s"-passout pass:" + KY_STORE_PASS      Seq("bash", "-c", keyStoreCommand) !!

I currently have 2 instances for testing: hydra-macos-test and hydra-linux-test (Linux is running in a VBox VM, Debian 12)

I have created a handshake over http between two instances that occurs when one is made aware of the other (through user input), detailed in the answer here.

The basics of this:

1. instance 1 creates a client actor, and requests a auth token via http from instance 2
2. instance 1 encrypts its self-signed certificate and sends it (using the auth token) to instance 2
3. instance 2 decrypts the certificate, and stores it into a custom trust store
4. instance 2 responds with its encrypted self-signed certificate
5. instance 1 decrypts the certificate and places it inside its custom trust store
6. https communications should now be able to take place

This process works very well, the certificates are added to the trust store on each side.

I'm setting the keystore/truststores like so:

    // set system keystore and truststore to our custom ones    System.setProperty("javax.net.ssl.trustStore", TRUST_STORE_PATH)    System.setProperty("javax.net.ssl.trustStorePassword", DatabaseUtil.hashString(SERVER_ID))    System.setProperty("javax.net.ssl.keyStore", KEY_STORE_PATH)    System.setProperty("javax.net.ssl.keyStorePassword", DatabaseUtil.hashString(SERVER_ID))

Creating the SSLContext for server and client connections within an instance:

  def createClientSSLContext: SSLContext = {    val sslContext = SSLContext.getInstance("TLS")    val keyStore = loadKeyStore(TRUST_STORE_PATH, KEY_STORE_PASS)    val keyManagerFactory: KeyManagerFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm)    keyManagerFactory.init(keyStore, KEY_STORE_PASS)    val tmf: TrustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm)    tmf.init(keyStore)    sslContext.init(keyManagerFactory.getKeyManagers, tmf.getTrustManagers, new SecureRandom)    sslContext  }  def createServerSSLContext: SSLContext = {    val sslContext = SSLContext.getInstance("TLS")    val keyStore = loadKeyStore(KEY_STORE_PATH, KEY_STORE_PASS)    val entry = keyStore.getEntry(ALIAS, new KeyStore.PasswordProtection(KEY_PASS))    val keyManagerFactory: KeyManagerFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm)    keyManagerFactory.init(keyStore, KEY_STORE_PASS)    val tmf: TrustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm)    tmf.init(keyStore)    sslContext.init(keyManagerFactory.getKeyManagers, tmf.getTrustManagers, new SecureRandom)    sslContext  }

Using SSLContext for the server:

val sslContext = SSLManager.createServerSSLContext    val https: HttpsConnectionContext = ConnectionContext.httpsServer(sslContext)    // create https Engine    ConnectionContext.httpsServer(() => {      val engine = sslContext.createSSLEngine()      engine.setUseClientMode(false)      engine.setNeedClientAuth(true)      engine    })    Http().newServerAt("localhost", 8443).enableHttps(https).bind(new HydraRoute(HttpsRoutes(clientManager)).masterRoute)      .onComplete {        case Success(binding) =>          val address = binding.localAddress          system.log.info(s"HTTPS Server is listening on ${address.getHostString}:${address.getPort}")        case Failure(ex) =>          system.log.error("HTTPS Server could not be started", ex)          stop()      }

Using SSLContext for the client:

    val connectionContext = ConnectionContext.httpsClient(SSLManager.createClientSSLContext)    http.singleRequest(request, connectionContext).pipeTo(self)

Problem

When instance 1 comes to send a httpspost request to instance 2, I get the error:

javax.net.ssl.SSLHandshakeException: (certificate_unknown) PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

The full output can be found here (with ssl:debug)

If there is anything else I can provide to help debug the issue please let me know and i'll be happy to do so. I've spent days setting this up and scratching my head trying to fix this.

Thanks in advance!