[Bug]: SSL handshake error on latest alpine jre image

[BalmungSan]

Please add the exact image (with tag) that you are using

eclipse-temurin:8u352-b08-jre-alpine

Please add the version of Docker you are running

Docker version 20.10.17, build 100c701

What happened?

Our Kotlin (Vert.x) application stopped working a couple of days ago.
After investigating the logs, we discovered that the problem is that the application couldn't establish a connection with the database (MongoDB hosted on Atlas) due an SSL handshake error.

After debugging we identified that the issue is related to the Docker image, since running outside of Docker (with the same JDK version) doesn't produce the error. Also, using the previous Docker image (eclipse-temurin:8u345-b01-jre-alpine) also fixes the problem.

Additional details

We tried to fix the problem running the following commands (both interactively inside a running container, as well as in the Dockerfile) but no combination of them fixed the problem.

apk --update upgrade && apk add openssl openssl-dev ca-certificates && update-ca-certificates

The JAR is run using

CMD ["java", "-server", "-XX:+PrintFlagsFinal", "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-jar", "app.jar"]

We also tried testing the TCP connection within the container using openssl with the following command:

openssl s_client -connect <atlas-cluster-dns>:27017

And it seemed to succeed correctly.

Relevant log output

Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
	at sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at sun.security.ssl.Alert.createSSLException(Alert.java:117)
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:311)
	at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
	at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185)
	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:152)
	at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:588)
	at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:544)
	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:411)
	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:390)
	at com.mongodb.internal.connection.tlschannel.impl.TlsChannelImpl.callEngineUnwrap(TlsChannelImpl.java:322)
	at com.mongodb.internal.connection.tlschannel.impl.TlsChannelImpl.unwrapLoop(TlsChannelImpl.java:288)
	at com.mongodb.internal.connection.tlschannel.impl.TlsChannelImpl.readAndUnwrap(TlsChannelImpl.java:619)
	at com.mongodb.internal.connection.tlschannel.impl.TlsChannelImpl.handshakeLoop(TlsChannelImpl.java:586)
	at com.mongodb.internal.connection.tlschannel.impl.TlsChannelImpl.handshake(TlsChannelImpl.java:561)
	at com.mongodb.internal.connection.tlschannel.impl.TlsChannelImpl.doHandshake(TlsChannelImpl.java:536)
	at com.mongodb.internal.connection.tlschannel.impl.TlsChannelImpl.handshake(TlsChannelImpl.java:522)
	at com.mongodb.internal.connection.tlschannel.impl.TlsChannelImpl.write(TlsChannelImpl.java:378)
	at com.mongodb.internal.connection.tlschannel.ClientTlsChannel.write(ClientTlsChannel.java:184)
	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.writeHandlingTasks(AsynchronousTlsChannelGroup.java:540)
	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.doWrite(AsynchronousTlsChannelGroup.java:498)
	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.lambda$processWrite$4(AsynchronousTlsChannelGroup.java:459)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 common frames omitted

[karianna]

@BalmungSan Do you know which certificate it is trying to connect with ?

[BalmungSan]

Do you know which certificate it is trying to connect with ?

@karianna Do you mean the public certificate from the server?

If so, I am sharing the information obtained using OpenSSL; including the certificate chain. Not sure how to obtain this information from the application itself, but AFAIK it should be the same.

Details

depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = *.skvec.mongodb.net
verify return:1
---
Certificate chain
 0 s:CN = *.skvec.mongodb.net
   i:C = US, O = Let's Encrypt, CN = R3
-----BEGIN CERTIFICATE-----
MIIGRzCCBS+gAwIBAgISA7KZpvfhEI86H8TfRqcB9GkzMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMjExMDEyMDMxMDJaFw0yMzAxMzAyMDMxMDFaMB4xHDAaBgNVBAMM
Eyouc2t2ZWMubW9uZ29kYi5uZXQwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK
AoICAQCbTWo3KjrzViavAE0nC6ih8sZDaq0f59iFvExbW0eZTinNVjVJlNTGuXQ8
OifONSa3VtssIVzwFFR4xvIxhyHM5V3WGdNqSxR16jb2I637IFuKM03BlFFOvijG
a91F6ssiZdlzumdLte+mibQ65DfKCDBFfQpHFbtLMkoiJwcl14G6NwWNIZY3RH3L
tIgRhiZQbNgMYSPm6IMqzDumkyUiwqxKIFpnFC9q0gwJgrUuIrx702egK6cvSggn
wgsxeSMi3Db9uboiC7btBriwprJz8m1we5k0886TMIglU4iwYgl9K/4WfEeFj+Xc
Q91VRtf8ia7s8mzPpH/Kk66fklpj3OlyyaQMmylBeNzYcJ8R++wIEmJ7Ay9cK0bW
pq6SZrV+4ZXxDGne6caunZR+uFd2uuJ5eOeysICd8ZeqTh6AQujm3SIVmDzWfCCl
7hK4v9eUS5yonbygv4bnlFbxfSVutcyCGRAJfDTL3z3RFZhdko5wFK7rNWzRwxrj
KQcNVQcWTnpl0eAzZLY9j8tHx2Lnmgmk4sTfIfEYhB84AElh7M8apMA3ZdVcFJc1
UF6pXS+9hgv6KT1NnPKMi9ueZdVv4GfbPNDKwl5+H3AwnWfg4gZxUDzq5l6MpXua
bjBe73J5gwcM5SzyAk4GYYPmq2Me32gCTFAD7J4lO1vFobgOfQIDAQABo4ICaTCC
AmUwDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcD
AjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBRIgxmbFUdG1VqwT9+d7W5n84qjqTAf
BgNVHSMEGDAWgBQULrMXt1hWy65QCUDmH6+dixTCxjBVBggrBgEFBQcBAQRJMEcw
IQYIKwYBBQUHMAGGFWh0dHA6Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYW
aHR0cDovL3IzLmkubGVuY3Iub3JnLzA4BgNVHREEMTAvghgqLnNrdmVjLm1lc2gu
bW9uZ29kYi5uZXSCEyouc2t2ZWMubW9uZ29kYi5uZXQwTAYDVR0gBEUwQzAIBgZn
gQwBAgEwNwYLKwYBBAGC3xMBAQEwKDAmBggrBgEFBQcCARYaaHR0cDovL2Nwcy5s
ZXRzZW5jcnlwdC5vcmcwggEFBgorBgEEAdZ5AgQCBIH2BIHzAPEAdgCt9776fP8Q
yIudPZwePhhqtGcpXc+xDCTKhYY069yCigAAAYQ1GtkJAAAEAwBHMEUCIAe0gMp5
1CWjCe12AtQGbewNDNfdsc6Pg6GFfmpFLBJAAiEA4uJcZ55m64iQEhliIHocJ2+S
zgw6NqCdogM6dEfl/ckAdwBvU3asMfAxGdiZAKRRFf93FRwR2QLBACkGjbIImjfZ
EwAAAYQ1Gtk1AAAEAwBIMEYCIQCheNIZPsFaiQkDa60SzM1s5H0i562jA50uWIU0
ZEwplgIhAJAVGjWjINIhlU3DddwNAd/M3p/xZLojp2p8SER1JKiCMA0GCSqGSIb3
DQEBCwUAA4IBAQCL2RPUtiJ/KsGal9RrQFm9SgnTHr52AllDCrYxtUZCgBzRvB52
6/6c0Ff0Vdqbxbkepj1WcnR401ebEmKE8U6fIYa5YJXHD4XwuZ0TV1uaBsiFHcuY
dm+GPrB5N0Fiavti8aJlHxZKdTSWkLOjbSGrmpq+RNKNUOieouJWX/8JOuYvLotT
xbF4LPqyvuOGJoSXVj7/JokZbM2VAWfhtcbmA0NTQiTWtl0gheRtNXEinvoJCajc
iCt4Lo/YB76YyKlgjcURIJhbS62JKVfLeTz0JUc6kXDFDJ1m6G0aSqTGzPAC6S5h
JC/dvKRn0XnE35H1qxUby8c1lJSaHRe4rrDB
-----END CERTIFICATE-----
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
/kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----
---
Server certificate
subject=CN = *.skvec.mongodb.net

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3785 bytes and written 428 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

(I double checked before posting, and it seems nothing there is sensitive. But, if I shared something I shouldn't, please let me know)

[dadez]

I'm facing the same issue, I have found that providing the truststore password (-Djavax.net.ssl.trustStorePassword=changeit ) on version 11-jre-alpine it works, but on 8-jre-alpine not.

PS: I'm using https://github.com/MichalHecko/SSLPoke for run the tests

[jerboaa]

This is likely due to using TLS 1.3 on the client side by default new in 8u352. See:
https://bugs.openjdk.org/browse/JDK-8245263

[BalmungSan]

This is likely due to using TLS 1.3 on the client side by default new in 8u352.

I also though that may be related.
However, that doesn't explain why it only happens inside Docker; remember I couldn't reproduce it locally using the same JDK version.

[jerboaa]

I also though that may be related. However, that doesn't explain why it only happens inside Docker; remember I couldn't reproduce it locally using the same JDK version.

I take it this was bare metal JDK 8 on alpine vs. JDK 8 in an alpine container?

[BalmungSan]

I take it this was bare metal JDK 8 on alpine vs. JDK 8 in an alpine container?

@jerboaa it was bare metal JDK 8 version 1.8.0_352-b08 on Ubuntu 20.04 VS eclipse-temurin:8u352-b08-jre-alpine Docker image.

[jerboaa]

@BalmungSan Does installing libgcc on the alpine image fix the problem? See adoptium/temurin-build#3002 (comment)

[BalmungSan]

@jerboaa I can confirm that installing libgcc does solve the issue: at least locally (using Docker).
If it happens again on a deployed service I will re-open the issue.

[jerboaa]

Good to know, thanks!