java.lang.UnsatisfiedLinkError linking libc

[nkconnor]

SBT Dependency

"io.valkey" % "valkey-glide" % "1.3.0" classifier "linux-x86_64"

Error

java.lang.UnsatisfiedLinkError: /tmp/nativeutils879005385204760/libglide_rs.so: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /tmp/nativeutils879005385204760/libglide_rs.so)

	at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method)
	at java.base/java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2442)
	at java.base/java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2498)
	at java.base/java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2694)
	at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2627)
	at java.base/java.lang.Runtime.load0(Runtime.java:768)
	at java.base/java.lang.System.load(System.java:1834)
	at glide.ffi.resolvers.NativeUtils.loadLibraryFromJar(NativeUtils.java:105)
	at glide.ffi.resolvers.NativeUtils.loadGlideLib(NativeUtils.java:44)
	at glide.ffi.resolvers.SocketListenerResolver.<clinit>(SocketListenerResolver.java:11)
	at glide.api.BaseClient.buildChannelHandler(BaseClient.java:472)
	at glide.api.BaseClient.createClient(BaseClient.java:367)
	at glide.api.GlideClusterClient.createClient(GlideClusterClient.java:105)

System Info

ldd --version
ldd (Ubuntu GLIBC 2.31-0ubuntu9) 2.31
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

lsb_release -a
Distributor ID: Pop
Description: Pop!_OS 20.04 LTS
Release: 20.04
Codename: focal

lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 20
On-line CPU(s) list: 0-19
Thread(s) per core: 2
Core(s) per socket: 10
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Core(TM) i9-9820X CPU @ 3.30GHz

[Yury-Fridlyand]

Hi
Thank you for reporting this issue!
GLIDE project has a native component and it was discovered that it doesn't work on some platforms like AL2 (amazon linux), but works on AL 2023 and many others.
I followed hivemind to see which GLIBC versions we're compatible with:

$ objdump -T target/release/libglide_rs.so | grep -Eo 'GLIBC_\S+' | sort -u
GLIBC_2.10)
GLIBC_2.14)
GLIBC_2.17)
GLIBC_2.18)
GLIBC_2.2.5)
GLIBC_2.25)
GLIBC_2.28)
GLIBC_2.29)
GLIBC_2.3)
GLIBC_2.3.2)
GLIBC_2.3.4)
GLIBC_2.33)
GLIBC_2.34)
GLIBC_2.4)
GLIBC_2.7)
GLIBC_2.9)

Unfortunately, GLIBC 2.31 that your machine has isn't in this list. That's pretty weird, because GLIBC 2.29 and 2.33 are supported.

According to this report, few symbols were removed in GLIBC 2.31 comparing to 2.30. One of them is pthread_attr_init and pthread_attr_destroy which is used by GLIDE, which makes this GLIBC version incompatible with GLIDE (or GLIDE incompatible with GLIBC 2.31).

It is very unclear for me, but these symbols weren't re-added back according to these reports, but GLIDE is compatible with GLIBC 2.34 and works fine there.

We need to track from which dependency these symbols (pthread_attr_init and pthread_attr_destroy) come from.

[avifenesh]

@nkconnor As @Yury-Fridlyand mentioned, Glide is based on a native component, and as such should be built for targets.
At this point, as we mentioned, we support ARM and X86 for Ubuntu, macOS Darwin, Alpine, and AL2023.
As for myself, I use Fedora and Debian, and it works with both.
If Pop!_OS has system differences (Such as missing symbols compared to the other GLIBC), we need to build it for the specific target.

If you're interested in having Glide supporting Pop!_OS LTS, can you replace this issue from a bug to a feature request, with the relevant information for the request?

Not to say that there is no user pain here, but specifically, it's not a distro we announce support for, hence it is not a bug.
But feature request will push us to implement it for Pop!_OS, we highly value users' needs, and users' pains.

[nkconnor]

@avifenesh I appreciate the guidance. I would offer that Pop!_OS isn't a fringe distribution and generally matches Ubuntu, one of the most popular distributions for server programming. Ubuntu 20.04 is a LTS release, and as far as I know, ships with the same version of GLIBC.

Pop! is also the default OS offered on System76 machines and very popular
in any rankings I found in a quick Google search.

I feel like it's a stretch to say this is a feature request, but, you can do what you want with it! I can tell you I was forwarded here as an AWS customer and I would hope that they adjust their documentation on this topic to avoid wasting customer's time in the future.

[avifenesh]

@nkconnor I'm sorry that you wasted your time.

You're right, ubuntu is popular, no questions, and we support it.
As said also AL2023, Alpine, and macos.
And without CI, mint, fedora and debian are also working well with glide, personal experience.

While Pop!_OS is an Ubuntu based distro, it has a customized version of the GNOME, called COSMIC, hence has differences in the libs and requires a targeted build, especially when running on a different machine type.

Pop!_OS might be ranked comparably high in the PC linux distros ranking, but the lists are not showing the reality, ubuntu takes 30-40% market share, fedora and debian 10-15% each, so while a distro might be 6 in the list, it still might have less than 1% user base.
And while this is the percentage for PC distros, when it comes to production servers or development machines, Pop!_OS is very unique, and we wouldn't think about targeting it without getting a request from user's.

Pop!_OS is a very interesting distro, especially because the system76 and I'm waiting for a while to try it.
But i didn't think about it as a "work" machine, but as a personal machine, and personal development.

All the above is just for explaining the reasoning why some distros are supported and some not.

It doesn't change the fact that we want to do what user's needs.
If someone opens a request for ALT linux or rasberian, we will do our best to fulfill the needs, and the project is centered around what the user needs.

We would be happy to add support, but we need to be aware of those requests.

For the AWS issue, the problem is that it isn't mentions there which distros are supported?

[nkconnor]

I just want to re-iterate this doesn't appear to be a Pop!_OS specific issue and as far as I can tell would effect all sorts of distributions including Ubuntu 20.04, which shares basically all the same libraries.

You can see a minimal example of this in the following repository:
https://github.com/nkconnor/glide-broken-ubuntu

Again, you can see the table above which shows some other distributions that share the same version of GLIBC

[avifenesh]

@nkconnor Thanks for the example, I will reproduce.
Did you try on Ubuntu?

[nkconnor]

@avifenesh Please see above example which reproduces issue on Ubuntu 20.04 LTS.

[Yury-Fridlyand]

We need to track which of our dependency use pthread_attr_init and pthread_attr_destroy and fix that (remove/replace/refactor/whatever) to extend our GLIBC compatibility.

[avifenesh]

@nkconnor I'm working on it, will update.

[avifenesh]

@Yury-Fridlyand @nkconnor
So, since when running v1.0 I don't see this issue.
I guess that the issue is simply the host we build on.
GH actions replaced a few months ago the Ubuntu versions and the latest is 24, and I guess that is just about the target we build for.
We don't see something similar in other languages, so I assume that java has some stronger dependency on the DL.

We need to understand if this is about the rust compilation, specifically, or is it the native code unique to Java.

We will need to check better, but the solution might be simple, but will require a release.

@nkconnor For the meanwhile, you can use 1.0, which is supported by glibc you have.
I noticed that 22 is already available for Pop!, you have a specific reason not to upgrade?
Last, I see that you have a lot of experience with Rust and Scala, so if you want you can also build from source, until we will release the support.

[avifenesh]

@Yury-Fridlyand @yipin-chen Can we release java 1.3.1 from a build on Ubuntu 20.04? Or adding it to the build configuration of the core?

[nkconnor]

I can confirm all tests pass when building locally on 20.04 (running ./gradlew :client:buildAll).

Is the note about musl incompatibility still accurate? It seems that could be a possible target to remedy this issue. My understanding is that this will continue to happen and cause future incompatibilities as the output will continue to select whatever version of GLIBC is available on the CI server.

[avifenesh]

@nkconnor the note about musl is still in place, and currently I'm exploring some options to reduce platform-specific dependency as a whole for the java client, by replacing some internal logic to an alternate design.
As long as we hard-code the version we build on, instead of using “latest” it will result in the same GLIBC.
The GLIBC is minimal, meaning, if you have a higher version, it will work for you, since each version is “includes” the previous.
But, when Ubuntu drops LTS, and move to ESM, we will likely drop support as well. It will be part of a major release with a noticeable announcement. We are trying hard to avoid breaking changes, and doing so, if necessary, will happen just as part of a major release.
But support means that we run a large matrix of valkey versions, language versions, and host versions.
So while some other targets may be supported, we can “state” just what we tested by the CI, and each version is inflating the CI a lot.
But we are talking on about 2–3 years from now, in case that the user itself is using at least 5-year-old OS version and doesn't want to upgrade. Which, in that case, we will be just pips in a line of incompatibility of the user machine.

[jamesx-improving]

The root cause of this issue might be the forward-incompatibility of GLIBC. Here's my finding:

Ubuntu LTS versions and their corresponding default GLIBC versions

• Ubuntu 20.04: GLIBC 2.31
• Ubuntu 22.04: GLIBC 2.35
• Ubuntu 24.04 (current ubuntu-latest): GLIBC 2.39

The "big change" of GLIBC 2.34

Here's a quote from the Release Note of GLIBC 2.34:

Major new features:

* In order to support smoother in-place-upgrades and to simplify
  the implementation of the runtime all functionality formerly
  implemented in the libraries libpthread, libdl, libutil, libanl has
  been integrated into libc.  New applications do not need to link with
  -lpthread, -ldl, -lutil, -lanl anymore.  For backwards compatibility,
  empty static archives libpthread.a, libdl.a, libutil.a, libanl.a are
  provided, so that the linker options keep working.  Applications which
  have been linked against glibc 2.33 or earlier continue to load the
  corresponding shared objects (which are now empty).  The integration
  of those libraries into libc means that additional symbols become
  available by default.

To put it simply:

• Binaries compiled with GLIBC version 2.33 or earlier, which has those linker flags, will WORK when running on 2.34 or later, thanks to those empty static archives.
• Binaries compiled with GLIBC version 2.34 or later, which do not have to link those extra libraries, will NOT WORK when running on 2.33 or earlier, as they can't find those functions in libc.

In the real world we might see some weird skipping of supported versions, as shown by @Yury-Fridlyand in his earlier comment, but I believe the big picture should be like so.

Conclusion

For compatibility consideration, we should seriously consider using Ubuntu20.04 / GLIBC 2.31 to compile our rust code for all bindings, if a significant number of our users are still using an earlier GLIBC version. But on the other hand, GitHub is planning to fully retire ubuntu-20.04 GH hosted runners on April 1st, 2025, which means if we choose to compile with GLIBC 2.31, we need alternative solution now. Also, there might be other trade-offs we haven't found out.

@avifenesh @jonathanl-bq @Yury-Fridlyand could you please weight in on this?

[avifenesh]

When you have glibc 33 a machine with glibc 35 will work because 35 "includes" 33.
So we need to build on 20.
We need to keep building on the lowest lts, or when building the rust to set the target which is possible with rust.
When I develop locally, i change the target to check if it compiles.
Another option is keeping the CI on what is available by GH, and running the CD on the ephemerals.

In each case we need to take this task and test it now, before we get to a real release.

[nkconnor]

You can build in Docker like the example repo I made and linked above. It would use the image glibc and make 20.04 and many others available I believe.

…

On Wed, Feb 26, 2025 at 3:37 AM Avi Fenesh ***@***.***> wrote: When you have glibc 33 a machine with glibc will work because 35 "includes" 33. So we need to build on 20. We need to keep building on the lowest lts, or when building the rust to set the target which is possible with rust. When I develop locally, i change the target to check if it compiles. Another option is keeping the CI on what is available by GH, and running the CD on the ephemerals. In each case we need to take this task and test it now, before we get to a real release. — Reply to this email directly, view it on GitHub <#3218 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABRMQUNJWNPYLRBYT2BAHWL2RV4NZAVCNFSM6AAAAABXRN2RN2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBUGI4DMOBRGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***> [image: avifenesh]*avifenesh* left a comment (valkey-io/valkey-glide#3218) <#3218 (comment)> When you have glibc 33 a machine with glibc will work because 35 "includes" 33. So we need to build on 20. We need to keep building on the lowest lts, or when building the rust to set the target which is possible with rust. When I develop locally, i change the target to check if it compiles. Another option is keeping the CI on what is available by GH, and running the CD on the ephemerals. In each case we need to take this task and test it now, before we get to a real release. — Reply to this email directly, view it on GitHub <#3218 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABRMQUNJWNPYLRBYT2BAHWL2RV4NZAVCNFSM6AAAAABXRN2RN2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBUGI4DMOBRGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[avifenesh]

Also an option