RFC: Slow dynamic linker resolution of shared libraries

[CajuM]

Presently when running a dynamically linked program, it searches its entire rpath for a given DSO with a SONAME. Such as:

nushell> strace -e openat /usr/bin/env out> /dev/null err>| grep 'libc.so'
openat(AT_FDCWD, "/nix/store/nrl5m466dwdscvavf1nqzdnqhszjyy9n-acl-2.3.2/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/nmq6s9l9zlym3qjx76zzrqbp05a63y90-attr-2.5.2/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/nxg637az00fnm705pinlrpjac9b4hgsc-gmp-with-cxx-6.3.0/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/glibc-hwcaps/x86-64-v3/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/glibc-hwcaps/x86-64-v2/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3

This causes unnecessary openat syscalls.

A proposed solution would be to add the -Wl,-soname=$out/lib/libname.so to every invocation of the linker for shared objects.

As an experiment, I compiled a share object with the above flag and compiled a test program linked to the previous shared object. The output was:

nushell> strace -e openat ./main
openat(AT_FDCWD, "$PWD/lib.so", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/glibc-hwcaps/x86-64-v3/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/glibc-hwcaps/x86-64-v2/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
+++ exited with 0 +++

As we can notice, lib.so is found immediately without a lookup in rpath.

If we run:

nushell> readelf -a main | grep lib.so                                                       12/23/2024 06:19:30 PM
 0x0000000000000001 (NEEDED)             Shared library: [$PWD/lib.so]

We notice immediately that SONAME has indeed the expected value.

Note: I replaced every instance of the current working directory with $PWD in the above code snippets for privacy reasons.

[CajuM]

I think adding a patchelf --set-soname to pkgs/build-support/setup-hooks/auto-patchelf.sh might be the easiest way to go.

[CajuM]

Actually, something like patchelf --replace-needed libc.so.6 --replace-needed libc.so.6 ${glibc}/lib/libc.so.6 exe may work better.

[CajuM]

I don't know what weird stuff changing the SONAME of every DSO may cause :)

[CajuM]

I guess patching pkgs/by-name/au/auto-patchelf/source/auto-patchelf.py to move dependencies from --set-rpath to --replace-needed should work. I'm waiting for feedback before making a PR.

[CajuM]

Pinging @layus and @Mic92 for comments.

[layus]

Thanks for pinging me ! Have you read <#24844> already ? Le 23 décembre 2024 18:34:09 GMT+01:00, "Mihai-Drosi Câju" ***@***.***> a écrit :

…

Pinging @layus and @Mic92 for comments. -- Reply to this email directly or view it on GitHub: #367683 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

[Mic92]

The idea to use absolute paths in soname is not new. I think the last time I saw that it being tried, it caused some crashes. In theory it should work but it would require a more intelligent linker wrapper. The wrapper would require to lookup the sonames within the provided linker paths. It would make patchelf for stdenv unnecessary. Also check out this issue: #362353

[CajuM]

The idea to use absolute paths in soname is not new. I think the last time I saw that it being tried, it caused some crashes. In theory it should work but it would require a more intelligent linker wrapper. The wrapper would require to lookup the sonames within the provided linker paths. It would make patchelf for stdenv unnecessary. Also check out this issue: #362353

The crashes that I saw were related to libc.so.6 IIRC. This was the bug and it got fixed: https://sourceware.org/bugzilla/show_bug.cgi?id=22787 . We don't use hwcaps directories, instead we link optimized symbols into libc directly, so we don't need additional rpaths.

An alternative would be a per-app ld.so.cache, it should be "easy" to implement. We could modify the dynamic linker to look for LD_SO_CACHE in an ELF symbol, including DSO's recursively. Adding this to DSO's is required as scripting languages such as Python import DSO's using dlopen. So, each ELF would have this symbol or note and we'd read ld.so.cache's cache path from it or we could just embed it in the symbol/note to avoid unnecessary stats/opens.

Or if we want to take from Guix(https://guix.gnu.org/en/blog/2021/taming-the-stat-storm-with-a-loader-cache/) we can look-up ld.so.cache in $out/etc/ld.so.cache. They use ${ORIGIN}/../etc/ld.so.cache. That would be a bad idea as executables can be in arbitrary locations compared to the root of the package. The root of the package can be easily found by looking-up the next directory after /nix/store in the exec's path this would cause some issues for those who use different store paths.

[CajuM]

Thanks for pinging me ! Have you read <#24844> already ? Le 23 décembre 2024 18:34:09 GMT+01:00, "Mihai-Drosi Câju" @.> a écrit :
…
Pinging @layus and @Mic92 for comments. -- Reply to this email directly or view it on GitHub: #367683 (comment) You are receiving this because you were mentioned. Message ID: @.>

I've just went through it. Sorry that I didn't check if this bug existed already. I got annoyed that Serpent OS used ldconfig and we didn't. I did not check if their implementation did anything innovative. But... knowing them, I guess it was a horrible hack.