Monolibtic koreader library

[benoit-pierre]

After fixing the libraries sonames on Android in #1907 (to fix koreader/koreader#12348), the dynamic linker on API level 18 (and 19 too I assume) craps out during symbol resolution (cf. comment).

The only way I could get around those type of issues on my meson branch was to bundle (almost) everything into one big library, and this PR does just that.

At first I tried a partial solution, not bundling in native Lua modules, but of course this just resulted in the dynamic loader having another symbol resolution fit when attempting to load common/ssl.so… So you get the full monty, native Lua modules included, no shared objects in assets.7z. The final list of native libraries is:

    71864  1981-01-01 01:01   lib/arm64-v8a/lib7z.so
  1014704  1981-01-01 01:01   lib/arm64-v8a/libc++_shared.so
     4728  1981-01-01 01:01   lib/arm64-v8a/libioctl.so
 18639648  1981-01-01 01:01   lib/arm64-v8a/libkoreader-monolibtic.so
    19584  1981-01-01 01:01   lib/arm64-v8a/libluajit-launcher.so
   689520  1981-01-01 01:01   lib/arm64-v8a/libluajit.so
  1908280  1981-01-01 01:01   lib/arm64-v8a/libsdcv.so

For now, monolibtic mode is only enabled for Android (but I tested the emulator and kindlepw2 builds too).

On Android, there's no need for the sdcv and libs symlinks in the application files directory anymore: both executable and libraries are directly loaded from the application native libraries directory (fixing this corner case).

Implementation

• build a bunch of static libraries
• link everything, taking advantage of utils/gen_linker_exports.sh and our FFI cdecls to only keep what we need
• use ffi.loadlib(…) instead of ffi.load to load our libraries (and a custom package loader to handle bundled native Lua modules)
• required changes in main project: koreader/koreader@master...benoit-pierre:koreader:4921e1f
• required changes in android-luajit-launcher: koreader/android-luajit-launcher@master...benoit-pierre:android-luajit-launcher:monolibtic

ffi.loadlib module

Apart from a host of commits to facilitate bundling everything, another big change is the ffi.loadlib helper, replacing ffi.load and util.ffiLoadCandidates for loading our libraries.

Example use:

function util.loadSDL2()
    if libSDL2 == nil then
        local ok
        ok, libSDL2 = pcall(ffi.loadlib,
            "SDL2-2.0", 0,
            "SDL2-2.0", nil,
            "SDL2", nil
        )
        if not ok then
            print("SDL2 not loaded:", libSDL2)
            libSDL2 = false
        end
    end
    return libSDL2 or nil
end

On Linux, the code will attempt to load libSDL2-2.0.so.0 in libs/ and then from the system, then libSDL2-2.0.so if that failed, and fallback to trying libSDL2.so.

In monolibtic mode, loading a bundled library will ffi.load the monolibtic library. A custom package loader handle the require("libs/libkoreader-lfs") case.

Code size

Impact on code size (bss+data+text, ignoring executables and the cURL library):

build	master	monolibtic
Android (ARM)	20.0 MB	15.1 MB (-4.9 MB, -24.5 %)
Android (ARM64)	28.2 MB	21.2 MB (-7.0 MB, -24.8 %)
Android (x86)	28.8 MB	25.0 MB (-3.8 MB, -13.2 %)
Android (x86_64)	31.5 MB	27.1 MB (-4.4 MB, -14.0 %)
AppImage	30.7 MB	26.7 MB (-4.0 MB, -13.0 %)
Kindle PW2	18.7 MB	16.1 MB (-2.7 MB, -13.9 %)

Impact on memory use: a slight increase at startup (starting with the FM, +0-2 MB), but a reduction after opening a few documents of different types (-2-8 MB).

TODO

If the solution is accepted, split the change into smaller PRs for merging: preliminary cleanups, ffi.loadlib module / helper, FFI cdecls cleanups, cURL static build, other static libraries support, final monolitic commit.

---

This change is 

[poire-z]

For now, monolibtic mode is only enabled for Android

I don't mind for Android, but I hope this won't go further - I would like to keep my kobo nightlies with smaller libraries I can quickly compile independantly without having to rebuild the whole thing.

(And on principle, I don't like the idea of a single monolithic library, but what do I know.)
Also, I remember that when I tested it months ago on my Kobo, it wasn't at all faster than a current nightly.

[benoit-pierre]

Test builds:

• Android ARM
• Android ARM64
• Android x86
• Android x86_64
• AppImage
• Kindle PW2

[NiLuJe]

No strong opinions about it as long as it stays Android-specific in practice ;).

[pazos]

It would be nice to have this on the next release. Android 18-22 seem broken since the unversioning.

[benoit-pierre]

It's not the unversioned libraries that caused the issue, but one or the combination of the libk2pfopt / mupdf / tesseract / wrap-mupdf optimizations (statically building / merging some of them).

[NiLuJe]

Reviewed 58 of 58 files at r1, all commit messages.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @benoit-pierre)

---

thirdparty/cmake_modules/koreader_thirdparty_libs.cmake line 199 at r1 (raw file):

target_link_libraries(pthread INTERFACE Threads::Threads)

# srell

Wrong comment for SQLite?

[benoit-pierre]

thirdparty/cmake_modules/koreader_thirdparty_libs.cmake line 199 at r1 (raw file):

target_link_libraries(pthread INTERFACE Threads::Threads)

# srell

Wrong comment for SQLite?

Yep, good catch.

[NiLuJe]

Reviewed 1 of 1 files at r2, all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @benoit-pierre)