[email protected]: Failed to load native module in Linux-x64 platform (redhat ubi8 minimal)

[praveen-em]

Software versions

• OS Image: redhat ubi8 minimal
• Pact Node version: 12.4.0
• Node Version: 18.19.1
• Other Versions: Running cat /etc/os-release:
  NAME="Red Hat Enterprise Linux"
  VERSION="8.9 (Ootpa)"
  ID="rhel"
  ID_LIKE="fedora"
  VERSION_ID="8.9"
  PLATFORM_ID="platform:el8"
  PRETTY_NAME="Red Hat Enterprise Linux 8.9 (Ootpa)"
  ANSI_COLOR="0;31"
  CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
  HOME_URL="https://www.redhat.com/"
  DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8"
  BUG_REPORT_URL="https://bugzilla.redhat.com/"
  REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
  REDHAT_BUGZILLA_PRODUCT_VERSION=8.9
  REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
  REDHAT_SUPPORT_PRODUCT_VERSION="8.9"

Issue Checklist

Please confirm the following:

• I have upgraded to the latest
• I have the read the FAQs in the Readme
• I have triple checked, that there are no unhandled promises in my code
• I have set my log level to debug and attached a log file showing the complete request/response cycle
• For bonus points and virtual high fives, I have created a reproduceable git repository (see below) to illustrate the problem

Expected behaviour

Pact tests should run successfully.

Actual behaviour

Pact tests fail with error Failed to load native module. The problem is similar to #472, but this one is for linux-x64 (redhat ubi8 minimal).

Steps to reproduce

Repo: https://github.com/praveen-em/pact-consumer-js.git (this example is based on the one used in pact js workshop)

This problem occurred initially while trying to run pact tests on Jenkins agents (pods) running on ocp (Openshift Container Platform). Those pods use redhat ubi8 minimal base image. See steps below to recreate the problem locally.

1. git clone https://github.com/praveen-em/pact-consumer-js.git
2. docker buildx build --platform linux/amd64 -t pactdemo:amd . --> This will create image pactdemo:amd
3. docker run --platform linux/amd64 pactdemo:amd --> using the image created in the previous step

Relevant log files

> [email protected] test
> cross-env LOG_LEVEL=trace CI=true react-scripts test

PASS src/api.spec.js
[16:15:19.990] DEBUG (101): [email protected]: We detected your platform as: 

 - linux-x64
FAIL src/api.pact.spec.js
  ● Test suite failed to run

    Failed to load native module, try setting LOG_LEVEL=debug for more info

       8 | const { eachLike, like } = MatchersV3;
       9 |
    > 10 | const provider = new PactV3({
         |                  ^
      11 |   consumer: "FrontendWebsite",
      12 |   provider: "ProductService",
      13 |   log: path.resolve(process.cwd(), "logs", "pact.log"),

      at initialiseFfi (node_modules/@pact-foundation/pact-core/src/ffi/index.ts:113:11)
      at getFfiLib (node_modules/@pact-foundation/pact-core/src/ffi/index.ts:125:11)
      at makeConsumerPact (node_modules/@pact-foundation/pact-core/src/consumer/index.ts:67:24)
      at PactV3.Object.<anonymous>.PactV3.setup (node_modules/@pact-foundation/src/v3/pact.ts:250:33)
      at new PactV3 (node_modules/@pact-foundation/src/v3/pact.ts:49:10)
      at Object.<anonymous> (src/api.pact.spec.js:10:18)

[16:15:20.379] TRACE (101): [email protected]: Initiliasing ffi for the first time
[16:15:20.381] DEBUG (101): [email protected]: Initalising native core at log level 'warn'
[16:15:20.381] DEBUG (101): [email protected]: binding path #0: : attempting to load native module from: 

 - /app/node_modules/@pact-foundation/pact-core/prebuilds/linux-x64 
   source: pact-js-core binding lookup 

 - You can override via PACT_PREBUILD_LOCATION

[16:15:20.387] DEBUG (101): [email protected]: binding path #1: : attempting to load native module from: 

 - /app/prebuilds/linux-x64 
   source: pact-js-core binding lookup 

 - You can override via PACT_PREBUILD_LOCATION

[16:15:20.391] DEBUG (101): [email protected]: Supported platforms are: 
 - darwin-arm64
 - darwin-x64
 - linux-arm64
 - linux-x64
 - win32-x64
[16:15:20.392] ERROR (101): [email protected]: Failed to find native module for linux-x64: TypeError: Cannot read properties of undefined (reading 'pactffiInitWithLogLevel')
[16:15:20.392] DEBUG (101): [email protected]: We looked for a supported build in this location /app/node_modules/@pact-foundation/pact-core/prebuilds/linux-x64
[16:15:20.394] DEBUG (101): [email protected]: We looked for a supported build in this location /app/prebuilds/linux-x64
[16:15:20.394] DEBUG (101): [email protected]: Tip: check there is a prebuild for linux-x64 

          check the path exists

      Wrong Path?: set the load path with PACT_PREBUILD_LOCATION ensuring that $PACT_PREBUILD_LOCATION/prebuilds/linux-x64 exists

      - Note: You dont need to include the prebuilds/linux-x64 part of the path, just the parent directory

      - Let us know: We can add more supported path lookups easily, chat to us on slack or raise an issue on github
Test Suites: 1 failed, 1 passed, 2 total
Tests:       2 passed, 2 total
Snapshots:   0 total
Time:        3.57 s
Ran all test suites.

[YOU54F]

the ffi is being built against ubuntu 20.04 for x86_64 which will link against glibc 2.31. rhel8 is 2.28

we will need to build with cross 0.2.5 for the x86_64 builds, as we are doing for aarch64

change will need making in the rust_ffi release.sh

[praveen-em]

Thanks for quick response @YOU54F . Here is the draft PR. Not sure if this covers everything. I tested locally with a bit of hack using libpact_ffi.so generated from the fix and it worked fine. Some observations and questions.

1. A test was failing (both before and after my change). It appears to be a known issue - related to Binary Content Matching is odd - cross platform/arch #447 and Matching binary payloads does not work consistently across OS/Platform pact-reference#171 ?

thread 'matching::tests::pactffi_matches_binary_value_test' panicked at /Users/praveen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/expectest-0.12.0/src/core/result.rs:81:13:
assertion failed: `expected to be true`, pact_ffi/src/matching.rs:360:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

failures:
    matching::tests::pactffi_matches_binary_value_test

test result: FAILED. 166 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s

2. Not sure if cross prebuild config is required for x64 similar to aarch64 ?

3. Running release-linux script locally couldn't pull cross v0.2.5 which I think is because of this open issue "no matching manifest" on cross v0.2.5 cross-rs/cross#1214. See error below. Have you seen this before? 0.2.4 worked fine locally.

Unable to find image 'ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5' locally
0.2.5: Pulling from cross-rs/x86_64-unknown-linux-gnu
docker: no matching manifest for linux/arm64/v8 in the manifest list entries.

[YOU54F]

0.2.5 of cross only has amd64 images so you’ll need to set docker default platform env var to linux/amd64 main has arm64 images but will link to a later glibc version

[praveen-em]

Looks like using cross 0.2.5 (glibc 2.23) for x86_64 breaks serde which comes with its own precompiled binaries (it uses glibc 2.25).

error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.25' not found (required by /target/debug/deps/libserde_derive-5f72523c3468f1ff.so)
   --> /cargo/registry/src/index.crates.io-6f17d22bba15001f/serde-1.0.198/src/lib.rs:328:1

[YOU54F]

cross seems to pick up dependencies built for other targets, when building, so the resolution I've found is rather destructive, but to perform a cargo clean prior to executing a run with cross

all green here pact-foundation/pact-reference#413 💚

ty for the pr and digging in, appreciate it

[praveen-em]

Excellent! Thanks @YOU54F. Looking forward to a new release.

[mefellows]

Leaving this open until we release the FFI changes in ^^.

Thanks for all of your help everyone!

[YOU54F]

Latest release is now out 14.3.6

https://github.com/pact-foundation/pact-js-core/releases/tag/v14.3.6

Lets test

1. move into a project folder with pact
   1. in my example, I've cloned https://github.com/pactflow/example-consumer-js-kafka
   2. updated the project to use the latest [email protected], which in turn brings in [email protected]
2. Grab a RHEL based docker image and mount your folder in /app
docker run --platform=linux/amd64 -v $PWD:/app --rm -it rockylinux:8
3. Check glibc version

   > rpm -q glibc
   glibc-2.28-236.el8_9.7.x86_64 

4. Install a modern version of node

   > dnf module enable -y nodejs:18
   > dnf module install -y nodejs
   > node -v
   v18.19.1

5. Test our app

   cd /app
   npm install
   npm test

Output

[root@7a1632f1316f app]#    npm test

> [email protected] test
> jest --testTimeout 30000

PASS  src/product/product.repository.test.js
[12:52:17.557] INFO (165): 0.4.20: pact native library successfully found, and the correct version
[12:52:17.620] INFO (165): [email protected]: Verifying message
2024-05-09T12:52:17.652716Z  WARN ThreadId(01) pact_models::pact: Note: Existing pact is an older specification version (V3), and will be upgraded
PASS  src/product/product.handler.pact.test.js
 ● Console

   console.log
     received product: {
       event: 'UPDATED',
       id: 'some-uuid-1234-5678',
       name: 'Some Product',
       type: 'Product Range',
       version: 'v1'
     }

     at log (src/product/product.handler.js:5:11)


Test Suites: 2 passed, 2 total
Tests:       2 passed, 2 total
Snapshots:   0 total
Time:        3.249 s

[praveen-em]

Thank you, the new version works fine.