Ship libdcsctp4j.so for ppc64le

[fitzsim]

While verifying jitsi/jitsi-srtp#49 I noticed that dcsctp4j does not yet have native objects for ppc64le:

$ dpkg -S /usr/share/jitsi-videobridge/lib/jitsi-dcsctp-1.0-3-gaf9d564.jar
jitsi-videobridge2: /usr/share/jitsi-videobridge/lib/jitsi-dcsctp-1.0-3-gaf9d564.jar
$ dpkg -l | grep jitsi-videobridge2
ii  jitsi-videobridge2  2.3-150-gb67f0c0d-1  all  WebRTC compatible Selective Forwarding Unit (SFU)
$ jar tf /usr/share/jitsi-videobridge/lib/jitsi-dcsctp-1.0-3-gaf9d564.jar | grep '.so$'
linux-aarch64/libdcsctp4j.so
linux-x86-64/libdcsctp4j.so

Are there plans to include ppc64le binaries at some point?

[JonathanLennox]

Unfortunately, the dcsctp library comes from Google's libwebrtc, which does not support building on PPC.

I know that Debian has a patchset to build libwebrtc on PPC, but I don't know how hard it would be to adapt to the build process we're using, which pulls from the main Google repo.

[fitzsim]

Ah, I see, thanks. Is there an upstream feature request for ppc64le support we can link to here? If `dcsctp4j` is mandatory for Jitsi now, I will try to figure out the best way to make it work on Debian ppc64le, in lieu of upstream support.

[aaronkvanmeerten]

As a note, it is still possible to use the usrsctp library via user configuration options, so we suggest users who require both SCTP and PPC use that method instead of dcsctp.

[JonathanLennox]

I think the WebRTC issue to support ppc is at https://issues.webrtc.org/issues/42224432, though it's currently marked "wontfix".

[fitzsim]

Thank you for the link; that looks like a patch for big-endian support. But the WebRTC maintainer seems receptive to a working patch for ppc64 and/or ppc64le. Jitsi would be a good "test case" to help validate a proposed upstream patch. (Not suggesting Jitsi maintainers work on this, just pointing it out to others who may read this.) Also good to know about the `usrsctp` configuration option, thank you.

[tpearson-ssc]

I'm the maintainer of the Debian patchset; we just hit this on multiple servers due to the Jitsi change to dcsctp. How can I help with support?

[fitzsim]

I am trying to get Jitsi working again on ppc64le, and I have now hit:

java.lang.NoClassDefFoundError: Could not initialize class org.jitsi.dcsctp4j.DcSctpOptions
[...]
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.UnsatisfiedLinkError: no dcsctp4j in java.library.path

I am attempting to build the dcsctp4 native binaries from source, and I hit:

$ resources/ubuntu-build-all.sh ~/DepotTools ~/WebRTC
[...]
/src/dcsctp4j/resources/checkout-webrtc.sh: line 54: fetch: command not found

At this point I will attempt to hack my checkout to build against the Debian WebRTC development package, so as to avoid the large checkout of a Google repository and to avoid the need to install (apparently custom?) fetch and gclient commands, which I do not have already-installed.

[JonathanLennox]

fetch and gclient are from Google's depot_tools, and are their standard ways of checking out the source to the webrtc native libraries.

checkout-webrtc.sh should have checked them out and added them to your PATH, so I'm surprised it's not finding them, assuming it didn't fail. Can you run with bash -x resources/ubuntu-build-all.sh ~/DepotTools ~/WebRTC and see if it's failing?

That said, the WebRTC library as shipped by Google doesn't support ppc64le, so the script ultimately won't work for you anyway. You may be able to build it using the Debian patched version, but I'm not sure.

Our build process automatically rebuilds the libraries on our build machines at each commit to main, so to get ppc support into main, we'd need to make it possible to apply the patches automatically, including being able to cross-build (since we don't have PPC build machines).

Given all that, a simpler fix would be to just change the default jitsi-videobridge configuration to use usrsctp rather than dcsctp on ppc64le, since that's still supported.

[fitzsim]

OK, thank you @JonathanLennox. Are you able to paste the configuration fragment required to use usrsctp instead?

Separately, if dcsctp4j is the "way forward" then I also would like to draw a line around the codee changes required for ppc64le, but I cannot find a standalone Debian libwebrtc package. I did find libtgowt-dev which has some of the correct headers, and does build some files successfully, but the build ultimately fails I guess because it is not the precise version required by libdcsctp4j.so.

Can someone please paste a URL to the libwebrtc Debian ppc64le patch set? I searched the web and Debian repositories but failed to find it. Thank you for the checkout-webrtc.sh advice; I am not sure why it failed, but I skipped over it for the reasons you pointed out, and am instead trying the approach of making the ppc64le library and headers available to the dcsctp4j build.

[JonathanLennox]

This pull request should automatically modify the videobridge config file on Debian/ppc64le to use usrsctp: jitsi/jitsi-videobridge#2214. Please let me know if that works, or if you see any issues with it.

If you want to modify the config file manually, the hocon config path is videobridge.sctp.use-usrsctp; set it to true.

The Debian patches for libwebrtc seem to be part of the Debian Chromium package. See https://salsa.debian.org/chromium-team/chromium/-/tree/master/debian/patches/ppc64le?ref_type=heads . (I think these patches may only support native ppc64le builds, though, not cross-compilation?)

[fitzsim]

Thank you! When I get the next chance to work on my configuration I will try installing Jitsi from scratch from the nightly apt source and let you know if the hocon logic works on ppc64le.

[tpearson-ssc]

Our build process automatically rebuilds the libraries on our build machines at each commit to main, so to get ppc support into main, we'd need to make it possible to apply the patches automatically, including being able to cross-build (since we don't have PPC build machines).

Given all that, a simpler fix would be to just change the default jitsi-videobridge configuration to use usrsctp rather than dcsctp on ppc64le, since that's still supported.

I'm very nervous about this; if we are the only user of usrsctp support could be dropped at any point with no warning. If I can assign an individual or team to maintain the ppc64el patches for dcsctp on your infrastructure (or provide native ppc64el build infrastructure) is that an option?

[fitzsim]

SBCL recently pushed a commit to use GitHub Actions to build on Debian ppc64le using QEMU: sbcl/sbcl@062128c. Maybe that approach could work for dcsctp4j's continuous integration.

[madscientist159]

Any progress on getting this to build? We can offer free access to a build server to help this along. Thanks!

[JonathanLennox]

As mentioned, the main blocker is getting libdcsctp from webrtc to build for ppc64le (ideally, cross-compiled from arm or x86).

Once that's done, compiling Jitsi's JNI wrappers for ppc64le should be trivial.

[fitzsim]

I attempted a native build on ppc64le. The checkout, at ~17 GiB almost filled up the rest of my build partition. Has anyone attempted to write a simple Makefile for WebRTC? I guess the feasibility depends on the depth of gn's logic.

+ gclient sync -r branch-heads/6422 -D
Errors:
  failed to resolve infra/3pp/tools/cpython3/linux-ppc64le@version:[email protected] (line 21): no such package: infra/3pp/tools/cpython3/linux-ppc64le
/build/depot_tools/bootstrap_python3: line 32: [email protected]_bin/python3/bin/python3: No such file or directory
Bundled Python 3.11 not found. Use VPYTHON_BYPASS if prebuilt cpython not available on this platform: open /build/depot_tools/.cipd_bin/.cipd/pkgs/0/1hf_0x7pYYqUEtV3epA4lxyXER3MHmqexkZ7bc0QXPIC/3.11/.versions/cpython3.cipd_version: no such file or directory

This got me further:

export VPYTHON_BYPASS="manually managed python not supported by chrome operations"

The directory basename needs to be depot_tools, not DepotTools. I did this:

resources/ubuntu-build-all.sh /build/depot_tools /build/WebRTC

There were no instructions about gclient configuration, so I guessed:

cd /build/WebRTC
/build/depot_tools/gclient config https://chromium.googlesource.com/external/webrtc/

After which the build attempt produces:

cd /build/dcsctp4j
resources/ubuntu-build-all.sh /build/depot_tools /build/WebRTC
Picked up _JAVA_OPTIONS: -Dawt.useSystemAAFontSettings=on
[INFO] Scanning for projects...
[INFO] 
[INFO] -----------------------< org.jitsi:jitsi-dcsctp >-----------------------
[INFO] Building Jitsi Java DCSCTP library 1.0-SNAPSHOT
[INFO] -------------------------------[ bundle ]-------------------------------
[INFO] 
[INFO] --- maven-resources-plugin:3.3.1:resources (default-resources) @ jitsi-dcsctp ---
[INFO] skip non existing resourceDirectory /build/dcsctp4j/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.12.1:compile (default-compile) @ jitsi-dcsctp ---
[INFO] Nothing to compile - all classes are up to date.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.906 s
[INFO] Finished at: 2025-02-04T13:04:12-05:00
[INFO] ------------------------------------------------------------------------
Errors:
  failed to resolve infra/3pp/tools/cpython3/linux-ppc64le@version:[email protected] (line 21): no such package: infra/3pp/tools/cpython3/linux-ppc64le
/build/depot_tools/bootstrap_python3: line 32: [email protected]_bin/python3/bin/python3: No such file or directory
Errors:
  failed to resolve infra/3pp/tools/cpython3/linux-ppc64le@version:[email protected] (line 21): no such package: infra/3pp/tools/cpython3/linux-ppc64le
/build/depot_tools/bootstrap_python3: line 32: [email protected]_bin/python3/bin/python3: No such file or directory
Syncing projects: 100% (52/52), done.                  

________ running 'cipd ensure -log-level error -root /build/WebRTC -ensure-file /tmp/tmp30h6f5v0.ensure' in '.'
Errors:
  failed to resolve gn/gn/linux-ppc64le@git_revision:d823fd85da3fb83146f734377da454473b93a2b2 (line 5): no such package: gn/gn/linux-ppc64le
  failed to resolve infra/rbe/client/linux-ppc64le@re_client_version:0.138.0.e854224-gomaip (line 8): no such package: infra/rbe/client/linux-ppc64le
  failed to resolve infra/3pp/tools/ninja/linux-ppc64le@version:[email protected] (line 11): no such package: infra/3pp/tools/ninja/linux-ppc64le
Error: Command 'cipd ensure -log-level error -root /build/WebRTC -ensure-file /tmp/tmp30h6f5v0.ensure' returned non-zero exit status 1
Errors:
  failed to resolve gn/gn/linux-ppc64le@git_revision:d823fd85da3fb83146f734377da454473b93a2b2 (line 5): no such package: gn/gn/linux-ppc64le
  failed to resolve infra/rbe/client/linux-ppc64le@re_client_version:0.138.0.e854224-gomaip (line 8): no such package: infra/rbe/client/linux-ppc64le
  failed to resolve infra/3pp/tools/ninja/linux-ppc64le@version:[email protected] (line 11): no such package: infra/3pp/tools/ninja/linux-ppc64le

I am not sure how to proceed from here.

One idea: can I download, from an x86-64 continuous integration run, the Makefiles generated via gn -> ninja -> CMake? Then I could check them for non-build-tool architecture dependence.

Does Jitsi build dcsctp4j using the gn-installed tools or Ubuntu's tools?

[madscientist159]

@fitzsim Thanks for trying that. If I had to guess, I'd say the Google tooling failures are spurious, the real problem is the build isn't even attempted. Looking in ubuntu-build-all.sh the arches are hard coded, I'd try editing line 4 to include only ppc64le and see if that helps anything. ubuntu-build.sh will also need updates to add environment variable setup for ppc64le; some of the hints in the old Chromium Wiki notes may help [1]. Note the actual live Chromium patches for new versions are maintained elsewhere [2], though I don't think they are needed here at least at a first glance.

As far as CI/CD goes, Raptor is willing to donate space on its Chromium public build hosts to support the dcsctp4j builds if there is a way to tie that in with the rest of the project infrastructure.

[1] https://wiki.raptorcs.com/wiki/Porting/Chromium_(New)
[2] https://gitlab.raptorengineering.com/raptor-engineering-public/chromium/openpower-patches

[fitzsim]

I got libdcsctp4j.so working on ppc64le.

jar tf /usr/share/jitsi-videobridge/lib/jitsi-dcsctp-1.0-3-gaf9d564.jar | grep ppc64le/lib
linux-ppc64le/libdcsctp4j.so

JVB 2025-02-05 05:46:34.270 INFO: [49] DcSctp4j.<clinit>#38: DcSctp4j lib loaded

I wrote a small Makefile and README and included them in a tarball with the subset of WebRTC source code (revision branch-heads/6422) required to build libdcsctp.a. As far as I can tell, the Google tools and ~17 GiB of downloads are used only to get this ~32MiB of source code and build libdcsctp.a to be consumed by libdcsctp4j.so. @madscientist159, can you confirm you can build libdcsctp.a on a ppc64le system using just this tarball?

Once you have libdcsctp.a built, you just need to hack the ubuntu-build-all.sh and ubuntu-build.sh scripts to assume ppc64le, comment out the fetch and gn steps, and point CMakeLists.txt at the newly-built libdcsctp.a.

[fitzsim]

Thank you for reviewing and merging #9, @JonathanLennox. Let me know if I can help with other pull requests, otherwise I will watch for a jitsi-dcsctp update in the https://download.jitsi.org unstable repository.

[fitzsim]

I just tested jitsi/jitsi-videobridge#2291 via the https://download.jitsi.org unstable/ apt repository and I can confirm that jitsi-videobridge2 2.3-209-gb5fbe618-1 fixes this issue on my Jitsi setup.

JVB 2025-02-24 15:59:07.874 INFO: [50] DcSctp4j.<clinit>#38: DcSctp4j lib loaded

@madscientist159, can you confirm it works on your Jitsi installation? Then we can close this issue.

[tpearson-ssc]

@fitzsim Responding from my SSC work account as that is where the newer Jitsi version is available for testing. I was actually able to use the upstream build script directly with a few modifications, and the library looks good.

For anyone else following, here's how I did that.

First, there's an error in the script that probably affects all distributions. ./resources/ubuntu-build-all.sh will only accept depot_tools as the name for the depot tools folder, any other name will fail due to the logic in the script.

gclient sync will raise a false positive failure and exit due to not being able to find Google binaries for various dependencies. Since random downloaded binaries are considered a security risk in many environments, including ours, I just added || true to the last line in resources/checkout-webrtc.sh

From there, it's a simple matter of running:

export VPYTHON_BYPASS="manually managed python not supported by chrome operations"
./resources/ubuntu-build-all.sh ~/depot_tools ~/WebRTC

Eventually you should see

[100%] Built target dcsctp4j
Install the project...
-- Install configuration: "RelWithDebInfo"
-- Installing: dcsctp4j/src/main/resources/linux-ppc64le/./libdcsctp4j.so

Library linkage if anyone is curious

ldd dcsctp4j/src/main/resources/linux-ppc64le/./libdcsctp4j.so
        linux-vdso64.so.1 (0x00007fff9bfc0000)
        libstdc++.so.6 => /lib/powerpc64le-linux-gnu/libstdc++.so.6 (0x00007fff9ba00000)
        libgcc_s.so.1 => /lib/powerpc64le-linux-gnu/libgcc_s.so.1 (0x00007fff9be60000)
        libc.so.6 => /lib/powerpc64le-linux-gnu/libc.so.6 (0x00007fff9b600000)
        /lib64/ld64.so.2 (0x00007fff9bfd0000)
        libm.so.6 => /lib/powerpc64le-linux-gnu/libm.so.6 (0x00007fff9bd30000)