building static libraries with shared library wrapper #105
Conversation
CMakeLists.txt
Outdated
| ############ | ||
| # Boost | ||
| ############ | ||
|
|
||
| set(Boost_NO_BOOST_CMAKE 1) | ||
| set(Boost_USE_MULTITHREADED ON) | ||
| set(Boost_USE_STATIC_LIBS ON) |
There was a problem hiding this comment.
On Ubuntu 20.0 and 22.04, I get this error with this field set to ON:
[100%] Linking CXX shared library libmonero-java.so
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libboost_serialization.a(basic_iarchive.o): warning: relocation against `_ZTVN5boost7archive6detail14basic_iarchiveE' in read-only section `.text'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libboost_chrono.a(chrono.o): relocation R_X86_64_PC32 against symbol `_ZN5boost6system6detail10cat_holderIvE24system_category_instanceE' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
But it works when set to OFF. Anything wrong with leaving it off?
There was a problem hiding this comment.
Hm, if I set it to OFF, it builds successfully, but the libraries built on Ubuntu 20.04 still can't be run on Ubuntu 22.04, showing an error from libboost_chrono.so.
There was a problem hiding this comment.
I think boost is compiled without -fPIC on ubuntu. I will look into libboost_chrono.so error.
There was a problem hiding this comment.
Yeah, maybe it's not possible to build libs compatible with both versions then.
There was a problem hiding this comment.
@woodser , does the actual build only run on Ubuntu 20.04?
Shared libs have different versions on Ubuntu 20.04 and Ubuntu 22.04.
Ex:
ubuntu-20-04:~$ ls /lib/x86_64-linux-gnu/libssl.*
/lib/x86_64-linux-gnu/libssl.a /lib/x86_64-linux-gnu/libssl.so /lib/x86_64-linux-gnu/libssl.so.1.1
ubuntu-22-04:~$ ls /lib/x86_64-linux-gnu/libssl.*
/lib/x86_64-linux-gnu/libssl.a /lib/x86_64-linux-gnu/libssl.so /lib/x86_64-linux-gnu/libssl.so.3
There was a problem hiding this comment.
I don't know that the same build has ever worked on both platforms.
Was hoping the static libs with a shared library wrapper (for JNI) would be cross compatible.
There was a problem hiding this comment.
Hi @woodser .
I got haveno working with native wallet on ubuntu 20 and 22 but I have had to:
- use openssl and unbound static lib
- clone boost (1.71.0) and build it static with -fPIC (maybe it will become a module in monero-java)
- disable Trezor hardware wallet support on monero build (USE_DEVICE_TREZOR=OFF). Because I have had problems to run haveno with protobuf built static with -fPIC. Is it a problem?
OBS: I built monero/monero-cpp/monero-java on ubuntu 20.
There was a problem hiding this comment.
Its ok to go in this way?
Yes I think we may have to go this way if we want this static/shared library wrapper idea to work, since the goal is to make the libraries standalone and supported in JNI. Maybe this is the best way?
There was a problem hiding this comment.
Yes I think we may have to go this way if we want this static/shared library wrapper idea to work, since the goal is to make the libraries standalone and supported in JNI. Maybe this is the best way?
I think so. This is done with the last commit.
PS. I only tested this on linux. I dont have how testing on others platforms.
|
I've applied this PR and woodser/monero-cpp#64. Looks promising, but I'm having trouble building it. On Ubuntu 20.04, I first get this error: Setting Then the build fails during linking: On macOS, I get this error: Changing |
|
Hoping the errors can be resolved because it would be really great to build to portable, static libraries. :) |
This is done in bin/build_libmonero_java.sh
OpenSSL will have to become a module built with -fPIC. I will look into this.
I will look into this |
@woodser , could you try to build again on Ubuntu 20.04 aarch64-linux-gnu? |
|
I was able to build successfully on Ubuntu 20.04 aarch64-linux-gnu, but I'm still having trouble building on macOS, first failing on this error: I'm confused why the build should work at all, including on Ubuntu, because By modifying CMakeLists.txt to |
|
I see that monero-cpp is using boost and openssl with these paths instead of the new submodules, wondering if that could be a problem: Perhaps boost and openssl should be included within ./external/monero-cpp/external instead and used to build monero-cpp as well? |
It works because you have boost installed. You are write, I have fixed boost include path. |
Whats the boost version installed on macOS? |
I have boost 1.84 installed through brew.
Wondering again then if boost and openssl should be a submodule within external/monero-cpp/external, which both monero-java and monero-cpp build from? This would be consistent with how monero-ts works, where boost [1] and openssl [2] are downloaded to external/monero-cpp/external. |
Ok. I will try. But monero-project also use boost and openssl and maybe we dont have control of it. |
Maybe it's not necessary, and the boost versions merely need adjusted to be compatible, but I would think monero-cpp should be built with the same version as monero-java.
Don't know if the |
|
@woodser could you try to build on macos again? |
|
Hm, I still end up with these errors building monero-java with the new changes to this and monero-cpp: I confirmed that the build succeeds from the master branches. |
|
Could you past here or send me the full build output? I want to know if build is using boost lib from submodule or system. |
|
|
@woodser could you try again on macos and put the log here please? |
|
It's failing to build boost with the latest changes to monero-java and monero-cpp: |
|
This is the error: |
|
Its related with the newer macOS plus boost 1.7x. See bambulab/BambuStudio#3957 |
|
Yeah I saw that. I just updated my OS to the latest version, but it didn't help. Using 1.8x sounds good, it's what I've been using in the libraries anyway. I don't have any experience emulating mac in linux unfortunately. |
|
Build failed on ubuntu-20.04 with boost 1.86. We need to update monero-project. See monero-project/monero#9313 |
|
Ok, I'm in the process of updating to monero-project v0.18.3.4 now. |
|
Ok it's all updated to monero-project v0.18.3.4. |
|
@woodser , build ok on ubuntu-20.04. Could you try again on macos? |
|
Sure, please resolve the merge conflict with master. :) |
|
With the PRs applied to monero-java and monero-cpp (and the merge conflict resolved), and using the latest monero-project v0.18.3.4 (pushed to my master at https://github.com/woodser/monero), I get these errors on building monero-project: Here's the full build log: monero-project succeeds to build if I run I confirmed that the build succeeds by switching to the latest master branch of all 3 projects. To ensure the build is completely fresh, I first deleted the |
|
@woodser , any problem keep on without hardware wallet support? Because this has new dependencies that we will have to build with static and fPIC. |
|
Ideally we can keep existing hardware wallet dependencies, unless they are not already included? In which case, no need to add them as part of this PR. |
No. if we keep existing hardware wallet dependencies (libs linked to libmonero-java.so), the host running monero-java application must have dependencies installed on. For example, haveno running on host with no hidapi lib installed and using monero-java (this PR) build with hidapi linked (your patch) gets this error: At the moment there is just this problem, mac build needs hidapi. I didnt understand why because |
|
I've released updates to monero-java and monero-cpp, and I'd like to retest this PR with the new updates. Maybe it could have fixed the previous error we saw? But can you rebase your PRs on master, please? |
|
Ok, I will do this and move openssl, boost and sodium build to monero-cpp. |
nsec1
force-pushed
the
sharedLibraryWrapper
branch
from
99925e5
to
bca404b
Compare
|
Done.
|
pom.xml
Outdated
| @@ -4,7 +4,7 @@ | |||
| <modelVersion>4.0.0</modelVersion> | |||
| <groupId>io.github.woodser</groupId> | |||
| <artifactId>monero-java</artifactId> | |||
| <version>0.8.33</version> | |||
| <version>0.8.33x</version> | |||
There was a problem hiding this comment.
Any reason to change the version number now? Normally I bump it just before release.
nsec1
force-pushed
the
sharedLibraryWrapper
branch
from
1ad252e
to
1e5b6c9
Compare
|
I have your latest changes applied to monero-java, monero-cpp, and monero-project, but I'm getting this linker error at the end of the build process: Here's the full build.log: build.log I confirmed that the build is working from the master branches and my release-v0.18 branch in monero-project, which includes a couple of custom commits. Confirming this change should still be applied to monero-project? Also I was wondering, is it necessary to checkout all of boost's submodules? There are so many of them :) |
It's the include path conflict. Monero project is using
Yes. It resolves include path conflict for zmq and sodium Could you apply this patch to monero-project and build again? It is for printing INCLUDE_DIRECTORIES list.
Did you use my monero fork? I can fork yours and apply my change.
It is out of my control, comes from git module update. I would have to track each module with its version. it doesn't worth. |
Yes please, if you can please submit a PR to my release branch here: https://github.com/woodser/monero/tree/release-v0.18. That's now the default for the libraries, since that's what they're based on. |
|
I applied your patch manually (it wasn't included in your PR). Here's the resulting log file: build.log |
It is for debug only, can be ignored. I make a fix on monero-cpp build script. |
|
I pulled in your commit to monero-cpp. Looks like I'm getting the same error, here's the full build log: build.log |
|
I don't know what to do. Monero project is using boost from |
|
I was able to build and run the libs, by uninstalling boost through brew, because as you said, it overrides and conflicts with the build boost. I find that I must build with |
| @@ -1,3 +1,4 @@ | |||
| [submodule "external/monero-cpp"] | |||
| path = external/monero-cpp | |||
| url = https://github.com/woodser/monero-cpp | |||
| url = https://github.com/nsec1/monero-cpp.git | |||
| branch = buildStaticOption | |||
There was a problem hiding this comment.
Can this branch be removed? We can assume your PR to monero-cpp is applied during the build process (as all PRs will be accepted at once).
There was a problem hiding this comment.
Yes, it can. It just makes tests easier. I will do this.
| if(STATIC AND NOT IOS) | ||
| if(UNIX) | ||
| set(OPENSSL_LIBRARIES "${OPENSSL_LIBRARIES};${CMAKE_DL_LIBS};${CMAKE_THREAD_LIBS_INIT}") | ||
| option(USE_DEVICE_TREZOR "Trezor hardware wallet suport" ON) |
There was a problem hiding this comment.
I get linker errors building with USE_DEVICE_TREZOR=OFF related to HIDAPI.
Can we remove this conditional and always find the package?
There was a problem hiding this comment.
@woodser other than this comment, is this PR ready to merge?
| export HOST_NCORES=${HOST_NCORES-$(nproc 2>/dev/null || shell nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 1)} | ||
| BUILD_DIR="$(pwd)/build" | ||
| export CMAKE_PREFIX_PATH="$(pwd)/external/monero-cpp/build/install"${CMAKE_PREFIX_PATH+:$CMAKE_PREFIX_PATH} | ||
| export USE_DEVICE_TREZOR=OFF |
There was a problem hiding this comment.
Can we remove this or does it cause a problem on your system?
There was a problem hiding this comment.
if removed, trezor deps would become dynamically linked an break the concept of "library wrapper". The problem begun with protobuf static -fPIC building. Follow the historic:
|
@woodser , just thoughts, why not make monero-java by passing monero-cpp and wallet2, and connect directly with monero daemon rpc? It would be java only, no jni, no portable builds. |
monero-cpp and wallet2 would still need to be built native for the OS, and monero-java communicates with monero-cpp over JNI which also uses C++, so I don't see how this would circumvent native builds. |
monero-java would not use monero-cpp nor wallet2, no C++, no jni, just java communicates directly to monero daemon rpc using http. Maybe have to re-implement some wallet2 features but maybe its worth. |
|
@nsec1 the concern is speed. Statically linked binaries run far faster than any RPC. |
Ok but wallet2 communicates to node daemon over rpc |
Ref. #74 haveno-dex/haveno#795
Now, there is only monero-java.so. It wraps monero, monero-cpp, boost etc.
Working with haveno on linux (--useNativeXmrWallet=true).
Depends woodser/monero-cpp#64