Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] add ROOT #7666

Closed
wants to merge 22 commits into from
Closed

[WIP] add ROOT #7666

wants to merge 22 commits into from

Conversation

Moelf
Copy link
Contributor

@Moelf Moelf commented Nov 13, 2023

@grasph this is the first step to making ROOT.jl usable again.

I've been using https://archlinux.org/packages/extra/x86_64/root/ as a reference for dependency and make dependency

Currently:

# you need BinaryBuilder.jl
julia --project=@hep build_tarballs.jl
-- Not building amdgpu-arch: hsa-runtime64 not found
CMake Error at tools/clang/CMakeLists.txt:556 (add_subdirectory):
  add_subdirectory given source "test" which is not an existing directory.


-- Registering Bye as a pass plugin (static build: OFF)
CMake Error: File /workspace/srcdir/root/interpreter/llvm/src/utils/lit/tests/lit.site.cfg.in does not exist.
CMake Error at cmake/modules/AddLLVM.cmake:1698 (configure_file):
  configure_file Problem configuring file
Call Stack (most recent call first):
  utils/lit/CMakeLists.txt:4 (configure_lit_site_cfg)


CMake Error at CMakeLists.txt:1022 (add_subdirectory):
  add_subdirectory given source "test" which is not an existing directory.


CMake Error at CMakeLists.txt:1023 (add_subdirectory):
  add_subdirectory given source "unittests" which is not an existing
  directory.


CMake Error at CMakeLists.txt:1025 (add_subdirectory):
  add_subdirectory given source "utils/unittest" which is not an existing
  directory.


-- Failed to find LLVM FileCheck
-- Version: 0.0.0
-- Performing Test HAVE_CXX_FLAG_STD_CXX11
-- Performing Test HAVE_CXX_FLAG_STD_CXX11 - Success
-- Performing Test HAVE_CXX_FLAG_WALL
-- Performing Test HAVE_CXX_FLAG_WALL - Success
-- Performing Test HAVE_CXX_FLAG_WEXTRA
-- Performing Test HAVE_CXX_FLAG_WEXTRA - Success
-- Performing Test HAVE_CXX_FLAG_WSHADOW
-- Performing Test HAVE_CXX_FLAG_WSHADOW - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC
-- Performing Test HAVE_CXX_FLAG_PEDANTIC - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS - Success
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32 - Failed
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL - Success
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_FNO_EXCEPTIONS
-- Performing Test HAVE_CXX_FLAG_FNO_EXCEPTIONS - Success
-- Performing Test HAVE_CXX_FLAG_WNO_SUGGEST_OVERRIDE
-- Performing Test HAVE_CXX_FLAG_WNO_SUGGEST_OVERRIDE - Success
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WD654
-- Performing Test HAVE_CXX_FLAG_WD654 - Failed
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY - Failed
-- Performing Test HAVE_CXX_FLAG_COVERAGE
-- Performing Test HAVE_CXX_FLAG_COVERAGE - Success
-- Performing Test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile
-- Performing Test HAVE_POSIX_REGEX
-- Performing Test HAVE_POSIX_REGEX
-- Performing Test HAVE_POSIX_REGEX -- success
-- Performing Test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK -- success
CMake Error at CMakeLists.txt:1165 (add_subdirectory):
  add_subdirectory given source "benchmarks" which is not an existing
  directory.

R/ROOT/build_tarballs.jl Outdated Show resolved Hide resolved
R/ROOT/build_tarballs.jl Outdated Show resolved Hide resolved
@Moelf
Copy link
Contributor Author

Moelf commented Nov 13, 2023

we could try to build with

-Dminimal=ON

Enable only required options by default

@Moelf
Copy link
Contributor Author

Moelf commented Nov 13, 2023

https://buildkite.com/julialang/yggdrasil/builds/6574#018bc96b-a76b-411b-b95e-a47be7b32c5b/6-5146 [ 52%] Built target root

but then
https://buildkite.com/julialang/yggdrasil/builds/6574#018bc96b-a76b-411b-b95e-a47be7b32c5b/6-5186

is this some weird stuff ROOT is doing that re-run CMake with another set of configs?

@giordano
Copy link
Member

Sounds like it's building LLVM? We should be using our own LLVM.

@Moelf
Copy link
Contributor Author

Moelf commented Nov 13, 2023

is LLVM_jll all we need?

@giordano
Copy link
Member

Not really, linking to LLVM is complicated, see Enzyme recipe

@Moelf
Copy link
Contributor Author

Moelf commented Nov 13, 2023

@giordano
Copy link
Member

It's using /opt/x86_64-linux-musl/bin/llvm-config, but that's the host llvm toolchain, not the LLVM for the JLL. You probably need to tell root buildsystem where to find llvm

@Moelf
Copy link
Contributor Author

Moelf commented Nov 13, 2023

where can I find the llvm-config from prefix? currently tried:

export LLVM_CONFIG="${prefix}/bin/llvm-config"

@Moelf
Copy link
Contributor Author

Moelf commented Nov 13, 2023

looks like I need

-DLLVM_CONFIG="${prefix}/bin/llvm-config"

but that path is wrong

@grasph
Copy link
Contributor

grasph commented Nov 13, 2023

@Moelf, you need to use clang code from ROOT, which differs from vanilla one in order to support Cling: see here.

In these conditions, I'm not sure you can use your own LLVM version, considering that standard build of clang is done by putting clang code in the LLVM source tree and build both together: see https://clang.llvm.org/get_started.html.

@Moelf
Copy link
Contributor Author

Moelf commented Nov 13, 2023

well, wtf. 😂

I think we can technically build without Cling though, maybe we should do that

@Moelf
Copy link
Contributor Author

Moelf commented Nov 13, 2023

Say we go back to use its own LLVM, how do I by pass that calling CMake within make ... I think it's not respecting our flags in that case

Maybe I will study Arch Linux recipe's patches more closely

@grasph
Copy link
Contributor

grasph commented Nov 13, 2023

I think we can technically build without Cling though, maybe we should do that

I think it's easier to follow the standard ROOT build instructions.

ROOT is also distributed as Conda, Snap, and docker packages. You may have a look at how these packages are built.

@Moelf
Copy link
Contributor Author

Moelf commented Nov 13, 2023

@Moelf
Copy link
Contributor Author

Moelf commented Nov 13, 2023

conda: https://anaconda.org/conda-forge/root/6.28.4/download/linux-aarch64/root-6.28.4-py311h2dc6514_1.conda

and there are shit load of patches as well

./recipe/parent/0001-Fix-using-external-nlohmann_json.patch
./recipe/parent/patches/root-x.xx.xx-copy-abspath-headers.patch
./recipe/parent/patches/root-x.xx.xx-osx-remove-hardcoded-sysroot.patch
./recipe/parent/patches/0001-Workaround-https-github.com-root-project-root-issues.patch
./recipe/parent/patches/root-x.xx.xx_patch-nbmain-for-conda.patch
./recipe/parent/patches/jupyroot.patch
./recipe/parent/patches/root-6.20.0-install-pyroot-headers.patch
./recipe/parent/patches/6503.diff
./recipe/parent/patches/root-6819.patch
./recipe/parent/patches/root-6.26.0-Do-not-link-libCoreEntitlements.patch
./recipe/parent/patches/0001-Patch-clad-build-for-use-with-external-clang.patch
./recipe/parent/patches/pr-11457.patch
./recipe/parent/patches/root-x.xx.xx_cling-runtime-sysroot.patch
./recipe/parent/patches/0002-Fix-DynamicLibraryManagerSymbol-for-external-LLVM.patch
./recipe/parent/patches/0001-Patch-for-apple-silicon.patch
./recipe/parent/patches/0001-Set-LLVM_REQUIRES_EH-for-Cling.patch
./recipe/parent/patches/0001-Use-find_package-for-llvm-clang.patch
./recipe/parent/patches/0001-Hacks.patch
./recipe/parent/0001-Add-missing-libs.patch
./recipe/parent/llvm-patches/0002-Make-datamember-protected.patch
./recipe/parent/llvm-patches/0001-Fix-the-compilation.patch
./recipe/parent/no-link-libpython.patch
./recipe/parent/0001-Support-standalone-pyroot.patch

@Moelf
Copy link
Contributor Author

Moelf commented Nov 14, 2023

I now suspect it's because the LLVM building step does not respect the CMAKE_TOOLCHAIN_FILE BinaryBuilder.jl set:

cmake -Dpyroot=OFF -DCMAKE_INSTALL_PREFIX=$prefix -DCMAKE_TOOLCHAIN_FILE=${CMAKE_TARGET_TOOLCHAIN} -DCMAKE_BUILD_TYPE=Release ../root-6.28.08

is there a way to verify this theory?

@Moelf
Copy link
Contributor Author

Moelf commented Nov 14, 2023

https://buildkite.com/julialang/yggdrasil/builds/6619#018bcea7-4b07-476a-9c1a-fccfb081f038/6-1790

removing TOOLCHAIN files doesn't work but got us much further.

@gbaraldi
Copy link
Contributor

If you need to build LLVM then i would look at the LLVM build we have to see what things need to be done

@vgvassilev
Copy link

Do we have stdlibc++ installed? Can you run rootcling with -v4 option for extra verbose output?

@giordano
Copy link
Member

Do we have stdlibc++ installed?

Note that we're trying to do a cross-compilation, so the host and the target platforms are in general strictly different. So if you ask "Do we have stdlibc++ installed?" you should specify for which platform. Also, in general we can't run foreign executables for the target platform (with the exception of a couple of platforms, but that's again the exception, not the rule).

@grasph
Copy link
Contributor

grasph commented Feb 20, 2024

@giordano for time being I'm trying using the same platform for the target and host, which does not exclude possible issues with paths. After this, I will probably need your input to define a strategy. E.g., make a package for x86_64-linux-musl only that will be used to build for the other architectures.

@vgvassilev stdc++ is installed. Here is the output with -v4:

Starting program: /workspace/srcdir/build/core/rootcling_stage1/src/rootcling_stage1 -v4 - -f G__Core.cxx -cxxmodule -s /workspace/srcdir/build/lib/libCore.so -excludePath /workspace/srcdir/root -excludePath /workspace/srcdir/build/ginclude -excludePath /workspace/srcdir/build/externals -excludePath /workspace/srcdir/build/builtins -writeEmptyRootPCM -m _Builtin_intrinsics -mByproduct _Builtin_intrinsics -mByproduct ROOT_Foundation_Stage1_NoRTTI -mByproduct ROOT_Foundation_C -mByproduct ROOT_Rtypes -D__STDC_NO_COMPLEX__ -D__COMPLEX_H__ -D_COMPLEX_H -D__CLANG_STDATOMIC_H -compilerI/opt/x86_64-linux-musl/x86_64-linux-musl/include/c++/8.1.0 -compilerI/opt/x86_64-linux-musl/x86_64-linux-musl/include/c++/8.1.0/x86_64-linux-musl -compilerI/opt/x86_64-linux-musl/x86_64-linux-musl/include/c++/8.1.0/backward -compilerI/opt/x86_64-linux-musl/x86_64-linux-musl/include -compilerI/opt/x86_64-linux-musl/lib/gcc/x86_64-linux-musl/8.1.0/include -compilerI/opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/local/include -compilerI/opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/include -compilerI/opt/x86_64-linux-musl/x86_64-linux-musl/include -compilerI/opt/x86_64-linux-musl/lib/gcc/x86_64-linux-musl/8.1.0/include -compilerI/opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/local/include -compilerI/opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/include -compilerI/opt/x86_64-linux-musl/include -I/opt/x86_64-linux-musl/include -I/workspace/srcdir/build/include -I/opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/include -I/opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/local/include -I/opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/include -I/workspace/srcdir/build/ginclude -I/workspace/srcdir/root/core/base/inc -I/workspace/srcdir/root/core/base/v7/inc -I/workspace/srcdir/root/core/clib/inc -I/workspace/srcdir/root/core/clingutils/inc -I/workspace/srcdir/root/core/clingutils/res -I/workspace/srcdir/root/core/cont/inc -I/workspace/srcdir/root/core/foundation/inc -I/workspace/srcdir/root/core/foundation/v7/inc -I/workspace/srcdir/root/core/foundation/res -I/workspace/srcdir/root/core/gui/inc -I/workspace/srcdir/root/core/meta/inc -I/workspace/srcdir/root/core/rint/inc -I/workspace/srcdir/root/core/textinput/inc -I/workspace/srcdir/root/core/textinput/src -I/workspace/srcdir/root/core/thread/inc -I/workspace/srcdir/root/core/zip/inc -I/workspace/srcdir/root/core/lzma/inc -I/opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/local/include -I/workspace/srcdir/root/core/lz4/inc -I/workspace/srcdir/root/core/zstd/inc -I/workspace/srcdir/root/core/unix/inc -I/workspace/srcdir/root/core/unix/../clib/res ROOT/TErrorDefaultHandler.hxx ROOT/TExecutorCRTP.hxx ROOT/TSequentialExecutor.hxx ROOT/StringConv.hxx Buttons.h Bytes.h Byteswap.h KeySymbols.h MessageTypes.h Riostream.h Rtypes.h TApplication.h TAtt3D.h TAttAxis.h TAttBBox2D.h TAttBBox.h TAttFill.h TAttLine.h TAttMarker.h TAttPad.h TAttText.h TBase64.h TBenchmark.h TBuffer3D.h TBuffer3DTypes.h TBuffer.h TColor.h TColorGradient.h TDatime.h TDirectory.h TEnv.h TException.h TExec.h TFileCollection.h TFileInfo.h TFolder.h TInetAddress.h TMacro.h TMathBase.h TMD5.h TMemberInspector.h TMessageHandler.h TNamed.h TNotifyLink.h TObject.h TObjString.h TParameter.h TPluginManager.h TPoint.h TPRegexp.h TProcessID.h TProcessUUID.h TQClass.h TQCommand.h TQConnection.h TQObject.h TRedirectOutputGuard.h TRefCnt.h TRef.h TRegexp.h TRemoteObject.h TROOT.h TRootIOCtor.h TStopwatch.h TStorage.h TString.h TStringLong.h TStyle.h TSysEvtHandler.h TSystemDirectory.h TSystemFile.h TSystem.h TTask.h TThreadSlots.h TTime.h TTimer.h TTimeStamp.h TUri.h TUrl.h TUUID.h TVersionCheck.h TVirtualAuth.h TVirtualFFT.h TVirtualGL.h TVirtualMonitoring.h TVirtualMutex.h TVirtualPadEditor.h TVirtualPad.h TVirtualPadPainter.h TVirtualPerfStats.h TVirtualPS.h TVirtualQConnection.h TVirtualRWMutex.h TVirtualTableInterface.h TVirtualViewer3D.h TVirtualX.h ROOT/RDirectoryEntry.hxx ROOT/RIndexIter.hxx strlcpy.h snprintf.h strtok.h root_std_complex.h ROOT/TSeq.hxx TArrayC.h TArrayD.h TArrayF.h TArray.h TArrayI.h TArrayL64.h TArrayL.h TArrayS.h TBits.h TBtree.h TClassTable.h TClonesArray.h TCollection.h TCollectionProxyInfo.h TExMap.h THashList.h THashTable.h TIterator.h TList.h TMap.h TObjArray.h TObjectTable.h TOrdCollection.h TRefArray.h TRefTable.h TSeqCollection.h TSortedList.h TVirtualCollectionProxy.h ESTLType.h RStringView.h TClassEdit.h TError.h ThreadLocalStorage.h ROOT/RLogger.hxx ROOT/RNotFn.hxx ROOT/RRangeCast.hxx ROOT/RSpan.hxx ROOT/RStringView.hxx ROOT/StringUtils.hxx ROOT/span.hxx ROOT/TypeTraits.hxx ROOT/RError.hxx GuiTypes.h TApplicationImp.h TBrowser.h TBrowserImp.h TCanvasImp.h TClassMenuItem.h TContextMenu.h TContextMenuImp.h TControlBarImp.h TGuiFactory.h TInspectorImp.h TObjectSpy.h TToggleGroup.h TToggle.h TBaseClass.h TClassGenerator.h TClass.h TClassRef.h TClassStreamer.h TDataMember.h TDataType.h TDictAttributeMap.h TDictionary.h TEnumConstant.h TEnum.h TFileMergeInfo.h TFunction.h TFunctionTemplate.h TGenericClassInfo.h TGlobal.h TInterpreter.h TInterpreterValue.h TIsAProxy.h TListOfDataMembers.h TListOfEnums.h TListOfEnumsWithLock.h TListOfFunctions.h TListOfFunctionTemplates.h TMemberStreamer.h TMethodArg.h TMethodCall.h TMethod.h TProtoClass.h TRealData.h TSchemaHelper.h TSchemaRule.h TSchemaRuleSet.h TStatusBitsChecker.h TStreamerElement.h TStreamer.h TVirtualIsAProxy.h TVirtualRefProxy.h TVirtualStreamerInfo.h TVirtualArray.h TVirtualObject.h Getline.h TUnixSystem.h /workspace/srcdir/root/core/base/inc/LinkDef.h
cling version ROOT_1.0~dev
Looking for C++ headers with:
  LC_ALL=C x86_64-linux-musl-g++   -xc++ -E -v /dev/null 2>&1 | sed -n -e '/^.include/,${' -e '/^ \/.*++/p' -e '}'
[Detaching after vfork from child process 593]
Found:
  /opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/8.1.0/../../../../x86_64-linux-musl/include/c++/8.1.0
  /opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/8.1.0/../../../../x86_64-linux-musl/include/c++/8.1.0/x86_64-linux-musl
  /opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/8.1.0/../../../../x86_64-linux-musl/include/c++/8.1.0/backward
clang version 13.0.0
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /workspace/srcdir/build/core/rootcling_stage1/src
ignoring nonexistent directory "/usr/local/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/include"
ignoring duplicate directory "/opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/local/include"
ignoring duplicate directory "/workspace/srcdir/build/include"
#include "..." search starts here:
 .
#include <...> search starts here:
 /opt/x86_64-linux-musl/include
 /workspace/srcdir/build/include
 /opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/include
 /opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/local/include
 /workspace/srcdir/build/ginclude
 /workspace/srcdir/root/core/base/inc
 /workspace/srcdir/root/core/base/v7/inc
 /workspace/srcdir/root/core/clib/inc
 /workspace/srcdir/root/core/clingutils/inc
 /workspace/srcdir/root/core/clingutils/res
 /workspace/srcdir/root/core/cont/inc
 /workspace/srcdir/root/core/foundation/inc
 /workspace/srcdir/root/core/foundation/v7/inc
 /workspace/srcdir/root/core/foundation/res
 /workspace/srcdir/root/core/gui/inc
 /workspace/srcdir/root/core/meta/inc
 /workspace/srcdir/root/core/rint/inc
 /workspace/srcdir/root/core/textinput/inc
 /workspace/srcdir/root/core/textinput/src
 /workspace/srcdir/root/core/thread/inc
 /workspace/srcdir/root/core/zip/inc
 /workspace/srcdir/root/core/lzma/inc
 /workspace/srcdir/root/core/lz4/inc
 /workspace/srcdir/root/core/zstd/inc
 /workspace/srcdir/root/core/unix/inc
 /workspace/srcdir/root/core/unix/../clib/res
 /workspace/srcdir/build/etc
 /opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/8.1.0/../../../../x86_64-linux-musl/include/c++/8.1.0
 /opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/8.1.0/../../../../x86_64-linux-musl/include/c++/8.1.0/x86_64-linux-musl
 /opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/8.1.0/../../../../x86_64-linux-musl/include/c++/8.1.0/backward
 /workspace/srcdir/build/etc/cling/lib/clang/13.0.0/include
 /usr/include
End of search list.
'/opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/include/module.modulemap' does not exist. Mounting '/workspace/srcdir/build/etc/cling/libc.modulemap' as '/opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/include/module.modulemap'
'/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/8.1.0/../../../../x86_64-linux-musl/include/c++/8.1.0/module.modulemap' does not exist. Mounting '/workspace/srcdir/build/etc/cling/std.modulemap' as '/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/8.1.0/../../../../x86_64-linux-musl/include/c++/8.1.0/module.modulemap'
modulemap.overlay.yaml
{
 'version': 0,
 'roots': [
{ 'name': '/opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/include', 'type': 'directory',
'contents': [
   { 'name': 'libc.modulemap', 'type': 'file',
  'external-contents': '/workspace/srcdir/build/etc/cling/libc.modulemap'
}
 ]
 },
{ 'name': '/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/8.1.0/../../../../x86_64-linux-musl/include/c++/8.1.0', 'type': 'directory',
'contents': [
   { 'name': 'std.modulemap', 'type': 'file',
  'external-contents': '/workspace/srcdir/build/etc/cling/std.modulemap'
}
 ]
 }]
 }
 ]
 }
Adding runtime include paths:
  ""
Added include paths:
Setting up system headers with clang:
ignoring nonexistent directory "/usr/local/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/include"
ignoring duplicate directory "/opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/local/include"
ignoring duplicate directory "/workspace/srcdir/build/include"
#include "..." search starts here:
 .
#include <...> search starts here:
 /opt/x86_64-linux-musl/include
 /workspace/srcdir/build/include
 /opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/include
 /opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/local/include
 /workspace/srcdir/build/ginclude
 /workspace/srcdir/root/core/base/inc
 /workspace/srcdir/root/core/base/v7/inc
 /workspace/srcdir/root/core/clib/inc
 /workspace/srcdir/root/core/clingutils/inc
 /workspace/srcdir/root/core/clingutils/res
 /workspace/srcdir/root/core/cont/inc
 /workspace/srcdir/root/core/foundation/inc
 /workspace/srcdir/root/core/foundation/v7/inc
 /workspace/srcdir/root/core/foundation/res
 /workspace/srcdir/root/core/gui/inc
 /workspace/srcdir/root/core/meta/inc
 /workspace/srcdir/root/core/rint/inc
 /workspace/srcdir/root/core/textinput/inc
 /workspace/srcdir/root/core/textinput/src
 /workspace/srcdir/root/core/thread/inc
 /workspace/srcdir/root/core/zip/inc
 /workspace/srcdir/root/core/lzma/inc
 /workspace/srcdir/root/core/lz4/inc
 /workspace/srcdir/root/core/zstd/inc
 /workspace/srcdir/root/core/unix/inc
 /workspace/srcdir/root/core/unix/../clib/res
 /workspace/srcdir/build/etc
 /opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/8.1.0/../../../../x86_64-linux-musl/include/c++/8.1.0
 /opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/8.1.0/../../../../x86_64-linux-musl/include/c++/8.1.0/x86_64-linux-musl
 /opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/8.1.0/../../../../x86_64-linux-musl/include/c++/8.1.0/backward
 /workspace/srcdir/build/etc/cling/lib/clang/13.0.0/include
 /usr/include
End of search list.

Program received signal SIGSEGV, Segmentation fault.
0x000000000091d694 in std::__shared_ptr<clang::PreprocessorOptions, (__gnu_cxx::_Lock_policy)2>::get (this=0x38) at /opt/x86_64-linux-musl/x86_64-linux-musl/include/c++/8.1.0/bits/shared_ptr_base.h:1292
1292	      { return _M_ptr; }

@vgvassilev
Copy link

Do we have stdlibc++ installed?

Note that we're trying to do a cross-compilation, so the host and the target platforms are in general strictly different. So if you ask "Do we have stdlibc++ installed?" you should specify for which platform. Also, in general we can't run foreign executables for the target platform (with the exception of a couple of platforms, but that's again the exception, not the rule).

The same version of libstdc++ needs to be installed when compiling and when running. I am not aware that ROOT supports cross-compilation. That might be a problem...

@Moelf
Copy link
Contributor Author

Moelf commented Feb 20, 2024

I am not aware that ROOT supports cross-compilation. That might be a problem...

now I wonder if ROOT can build on Alpine linux to begin with, ignore cross-compilation for a second, we should be able to build on Alpine linux to generic x86 binary and run on any linux x86, or not?

@grasph
Copy link
Contributor

grasph commented Feb 20, 2024

@Moelf ROOT builds on Alpine.

@vgvassilev There several copies of stdc++ libraries, all 6.0 and only the patch number differ. The version that must be the same are the one rootcling is linked to when we run it (output of ldd) and the one the library passed with the -s option was built for ?

@grasph
Copy link
Contributor

grasph commented Feb 20, 2024

@giordano is it possible in the container to chroot to a system with either the build or the target dependencies ? There are sys_root directories, but they contains symbolic link to directories that would be out-of-scope in the chroot.

@giordano
Copy link
Member

is it possible in the container to chroot to a system with either the build or the target dependencies ?

I don't think so because you'd be missing the entire operating system, can't do much with just what you find under ${prefix} or ${host_prefix}.

@grasph
Copy link
Contributor

grasph commented Feb 22, 2024

I did not manage to solve the issue. The crash of rootcling_stage1 comes from a std::make_shared() call that returns a null pointer, which should never happen. Memory usage is small, so it's not an out-of-memory issue.

I suspected some consistency issue in the built LLVM libraries and rootcling_stage1 code, but I could not find any. There are few small inconsistencies when using the build from the committed script (c++14/c++17, copy of the libc) but removing them did not help. @giordano if you find some time to try, maybe you will be luckier than me.

@gbaraldi
Copy link
Contributor

We've seen musl blow up with LLVM in interesting ways. One option would be to use x86-gnu instead of musl.

@grasph
Copy link
Contributor

grasph commented Feb 28, 2024

Thanks @gbaraldi for the suggestion. The idea of starting with musl was the need to run a build product during the build process, but I was also thinking of another approach: build it for glibc and use the possibility to run glibc-based executable on Alpine.

I have made some progress, thanks to inputs of a colleague with experience with building ROOT for Alpine. Apparently for this the runtime_cxx option must be disabled. @vgvassilev what is this feature for? Are you aware of this issue with musl?

So with the build configuration -Druntime_cxx=OFF option, rootcling does not crash anymore, but I get another problem: it does not find stdlib.h:

/opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/8.1.0/../../../../x86_64-linux-musl/include/c++/8.1.0/cstdlib:75:15: fatal error: 'stdlib.h' file not found
#include_next <stdlib.h>
              ^~~~~~~~~~
Error: Error loading the default rootcling header files.

I expect it is link to the -isystem gcc issue, the -compilerI option of rootcling was introduced to work with. There are three occurence of the stdlib.h file in the file system:

/opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/include/stdlib.h
/opt/x86_64-linux-musl/x86_64-linux-musl/include/c++/8.1.0/stdlib.h
/opt/x86_64-linux-musl/x86_64-linux-musl/include/c++/8.1.0/tr1/stdlib.h
/opt/x86_64-linux-musl/include/c++/v1/stdlib.h

The first one is excluded from -isystem with a -compilerI option and passed as a -I option. If I remove the -I, then it is assert.h which is not found from a std #include.

The second and last one are included as -compilerI only, the 3rd is not included in the command line, adding it as -compileI does not help (same error as with original command line).

I have also tried the -isysroot /opt/x86_64-linux-musl/x86_64-linux-musl/sys-root option with no success. Maybe there are some cross-compilation related options rootcling needs to pass to clang?

Philippe.

@grasph
Copy link
Contributor

grasph commented Feb 29, 2024

update: build for x86_64-linux-gnu (glibc instead of musl) and use of Alpine gcompat package to run glibc based executable lead to same errors: rootcling_stage1 crash if runtime_cxxmodules is enabled and stdlib.h not found with #include_next .

@grasph
Copy link
Contributor

grasph commented Jul 1, 2024

I've updated the build script. It compiles for x86 with musl. It doesn't with glibc. I failed to instruct the rootcling_stage1 built with musl to use the include path needed for the target setup (x86 + glibc).

@vgvassilev and @peremato could you have a look ? Command to run the build:

julia --project=. build_tarballs.jl --debug=error --verbose

It will open a shell in the Alpine container on failure. The file script.sh contains the command run in the container.

For a build with musl, append x86-linux-musl to the command line. I have not looked at macOS at all.

Philippe.

@giordano
Copy link
Member

giordano commented Jul 1, 2024

The git history of this branch is now quite messed up, with many unrelated changes showing up in the diff

@grasph
Copy link
Contributor

grasph commented Jul 1, 2024

The git history of this branch is now quite messed up, with many unrelated changes showing up in the diff

Fixed.

@giordano
Copy link
Member

giordano commented Jul 1, 2024

Ok, the diff is now resolved, thanks. Is there a way not to see 396 commits in the history though? GitHub even refuses to show most of them 🙂

. Works for x86_64-linux-musl, but of little use, due to the general
dependency library loard issue for .jll packages with musl

. Fails for x86_64-linux-gnu: misses to instruct the cross-compiled
  rootcling_stage1 to use the header files of the sysroot of the
  target host.
@grasph
Copy link
Contributor

grasph commented Jul 1, 2024

history should also be fixed now.

@grasph
Copy link
Contributor

grasph commented Jul 1, 2024

With above change ac99958, the rootcling_stage1 error is:

In file included from input_line_3:2:
In file included from /workspace/srcdir/root/core/base/inc/Rtypes.h:193:
In file included from /workspace/srcdir/root/core/meta/inc/TGenericClassInfo.h:21:
In file included from /workspace/srcdir/root/core/meta/inc/TSchemaHelper.h:17:
In file included from /opt/x86_64-linux-gnu/bin/../lib/gcc/x86_64-linux-gnu/11.1.0/../../../../x86_64-linux-gnu/include/c++/11.1.0/string:55:                              
In file included from /opt/x86_64-linux-gnu/bin/../lib/gcc/x86_64-linux-gnu/11.1.0/../../../../x86_64-linux-gnu/include/c++/11.1.0/bits/basic_string.h:6594:               
In file included from /opt/x86_64-linux-gnu/bin/../lib/gcc/x86_64-linux-gnu/11.1.0/../../../../x86_64-linux-gnu/include/c++/11.1.0/ext/string_conversions.h:41:            
In file included from /opt/x86_64-linux-gnu/bin/../lib/gcc/x86_64-linux-gnu/11.1.0/../../../../x86_64-linux-gnu/include/c++/11.1.0/cstdlib:75:                             
/opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/include/stdlib.h:518:12: error: exception specification in declaration does not match previous declaration             
extern int atexit (void (*__func) (void)) __THROW __nonnull ((1));
           ^
input_line_2:3:16: note: previous declaration is here
extern "C" int atexit(void(*f)()) ;

followed by a similar error for the function at_quick_exit.

I couldn't get what input_line_2:3:16: note: previous declaration is here refers to.

It looks like we mix up two versions of the stdlib.h file.

The rootcling_stage1 line run by the build is quite long, but the error can be reproduced by running the simplified one,

/workspace/NATIVE/core/rootcling_stage1/src/rootcling_stage1 -I/workspace/srcdir/root/core/base/inc -I/workspace/build/ginclude Rtypes.h /workspace/srcdir/root/core/base/inc/LinkDef.h

System header directory path list is set before running rootcling_stage1 using the following environment variable:

export CPLUS_INCLUDE_PATH="`/opt/bin/x86_64-linux-gnu-libgfortran5-cxx11/x86_64-linux-gnu-g++ --sysroot="/opt/$target/$target/sys-root" -E -x c++ -v /dev/null  2>&1  | awk '{gsub("^ ", "")} /End of search list/{a=0} {if(a==1){s=s d $0;d=":"}} /#include <...> search starts here/{a=1} END{print s}'`"

It copies the path from the cross-compiler.

@peremato
Copy link
Contributor

peremato commented Jul 1, 2024

My understanding is that the strategy for cross-compiling is:

  • build rootcling_stage1 for the native platform x86_64-linux-musl and be able to run it
  • patch ROOT to use this command instead of the one built for the target platform.
  • eventually use LLVM taken from LLVM_jll (I think I did manage with Julia 1.11 with minor patch to installation of LLVM_jll)
    But the pre-requisite is to be able to build for x86_64-linux-musl which is not the case since it fails not finding some system header files.

@grasph
Copy link
Contributor

grasph commented Jul 1, 2024

It does compile for x86_64-linux-musl.

@peremato
Copy link
Contributor

Continued in a new PR #9300

@Moelf Moelf closed this Aug 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants