Help with installing MOOSE on HPC

[smpark7]

I'm trying to install MOOSE on the ALCF Polaris HPC. I have the following environment:

module swap PrgEnv-nvhpc PrgEnv-gnu/8.3.3
module load cray-hdf5-parallel/1.12.2.3
module load cray-fftw/3.3.10.3
module load cray-python/3.9.13.1
module load cmake/3.23.2
export CC=mpicc CXX=mpicxx FC=mpif90 F90=mpif90 F77=mpif77

When running the update_and_rebuild_petsc.sh script, the installation appears to stall indefinitely at the step Configuring SUPERLU_DIST with CMake; this may take several minutes. Here's the config log after terminating the PETSc installation.

configure.log

The installation appears to be stuck while running the following command:

Executing: /soft/buildtools/cmake/cmake-3.23.2/cmake-3.23.2-linux-x86_64/bin/cmake .. -DCMAKE_INSTALL_PREFIX=/eagle/Hybrid-Sn-Diffusion/projects/moltres/moose/petsc/arch-moose -DCMAKE_INSTALL_NAME_DIR:STRING="/eagle/Hybrid-Sn-Diffusion/projects/moltres/moose/petsc/arch-moose/lib" -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1 -DCMAKE_BUILD_TYPE=Release -DCMAKE_AR="/usr/bin/ar" -DCMAKE_C_COMPILER="mpicc" -DMPI_C_COMPILER="/opt/cray/pe/mpich/8.1.25/ofi/gnu/9.1/bin/mpicc" -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC -Wno-lto-type-mismatch -Wno-stringop-overflow -g -O -fopenmp" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Wno-lto-type-mismatch -Wno-stringop-overflow -g -O -fopenmp" -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Wno-lto-type-mismatch -Wno-stringop-overflow -g -O -fopenmp" -DCMAKE_CXX_COMPILER="mpicxx" -DMPI_CXX_COMPILER="/opt/cray/pe/mpich/8.1.25/ofi/gnu/9.1/bin/mpicxx" -DCMAKE_CXX_FLAGS:STRING="-Wno-lto-type-mismatch -Wno-psabi -g -O -std=c++17 -fPIC -fopenmp" -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Wno-lto-type-mismatch -Wno-psabi -g -O -std=c++17 -fPIC -fopenmp" -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Wno-lto-type-mismatch -Wno-psabi -g -O -std=c++17 -fPIC -fopenmp" -DCMAKE_Fortran_COMPILER="mpif90" -DMPI_Fortran_COMPILER="/opt/cray/pe/mpich/8.1.25/ofi/gnu/9.1/bin/mpif90" -DCMAKE_Fortran_FLAGS:STRING="-fPIC -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -g -O -fopenmp -fallow-argument-mismatch" -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -g -O -fopenmp -fallow-argument-mismatch" -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -g -O -fopenmp -fallow-argument-mismatch" -DCMAKE_EXE_LINKER_FLAGS:STRING=" -fopenmp" -DBUILD_SHARED_LIBS:BOOL=ON -DUSE_XSDK_DEFAULTS=YES -DTPL_BLAS_LIBRARIES="-Wl,-rpath,/eagle/Hybrid-Sn-Diffusion/projects/moltres/moose/petsc/arch-moose/lib -L/eagle/Hybrid-Sn-Diffusion/projects/moltres/moose/petsc/arch-moose/lib -lflapack -Wl,-rpath,/eagle/Hybrid-Sn-Diffusion/projects/moltres/moose/petsc/arch-moose/lib -L/eagle/Hybrid-Sn-Diffusion/projects/moltres/moose/petsc/arch-moose/lib -lfblas -lm -ldl -Wl,-rpath,/opt/cray/pe/mpich/8.1.25/ofi/gnu/9.1/lib -L/opt/cray/pe/mpich/8.1.25/ofi/gnu/9.1/lib -lmpifort_gnu_91 -lmpi_gnu_91 -lgfortran -lm -Wl,-rpath,/opt/cray/pe/gcc/12.2.0/snos/lib/gcc/x86_64-suse-linux/12.2.0 -L/opt/cray/pe/gcc/12.2.0/snos/lib/gcc/x86_64-suse-linux/12.2.0 -Wl,-rpath,/opt/cray/pe/gcc/12.2.0/snos/lib64 -L/opt/cray/pe/gcc/12.2.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/gcc/12.2.0/snos/lib -L/opt/cray/pe/gcc/12.2.0/snos/lib -lgfortran -lm -lgcc_s -lquadmath" -DTPL_LAPACK_LIBRARIES="-Wl,-rpath,/eagle/Hybrid-Sn-Diffusion/projects/moltres/moose/petsc/arch-moose/lib -L/eagle/Hybrid-Sn-Diffusion/projects/moltres/moose/petsc/arch-moose/lib -lflapack -Wl,-rpath,/eagle/Hybrid-Sn-Diffusion/projects/moltres/moose/petsc/arch-moose/lib -L/eagle/Hybrid-Sn-Diffusion/projects/moltres/moose/petsc/arch-moose/lib -lfblas -lm -ldl -Wl,-rpath,/opt/cray/pe/mpich/8.1.25/ofi/gnu/9.1/lib -L/opt/cray/pe/mpich/8.1.25/ofi/gnu/9.1/lib -lmpifort_gnu_91 -lmpi_gnu_91 -lgfortran -lm -Wl,-rpath,/opt/cray/pe/gcc/12.2.0/snos/lib/gcc/x86_64-suse-linux/12.2.0 -L/opt/cray/pe/gcc/12.2.0/snos/lib/gcc/x86_64-suse-linux/12.2.0 -Wl,-rpath,/opt/cray/pe/gcc/12.2.0/snos/lib64 -L/opt/cray/pe/gcc/12.2.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/gcc/12.2.0/snos/lib -L/opt/cray/pe/gcc/12.2.0/snos/lib -lgfortran -lm -lgcc_s -lquadmath" -DTPL_PARMETIS_INCLUDE_DIRS="/eagle/Hybrid-Sn-Diffusion/projects/moltres/moose/petsc/arch-moose/include;/opt/cray/pe/mpich/8.1.25/ofi/gnu/9.1/include" -DTPL_PARMETIS_LIBRARIES="-Wl,-rpath,/eagle/Hybrid-Sn-Diffusion/projects/moltres/moose/petsc/arch-moose/lib -L/eagle/Hybrid-Sn-Diffusion/projects/moltres/moose/petsc/arch-moose/lib -lparmetis -Wl,-rpath,/eagle/Hybrid-Sn-Diffusion/projects/moltres/moose/petsc/arch-moose/lib -L/eagle/Hybrid-Sn-Diffusion/projects/moltres/moose/petsc/arch-moose/lib -lmetis -lm" -DXSDK_INDEX_SIZE=64 -DXSDK_ENABLE_Fortran=ON -Denable_tests=0 -Denable_examples=0 -DMPI_C_COMPILE_FLAGS:STRING="" -DMPI_C_INCLUDE_PATH:STRING="" -DMPI_C_HEADER_DIR:STRING="" -DMPI_C_LIBRARIES:STRING=""

[GiudGiud]

Hello

Does it consistently get stuck there?
@milljm

Guillaume

[milljm]

Posting, in case you end up running into an error. For all the directories I listed, please double check the paths! And/Or choose where you wish to install some of the libraries we will be building.

Step 1

Partially build PETSc. This will save us some time, as we leverage PETSc build system to build a couple dependencies that SuperLU_Dist requires.

cd moose/petsc
python3 ./configure --with-64-bit-indices \
  --with-cxx-dialect=C++17 \
  --with-debugging=no \
  --with-fortran-bindings=0 \
  --with-mpi=1 \
  --with-openmp=1 \
  --with-shared-libraries=1 \
  --with-sowing=0 \
  --download-fblaslapack=1 \
  --download-hypre=1 \
  --download-metis=1 \
  --download-mumps=1 \
  --download-ptscotch=1 \
  --download-parmetis=1 \
  --download-scalapack=1 \
  --download-slepc=1 \
  --download-strumpack=1 \
  --with-hdf5-dir=/opt/cray/pe/hdf5-parallel/1.12.2.3/gnu/9.1 \
  --prefix=/lus/eagle/projects/Hybrid-Sn-Diffusion/projects/moltres/petsc-3.20.3-installed

When you see this (STOP, and go to Step 2):

xxx=======================================================================================xxx
 Configure stage complete. Now build PETSc libraries with:
   make PETSC_DIR=/data/milljm/projects/moose/petsc PETSC_ARCH=arch-linux-c-opt all
xxx=======================================================================================xxx

Do not run the above. We first need to use all the dependencies that PETSc built for us, to build SuperLU_dist below

Step 2

Clone the SuperLU Dist repository, configure it, build and install it.

cd ../../
git clone https://github.com/xiaoyeli/superlu_dist
mkdir superlu_dist/build; cd superlu_dist/build
cmake .. \
  -DBUILD_SHARED_LIBS=ON \
  -DBUILD_STATIC_LIBS=ON \
  -DCMAKE_BUILD_TYPE=Release \
  -DTPL_ENABLE_PARMETISLIB=ON \
  -DTPL_LAPACK_LIBRARIES=OFF \
  -DTPL_PARMETIS_INCLUDE_DIRS=OFF \
  -DXSDK_ENABLE_Fortran=ON \
  -DXSDK_INDEX_SIZE=64 \
  -DTPL_BLAS_LIBRARIES=/lus/eagle/projects/Hybrid-Sn-Diffusion/projects/moltres/petsc-3.20.3-installed/lib/libfblas.a \
  -DTPL_PARMETIS_INCLUDE_DIRS=/lus/eagle/projects/Hybrid-Sn-Diffusion/projects/moltres/petsc-3.20.3-installed/include \
  -DTPL_PARMETIS_LIBRARIES="/lus/eagle/projects/Hybrid-Sn-Diffusion/projects/moltres/petsc-3.20.3-installed/lib/libparmetis.so;/lus/eagle/projects/Hybrid-Sn-Diffusion/projects/moltres/petsc-3.20.3-installed/lib/libmetis.so" \
  -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_CXX_COMPILER=mpicxx \
  -DCMAKE_Fortran_COMPILER=mpif90 \
  -DCMAKE_CXX_FLAGS="-std=c++11" \
  -DCMAKE_INSTALL_PREFIX=/lus/eagle/projects/Hybrid-Sn-Diffusion/projects/moltres/superlu_dist-installed

I am curious if we run into a hang here like it does when invoked from PETSc configure system! If configure completes, move on to building/installing it:

make -j 6
make install

Step 3

Now go back to PETSc, clean up the old build, and start again, but this time we tell it where SuperLU_Dist is, and we let it complete.

cd ../../moose/petsc
git clean -xfd
rm -rf arch-*
python3 ./configure --with-64-bit-indices \
  --with-cxx-dialect=C++17 \
  --with-debugging=no \
  --with-fortran-bindings=0 \
  --with-mpi=1 \
  --with-openmp=1 \
  --with-shared-libraries=1 \
  --with-sowing=0 \
  --download-fblaslapack=1 \
  --download-hypre=1 \
  --download-metis=1 \
  --download-mumps=1 \
  --download-ptscotch=1 \
  --download-parmetis=1 \
  --download-scalapack=1 \
  --download-slepc=1 \
  --download-strumpack=1 \
  --with-hdf5-dir=/opt/cray/pe/hdf5-parallel/1.12.2.3/gnu/9.1 \
  --with-superlu_dist-dir=/lus/eagle/projects/Hybrid-Sn-Diffusion/projects/moltres/superlu_dist-installed \
  --prefix=/lus/eagle/projects/Hybrid-Sn-Diffusion/projects/moltres/petsc-3.20.3-installed

When you see this, do it

xxx=======================================================================================xxx
 Configure stage complete. Now build PETSc libraries with:
   make PETSC_DIR=/data/milljm/projects/moose/petsc PETSC_ARCH=arch-linux-c-opt all
xxx=======================================================================================xxx

Your make line will look different than mine. Copy and paste what it says to do

When you see this, do it

=========================================
Now to install the libraries do:
make PETSC_DIR=/data/milljm/projects/moose/petsc PETSC_ARCH=arch-linux-c-opt install
=========================================

Your make line will look different than mine. Copy and paste what it says to do

Step 4 Build libMesh

From this point on, you're basically following the normal installation procedures. Aside from the fact that we need to now explain where we officially installed PETSc to (note the export PETSC_DIR line):

export PETSC_DIR=/lus/eagle/projects/Hybrid-Sn-Diffusion/projects/moltres/petsc-3.20.3-installed
export MOOSE_JOBS=6 METHODS=opt METHOD=opt
cd ../scripts
./update_and_rebuild_libmesh.sh

Step 5 Build WASP

./update_and_rebuild_wasp.sh

Step 6 Build MOOSE

cd ../test
make -j 6

Post notes: Because PETSc is in a non-standard directory MOOSE looks for, you will need to export PETSC_DIR when ever you work on MOOSE related things. Might be best to simply add this export to your bash profile:

export PETSC_DIR=/lus/eagle/projects/Hybrid-Sn-Diffusion/projects/moltres/petsc-3.20.3-installed

[smpark7]

Thanks for the detailed instructions! I have not had time to attempt the superlu_dist manual build today, but I'll give it a shot tomorrow.

When I tried the PETSc team's recommendation (commented earlier today), the petsc build appeared to be successful. However, a bunch of tests failed when I ran make check. I have relayed this info to them for further help.

In the meantime, I ran ./update_and_rebuild_libmesh.sh just to see how far I could go. I got the error:

/opt/cray/pe/gcc-libs/libstdc++.so.6: version `GLIBCXX_3.4.29' not found

which I encountered when attempting to install cmake. Is this an error that is independent of the petsc build issue I am facing?

For this instance, I did not export the mpi wrapper compilers. If I export the mpi wrapper compilers, the libmesh build runs into a cross compile error. I think it's because of the --with-cc=cc --with-fc=ftn --with-cxx=CC petsc configure arguments I received from the petsc team? I'm quite confused now because I assumed the initial output from configuring petsc implies that they ignore my mpi compiler exports.

[smpark7]

It turned out that I just had to run the petsc tests on a compute node rather than a login node for mpiexec to work properly. I also fixed the libmesh configure error by exporting /opt/cray/pe/gcc/12.2.0/snos/lib64/libstdc++.so.6 to LD_LIBRARY_PATH.

Now I'm encountering another error later in the libmesh build with the following error:

g++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
make[1]: *** [Makefile:27706: src/mesh/libmesh_opt_la-mesh_iterators.lo] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: Leaving directory '/lus/eagle/projects/Hybrid-Sn-Diffusion/projects/alt-moltres/moose/libmesh/build'
make: *** [Makefile:33431: all-recursive] Error 1

Here's the config log for libmesh:
config.log

[GiudGiud]

hello

you are running out of memory compiling libmesh

reduce MOOSE_JOBS (the environment variable) to 1 and compile again

[smpark7]

Ah I was a little overzealous with that variable. I managed to build everything and all the tests passed!

Something odd I noticed was that moose make would terminate with an error pointing to line 150 of moose/framework/build.mk when I'm on a login node, but I didn't encounter this error when running make on a compute node. This wasn't an issue with other clusters I've worked with.