Building PeleLM

Summit, cokurt branch build with ascent with genten support:

python3 scripts/uberenv/uberenv.py  --spec="%gcc+genten" --spack-config-dir=scripts/uberenv_configs/spack_configs/configs/olcf/summit_gcc_9.3.0/ 

ml gcc/6.4.0
ml cmake/3.14.2

git clone --recursive [email protected]:Alpine-DAV/PeleLM.git
cd PeleLM
git checkout ascent
cd ..

git clone [email protected]:Alpine-DAV/IAMR.git
cd IAMR
cd ..

git clone https://github.com/Alpine-DAV/amrex.git
cd amrex
git checkout ascent
cd ..
git clone https://github.com/AMReX-Combustion/PelePhysics.git
cd PelePhysics
git checkout cd6fe50f048635b92a9884ac675dfd79ff9150a9

PelePhysics needs sundials(if not building with cuda set USE_CUDA=FALSE)

cd ThirdParty
make USE_MPI=TRUE USE_CUDA=TRUE AMREX_HOME=../../amrex
cd ../..

cd PeleLM/Exec/RegTests/FlameSheet
make AMREX_HOME=/gpfs/alpine/proj-shared/csc340/larsen/pele_test_build/amrex/

Note: if building with CUDA, change line 18 of the make file to USE_CUDA=TRUE Set the ASCENT_HOME on line 38 to the install directory of Ascent.

Note: AMREX_HOME must be specified because some paths in the Pele makefiles don't respect the defaults. For me it built without it, but just to avoid warnings you should specify it.

Next you will need to edit the make file to point at your ascent install on line 39.

Additionally, sundials linking is not rpath'd, so you will have to put the sundials lib path into your LD_LIBRARY_PATH. For me, it was located here:

$(PELE_PHYSICS_HOME)/ThirdParty/INSTALL/gcc/lib/

Ascent is hooked into the same path as the checkpoint calls, so it will execute at the same frequency.

Running:

jsrun -n 4 ./PeleLM3d.gnu.MPI.ex inputs.3d-regt

Note: amrex adds system signal handlers for floating point errors. This means that any floating point exception, which are normally ignored, cause the entire run to crash. You can override this behavior by adding the following line to the input deck:

amrex.signal_handling = 0