Merge pull request #4 from barbagroup/rescience-rev

ReScience revision

README.md

@@ -30,11 +30,45 @@ This repository contains a computational replication of the scientific findings
 * [PetibmPy](https://github.com/mesnardo/petibmpy) (0.2)
 * [VisIt](https://wci.llnl.gov/simulation/computer-codes/visit) (2.12.3)
 
+## Container images
+
+* Olivier Mesnard & Lorena A. Barba (2021). [Re] Three-dimensional wake topology and propulsive performance of low-aspect-ratio pitching-rolling plates (container images). [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.5090342.svg)](https://doi.org/10.5281/zenodo.5090342)
+
+The deposit (4.7GB) contains the Docker and Singularity images used for the replication:
+
+* `petibm-rollingpitching_PetIBM0.5.1-xenial.tar`: tar archive of the Docker image `mesnardo/petibm-rollingpitching:PetIBM0.5.1-xenial` which includes the PetIBM application for the 3D rolling and pitching wing
+* `petibm-rollingpitching_PetIBM0.5.1-xenial.sif`: Singularity image (build from the Docker image) that was used to run the simulations on the `pegasus` HPC cluster (at the George Washington University)
+* `petibm-rollingpitching_prepost.tar`: tar archive of the Docker image `mesnardo/petibm-rollingpitching:prepost` which contains all tools to pre- and post-process the numerical outputs of the simulations
+
+We used Docker 1.39 (see [version info](docker/Docker.version)) and Singularity 3.4.2.
+
+## Software/Hardware requirements
+
+The following list of libraries and tools should be installed on the host machine to be able to run the Singularity image:
+
+* CUDA Toolkit 10.1
+* Singularity 3.4+
+* OpenMPI 3.1.4
+
+The NVIDIA AmgX library contained in the Docker image `mesnardo/petibm-rollingpitching:PetIBM0.5.1-xenial` was compiled with CUDA Toolkit 10.1 to target NVIDIA GPUs with a compute capability of 3.5 (Kepler - Tesla K20, K40), 3.7 (Kepler - Tesla K80), 6.0 (Pascal - Tesla P100), and 7.0 (Volta - Tesla V100).
+
+Note we used NVIDIA V100 GPUs to solve the pressure Poisson system in our simulations; the two other linear systems were solved on CPUs with the PETSc library.
+If no GPUs are available on the host machine, one should still be able to use the same Docker or Singularity image and solve the Poisson system on CPUs.
+Otherwise, one should be able to launch and configure an AWS EC2 instance following [these instructions](misc/running-on-aws.md).
+
+## Hardware configuration and run times
+
+See [this README section](runs/README.md/#hardware-configuration-and-run-times).
+
 ## Reproducibility packages
 
 * Olivier Mesnard & Lorena A. Barba (2021). [Re] Three-dimensional wake topology and propulsive performance of low-aspect-ratio pitching-rolling plates (repro-packs). [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.4732946.svg)](https://doi.org/10.5281/zenodo.4732946)
 
-Download the Zenodo archive (20G) with the primary data output from PetIBM.
+Download the Zenodo archive (18.6 GB) with the primary data output from PetIBM.
+
+| :warning: WARNING |
+|:-|
+| Generating secondary data (e.g., 3D vorticity field) requires a machine with more than 16 GB of memory (32 GB is recommended). If you do not have access to such machine, one solution is to launch a `t3.2xlarge` EC2 instance on AWS and run the Docker container there (see [instructions](misc/repro-packs-on-aws.md)). |
 
 Generate all secondary data and figures of the manuscript (~20 minutes):
 

docker/Docker.version

@@ -0,0 +1,19 @@
+# docker version on phantom
+Client: Docker Engine - Community
+ Version:           19.03.10
+ API version:       1.39
+ Go version:        go1.13.10
+ Git commit:        9424aeaee9
+ Built:             Thu May 28 22:17:05 2020
+ OS/Arch:           linux/amd64
+ Experimental:      false
+
+Server: Docker Engine - Community
+ Engine:
+  Version:          18.09.7
+  API version:      1.39 (minimum version 1.12)
+  Go version:       go1.10.8
+  Git commit:       2d0083d
+  Built:            Thu Jun 27 17:23:02 2019
+  OS/Arch:          linux/amd64
+  Experimental:     false

misc/repro-packs-on-aws.md

@@ -0,0 +1,63 @@
+# Repro-Packs on AWS
+
+1. Sign-in to your AWS account
+2. Go to EC2, switch to your preferred Region, and click "Launch instance"
+3. Choose one of the Ubuntu AMIs (e.g., "Ubuntu Server 20.04 LTS (HVM), SSD Volume Type")
+4. Choose the `t3.2xlarge` instance type (or a similar one with 32 GB of memory)
+5. Use a Bootstrap script to install and start Docker on the instance:
+
+   ```bash
+   #!/bin/bash
+   sudo snap install docker
+   sudo addgroup --system docker
+   sudo adduser ubuntu docker
+   newgrp docker
+   sudo snap disable docker
+   sudo snap enable docker
+   ```
+
+6. Request 60 GB for the Root volume (gp2 volume type is fine)
+7. Use a Security Group with SSH to your IP address
+8. Review, set your Permission Key-Pair, and Launch
+9. Once the instance is provisioned, connect to it
+
+   ```shell
+   ssh -i <pem-path> ubuntu@<public-ipv4-dns-or-address>
+   ```
+
+10. Install `unzip`
+
+    ```shell
+    sudo apt install unzip
+    ```
+
+11. Download the repro-packs from Zenodo and unzip the archive
+
+    ```shell
+    ARCHIVE=petibm-rollingpitching_repro-packs.zip
+    curl --output $ARCHIVE https://zenodo.org/record/4732946/files/$ARCHIVE
+    unzip $ARCHIVE
+    rm -f $ARCHIVE
+    ```
+
+12. Download the container images from Zenodo and unzip the archive
+
+    ```shell
+    ARCHIVE=petibm-rollingpitching-images.zip
+    curl --output $ARCHIVE https://zenodo.org/record/5090342/files/$ARCHIVE
+    unzip $ARCHIVE
+    rm -f $ARCHIVE
+    ```
+
+13. Load the Docker image for post-processing the data
+
+    ```shell
+    docker load -i petibm-rollingpitching-images/petibm-rollingpitching_prepost.tar
+    ```
+
+14. Run a Docker container to generate the secondary data and figures
+
+    ```shell
+    cd repro-packs
+    docker run --rm -it -v $(pwd):/postprocessing mesnardo/petibm-rollingpitching:prepost /bin/bash /postprocessing/scripts/generate_all_figures.sh > repro-packs.log 2>&1
+    ```

misc/running-on-aws.md

@@ -0,0 +1,109 @@
+# Running simulations on AWS
+
+1. Sign-in to your AWS account
+2. Go to EC2, switch to your preferred Region, and click "Launch instance"
+3. Choose one the Ubuntu 18.04 AMI
+4. Choose the `p2.8xlarge` instance type (8 NVIDIA K80 GPUs, 32 vCPUs, 488 GiB Mem, 96 GiB GPU Mem). (`p2.xlarge` does not have enough memory for the base case simulation.)
+5. Use a Bootstrap script to install and start Docker on the instance:
+
+   ```bash
+   #!/bin/bash
+   sudo snap install docker
+   sudo addgroup --system docker
+   sudo adduser ubuntu docker
+   newgrp docker
+   sudo snap disable docker
+   sudo snap enable docker
+   ```
+
+6. Request 100 GB for the Root volume (gp2 volume type is fine)
+7. Use a Security Group with SSH to your IP address
+8. Review, set your Permission Key-Pair, and Launch
+9. Once the instance is provisioned, connect to it
+
+   ```shell
+   ssh -i <pem-path> ubuntu@<public-ipv4-dns-or-address>
+   ```
+
+10. On the EC2 instance, install dependencies for Singularity
+
+    ```shell
+    sudo apt update
+    sudo apt install -y build-essential libssl-dev uuid-dev libgpgme11-dev libseccomp-dev wget pkg-config git
+    ```
+
+11. Install CUDA Toolkit 10.1
+
+    ```shell
+    wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
+    sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
+    sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
+    sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /"
+    sudo apt update
+    sudo apt -y install cuda-10-1
+    ```
+
+12. Install GO (required for Singularity)
+
+    ```shell
+    export VERSION=1.14.12 OS=linux ARCH=amd64
+    wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz
+    sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz
+    rm -f go$VERSION.$OS-$ARCH.tar.gz
+    export PATH=/usr/local/go/bin:$PATH
+    ```
+
+13. Install Singularity
+
+    ```shell
+    export VERSION=3.8.0
+    wget https://github.com/hpcng/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz
+    tar -xzf singularity-${VERSION}.tar.gz
+    rm -f singularity-${VERSION}.tar.gz
+    cd singularity-${VERSION}
+    ./mconfig -V ${VERSION}
+    make -C builddir
+    sudo make -C builddir install
+    ```
+
+14. Install nvidia-container-cli
+
+    ```shell
+    DIST=$(. /etc/os-release; echo $ID$VERSION_ID)
+    curl -s -L https://nvidia.github.io/libnvidia-container/gpgkey | sudo apt-key add -
+    curl -s -L https://nvidia.github.io/libnvidia-container/${DIST}/libnvidia-container.list | sudo tee /etc/apt/sources.list.d/libnvidia-container.list
+    sudo apt update
+    sudo apt install libnvidia-container-tools
+    ```
+
+15. Download the Zenodo archive with container images
+
+    ```shell
+    ARCHIVE=petibm-rollingpitching-images.zip
+    curl --output $ARCHIVE https://zenodo.org/record/5090342/files/$ARCHIVE
+    unzip $ARCHIVE
+    rm -f $ARCHIVE
+    ```
+
+16. Load the pre-processing Docker image
+
+    ```shell
+    docker load -i petibm-rollingpitching-images/petibm-rollingpitching_prepost.tar
+    ```
+
+17. Download the Zenodo archive with the application input data
+
+    ```shell
+    ARCHIVE=petibm-rollingpitching-2021.05.02.zip
+    curl --output $ARCHIVE https://zenodo.org/record/4733323/files/$ARCHIVE
+    unzip $ARCHIVE
+    rm -f $ARCHIVE
+    ```
+
+18. Run the base case simulation
+
+    ```shell
+    cd petibm-rollingpitching-2021.05.02/runs/Re200_St0.6_AR1.27_psi90
+    docker run --rm -v $(pwd):/volume mesnardo/petibm-rollingpitching:prepost python /volume/scripts/create_body.py
+    SINGULARITY_CUDA_VISIBLE_DEVICES=0,1,2,3 singularity exec --nv ~/petibm-rollingpitching-images/petibm-rollingpitching_petibm0.5.1-xenial.sif mpirun --use-hwthread-cpus petibm-rollingpitching -probes probes.yaml -options_left -log_view ascii:view.log
+    ```

runs/README.md

@@ -10,43 +10,47 @@ The present directory contains all the input files and post-processing scripts,
 * `Re*_St*_AR*_psi*`: simulation folders of the parametric study; for example:
   * `Re200_St0.6_AR1.27_psi90`: simulation folder to compute the three-dimensional flow around a pitching and rolling flat plate with aspect ratio `AR=1.27` at Reynolds number `Re=200`, Strouhal number `St=0.6`, with a phase difference of `psi=90` degrees between the rolling and pitching motions.
 
-## Runtimes
+## Hardware configuration and run times
 
-Each simulation job was submitted to SLURM scheduling system on Pegasus (HPC cluster at GWU), using nodes of the `small-gpu` partition.
+Each simulation job was submitted to the SLURM scheduling system on Pegasus (HPC cluster at the George Washington University), using nodes of the `small-gpu` partition.
+(Note: The SLURM submission scripts were designed for the user `mesnardo` to run on Pegasus; do not use them as such; use them as an example to develop your own submission script.)
 
-Hardware configuration of small GPU nodes:
+Hardware configuration of the nodes on the `small-gpu` partition:
 
 * Dell PowerEdge R740 server
-* (2) NVIDIA Tesla V100 GPU
+* (2) NVIDIA Tesla V100 GPU (Memory size: 16GB)
 * Dual 20-Core 3.70GHz Intel Xeon Gold 6148 processors
 * 192GB of 2666MHz DDR4 ECC Register DRAM
 * 800 GB SSD onboard storage (used for boot and local scratch space)
 * Mellanox EDR Infiniband controller to 100GB fabric
 
+For each simulation, we used 20 CPUs and 2 GPU devices per node.
+Information about the number of nodes, the number of time steps computed, the mesh size, and the run time for each simulation are displayed in the following table.
+
 | Simulation | # time steps | # grid cells (10^6) | # nodes | runtime (hr) |
 |:-|:-:|:-:|:-:|:-:|
-| `independence/run1` | 10000 | 5.1 | 1 | 3.0 |
-| `independence/run2` | 10000 | 11.5 | 1 | 7.4 |
-| `independence/run3` | 10000 | 21.1 | 2 | 8.4 |
-| `independence/run4` | 10000 | 72.3 | 6 | 23.7 |
-| `independence/run5` | 10000 | 21.1 | 2 | 10.8 |
-| `independence/run6` | 5000 | 21.1 | 2 | 5.1 |
-| `independence/run7` | 10000 | 21.1 | 2 | 12.8 |
-| `Re100_St0.6_AR1.27_psi90` | 10000 | 21.1 | 2 | 11.0 |
-| `Re200_St0.4_AR1.27_psi90` | 10000 | 21.1 | 2 | 9.8 |
-| `Re200_St0.6_AR1.27_psi100` | 10000 | 21.1 | 2 | 9.1 |
-| `Re200_St0.6_AR1.27_psi110` | 10000 | 21.1 | 2 | 8.1 |
-| `Re200_St0.6_AR1.27_psi120` | 10000 | 21.1 | 2 | 9.0 |
-| `Re200_St0.6_AR1.27_psi60` | 10000 | 21.1 | 2 | 8.1 |
-| `Re200_St0.6_AR1.27_psi70` | 10000 | 21.1 | 2 | 9.7 |
-| `Re200_St0.6_AR1.27_psi80` | 10000 | 21.1 | 2 | 9.7 |
-| `Re200_St0.6_AR1.27_psi90` | 10000 | 21.1 | 2 | 13.8 |
-| `Re200_St0.6_AR1.91_psi90` | 10000 | 29.9 | 2 | 9.0 |
-| `Re200_St0.6_AR2.55_psi90` | 10000 | 42.6 | 2 | 12.8 |
-| `Re200_St0.8_AR1.27_psi90` | 10000 | 21.1 | 2 | 9.2 |
-| `Re200_St1.0_AR1.27_psi90` | 10000 | 21.1 | 2 | 8.2 |
-| `Re200_St1.2_AR1.27_psi90` | 10000 | 21.1 | 2 | 12.2 |
-| `Re400_St0.6_AR1.27_psi90` | 10000 | 21.1 | 2 | 11.0 |
+| `independence/run1` | 10,000 | 5.1 | 1 | 3.0 |
+| `independence/run2` | 10,000 | 11.5 | 1 | 7.4 |
+| `independence/run3` | 10,000 | 21.1 | 2 | 8.4 |
+| `independence/run4` | 10,000 | 72.3 | 6 | 23.7 |
+| `independence/run5` | 10,000 | 21.1 | 2 | 10.8 |
+| `independence/run6` | 5,000 | 21.1 | 2 | 5.1 |
+| `independence/run7` | 10,000 | 21.1 | 2 | 12.8 |
+| `Re100_St0.6_AR1.27_psi90` | 10,000 | 21.1 | 2 | 11.0 |
+| `Re200_St0.4_AR1.27_psi90` | 10,000 | 21.1 | 2 | 9.8 |
+| `Re200_St0.6_AR1.27_psi100` | 10,000 | 21.1 | 2 | 9.1 |
+| `Re200_St0.6_AR1.27_psi110` | 10,000 | 21.1 | 2 | 8.1 |
+| `Re200_St0.6_AR1.27_psi120` | 10,000 | 21.1 | 2 | 9.0 |
+| `Re200_St0.6_AR1.27_psi60` | 10,000 | 21.1 | 2 | 8.1 |
+| `Re200_St0.6_AR1.27_psi70` | 10,000 | 21.1 | 2 | 9.7 |
+| `Re200_St0.6_AR1.27_psi80` | 10,000 | 21.1 | 2 | 9.7 |
+| `Re200_St0.6_AR1.27_psi90` | 10,000 | 21.1 | 2 | 13.8 |
+| `Re200_St0.6_AR1.91_psi90` | 10,000 | 29.9 | 4 | 9.0 |
+| `Re200_St0.6_AR2.55_psi90` | 10,000 | 42.6 | 4 | 12.8 |
+| `Re200_St0.8_AR1.27_psi90` | 10,000 | 21.1 | 2 | 9.2 |
+| `Re200_St1.0_AR1.27_psi90` | 10,000 | 21.1 | 2 | 8.2 |
+| `Re200_St1.2_AR1.27_psi90` | 10,000 | 21.1 | 2 | 12.2 |
+| `Re400_St0.6_AR1.27_psi90` | 10,000 | 21.1 | 2 | 11.0 |
 
 ## Post-processing steps
 

runs/Re100_St0.6_AR1.27_psi90/README.md

@@ -36,6 +36,7 @@ and run the instructions displayed in the following sub-sections.
 ## Submit the simulation job
 
 The simulation job was submitted to SLURM scheduling system on Pegasus (HPC cluster at GWU), requesting 2 nodes of the `small-gpu` partition; CLI: `sbatch pegasus.slurm`.
+(Note: The SLURM submission script was designed for the user `mesnardo` to run on Pegasus; do not use it as such; use it as an example to develop your own submission script.)
 
 Hardware configuration of small GPU nodes:
 

runs/Re200_St0.4_AR1.27_psi90/README.md

@@ -36,6 +36,7 @@ and run the instructions displayed in the following sub-sections.
 ## Submit the simulation job
 
 The simulation job was submitted to SLURM scheduling system on Pegasus (HPC cluster at GWU), requesting 2 nodes of the `small-gpu` partition; CLI: `sbatch pegasus.slurm`.
+(Note: The SLURM submission script was designed for the user `mesnardo` to run on Pegasus; do not use it as such; use it as an example to develop your own submission script.)
 
 Hardware configuration of small GPU nodes:
 

runs/Re200_St0.6_AR1.27_psi100/README.md

@@ -36,6 +36,7 @@ and run the instructions displayed in the following sub-sections.
 ## Submit the simulation job
 
 The simulation job was submitted to SLURM scheduling system on Pegasus (HPC cluster at GWU), requesting 2 nodes of the `small-gpu` partition; CLI: `sbatch pegasus.slurm`.
+(Note: The SLURM submission script was designed for the user `mesnardo` to run on Pegasus; do not use it as such; use it as an example to develop your own submission script.)
 
 Hardware configuration of small GPU nodes:
 

runs/Re200_St0.6_AR1.27_psi110/README.md

@@ -36,6 +36,7 @@ and run the instructions displayed in the following sub-sections.
 ## Submit the simulation job
 
 The simulation job was submitted to SLURM scheduling system on Pegasus (HPC cluster at GWU), requesting 2 nodes of the `small-gpu` partition; CLI: `sbatch pegasus.slurm`.
+(Note: The SLURM submission script was designed for the user `mesnardo` to run on Pegasus; do not use it as such; use it as an example to develop your own submission script.)
 
 Hardware configuration of small GPU nodes:
 

runs/Re200_St0.6_AR1.27_psi120/README.md

@@ -36,6 +36,7 @@ and run the instructions displayed in the following sub-sections.
 ## Submit the simulation job
 
 The simulation job was submitted to SLURM scheduling system on Pegasus (HPC cluster at GWU), requesting 2 nodes of the `small-gpu` partition; CLI: `sbatch pegasus.slurm`.
+(Note: The SLURM submission script was designed for the user `mesnardo` to run on Pegasus; do not use it as such; use it as an example to develop your own submission script.)
 
 Hardware configuration of small GPU nodes:
 

runs/Re200_St0.6_AR1.27_psi60/README.md

@@ -36,6 +36,7 @@ and run the instructions displayed in the following sub-sections.
 ## Submit the simulation job
 
 The simulation job was submitted to SLURM scheduling system on Pegasus (HPC cluster at GWU), requesting 2 nodes of the `small-gpu` partition; CLI: `sbatch pegasus.slurm`.
+(Note: The SLURM submission script was designed for the user `mesnardo` to run on Pegasus; do not use it as such; use it as an example to develop your own submission script.)
 
 Hardware configuration of small GPU nodes:
 

runs/Re200_St0.6_AR1.27_psi70/README.md

@@ -36,6 +36,7 @@ and run the instructions displayed in the following sub-sections.
 ## Submit the simulation job
 
 The simulation job was submitted to SLURM scheduling system on Pegasus (HPC cluster at GWU), requesting 2 nodes of the `small-gpu` partition; CLI: `sbatch pegasus.slurm`.
+(Note: The SLURM submission script was designed for the user `mesnardo` to run on Pegasus; do not use it as such; use it as an example to develop your own submission script.)
 
 Hardware configuration of small GPU nodes:
 

runs/Re200_St0.6_AR1.27_psi80/README.md

@@ -36,6 +36,7 @@ and run the instructions displayed in the following sub-sections.
 ## Submit the simulation job
 
 The simulation job was submitted to SLURM scheduling system on Pegasus (HPC cluster at GWU), requesting 2 nodes of the `small-gpu` partition; CLI: `sbatch pegasus.slurm`.
+(Note: The SLURM submission script was designed for the user `mesnardo` to run on Pegasus; do not use it as such; use it as an example to develop your own submission script.)
 
 Hardware configuration of small GPU nodes:
 

runs/Re200_St0.6_AR1.27_psi90/README.md

@@ -36,6 +36,7 @@ and run the instructions displayed in the following sub-sections.
 ## Submit the simulation job
 
 The simulation job was submitted to SLURM scheduling system on Pegasus (HPC cluster at GWU), requesting 2 nodes of the `small-gpu` partition; CLI: `sbatch pegasus.slurm`.
+(Note: The SLURM submission script was designed for the user `mesnardo` to run on Pegasus; do not use it as such; use it as an example to develop your own submission script.)
 
 Hardware configuration of small GPU nodes: