Command to launch MPI job using Parsl's FluxExecutor

[christopherwharrop-noaa]

I am trying to use Parsl's FluxExecutor to schedule and run MPI jobs within a Parsl application. Unfortunately, there is very little practical information to be found about how to use the FluxExecutor, and it is not at all clear how MPI jobs should be launched from a Parsl App when using the FluxExecutor. I would like to use my HPC's local installation of Intel oneAPI.

I have the following Parsl Config:

config = Config(
    executors=[
        FluxExecutor(
            provider=SlurmProvider(
                channel=LocalChannel(),
                nodes_per_block=2,
                init_blocks=1,
                min_blocks=1,
                max_blocks=1,
                partition='hera',
                account='gsd-hpcs',
                walltime='00:10:00',
                launcher=SrunLauncher(),
                worker_init='''                                                                                                                                                                                                                                               
. /opt/spack/share/spack/setup-env.sh                                                                                                                                                                                              
spack env activate flux                                                                                                                                                                                                                                                       
module load intel/2022.3.0                                                                                                                                                                                                                                                    
module load impi/2022.3.0                                                                                                                                                                                                                                                     
module list                                                                                                                                                                                                                                                                   
''',
            ),
        )
    ],
)

And I am trying to use the following Bash App to run my simple test program:

@bash_app
def compute_pi(dirpath, app="./mpi_pi.exe", outputs=[], parsl_resource_specification={'tasks': 1}):
    return "cd {0}; module load intel/2022.3.0 impi/2022.3.0; srun -n 1 {1} &> {2};".format(dirpath, app, outputs[0])

I am calling the app as follows:

mpi_pi = compute_pi(dirpath=shared_dir,
                    outputs=[File(os.path.join(shared_dir, "mpi_pi.txt"))], parsl_resource_specification={'tasks': 1})
output_file = mpi_pi.outputs[0].result()

The problem with the above is that the srun command (which runs inside the Parsl pilot job where Flux is set up) asks Slurm to run the job somewhere, and it runs it outside of the Parsl pilot job altogether, and not on a host controlled by Parsl or Flux.

I also tried:

@bash_app
def compute_pi(dirpath, app="./mpi_pi.exe", outputs=[], parsl_resource_specification={'tasks': 1}):
    return "cd {0}; module load intel/2022.3.0 impi/2022.3.0; flux run -n 1 {1} &>> {2};".format(dirpath, app, outputs[0])

But that just hung forever

$ more runinfo/001/submit_scripts/parsl.slurm.1684182502.53311.submit.stderr 
0: 2023-05-15 20:28:46,677 [DEBUG] Flux package path sent.
0: 2023-05-15 20:28:46,688 [DEBUG] Flux URI is ssh://h20c22/tmp/flux-YbQdYN/local-0
0: 2023-05-15 20:28:46,955 [DEBUG] Received acknowledgment b'ack'
0: 2023-05-15 20:28:46,955 [DEBUG] URI sent. Blocking for response...

Then looking at another question from this discussion forum, I tried:

@bash_app
def compute_pi(dirpath, app="./mpi_pi.exe", outputs=[], parsl_resource_specification={'tasks': 1}):
    return "cd {0}; module load intel/2022.3.0 impi/2022.3.0; I_MPI_FABRICS=shm flux run -n 1 {1} &>> {2};".format(dirpath, app, outputs[0])

But that also hangs forever.

I have also tried leaving out a launch command and calling the executable directly:

@bash_app
def compute_pi(dirpath, app="./mpi_pi.exe", outputs=[], parsl_resource_specification={'tasks': 1}):
    return "cd {0}; module load intel/2022.3.0 impi/2022.3.0; I_MPI_FABRICS=shm {1} &>> {2};".format(dirpath, app, outputs[0])

But that gives me MPI_Init() errors:

Abort(1090831) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(176): 
MPID_Init(1430)......: 
MPIR_pmi_init(162)...: PMI_Init returned 14

If anyone on this discussion board has any tips for the practical details of how an MPI job should be launched when using Parsl's FluxExecutor, I'd be very grateful.