SLURM Increasing CPU Usage Efficiency

[kmlefran]

Hi, trying to tune my pysr code on slurm, and just want to know if there are any tweaks I can do to increase CPU usage efficiency. I'm running a whole node job with 32 cores. Only 23% of CPU time was used. Head worker occupation was 6.9%. The start of the .sh file and my defined PySRRegressor shown below.

Are there any things in PySRRegressor that I can tweak that might increase the CPU usage?

#!/bin/bash
#SBATCH --time=00-01:00:00
#SBATCH --mem=0
#SBATCH --ntasks-per-node=32

model = PySRRegressor(
    model_selection="accuracy",
    binary_operators=["+", "-", "*", "/", "^"],
    unary_operators=["sin", "log"],
    constraints={'^':(-1,1)},
    nested_constraints={'sin':{'sin':0}},
    niterations=10000000,
    ncycles_per_iteration=10000,
    maxsize=35,
    timeout_in_seconds=loop_length_in_seconds,
    turbo=True,
    procs=32,
    cluster_manager="slurm",
    populations=96,
    multithreading=False,
    verbosity=1
)

[MilesCranmer]

First, just to check, make sure to not launch the script 32 times on the node. This might be happening when you do --ntasks-per-node=32. It should just be launched once with procs=32. i.e., this will create new processes from within PySR, not like MPI where if you run the script from 32 processes, they will communicate with eachother.

Also, if it is only a single node, you don't need to use slurm cluster manager; you can just use regular distributed computing with cluster_manager=None.

If everything is good above, the next thing to try is increase the population size. The default is about 30. I would go up to 300. That should let each worker work for much longer without communicating with the head worker.

Additional tweaks to try would be to increase weight_optimize, maybe to 0.01?