8 procs not used

[Nakul9621]

Dear Miles

I have a dataset of 2000 training points. I have set the number of iterations to 100, procs=8 / cpu.count() and multithreading=True. When I am looking at the CPU usage, I see all 8 of them are at 50-70% during PySr's training. Even if I saturate procs to 32, this issue still remains. What should I change here?

Although I get the answer in less than 2 minutes even with these settings, I want to make use of all 8 CPUs.

If I run for a higher number of iterations=500 and populations=3*32 over a "pbs" cluster of procs=32, I see the same issue there. Only one CPU is getting used. The settings here were: procs=32, cluster_manager="pbs"

[MilesCranmer]

Does it say the head occupation %? The lower that is, the better. If that number is high, the head worker basically can’t give out work quickly enough.

To improve this you should increase population size and/or ncyclesperiteration.

Alternatively you can use multiprocessing instead of multithreading by setting multithreading=False. I find sometimes that improves usage of cores for whatever reason.

[Nakul9621]

Thank you for your reply and this information. I tried changing these settings on my local computer with 8 CPUs and it did not improve the CPU usage.

After this, I tried on the cluster.

Here are the cluster settings for a training dataset of 4000 points. The cluster has 32 cores per node.

--- niterations=5000, procs=32, cluster_manager="pbs", populations=96, population_size=50, multithreading=False,
ncyclesperiteration=1000

The cluster is still using only one CPU (cluster_run.jpeg).

[MilesCranmer]

What version of PBS are you using? I know that some people have run into issues on multi-node processing with PBS due to ClusterManagers.jl not yet supporting the latest qsub command line arguments. See JuliaParallel/ClusterManagers.jl#179.

So your current options are to:

1. Use single-node multiprocessing (cluster_manager=None, multithreading=False, procs=32).
2. Use a slurm cluster manager, if available, to get working multi-node multiprocessing.
3. Help us fix the issue on pbs error JuliaParallel/ClusterManagers.jl#179. It's a bit hard for me to fix this myself since I don't have a PBS cluster I can test it on (all my institutes use slurm). So any help is much appreciated.

[Nakul9621]

Thank you so much for this information.

[MilesCranmer]

@Nakul9621 also this issue is highly relevant! #419

[Nakul9621]

I will look into this.