pykdtree requires OMP_NUM_THREADS flag on Linux

[schlegelp]

Just to leave a paper trail for this observation:

I was setting up a large NBLAST on a Linux (Ubuntu 16.04) server and noticed that it was running suspiciously slow. Some digging flagged pykdtree as the culprit: it was much slower than scipy's KDTree solution (order 5-10 fold). This is weird because on my OSX laptop it's the other way around.

A quick glance at the pykdtree Github brought up the OMP_NUM_THREADS environment variable (which I never bothered with on OSX). Setting OMP_NUM_THREADS=4 indeed brings up pykdtree to the expected speed.

FYI: @clbarnes @sdorkenw

[schlegelp]

Another breadcrumb to chase up: while individual KNN queries with pykdtree are faster, it looks like there is some blocking when used with multiprocessing that makes it slower than scipy.

[clbarnes]

I think multiprocessing can misbehave if there's more than one layer of concurrency - I think it depends on how everything is implemented (and can be platform-dependent too because the default forking behaviour is different for different OSs). Something like if different multiprocessing processes are sharing some object which has its own locks, they end up sharing that lock too, unless due care is taken.

[schlegelp]

Good point!

Confusingly, I managed to get pykdtree to behave similar to on my Mac in both concurrent and non-concurrent contexts by setting OMP_NUM_THREADS=1.

On a related note: I've been working on another NBLAST variant that uses mostly matrix operations via numpy and I observed a similar behaviour. At a certain point throwing more cores at this NBLAST makes it go slower than faster - presumably because numpy already employs some level of concurrency.

[schlegelp]

@jefferis had a very good explanation for this behaviour: Apple's clang compiler doesn't support openmp which is why that flag might not need to be set on OSX.

[jefferis]

NB if you have homebrew clang, that may support openmp.

[schlegelp]

Just to leave a note that threadpoolctl might be handy here to avoid multiple layers of concurrency. Will see if it does the trick.

[schlegelp]

Leaving another breadcrumb: on ARM Macs, pykdtree seems to be compiled with openmp support.

Also: I tried threadpoolctl in the past and it didn't work 🤷

[schlegelp]

Just to leave a note that since version 1.5.0 (July 2023), navis is automatically setting the OMP_NUM_THREADS flag before spawning the child processes. By default it is set to 1 but that can be changed:

# Increase limit to 2
navis.nbl.nblast_funcs.OMP_NUM_THREADS_LIMIT = 2