Defaults: affinity

[trws]

There has been a request from a couple of power users that we turn on the cpu-affinity=per-task and gpu-affinity=per-task options by default. What do people think about that? Given how mpi and OpenMP rely on tasksets and similar for reasonable usage, it seems pretty reasonable to me, but is there a downside I'm not thinking of?

[grondo]

I can't think of one off hand, though I thought the long-term plan was to use mpibind instead.

This can be done per-cluster by dropping something like this into a file in /etc/flux/shell/lua.d or adding to /etc/flux/shell/initrc.lua, so I don't think that any changes to flux-core default behavior will be necessary.
(I think @ryanday36 already experimented with this on corona?)

-- turn on per-task affinity if not set by user:
if shell.options['cpu-affinity'] == nil then
   shell.options['cpu-affinity'] = 'per-task'
end
if shell.options['gpu-affinity'] == nil then
   shell.options['gpu-affinity'] = 'per-task'
end

[ryanday36]

Yeah, I had those settings in, but removed them due to a (now fixed) bug in the cpu bindings and the GPU affinity not being topology aware. We are using mpibind by default now. I haven't looked at how nicely that will play with the cpu-affinity and gpu-affinity plugins.

[trws]

Sure, but my point is that the default shell config should probably use our affinity. If we want to turn off or replace that with mpibind for livermore we can do that, but then at least users get affinity by default.

[grondo]

Ok, that sounds reasonable. The only reason it isn't on by default now is because we were a bit afraid of getting it wrong by default. And I don't think Slurm ever operated that way, you either had to install the auto-affinity or mpibind plugin or turn on some config.

We might also need to add a way to force it back to setting affinity for the job shell only.

[trws]

That's probably a good idea. It's interesting you mention that slurm doesn't do it by default, I'm sure you're right about that but I haven't actually run on a slurm instance that didn't since... 2009? It's become so widely expected I kinda assumed slurm did too 🤷 . Either way, having that and maybe an easy argument for it on mini seem like good things. We'll want to have more precise control over binding at some point anyway, so adding --affinity or --binding or similar with off and auto or some such thing that we can extend with more options later seems pretty reasonable to me.

In the end it's one more thing where we have the opportunity to do the "right thing" by default. I still get people emailing me about why OpenMP is terrible when they run with mpirun (and it's because OpenMP gives every rank all the cores, and pain ensues 😥).

[grondo]

Yeah, so far we've avoided adding too many new options to the flux mini commands to avoid --I've-got-an-option-or-two-for-that syndrome exhibited by some commands, but your suggestion definitely seems reasonable.

[trws]

Yeah, having too many can definitely be an issue, and I love the general availability of options and attributes, but especially after dealing with jsrun and similar I have a feeling affinity will be a normal thing for users to reach for. Having up-front support and command help docs for it seems worthwhile. The question I think will be if we want to try to match the slurm argument there, I need to dig back into that but I have a feeling we might not want to do that. Worth really thinking about before we add anything other than the basic binary on/off.