Excluding particular nodes from a resource request

[SteVwonder]

How do we want to support the equivalent to Slurm's --exclude flag? Defined as:

Request that a specific list of hosts not be included in the resources allocated to this job.

I suspect this will require design considerations at the CLI, Jobspec, and scheduler-levels.

[grondo]

Yeah, especially since Flux isn't necessarily a node-level scheduler this one might be tricky. Do we even have a way to add exclusions to canonical jobspec? We probably have to start with figuring out how this kind of thing can be expressed in josbpec and then go from there.

If we had a way to explicitly request resources by name (e.g. a set of hostnames), then could we add an exclude: section to jobspec which would generically exclude all matching resources? Then a scheduler could first prune the resources that match the excluded set before attempting to find a match for the main resources section of jobspec. This might be fairly trivial in the "hostnames" case, and would also open the door to some other kinds of exclusions.

[SteVwonder]

This might be fairly trivial in the "hostnames" case, and would also open the door to some other kinds of exclusions.

Agreed! One use case that comes to mind is K8s anti-affinity, which basically let's you say "I don't want to be co-scheduled on the same node (or availability zone) with any other containers/pods that have the label X".