Running DATs in flux

[ryanday36]

I wanted to start a discussion, or at least a place for discussion, on how we're going to run DATs / DSTs (giving one-ish user exclusive access to a set of nodes for a given time) under flux. The traditional way we do this is by creating an advance reservation, which seems preferable from an operational perspective, but I could also see a possibility of doing something complicated with queues / partitions.

[garlick]

DATs were an early use case that motivated Flux's recursive design. The idea is that the DAT owner would be the instance owner of a Flux instance submitted with a reservation to run for some specific time period. I think we do need to circle back and nail down some details, e.g.:

• Right now only flux instances running as the the flux user id are allowed to launch work as other users. A DAT would be launched as a user who is privileged within the DAT instance (hotline staff? compute coordinator?)
• What would be the DAT's initial program?
• How does a DAT user get "invited " (allowed to submit jobs)?
• How does a DAT user find out the FLUX_URI of the DAT instance?

I think we know how to set up reservations but the Fluxion team may need to respond to that one.

[ryanday36]

I guess if I can wrap my head around user jobs being single-user flux instances inside of a multi-user instance, I can wrap my head around DATs being a multi-user flux instance inside of a multi-user instance :)

The question that I'd like to add to the one's that @garlick asked above is how do we do the 'advance' part of the advance reservation. i.e. can we tell flux that a job / instance needs to start on a certain set of resources at a 5:00 tomorrow, so don't schedule anything else on those resources that'll still be running then regardless of priority? I guess this might be just a combination of --begin and an 'expedite' qos (make this job/instance the highest priority) on the DAT instance.

[SteVwonder]

I guess this might be just a combination of --begin and an 'expedite' qos (make this job/instance the highest priority) on the DAT instance.

At the risk of getting in the weeds a bit, if we implement --begin as something the dependency module handles (discussed in #3243 (comment)), then we will want to go with something slightly different. At least in the flux-sched Fluxion scheduler, it should not be difficult at all to create an advance reservation. A lot of the hard logic is already there, we would just need to expose it via front-end tools/APIs. We also probably need to think about what level of privilege you need to ask for a reservation to be made. I'm assuming it would be something reserved for sys admins and the scheduler integration team (i.e., not typical users).

On the note about front-end tools, would we want advance reservations to be a flux-core interface/front-end tool? Or should it be a flux-sched interface? I kind of doubt sched-simple will ever implement reservations, but we've traditionally tried to make those interfaces generic enough so that we don't become too flux-sched-centric.

[dongahn]

Yap. This is correct. This is called advanced reservation (AR) and we design our scheduler to be able to do this easily.

We just need to extend Fluxion slightly to treat the job with --begin with match_allocate at a specific time point.

However, we still need to discuss some fine points about this. If there are currently running jobs that disallow the DAT job to be scheduled at time T, what should be the right behavior?

[dongahn]

On the note about front-end tools, would we want advance reservations to be a flux-core interface/front-end tool?

It will be best if the flux-core interface allows it to go into the resulting jobspec. If the underlying scheduler (flux-sched) doesn't support it, this can be rejected with something like errno= EPROTONOSUPPORT. Speaking of our jobspec, there is currently no begin key. I think we can extend attributes.system dictionary to have begin key. Then, this will likely decouple the front end discussion with the implementation of the AR within Fluxion. We develop the multi queue support within Fluxion this way. To test, one can simply add --setattr system.begin=<future time> with flux mini interface.

[ryanday36]

We also probably need to think about what level of privilege you need to ask for a reservation to be made. I'm assuming it would be something reserved for sys admins and the scheduler integration team (i.e., not typical users).

This is correct. Normal users should not be able to create advance reservations.

[SteVwonder]

A few more details/questions I think we need to answer about DATs:

• Do we need to support the ability to pre-empt running jobs to run a DAT? Or are we mainly focused on advanced reservations that will cause a gradual draining of the system to make room for the DAT?
• How important is it for users to be able to submit jobs and queue up work for their DAT before the DAT begins? If it is important, how do we want to enable this in Flux?

[ryanday36]

Regarding @SteVwonder's questions at the top of this piece of the thread, for small DATs (less than the max number of nodes for a normal job), we usually try to set up the reservation far enough in advance to allow the queue to drain for those nodes. For larger DATs (e.g. full machine, more than the max nodes per job), we create the reservation an hour or so before the DAT start time and then kill all the jobs that overlap it at the DAT start time. LC's policy / stance is that users should be able to restart and this avoids idling the cluster for a significant amount of time. So, we need to be able to create a reservation that overlaps existing jobs.

Despite the short time window between creating the reservation and the DAT starting, users do often submit jobs to the reservation before the DAT officially begins. Another common-ish case, especially with people's limited ability to get on the SCF these days, is for the user to submit a held job before the reservation has been created that we (hotline) add to the DAT reservation after the reservation is created.

[dongahn]

For larger DATs (e.g. full machine, more than the max nodes per job), we create the reservation an hour or so before the DAT start time and then kill all the jobs that overlap it at the DAT start time.

Interesting. What partition does the DAT reservation go in?

Do you identify the running jobs that overlap with the DAT manually and kill them? And there is a guarantee built into the DAT job allocation such that other high priority jobs that will delay the starting of the DAT jobs, correct? But backfilling can still be done?

[ryanday36]

Interesting. What partition does the DAT reservation go in?

For big-but-not-full-machine, they go in pbatch. We usually also have an inactive 'pall' partition in the slurm.conf that we can activate for true full system DATs so that the users can get the pdebug nodes too.

Do you identify the running jobs that overlap with the DAT manually and kill them?

If by "manually" you mean "with a script", then yes. For Slurm, we actually have to pull the node list out of the reservation, then do an 'squeue -n ' to see which jobs are running on the reserved nodes. LSF will 'suspend' jobs that are running in the reservation when the first DAT job starts, which generally has the effect of killing them for us. It can be awfully slow though, so we have a script to help it along too.

And there is a guarantee built into the DAT job allocation such that other high priority jobs that will delay the starting of the DAT jobs, correct? But backfilling can still be done?

Once a DAT starts, only jobs from the users that we list in the DAT reservation that also specify the reservation name will start on the nodes in the reservation. The nodes do not get non-DAT jobs backfilled on them. If there are already high priority jobs running that we want to allow to continue, we would have to manually exclude the nodes that those jobs are running on from the DAT reservation or modify our job killing scripts to spare those jobs. In a sense, DAT jobs are always the highest priority jobs on the system for the period of the DAT reservation.

[dongahn]

For big-but-not-full-machine, they go in pbatch. We usually also have an inactive 'pall' partition in the slurm.conf that we can activate for true full system DATs so that the users can get the pdebug nodes too.

OK. This will require multiqueue support. #3228 (comment). If we can annotate RV1 with queue, supporting multiqueue can be done at the Fluxion level.

Once a DAT starts, only jobs from the users that we list in the DAT reservation that also specify the reservation name will start on the nodes in the reservation. The nodes do not get non-DAT jobs backfilled on them. If there are already high priority jobs running that we want to allow to continue, we would have to manually exclude the nodes that those jobs are running on from the DAT reservation or modify our job killing scripts to spare those jobs. In a sense, DAT jobs are always the highest priority jobs on the system for the period of the DAT reservation.

Sorry I wasn't clear enough. I was wondering if the backfilling is allow for those reserved nodes before the DAT starts.

[ryanday36]

Sorry I wasn't clear enough. I was wondering if the backfilling is allow for those reserved nodes before the DAT starts.

Ah. That makes more sense. Yes, jobs can be backfilled onto the reserved nodes before the DAT starts.

[dongahn]

Do we need to support the ability to pre-empt running jobs to run a DAT? Or are we mainly focused on advanced reservations that will cause a gradual draining of the system to make room for the DAT?

Once Fluxion will be able to create "allocation" into the future, jobs that cannot be backfilled cannot run. I think we are getting this for free.

[dongahn]

How important is it for users to be able to submit jobs and queue up work for their DAT before the DAT begins? If it is important, how do we want to enable this in Flux?

For the first cut, I would say, we look at what users do and try to match that as the baseline. We can later extend it.

Perhaps, users can do is to pass their batch jobs as the initial program to this DAT job, which would be equivalent to "presubmitting" their jobs? The initial program should not drain the queue so if users want to submit more jobs to it, they can do that. They will need to point FLUX_URI to this DAT instance though... This can be communicated to the users either via front end trick or via hotline protocol... just a though,.

[grondo]

This is what I was going to suggest. However, I was wondering what abilities a user with an upcoming scheduled DAT has in our current environment. I.e., once the reservation is active in Slurm can users actually submit work before the reservation is active now?

One potential gotcha - if a member of staff has to create the DAT "reservation" job on behalf of the DAT user(s), how are those users going to submit additions or changes to the initial program of the job?

[dongahn]

One potential gotcha - if a member of staff has to create the DAT "reservation" job on behalf of the DAT user(s), how are those users going to submit additions or changes to the initial program of the job?

Interesting question.

If a member of staff has to create the DAT job, there would be no user provided initial program so users will just have to wait until the DAT instance comes on line before being able to submit their jobs? They are used to this under SLURM perhaps?

If users can submit a DAT job and a member of staff will just make it allocated by bumping up the system priority, this wouldn't be a problem... I'd be a bit worried about any mistake users might introduce into the initial program and the impact of that mistake to the DAT job. But at its core, we are treating a DAT job as the same as other batch jobs so how we treat a failure with the DAT job is the same as regular batch jobs...

[grondo]

I'd be a bit worried about any mistake users might introduce into the initial program and the impact of that mistake to the DAT job. But at its core, we are treating a DAT job as the same as other batch jobs so how we treat a failure with the DAT job is the same as regular batch jobs...

A DAT job would be similar to a normal batch job, except we want to make sure the job stays around for the entire DAT time as you stated above, so that would require some specific DAT functionality from the initial program (e.g. sleep inf or some other mechanism that keeps the initial program from exiting before the DAT is complete)

[dongahn]

@grondo: yeah I was thinking for users to add that into the initial program as the first step. If it is easy enough to add such "guarding" mechanisms into flux-core, this will be extremely helpful I think.

[grondo]

Another constraint for running a DAT is that we would, at least at first, want the DAT instance to have a TBON with all compute nodes as leaves (i.e. similar to the system instance), so that the whole DAT job is not terminated if one compute node goes down.

So maybe a command specific to starting DATs is warranted as I think you hinted above: e.g. flux mini dat 😄.

[dongahn]

I guess if I can wrap my head around user jobs being single-user flux instances inside of a multi-user instance, I can wrap my head around DATs being a multi-user flux instance inside of a multi-user instance :)

@ryanday36: Under SLURM, do you give multiple users (each user with distinct Unix id) to be able to run during DAT by permitting them with saccount or similar? If the main use case is users use the same "shared Unix account" and run their DAT jobs that way, the single user can still support this...

[ryanday36]

The Slurm tools for creating a reservation allow you to give access to multiple users or accounts (banks) and we almost always give access to multiple users. There aren't really that many LC users who run with shared Unix accounts, and it's also often useful to give the LC staff member who is setting things up the ability to submit jobs to the reservation.

[dongahn]

Yeah, this is great info and make sense. This really does sound like multi-user within multi-user.

[dongahn]

@ryanday36: how do you compare the priority of this capability with other scheduler items? Probably soon we will want to pull the issues that we want to fix for our Oct release and knowing priorities will be helpful.

[ryanday36]

@ryanday36: how do you compare the priority of this capability with other scheduler items? Probably soon we will want to pull the issues that we want to fix for our Oct release and knowing priorities will be helpful.

On the Kanban that @SteVwonder put up on Thursday (https://github.com/orgs/flux-framework/projects/27), I put this in the medium priorities column. We should have a larger discussion of this at some point, but I basically prioritized things that relate to configuring clusters (node exclusivity, job / user limits) and basic functionality (the ability to modify jobs after submission) as high priority, then job submission options that get used often (--depend, --begin) as medium priority. My feeling is that we definitely want the medium priority things done before we roll out to more than friendly users as they're going to significantly color users first impression of flux.

[grondo]

Thanks @ryanday36! It is very helpful to know that the ability to modify jobs after submission is higher priority than --depend/--begin/--signal. Modification of jobspec after submission is going to be tricky for us, so we'll have to come with a strategy (and an interface) for that functionality asap.

[ryanday36]

No problem @grondo. It would still be good to have an overall discussion about priorities though. I feel like I use job modification tools a fair bit in the hotline, etc, but I'm not sure how much users use them, and I'm also not sure that it would really be all that bad to just make people resubmit the job (or maybe that's what the job modification tools will actually be doing). I prioritized things based mostly on what I'd like to have in order to administer flux on clusters. From a user experience perspective, getting those job submission flags sorted out might be a higher priority.