Supporting WCKey (or equivalent)

[SteVwonder]

Slurm has a feature --wckey that AFAIK exists solely for LLNL. The documentation for it states:

Specify wckey to be used with job. If TrackWCKey=no (default) in the slurm.conf this value is ignored. This option applies to job allocations.

IIUC, the WCKey is a piece of metadata associated with a job that is used for categorizing jobs. These categories are used when creating job usage reports. It appears that most users do not set this flag themselves, but the metadata is added by admins like Greg and @ryanday36 on the backend.

How do we want to go about supporting this within the Flux architecture?

[SteVwonder]

My first thought was that to handle the case where users request a specific WCKey be added to their job, we can leverage jobspec attributes. Maybe something like attributes.user.wckey?

Then the job-archive (or some other module) could be programmed to look for that key and insert it into the job database.

Adding the key on the backend then just becomes some SQL queries against the job database (which we could hide behind a front-end tool if need be). Before going too deep into design that particular piece though, I think we need some more input from Greg or @ryanday36 as to what criteria then use when tagging jobs with different WCKeys. Is it an entirely manual process? Or it is some automatable combination of user, account, and job size?

[ryanday36]

Greg or Ines set a default wckey for most users, and all of their jobs get tagged with that wckey so that we (Greg) can generate reports on usage by wckey. There may be some users who do work for multiple projects and have to remember to manually set a wckey on some of their jobs, but I believe that's more rare. I'll double check with Greg on that though. The other thing that inevitably happens is that, when Greg runs the report, there are some users who didn't have a default wckey set or had an old wckey, and then we have to go back in and retroactively set a wckey for the user. In those cases the wckey is basically set by who ran the job and when the job ran (they're set quarterly).

[grondo]

Would it make sense to have the wckey be an accounting database only thing, and not propagate it up to the user tools? The user->wckey mapping could be kept in the database instead of associated with each job. If the wckey needs to be changed quarterly I presume there is some database way to do this as well?

We'd still need a way for users to "characterize" their jobs if they are submitting work under multiple projects, but maybe a "project" attribute in jobspec would make more sense there? (and be less LLNL-specific)

[ryanday36]

Here's Greg's reply to my email asking how often users are manually setting the wckey on their job:

In practice, we probably have people who should be setting a wckey that’s different than their default for some percentage of the jobs they’re running. But the main source of error is not having up-to-date default wckeys for people, so that’s where we focus our time and energy.

This is not true on Sierra, where we have people running across multiple campaigns with different wckeys. But on that system we just force the wckey to be appropriate for the campaign, since there’s a one-to-one mapping between campaign/bank/wckey in that case.

-Greg

My feeling is that right now it would be fine if it was just something in the database, but it would be better if there was something that users could set in the job metadata. Calling it a "project" would be fine. That's what we use in LSF.