Add ability to overwrite file retention level for specific executables from config

[GarethCabournDavies]

Use case:

• We are reusing a large bank / phase-time-amp files for many analyses

Problem:

• We don't want to copy the file into every run directory as it is wasting disk space, and removing the "single source of truth" by having multiple copies of the same information

Solution:

• Add an option in the pegasus_profile section for the executables to overwrite the retention level with the pycbc|retention_level flag, i.e.:

[pegasus_profile-bank2hdf]
pycbc|retention_level = do_not_keep

• Add an extra retention level into the choices which is 0, and therefore less than the global_retention_threshold, so the file is not retained (this comparison is False)

Testing:

• File is not retained in the standard offline workflow
• File is not retained from a --cache-file during generation

[spxiwh]

Note that there are two distinct "values" here, which are somewhat confused and confusing. See also the declarations on lines 73 - 76.

There's the global retention level "This workflow will keep jobs if they are level X or higher". For the global retention level (which these options are), lower numbers mean "keep more", so the level 0 choice here doesn't really make much sense, as level 0 should be "keep everything" at the global level.

Then there's the local level (lines 73-76), which is "How important are the files produced by this Job?". In that case, higher means more likely to be stored. (From 4=always store me, to 0=never store me). I think a DO_NOT_KEEP level 0 option might belong at line 72, but not here.

Then I just need one more comment on a later line .....

[spxiwh]

This is the local not the global retention level. It should therefore refer to the stuff in lines 73-76. Be aware that there is a "level 5" for the local retention level, which is "user needs to set this, warn them and set to 1" (see line 497, 95 and 100). This should be moved to 6 if you wanted a "NEVER_KEEP" possibility.

[spxiwh]

... Rethinking that, as level 4 is "ALWAYS_KEEP", so "NEVER_KEEP" would be 0 and the default remains the special case at 5.

[GarethCabournDavies]

I have update to use the local level rather than global - (this actually seems a lot neater)

[titodalcanton]

Seems functionally ok to me. I am assuming @spxiwh's comments above have been addressed/answered. You could remove some duplication by introducing a variable

retention_opt_args = (
    f'pegasus_profile-{name}',
    'pycbc|retention_level',
    tags
)

and then passing *retention_opt_args to the two functions that need them.

[maxtrevor]

All looks good to me

[maxtrevor]

All looks good to me

Never mind, I see this was already reviewed and merged by Tito lol