Allow manual trigger when workflow paused.

[hjoliver]

Close #5963

I've added the "question" label because there's some disagreement on the issue itself:

@oliver-sanders has argued that manual triggering (i.e., with immediate effect) should only be allowed with cylc hold */* (hold all tasks individually) and not cylc pause (pause the workflow).

However:

• we haven't implemented cylc hold */* yet, and it may not be trivial to do it
• the only solution at the moment is "hold after cycle point", which is not intuitive at all
• two primary Cylc sites have highlighted this as a must-have for operational migration - @dwsutherland, @ColemanTom
• this is very easy to implement (+13/-7 code lines without tests) so at the very least it provides a working interim solution
• and finally, I cannot see any problem with allowing manual triggering in a paused workflow - there's nothing wrong with defining workflow pause as pausing all automatic job submission

Check List

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
• Tests are included (or explain why tests are not needed).
• CHANGES.md entry included if this is a change that can affect users
• Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
• If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

[oliver-sanders]

I would prefer to have a clean separation of play/pause and hold/release rather than crossing the concepts. I'm also not convinced by the "operational blocker" statement, especially as there is a documented workaround (i.e. this is a preference not a requirement).

However, I am quite happy to concede if this is desired. Removing the question label.

This change shouldn't really go into a bugfix release, but it's small enough that I think it's ok to sneak it in if we really want to.

[oliver-sanders]

From a quick functional test, this works as expected:

• cylc trigger causes a task to submit
• cylc set --pre=all does not.

[oliver-sanders]

We would expect this functionality to work the same across all queue implementations, so we could really do with abstracting it into the base class rather than relying on each queue to implement it separately.

Note: If we could bypass the queue entirely that would be ideal.

Idea (for future work), write some abstract tests that should hold for all queue implementations.

[oliver-sanders]

there's nothing wrong with defining workflow pause as pausing all automatic job submission

This has gone a little further than that one word change above as we are now continuing to process the queues whilst the workflow is paused making the paused state a little more dynamic than before.

I don't think we actually need to process the entire queue to achieve this, we just need to skip any manually triggered tasks through the queue (ideally bypassing the queue itself if possible).

If we pass self.is_paused through to release_queued_tasks we could reduce the scope of this override such that it only impacts manually triggered tasks.

[oliver-sanders]

Needs tests (could do with covering the behaviour w/ set --pre=all and trigger), CLI help and a cylc-doc PR to update the interventions page & glossary.

[hjoliver]

(Will undraft again soon, once I've tried your suggestion re queue processing @oliver-sanders )

[hjoliver]

(Damn it, thought I had completed this ... it's wanted at NIWA ... coming back to it soon).

[ColemanTom]

One question sprung to mind just now @hjoliver - would this proposed change break the workaround for #6308 ?