trigger --flow=new doesn't trigger downstream tasks with off-flow dependencies #6265
Comments
ColemanTom
changed the title
|
NOTE: I'm only talking about cross-cycle dependency, but I imagine if you have |
|
For me, unless there is a workaround, this is a big problem as I can't see how to trigger a new flow correctly when tasks have extra dependencies other than the task you have triggered. |
|
Trying with 8.2.3 and 8.2.7, there was an oddity back then, but it didn't stall a workflow. The b task didn't ever rerun, it just got removed from the graph. So, that was a bug too I think. It has just evolved and become more visible. The old version was probably more dangerous as it gave false confidence tasks were run. |
Prior to 8.3.0 tasks with partially satisfied prerequisites were "hidden" to make it look as if they got spawned (into the active task pool) once all their prerequisites were satisfied, rather than when the first prerequisite was satisfied. This was a decision made early in implementation of the Cylc 8 scheduling algorithm that we later decided was a mistake. |
|
This is not a bug, but you've highlighted something that we really need to hammer home in documenting manual interventions in Cylc 8. A flow is a self-consistent, self-sustaining run through the graph, and as such the prerequisites of all affected tasks must be satisfied within the flow. It has to be that way because if you start a new flow in the past then technically ALL of the prerequisites (in the past graph) have already been satisfied by the original flow - so if we allowed old-flow outputs to satisfy new-flow prerequisites then all of the new-flow tasks could run at once without waiting on any dependencies. Also, given that graph edges typically represent real (file) dependencies, it's perfectly possible that you would not want your next-cycle task All you need to do is manually set Note we have recently discussed an easier way to automatically detect off-flow prerequisites, for cases when you can easily identify all the tasks that should run in the new flow, but I can't find the issue right now (maybe it was in Element chat...) ... I'll make sure that's captured in an issue. |
|
@ColemanTom - if you're happy with my explanation, I'll change the title of this issue (the cross-cycle dependence is not relevant) and close it. |
|
Yes, happy for the title to be changed but I think this is a major flaw. For complex graphs, that could be many |
hjoliver
changed the title
|
I disagree that it's a flaw, let alone a major flaw. If you want to run arbitrary new flows within the same graph then dealing with off-flow prerequisites is unavoidable - when the new flow spawns a new task, how should we determine which of its other prerequisites must wait on parents within the new flow and which to take as magically satisfied by the old flow? This applies to Cylc 7 too, although in simple cases you could get away with not really understanding that if all the affected tasks remained in the task pool, and if you knew which tasks they were so you could manually reset them all to waiting to set up the flow. (Arbitrary new flows from any point in the graph were basically impossible pre Cylc 8). The good news is (the discussion/issue I referred to just above) those cases (the ones where you could fake a limited new flow with a bunch of manual state resetting to set it up) will soon be much easier in Cylc 8. Not by means of fixing a flaw, but by making it easy to say "here's all the tasks that can participate in the new flow, and any dependence on tasks outside of that group should be ignored" (where it is appropriate to ignore off-flow dependence, which as I said is not always the case). |
In Cylc 7 that required a warm start, which automatically ignores off-flow prerequisites (which in this case means previous-cycle dependence). But if you want to do that in a running Cylc 7 workflow, good luck! |
In cylc 7 I could just reset the whole cycle to waiting and it worked. I know you have said in the past that is just coincidence, but with how the graphs were setup, it was a safe assumption that it would work for the models I'm talking about. I only work in ops, I don't reset past the current cycle. I'm never discussing rewinding more than the current cycle. |
|
Oh, OK, understood - I'll partially retract my previous "good luck!" comment then! However, that working relies on ALL the tasks in the entire cycle point that you want to rerun remaining in the Cylc 7 task pool. Rerunning an entire cycle actually tends to be simpler than the general case - for full-cycles we can be confident that off-flow prerequisites are (a) easily identified, and (b) the user definitely wants to ignore them. In Cylc 8 you'll soon be able to do that too, with something like "cylc trigger --cycle=x". But for the moment, you have to do it the general way:
|
That is a very fair comment that I had not considered. It just is making my job very difficult right now as I'm trying to figure out how to adjust support instructions to people who will not be able to understand Cylc graphs and have limited time to do lots of If you remember the |
|
Quick question with partial rewind of a cycle If I have processed up to hour=4 and want to rewind to hour=2. If I trigger a new flow from |
That's a fair comment too - we've had exactly the same from the UK support people. Initially my opinion was it's just a matter of getting used to a better - albeit new and different! - way to do things, but I've grudgingly conceded that it may genuinely be easier to do some kinds of restricted runtime interventions in Cylc 7 compared to Cylc 8 as it is right now. Power and generality will win out in the end, but unfortunately that doesn't help you much in the present. |
The upcoming task metadata UI view is a very easy win at this point (the data is available to the UI and @oliver-sanders already mocked up an example), but we've not had time to prioritize it just yet. Maybe something your site could look at? |
@ScottWales I'm guessing you wouldn't have time to do this? |
|
Having seen @ScottWales multi-user GUI, he would find this super-easy 😁 |
Can you just confirm that the two different parameterized forms of |
Updated. They were mistakes. Initially I had |
Yes.
No (good news!) - you can set those prerequisites ahead of time, at the same time you trigger the new flow (note for multiple ops like this you can only use Equivalently, in this case, you can set the upstream outputs of the
|
I can have a bit of a look, can't put that much time into it though
Is there a scriptable way to get the flow number that |
Sometimes managers just need a bit of subtle persuasion :-)
We plan to expose this in the UI soon. For the moment:
Log: DB: $ sqlite3 -header ~/cylc-run/tom/runN/log/db 'select cycle, name, flow_nums, status from task_states'
cycle|name|flow_nums|status
1|wait_hour0|[1]|succeeded
1|wait_hour1|[1]|succeeded
1|task1_hour0|[1]|succeeded
1|task1_hour1|[1]|succeeded
1|wait_hour2|[1]|succeeded
1|task2_hour0|[1]|succeeded
1|task2_hour1|[1]|succeeded
... |
|
I'll close this issue. Summary:
|
See #5416 |
|
FYI @ColemanTom -
|
How does one pre-set the prereqs? |
|
Just use |
Description
This is only tested with CYLC_VERSION=8.3.3
UPDATE: Tested in 8.2.3, 8.2.7 and a similar (perhaps more dangerous version) of the bug was found there too.
When I have a graph like shown below, if I trigger a new flow for the
atask for a cycle, thebtask never gets rerun. I thought it would be because it should be part of the flow for that graph. Instead, it gets stuck as waiting. I'm not sure, but this may lead to longer-term issues with subsequent cycles?Reproducible Example
flow.cylc for a workflow I've called
basic-workflow:cylc trigger --flow=new basic-workflow/run3 //20240729/aExpected Behaviour
btask should run as it is a brand new flow. That the other tasks it relies on didn't get rerun as a new flow I don't think should matter.The text was updated successfully, but these errors were encountered: