Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

handle job log directory deleted for active task #6425

Open
oliver-sanders opened this issue Oct 17, 2024 · 0 comments
Open

handle job log directory deleted for active task #6425

oliver-sanders opened this issue Oct 17, 2024 · 0 comments
Labels
bug Something is wrong :(
Milestone

Comments

@oliver-sanders
Copy link
Member

Spotted in the wild!

If you delete the job log directory for an active task, Cylc will preserve its last known status indefinitely. I.e, Cylc will consider the job to be submitted/running forever.

In this case it was caused by a housekeep task being triggered whilst other tasks in the cycle were still running. The housekeep task tarred up the log/job/<cycle> dir removing the job status files in the process.

This situation should be handled similarly to the job no longer appearing in the queue, i.e, the job is dead, long live the job. Stick it into the failed/submit-failed state as appropriate.

@oliver-sanders oliver-sanders added the bug Something is wrong :( label Oct 17, 2024
@oliver-sanders oliver-sanders added this to the 8.3.x milestone Oct 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something is wrong :(
Projects
None yet
Development

No branches or pull requests

1 participant