Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limit total CI run time to 24-ish hours. #331

Open
hategan opened this issue Jan 11, 2023 · 0 comments
Open

Limit total CI run time to 24-ish hours. #331

hategan opened this issue Jan 11, 2023 · 0 comments

Comments

@hategan
Copy link
Collaborator

hategan commented Jan 11, 2023

The CI tests do not currently have a time limit. Some of the tests that involve queing systems do have some explicit timeouts, but that's rather specific.

It is possible for certain tests to be buggy and trigger a deadlock. This can happen in a PR, so a faulty PR that may not even be merged could cause a CI run to never finish. On some machines with NFS mounts, this may lead to some file locking issues that could prevent subsequent CI runs from happening. This, in turn, means that one needs to manually log into test machines and kill the deadlocked process. This is less than ideal.

Since CI runs are technically supposed to run every 24 hours, there is somewhat of an implicit assumption that they should last less than that. Exceeding that can be an issue via the NFS problem mentioned above and also for @at, since a new run is scheduled only after the previous run finishes.

Anyway, we should put a cap of, say, 23 hours on a CI run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant