You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The CI tests do not currently have a time limit. Some of the tests that involve queing systems do have some explicit timeouts, but that's rather specific.
It is possible for certain tests to be buggy and trigger a deadlock. This can happen in a PR, so a faulty PR that may not even be merged could cause a CI run to never finish. On some machines with NFS mounts, this may lead to some file locking issues that could prevent subsequent CI runs from happening. This, in turn, means that one needs to manually log into test machines and kill the deadlocked process. This is less than ideal.
Since CI runs are technically supposed to run every 24 hours, there is somewhat of an implicit assumption that they should last less than that. Exceeding that can be an issue via the NFS problem mentioned above and also for @at, since a new run is scheduled only after the previous run finishes.
Anyway, we should put a cap of, say, 23 hours on a CI run.
The text was updated successfully, but these errors were encountered:
The CI tests do not currently have a time limit. Some of the tests that involve queing systems do have some explicit timeouts, but that's rather specific.
It is possible for certain tests to be buggy and trigger a deadlock. This can happen in a PR, so a faulty PR that may not even be merged could cause a CI run to never finish. On some machines with NFS mounts, this may lead to some file locking issues that could prevent subsequent CI runs from happening. This, in turn, means that one needs to manually log into test machines and kill the deadlocked process. This is less than ideal.
Since CI runs are technically supposed to run every 24 hours, there is somewhat of an implicit assumption that they should last less than that. Exceeding that can be an issue via the NFS problem mentioned above and also for @at, since a new run is scheduled only after the previous run finishes.
Anyway, we should put a cap of, say, 23 hours on a CI run.
The text was updated successfully, but these errors were encountered: