Change the unit of delay in stages from milliseconds to seconds

[yuanchen8911]

The delay field in a stage file is specified in milliseconds. I suggest changing it to seconds.

• Do we really need this level of granularity?
• We want to set a very large delay, but the default maximum value in YAML is 999,999. 999,999 milliseconds is too short for our tests.
  /cc @dmitsh

[yuanchen8911]

/cc @wzshiming

[dmitsh]

The problem I'm trying to solve is not related to the delay.
I'm trying to start a pod and keep it running until I decide to terminate it.
With the current stages, the pod becomes "Completed" right after "Running".

[yuanchen8911]

The delay parameters in the pod-complete stage determine how fast the pod will go from Running to Completed, aren't they? That's why I proposed the introduction of configurable delays.

[dmitsh]

Sometimes it is not possible to estimate how long the pod will run, especially for complex scenarios such as preemption.
We solved this issue by removing Job kind from the owner reference in pod-complete stage.

https://github.com/NVIDIA/knavigator/blob/main/charts/overrides/kwok/pod-complete.yml

[yuanchen8911]

We want to keep a job running and control when to stop it. This can be achieved by setting a very large duration, e.g., 1 year.

As we discussed internally, there may be a better solution for this use case, but KWOK needs to stay general and flexible for different use cases.

[dmitsh]

Setting a one-year expiration on a pod to allow indefinite operation is neither convenient nor user-friendly.
Additionally, the behavior of 'pod-complete' stage is inconsistent. It only completes the pod if it originates from a job. If the pod originates from another object, such as a replica set, it remains in the running state.
Nevertheless, we have devised a workaround.

[yuanchen8911]

To address @dmitsh's concerns regarding "neither convenient nor user-friendly", we have three options:

• Options 1: if KWOK stages can function as parametrizable templates, we can customize the default delay in the pod-complete stage with an arbitrary value. Thus, we won't need a change (i.e., adding annotations) to a pod. Would this be possible, @wzshiming?

• Option 2: kwok adds an additional stage file pod-never-complete.yaml and allows a user to specify which stage(pod-never-complete.yaml or pod-never-complete.yaml) to deploy? Can helm chart support it?

• Options 3: a user can always take the current pod-complete.yaml and set the default delay to a different value in their internal use. In our case, is this a simpler and safer workaround than removing the ownerReference? @dmitsh

Regardless, the current pod annotation approach works. We just need to add an additional pod annotation to the job templates.

The pod-complete stage only applies to a pod with the ownerReferences.kind: job. Therefore, a pod from a deployment will remain in the "Running" state until deletion.

[wzshiming]

The delay field in a stage file is specified in milliseconds. I suggest changing it to seconds.

• Do we really need this level of granularity?

You have the same idea as my original. 😄

The millisecond makes sense to achieve more fine-grained control over resource behavior, although it is not needed in most cases.

Here's the previous discussion
#74 (comment)

This was when Milliseconds was finally chosen for convenience.

I actually think that's probably a better definition, there wasn't enough time.
ref kubernetes/kubernetes#76951 (comment)

delayDuration:
  value: <uint64>
  valueFrom:
    expression: <expression-string>
  unit: <unit-string>

• We want to set a very large delay, but the default maximum value in YAML is 999,999. 999,999 milliseconds is too short for our tests.

YAML should have no limit on a number, and even if it did, it would be the maximum of uint64 9223372036854775808 (1<<64-1)

[yuanchen8911]

You are right. The maximum value is great enough. I just tried a value 31536000000 (1 year). It seems to be working.

 pod-complete.stage.kwok.x-k8s.io/delay: "31536000000"
  pod-complete.stage.kwok.x-k8s.io/jitter: "31536000000"