Slow operation

[sheepdragon]

Why is such a simple example so time-consuming

[jrbe228]

This minimum ~10s delay may be the biggest block from our organization adopting ArgoWF at larger scale. There may be no easy solution given the delays fundamental to Kubernetes itself. It's frustrating when compared to docker run hello-world taking milliseconds.

I tried the WASM plugin as a potential workaround, but got even slower results - Shark/wasm-workflows-plugin#16, at least initially.

I see other discussions suggesting the argo-dataflow (now numaflow) project for data processing workflows. Is this the only known alternative? How would we adapt Argo workflows with legacy containers to a data-centric framework?

[jswxstw]

I initially tried the HTTP Template, but its functionality was quite limited, so later I implemented a custom CLI-based plugin that serves HTTP requests containing CLI parameters.

[jrbe228]

Interesting. Reducing the Env Vars ARGO_AGENT_PATCH_RATE and DEFAULT_REQUEUE_TIME did improve performance. They are set to 10s by default. Reducing to 1s caused approx 30% improvement (10s to 7s) in completion time for trivial workflows like hello-world. There was no further benefit from reducing the vars into the millisecond range.

I also tried the suggested HTTP Template workflow. The core workflow step "get-google-homepage" completes in 3 to 5 seconds. For some reason the parent workflow continues to "spin" for several minutes , as if still executing.

Lastly, I retried WASM plugin execution with the new Env Var config. Execution time dropped from ~30 seconds I described in this issue to ~5 seconds. A very encouraging result!

[jrbe228]

@jswxstw - for your custom CLI-based HTTP plugin, how fast do minimal workflows execute? The best results I see from the WASM plugin around 4 seconds:

[jswxstw]

Argo is not inherently a low-latency task scheduling system, what you've measured now is already almost the minimum time consumption (the agent pod creation itself takes several seconds).
Also, reducing DEFAULT_REQUEUE_TIME may increase the load of APIServer, as the scale of the workflow increases, the time taken for pod creation and various reconciliation will significantly increase, which is uncontrollable.

[jrbe228]

Some latency appears fundamental to Kubernetes. Testing minimal pods takes around 4 seconds on AKS -
kubectl run hello-world --image=hello-world --restart=Never --rm -it

I imagine a bare-metal K8s distro could improve somewhat on latency. One less layer of virtualization.

Otherwise I would need some kind of server/client framework using WASM pods. I tried to describe it here.