[Bug] Benchmarking script failing to capture logs in EKS

[srinarayan-srikanthan]

I am using this script (https://github.com/opea-project/GenAIEval/tree/main/evals/benchmark) to capture data for chatqna. It works on baremetal but failing with the following error on EKS.

Exception: Tried to stop User in an unexpected state: stopping. This should never happen.

2024-11-27T00:39:12Z <Greenlet at 0x7fb4215eac20: <bound method User.stop of <aistress.AiStressUser object at 0x7fb428353610>>(force=False)> failed with Exception


[]

[]

[]

Exception when retrieving ConfigMap: (404)

Reason: Not Found

HTTP response headers: HTTPHeaderDict({'Audit-Id': 'bc671b52-6979-4a63-a49b-f62d27d59f6d', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '4916798f-ba14-43f8-a168-6ef5d36d86e2', 'X-Kubernetes-Pf-Prioritylevel-Uid': '22c70de0-7569-474e-b81b-bf3e48a02906', 'Date': 'Wed, 27 Nov 2024 00:39:54 GMT', 'Content-Length': '208'})

HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"kubelet-config\" not found","reason":"NotFound","details":{"name":"kubelet-config","kind":"configmaps"},"code":404} 

[eero-t]

kubelet-config configMap seems to be created by kubeadm init when cluster is created: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/kubelet-integration/#workflow-when-using-kubeadm-init

I assume it's missing when something else is used to create the cluster.

Failing benchmark code should handle such case better:
https://github.com/opea-project/GenAIEval/blob/main/evals/benchmark/stresscli/commands/utils.py

Btw. Another bug in that same code is it assuming cluster to be using static policy when policy is != none, instead of using the policy name returned by the called function...

[poussa]

You don't have kubelet-config configmap in EKS cluster. The benchmark script needs to handle that case and not to fail as it is doing now.