Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error updating webhook with certificate {"error": "Operation cannot be fulfilled on apiservices.apiregistration.k8s.io \"v1beta1.external.metrics.k8s.io\": the object has been modified; please apply your changes to the latest version and try again"} #6551

Open
zoechou opened this issue Feb 14, 2025 · 7 comments
Open

Error updating webhook with certificate {"error": "Operation cannot be fulfilled on apiservices.apiregistration.k8s.io \"v1beta1.external.metrics.k8s.io\": the object has been modified; please apply your changes to the latest version and try again"} #6551

zoechou opened this issue Feb 14, 2025 · 7 comments
Labels
bug Something isn't working

Comments

@zoechou
Copy link

zoechou commented Feb 14, 2025

Report

Hi support,
we upgraded our keda to 2.16 on 13 Feb 2025 with keda helm-chart ,
and found keda-operator would stop providing service if encounter cert-rotation issue.
didn't catch the keda-metrics-api-server log, but just error that cannot establish connection with keda-operator.

[hpa events]

Warning   FailedGetExternalMetric   horizontalpodautoscaler/workers   unable to get external metric insert/s1-prometheus/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: wokers,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server is currently unable to handle the request (get s1-prometheus.external.metrics.k8s.io)

Expected Behavior

keda-operator can rotate the cert successfully and provide service

Actual Behavior

keda-operator failed to rotate the cert and stop working, keda-metrics-api-server was affected.

Steps to Reproduce the Problem

We just wait for one day, and we'll encounter issue on cert-rotation

Logs from KEDA operator

2025-02-14T22:17:39Z    INFO    cert-rotation    no cert refresh needed
2025-02-14T22:17:39Z    INFO    cert-rotation    Ensuring CA cert    {"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2025-02-14T22:17:39Z    INFO    cert-rotation    Ensuring CA cert    {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2025-02-14T22:17:39Z    INFO    cert-rotation    no cert refresh needed
2025-02-14T22:17:39Z    INFO    cert-rotation    Ensuring CA cert    {"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2025-02-14T22:17:39Z    INFO    cert-rotation    Ensuring CA cert    {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2025-02-14T22:33:51Z    INFO    cert-rotation    no cert refresh needed
2025-02-14T22:33:51Z    INFO    cert-rotation    Ensuring CA cert    {"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2025-02-14T22:33:51Z    INFO    cert-rotation    Ensuring CA cert    {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2025-02-14T22:33:51Z    INFO    cert-rotation    no cert refresh needed
2025-02-14T22:33:51Z    INFO    cert-rotation    Ensuring CA cert    {"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2025-02-14T22:33:51Z    INFO    cert-rotation    Ensuring CA cert    {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2025-02-14T22:40:21Z    INFO    cert-rotation    no cert refresh needed
2025-02-14T22:40:21Z    INFO    cert-rotation    Ensuring CA cert    {"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2025-02-14T22:40:21Z    INFO    cert-rotation    Ensuring CA cert    {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2025-02-14T22:40:21Z    ERROR    cert-rotation    Error updating webhook with certificate    {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "error": "Operation cannot be fulfilled on apiservices.apiregistration.k8s.io \"v1beta1.external.metrics.k8s.io\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).ensureCerts
    /workspace/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:845
github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).Reconcile
    /workspace/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:791
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:116
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:224
2025-02-14T22:40:21Z    ERROR    Reconciler error    {"controller": "cert-rotator", "object": {"name":"kedaorg-certs","namespace":"keda"}, "namespace": "keda", "name": "kedaorg-certs", "reconcileID": "7367f7ca-aee4-4c85-a227-2b183d48eeab", "error": "Operation cannot be fulfilled on apiservices.apiregistration.k8s.io \"v1beta1.external.metrics.k8s.io\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:224
2025-02-14T22:40:21Z    INFO    cert-rotation    no cert refresh needed
2025-02-14T22:40:21Z    INFO    cert-rotation    Ensuring CA cert    {"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2025-02-14T22:40:21Z    INFO    cert-rotation    Ensuring CA cert    {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2025-02-14T22:40:21Z    INFO    cert-rotation    no cert refresh needed
2025-02-14T22:40:21Z    INFO    cert-rotation    Ensuring CA cert    {"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2025-02-14T22:40:21Z    INFO    cert-rotation    Ensuring CA cert    {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}

KEDA Version

2.16.1

Kubernetes Version

< 1.29

Platform

Amazon Web Services

Scaler Details

AWS SQS scaler, prometheus scaler

Anything else?

NA
@zoechou zoechou added the bug Something isn't working label Feb 14, 2025
@JorTurFer
Copy link
Member

Hello
That background process can sometimes fail but it shouldn't stop the service as it works in parallel with other controllers. Has your certificate expired and KEDA is trying to renew it or so? cert-controller (the used tool) generates 10 years certificate, so it shouldn't have expired and when the certificate is valid, the service shouldn't be affected
Has your service been affected in terms of scaling?

@zoechou
Copy link
Author

zoechou commented Feb 18, 2025

Hello @JorTurFer ,
The certificate is still valid, as you can from logs no cert refresh needed.
But it still record

2025-02-14T22:40:21Z    ERROR    cert-rotation    Error updating webhook with certificate    {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "error": "Operation cannot be fulfilled on apiservices.apiregistration.k8s.io \"v1beta1.external.metrics.k8s.io\": the object has been modified; please apply your changes to the latest version and try again"}

I found when keda-operator kept reporting this error per 6 minutes, keda-metrics-api-server will be failed to connect to keda-operator, and service been affected in terms of scaling.

We need to restart keda-operator to let it back to normal, thanks.

I'm trying to disable cert rotation and see if we'll encounter the issue again, thanks.

--enable-cert-rotation=false

@JorTurFer
Copy link
Member

if you're deploying KEDA using helm chart and you use cert-manager in your cluster too, you can delegate the certificate management to cert-manager (which is the best idea if you are already using KEDA's helm chart and cert-manager)
https://github.com/kedacore/charts/blob/main/keda/values.yaml#L799

@zoechou
Copy link
Author

zoechou commented Feb 21, 2025

No, we didn't deploy cert-manager in our cluster.

But I still think it's part of issue on keda-operator, can you help check?
thanks.

We've disabled cert-rotation for now, it works as expect for at least 5 days...

@JorTurFer
Copy link
Member

JorTurFer commented Feb 21, 2025 •
edited
Loading

There is nothing that we can do because the error is related with the object being modified at the same time as other (are you using ArgoCD/Flux syncing the chart?)

2025-02-14T22:40:21Z    ERROR    Reconciler error    {"controller": "cert-rotator", "object": {"name":"kedaorg-certs","namespace":"keda"}, "namespace": "keda", "name": "kedaorg-certs", "reconcileID": "7367f7ca-aee4-4c85-a227-2b183d48eeab", "error": "Operation cannot be fulfilled on apiservices.apiregistration.k8s.io \"v1beta1.external.metrics.k8s.io\": the object has been modified; please apply your changes to the latest version and try again"}

Once the operator starts, that process doesn't stop the communication between metrics server and the operator AFAIK. That failing process can stop the startup but not the process ongoing.

You can keep it disabled if you want because the signed certificate is for 10 years, so if you don't change it, you won't see any issue in the future

@zoechou
Copy link
Author

zoechou commented Feb 25, 2025

We have argoCD in our cluster, but we didn't maintain core keda component with argoCD.
I think it'll not be affected if we manage scaledObject with Argo?
thanks

@yo-l1982
Copy link

There is nothing that we can do because the error is related with the object being modified at the same time as other (are you using ArgoCD/Flux syncing the chart?)

2025-02-14T22:40:21Z ERROR Reconciler error {"controller": "cert-rotator", "object": {"name":"kedaorg-certs","namespace":"keda"}, "namespace": "keda", "name": "kedaorg-certs", "reconcileID": "7367f7ca-aee4-4c85-a227-2b183d48eeab", "error": "Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.external.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again"}

Once the operator starts, that process doesn't stop the communication between metrics server and the operator AFAIK. That failing process can stop the startup but not the process ongoing.

You can keep it disabled if you want because the signed certificate is for 10 years, so if you don't change it, you won't see any issue in the future

Hello!
I got the same issue and am using ArgoCD.
Im not entirely sure I understand this, is it some resource I should exclude from getting synced by Argo?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
Roadmap - KEDA Core
Status: To Triage
Milestone
No milestone
Development

No branches or pull requests
3 participants
@yo-l1982 @zoechou @JorTurFer