Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FailedDiscoveryCheck for v1beta1.external.metrics.k8s.io APIService #6572

Open
patrostkowski opened this issue Feb 26, 2025 · 0 comments
Open

FailedDiscoveryCheck for v1beta1.external.metrics.k8s.io APIService #6572

patrostkowski opened this issue Feb 26, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@patrostkowski
Copy link

Report

I am trying to install KEDA but APIService v1beta1.external.metrics.k8s.io is not available, which results in broken KEDA.

Expected Behavior

v1beta1.external.metrics.k8s.io APIService is available and ScaledObject actually scales the deployment.

Actual Behavior

When I create any ScaledObject CRD, its created HPA reports:

$ k describe hpa keda-hpa-communications-service 
Name:                                      keda-hpa-communications-service
Namespace:                                 stage
Labels:                                    app.kubernetes.io/managed-by=keda-operator
                                           app.kubernetes.io/name=keda-hpa-communications-service
                                           app.kubernetes.io/part-of=communications-service
                                           app.kubernetes.io/version=2.16.1
                                           scaledobject.keda.sh/name=communications-service
Annotations:                               <none>
CreationTimestamp:                         Wed, 26 Feb 2025 14:07:30 +0100
Reference:                                 Deployment/communications-service
Metrics:                                   ( current / target )
  "s0-prometheus" (target average value):  <unknown> / 10
Min replicas:                              1
Max replicas:                              10
Deployment pods:                           3 current / 3 desired
Conditions:
  Type            Status  Reason                   Message
  ----            ------  ------                   -------
  AbleToScale     True    ReadyForNewScale         recommended size matches current size
  ScalingActive   False   FailedGetExternalMetric  the HPA was unable to compute the replica count: unable to get external metric stage/s0-prometheus/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: communications-service,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server is currently unable to handle the request (get s0-prometheus.external.metrics.k8s.io)
  ScalingLimited  False   DesiredWithinRange       the desired count is within the acceptable range
Events:
  Type     Reason                   Age   From                       Message
  ----     ------                   ----  ----                       -------
  Warning  FailedGetExternalMetric  14s   horizontal-pod-autoscaler  unable to get external metric stage/s0-prometheus/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: communications-service,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server is currently unable to handle the request (get s0-prometheus.external.metrics.k8s.io)

$ kubectl get apiservices v1beta1.external.metrics.k8s.io
NAME                              SERVICE                                AVAILABLE                      AGE
v1beta1.external.metrics.k8s.io   keda/keda-operator-metrics-apiserver   False (FailedDiscoveryCheck)   2m58s

$ kubectl get apiservices v1beta1.external.metrics.k8s.io -o yaml
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  annotations:
    meta.helm.sh/release-name: keda
    meta.helm.sh/release-namespace: keda
  creationTimestamp: "2025-02-26T13:01:39Z"
  labels:
    app.kubernetes.io/component: operator
    app.kubernetes.io/instance: keda
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: v1beta1.external.metrics.k8s.io
    app.kubernetes.io/part-of: keda-operator
    app.kubernetes.io/version: 2.16.1
    helm.sh/chart: keda-2.16.1
  name: v1beta1.external.metrics.k8s.io
  resourceVersion: "371635203"
  uid: 43665587-6141-486d-8779-e521a4abcf20
spec:
  caBundle: ...
  group: external.metrics.k8s.io
  groupPriorityMinimum: 100
  service:
    name: keda-operator-metrics-apiserver
    namespace: keda
    port: 443
  version: v1beta1
  versionPriority: 100
status:
  conditions:
  - lastTransitionTime: "2025-02-26T13:01:39Z"
    message: 'failing or missing response from https://10.102.2.41:6443/apis/external.metrics.k8s.io/v1beta1:
      Get "https://10.102.2.41:6443/apis/external.metrics.k8s.io/v1beta1": net/http:
      request canceled while waiting for connection (Client.Timeout exceeded while
      awaiting headers)'
    reason: FailedDiscoveryCheck
    status: "False"
    type: Available

This results in broken scaling, which never occurs.

Steps to Reproduce the Problem

  1. get GKE cluster (v1.30.9-gke.1009000)
  2. helm install keda kedacore/keda --namespace keda --create-namespace
  3. kubectl describe apiservices v1beta1.external.metrics.k8s.io

Logs from KEDA operator

Operator:

2025-02-26T13:16:19Z	INFO	setup	maxprocs: Updating GOMAXPROCS=1: determined from CPU quota
2025-02-26T13:16:19Z	DEBUG	setup	setting up cert rotation
2025-02-26T13:16:19Z	INFO	setup	Starting manager
2025-02-26T13:16:19Z	INFO	setup	KEDA Version: 2.16.1
2025-02-26T13:16:19Z	INFO	setup	Git Commit: ce14b239e0300f388b0425aef68154d8070cd66f
2025-02-26T13:16:19Z	INFO	setup	Go Version: go1.23.4
2025-02-26T13:16:19Z	INFO	setup	Go OS/Arch: linux/amd64
2025-02-26T13:16:19Z	INFO	setup	Running on Kubernetes 1.30	{"version": "v1.30.9-gke.1009000"}
2025-02-26T13:16:19Z	INFO	starting server	{"name": "health probe", "addr": "[::]:8081"}
I0226 13:16:19.357413       1 leaderelection.go:254] attempting to acquire leader lease keda/operator.keda.sh...
I0226 13:17:00.039318       1 leaderelection.go:268] successfully acquired lease keda/operator.keda.sh
2025-02-26T13:17:00Z	DEBUG	events	keda-operator-586974ffdb-59sl2_0235cb71-7cbe-4c37-8917-c59eab7df873 became leader	{"type": "Normal", "object": {"kind":"Lease","namespace":"keda","name":"operator.keda.sh","uid":"23ee039c-49a9-4f4b-bb1d-380e0e2e68a9","apiVersion":"coordination.k8s.io/v1","resourceVersion":"371645617"}, "reason": "LeaderElection"}
2025-02-26T13:17:00Z	INFO	Starting EventSource	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "source": "kind source: *v1alpha1.ScaledObject"}
2025-02-26T13:17:00Z	INFO	Starting EventSource	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "source": "kind source: *v2.HorizontalPodAutoscaler"}
2025-02-26T13:17:00Z	INFO	Starting Controller	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject"}
2025-02-26T13:17:00Z	INFO	Starting EventSource	{"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication", "source": "kind source: *v1alpha1.TriggerAuthentication"}
2025-02-26T13:17:00Z	INFO	Starting Controller	{"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication"}
2025-02-26T13:17:00Z	INFO	Starting EventSource	{"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "source": "kind source: *v1alpha1.ScaledJob"}
2025-02-26T13:17:00Z	INFO	Starting Controller	{"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob"}
2025-02-26T13:17:00Z	INFO	Starting EventSource	{"controller": "cloudeventsource", "controllerGroup": "eventing.keda.sh", "controllerKind": "CloudEventSource", "source": "kind source: *v1alpha1.CloudEventSource"}
2025-02-26T13:17:00Z	INFO	Starting Controller	{"controller": "cloudeventsource", "controllerGroup": "eventing.keda.sh", "controllerKind": "CloudEventSource"}
2025-02-26T13:17:00Z	INFO	Starting EventSource	{"controller": "clustertriggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "ClusterTriggerAuthentication", "source": "kind source: *v1alpha1.ClusterTriggerAuthentication"}
2025-02-26T13:17:00Z	INFO	Starting Controller	{"controller": "clustertriggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "ClusterTriggerAuthentication"}
2025-02-26T13:17:00Z	INFO	Starting EventSource	{"controller": "clustercloudeventsource", "controllerGroup": "eventing.keda.sh", "controllerKind": "ClusterCloudEventSource", "source": "kind source: *v1alpha1.ClusterCloudEventSource"}
2025-02-26T13:17:00Z	INFO	Starting Controller	{"controller": "clustercloudeventsource", "controllerGroup": "eventing.keda.sh", "controllerKind": "ClusterCloudEventSource"}
2025-02-26T13:17:00Z	INFO	Starting EventSource	{"controller": "cert-rotator", "source": "kind source: *v1.Secret"}
2025-02-26T13:17:00Z	INFO	Starting EventSource	{"controller": "cert-rotator", "source": "kind source: *unstructured.Unstructured"}
2025-02-26T13:17:00Z	INFO	Starting EventSource	{"controller": "cert-rotator", "source": "kind source: *unstructured.Unstructured"}
2025-02-26T13:17:00Z	INFO	Starting Controller	{"controller": "cert-rotator"}
2025-02-26T13:17:00Z	INFO	Starting workers	{"controller": "cert-rotator", "worker count": 1}
2025-02-26T13:17:00Z	INFO	Starting workers	{"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication", "worker count": 1}
2025-02-26T13:17:00Z	INFO	Starting workers	{"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "worker count": 1}
2025-02-26T13:17:00Z	INFO	Starting workers	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "worker count": 5}
2025-02-26T13:17:00Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"communications-service","namespace":"stage"}, "namespace": "stage", "name": "communications-service", "reconcileID": "5ebd9bc9-67d3-40ad-9fd2-2a4e1ac96d48"}
2025-02-26T13:17:00Z	DEBUG	Parsed Group, Version, Kind, Resource	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"communications-service","namespace":"stage"}, "namespace": "stage", "name": "communications-service", "reconcileID": "5ebd9bc9-67d3-40ad-9fd2-2a4e1ac96d48", "GVK": "apps/v1.Deployment", "Resource": "deployments"}
2025-02-26T13:17:00Z	DEBUG	Status Group, Version, Kind, Resource	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"communications-service","namespace":"stage"}, "namespace": "stage", "name": "communications-service", "reconcileID": "5ebd9bc9-67d3-40ad-9fd2-2a4e1ac96d48", "GVK": "/v1.Deployment", "Resource": "deployments"}
2025-02-26T13:17:00Z	INFO	Starting workers	{"controller": "cloudeventsource", "controllerGroup": "eventing.keda.sh", "controllerKind": "CloudEventSource", "worker count": 1}
2025-02-26T13:17:00Z	INFO	Starting workers	{"controller": "clustercloudeventsource", "controllerGroup": "eventing.keda.sh", "controllerKind": "ClusterCloudEventSource", "worker count": 1}
2025-02-26T13:17:00Z	INFO	Starting workers	{"controller": "clustertriggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "ClusterTriggerAuthentication", "worker count": 1}
2025-02-26T13:17:00Z	INFO	Detected resource targeted for scaling	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"communications-service","namespace":"stage"}, "namespace": "stage", "name": "communications-service", "reconcileID": "5ebd9bc9-67d3-40ad-9fd2-2a4e1ac96d48", "resource": "apps/v1.Deployment", "name": "communications-service"}
2025-02-26T13:17:00Z	DEBUG	Updating ScaledObject status with triggers and authentications types	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"communications-service","namespace":"stage"}, "namespace": "stage", "name": "communications-service", "reconcileID": "5ebd9bc9-67d3-40ad-9fd2-2a4e1ac96d48", "triggersTypes": "prometheus", "authenticationsTypes": ""}
2025-02-26T13:17:00Z	INFO	KubeAPIWarningLogger	unknown field "status.authenticationsTypes"
2025-02-26T13:17:00Z	INFO	KubeAPIWarningLogger	unknown field "status.triggersTypes"
2025-02-26T13:17:00Z	DEBUG	certificates	the path /custom/ca doesn't exist, skipping custom CA registrations
2025-02-26T13:17:00Z	DEBUG	events	Scaler prometheus is built.	{"type": "Normal", "object": {"kind":"ScaledObject","namespace":"stage","name":"communications-service","uid":"7aa98852-8d87-41c2-82a7-99fdd47fe4e0","apiVersion":"keda.sh/v1alpha1","resourceVersion":"371644246"}, "reason": "KEDAScalersStarted"}
2025-02-26T13:17:00Z	DEBUG	Notify scaleHandler of an update in scaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"communications-service","namespace":"stage"}, "namespace": "stage", "name": "communications-service", "reconcileID": "5ebd9bc9-67d3-40ad-9fd2-2a4e1ac96d48"}
2025-02-26T13:17:00Z	INFO	Initializing Scaling logic according to ScaledObject Specification	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"communications-service","namespace":"stage"}, "namespace": "stage", "name": "communications-service", "reconcileID": "5ebd9bc9-67d3-40ad-9fd2-2a4e1ac96d48"}
2025-02-26T13:17:00Z	DEBUG	ScaledObject is defined correctly and is ready for scaling	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"communications-service","namespace":"stage"}, "namespace": "stage", "name": "communications-service", "reconcileID": "5ebd9bc9-67d3-40ad-9fd2-2a4e1ac96d48"}
2025-02-26T13:17:00Z	DEBUG	events	Started scalers watch	{"type": "Normal", "object": {"kind":"ScaledObject","namespace":"stage","name":"communications-service","uid":"7aa98852-8d87-41c2-82a7-99fdd47fe4e0","apiVersion":"keda.sh/v1alpha1","resourceVersion":"371644246"}, "reason": "KEDAScalersStarted"}
2025-02-26T13:17:00Z	DEBUG	scale_handler	Watching with pollingInterval	{"type": "ScaledObject", "namespace": "stage", "name": "communications-service", "PollingInterval": "10s"}
2025-02-26T13:17:00Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"communications-service","namespace":"stage"}, "namespace": "stage", "name": "communications-service", "reconcileID": "5ed1373d-e279-42d9-8808-d48a3615d3c4"}
2025-02-26T13:17:00Z	DEBUG	Parsed Group, Version, Kind, Resource	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"communications-service","namespace":"stage"}, "namespace": "stage", "name": "communications-service", "reconcileID": "5ed1373d-e279-42d9-8808-d48a3615d3c4", "GVK": "apps/v1.Deployment", "Resource": "deployments"}
2025-02-26T13:17:00Z	DEBUG	Status Group, Version, Kind, Resource	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"communications-service","namespace":"stage"}, "namespace": "stage", "name": "communications-service", "reconcileID": "5ed1373d-e279-42d9-8808-d48a3615d3c4", "GVK": "/v1.Deployment", "Resource": "deployments"}
2025-02-26T13:17:00Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "stage", "scaledObject.Name": "communications-service", "scaler": "prometheusScaler", "metricName": "s0-prometheus", "metrics": [{"metricName":"s0-prometheus","metricLabels":null,"timestamp":"2025-02-26T13:17:00Z","value":"0"}], "activity": false, "scalerError": null}
2025-02-26T13:17:00Z	DEBUG	scaleexecutor	ScaleTarget no change	{"scaledobject.Name": "communications-service", "scaledObject.Namespace": "stage", "scaleTarget.Name": "communications-service"}
2025-02-26T13:17:00Z	INFO	Detected resource targeted for scaling	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"communications-service","namespace":"stage"}, "namespace": "stage", "name": "communications-service", "reconcileID": "5ed1373d-e279-42d9-8808-d48a3615d3c4", "resource": "apps/v1.Deployment", "name": "communications-service"}
2025-02-26T13:17:00Z	DEBUG	Updating ScaledObject status with triggers and authentications types	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"communications-service","namespace":"stage"}, "namespace": "stage", "name": "communications-service", "reconcileID": "5ed1373d-e279-42d9-8808-d48a3615d3c4", "triggersTypes": "prometheus", "authenticationsTypes": ""}
2025-02-26T13:17:00Z	DEBUG	ScaledObject is defined correctly and is ready for scaling	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"communications-service","namespace":"stage"}, "namespace": "stage", "name": "communications-service", "reconcileID": "5ed1373d-e279-42d9-8808-d48a3615d3c4"}
2025-02-26T13:17:01Z	INFO	grpc_server	Starting Metrics Service gRPC Server	{"address": ":9666"}
2025-02-26T13:17:10Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "stage", "scaledObject.Name": "communications-service", "scaler": "prometheusScaler", "metricName": "s0-prometheus", "metrics": [{"metricName":"s0-prometheus","metricLabels":null,"timestamp":"2025-02-26T13:17:10Z","value":"0"}], "activity": false, "scalerError": null}
2025-02-26T13:17:10Z	DEBUG	scaleexecutor	ScaleTarget no change	{"scaledobject.Name": "communications-service", "scaledObject.Namespace": "stage", "scaleTarget.Name": "communications-service"}
2025-02-26T13:17:20Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "stage", "scaledObject.Name": "communications-service", "scaler": "prometheusScaler", "metricName": "s0-prometheus", "metrics": [{"metricName":"s0-prometheus","metricLabels":null,"timestamp":"2025-02-26T13:17:20Z","value":"0"}], "activity": false, "scalerError": null}
2025-02-26T13:17:20Z	DEBUG	scaleexecutor	ScaleTarget no change	{"scaledobject.Name": "communications-service", "scaledObject.Namespace": "stage", "scaleTarget.Name": "communications-service"}
2025-02-26T13:17:30Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "stage", "scaledObject.Name": "communications-service", "scaler": "prometheusScaler", "metricName": "s0-prometheus", "metrics": [{"metricName":"s0-prometheus","metricLabels":null,"timestamp":"2025-02-26T13:17:30Z","value":"0"}], "activity": false, "scalerError": null}
2025-02-26T13:17:30Z	DEBUG	scaleexecutor	ScaleTarget no change	{"scaledobject.Name": "communications-service", "scaledObject.Namespace": "stage", "scaleTarget.Name": "communications-service"}
2025-02-26T13:17:40Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "stage", "scaledObject.Name": "communications-service", "scaler": "prometheusScaler", "metricName": "s0-prometheus", "metrics": [{"metricName":"s0-prometheus","metricLabels":null,"timestamp":"2025-02-26T13:17:40Z","value":"0"}], "activity": false, "scalerError": null}
2025-02-26T13:17:40Z	DEBUG	scaleexecutor	ScaleTarget no change	{"scaledobject.Name": "communications-service", "scaledObject.Namespace": "stage", "scaleTarget.Name": "communications-service"}

apiserver metrics

I0226 13:17:26.412358       1 welcome.go:34] "msg"="Starting metrics server" "logger"="keda_metrics_adapter"
I0226 13:17:26.412559       1 welcome.go:35] "msg"="KEDA Version: 2.16.1" "logger"="keda_metrics_adapter"
I0226 13:17:26.412568       1 welcome.go:36] "msg"="Git Commit: ce14b239e0300f388b0425aef68154d8070cd66f" "logger"="keda_metrics_adapter"
I0226 13:17:26.412576       1 welcome.go:37] "msg"="Go Version: go1.23.4" "logger"="keda_metrics_adapter"
I0226 13:17:26.412583       1 welcome.go:38] "msg"="Go OS/Arch: linux/amd64" "logger"="keda_metrics_adapter"
I0226 13:17:26.412622       1 welcome.go:39] "msg"="Running on Kubernetes 1.30" "logger"="keda_metrics_adapter" "version"={"major":"1","minor":"30","gitVersion":"v1.30.9-gke.1009000","gitCommit":"bd22bb26d389ceda2b24067faf3a6346c722a21d","gitTreeState":"clean","buildDate":"2025-01-15T22:50:02Z","goVersion":"go1.22.10 X:boringcrypto","compiler":"gc","platform":"linux/amd64"}
I0226 13:17:26.413020       1 maxprocs.go:31] "msg"="maxprocs: Updating GOMAXPROCS=1: determined from CPU quota" "logger"="keda_metrics_adapter"
I0226 13:17:26.413831       1 main.go:111] "msg"="Connecting Metrics Service gRPC client to the server" "address"="keda-operator.keda.svc.cluster.local:9666" "logger"="keda_metrics_adapter"
I0226 13:17:26.450215       1 provider.go:56] "msg"="starting" "logger"="keda_metrics_adapter.provider"
I0226 13:17:26.450261       1 main.go:272] "msg"="starting adapter..." "logger"="keda_metrics_adapter"
I0226 13:17:26.459286       1 dynamic_serving_content.go:116] "Loaded a new cert/key pair" name="serving-cert::/certs/tls.crt::/certs/tls.key"
I0226 13:17:26.475945       1 client.go:103] "msg"="Waiting for establishing a gRPC connection to KEDA Metrics Server" "logger"="keda_metrics_adapter.provider"
I0226 13:17:26.476100       1 main.go:184] "msg"="starting /metrics server endpoint" "logger"="keda_metrics_adapter"
I0226 13:17:26.787861       1 dynamic_cafile_content.go:122] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/certs/ca.crt"
I0226 13:17:26.817393       1 requestheader_controller.go:247] Loaded a new request header values for RequestHeaderAuthRequestController
I0226 13:17:26.819594       1 maxinflight.go:139] "Initialized nonMutatingChan" len=400
I0226 13:17:26.819627       1 maxinflight.go:145] "Initialized mutatingChan" len=200
I0226 13:17:26.819660       1 timing_ratio_histogram.go:203] "TimingRatioHistogramVec.NewForLabelValuesSafe hit the inefficient case" fqName="apiserver_flowcontrol_read_vs_write_current_requests" labelValues=["executing","readOnly"]
I0226 13:17:26.819679       1 timing_ratio_histogram.go:203] "TimingRatioHistogramVec.NewForLabelValuesSafe hit the inefficient case" fqName="apiserver_flowcontrol_read_vs_write_current_requests" labelValues=["executing","mutating"]
I0226 13:17:26.819698       1 maxinflight.go:116] "Set denominator for readonly requests" limit=400
I0226 13:17:26.819704       1 maxinflight.go:120] "Set denominator for mutating requests" limit=200
I0226 13:17:26.820204       1 config.go:916] Not requested to run hook priority-and-fairness-config-consumer
I0226 13:17:26.828831       1 genericapiserver.go:533] MuxAndDiscoveryComplete has all endpoints registered and discovery information is complete
I0226 13:17:26.921761       1 tlsconfig.go:181] "Loaded client CA" index=0 certName="client-ca-bundle::/certs/ca.crt,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"KEDA\" [] groups=[KEDAORG] validServingFor=[KEDA] issuer=\"<self>\" (2025-02-24 08:22:42 +0000 UTC to 2035-02-22 09:22:42 +0000 UTC (now=2025-02-26 13:17:26.921690236 +0000 UTC))"
I0226 13:17:26.922091       1 tlsconfig.go:203] "Loaded serving cert" certName="serving-cert::/certs/tls.crt::/certs/tls.key" certDetail="\"keda-operator\" [serving,client] validServingFor=[keda-operator,keda-operator,keda-operator.keda,keda-operator.keda.svc,keda-operator.keda.svc.cluster.local,keda-admission-webhooks,keda-admission-webhooks.keda,keda-admission-webhooks.keda.svc,keda-admission-webhooks.keda.svc.cluster.local,keda-operator-metrics-apiserver,keda-operator-metrics-apiserver.keda,keda-operator-metrics-apiserver.keda.svc,keda-operator-metrics-apiserver.keda.svc.cluster.local] issuer=\"KEDA\" (2025-02-24 08:22:42 +0000 UTC to 2026-02-24 09:22:42 +0000 UTC (now=2025-02-26 13:17:26.922051096 +0000 UTC))"
I0226 13:17:26.922345       1 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1740575846\" [serving] validServingFor=[apiserver-loopback-client] issuer=\"apiserver-loopback-client-ca@1740575846\" (2025-02-26 12:17:26 +0000 UTC to 2026-02-26 12:17:26 +0000 UTC (now=2025-02-26 13:17:26.922306303 +0000 UTC))"
I0226 13:17:26.922398       1 secure_serving.go:213] Serving securely on [::]:6443
I0226 13:17:26.922502       1 genericapiserver.go:683] [graceful-termination] waiting for shutdown to be initiated
I0226 13:17:26.922549       1 requestheader_controller.go:172] Starting RequestHeaderAuthRequestController
I0226 13:17:26.922590       1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController
I0226 13:17:26.922653       1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/certs/tls.crt::/certs/tls.key"
I0226 13:17:26.922932       1 tlsconfig.go:243] "Starting DynamicServingCertificateController"
I0226 13:17:26.923188       1 dynamic_cafile_content.go:160] "Starting controller" name="client-ca-bundle::/certs/ca.crt"
I0226 13:17:26.923378       1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0226 13:17:26.923399       1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0226 13:17:26.923756       1 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0226 13:17:26.923794       1 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0226 13:17:26.923822       1 reflector.go:305] Starting reflector *v1.ConfigMap (12h0m0s) from k8s.io/apiserver/pkg/authentication/request/headerrequest/requestheader_controller.go:175
I0226 13:17:26.923842       1 reflector.go:341] Listing and watching *v1.ConfigMap from k8s.io/apiserver/pkg/authentication/request/headerrequest/requestheader_controller.go:175
I0226 13:17:26.924627       1 reflector.go:305] Starting reflector *v1.ConfigMap (12h0m0s) from k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:209
I0226 13:17:26.924646       1 reflector.go:341] Listing and watching *v1.ConfigMap from k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:209
I0226 13:17:26.943125       1 reflector.go:368] Caches populated for *v1.ConfigMap from k8s.io/apiserver/pkg/authentication/request/headerrequest/requestheader_controller.go:175
I0226 13:17:26.945753       1 reflector.go:368] Caches populated for *v1.ConfigMap from k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:209
I0226 13:17:26.977129       1 provider.go:64] "msg"="Connection to KEDA Metrics Service gRPC server has been successfully established" "logger"="keda_metrics_adapter.provider" "server"="keda-operator.keda.svc.cluster.local:9666"
I0226 13:17:27.023471       1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0226 13:17:27.023495       1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController
I0226 13:17:27.024052       1 tlsconfig.go:181] "Loaded client CA" index=0 certName="client-ca-bundle::/certs/ca.crt,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"KEDA\" [] groups=[KEDAORG] validServingFor=[KEDA] issuer=\"<self>\" (2025-02-24 08:22:42 +0000 UTC to 2035-02-22 09:22:42 +0000 UTC (now=2025-02-26 13:17:27.023982492 +0000 UTC))"
I0226 13:17:27.024153       1 tlsconfig.go:181] "Loaded client CA" index=1 certName="client-ca-bundle::/certs/ca.crt,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"75219ad4-e2a4-4cfb-a353-3d405eac7c65\" [] issuer=\"<self>\" (2024-02-15 08:00:45 +0000 UTC to 2054-02-07 09:00:45 +0000 UTC (now=2025-02-26 13:17:27.024110765 +0000 UTC))"
I0226 13:17:27.024453       1 tlsconfig.go:203] "Loaded serving cert" certName="serving-cert::/certs/tls.crt::/certs/tls.key" certDetail="\"keda-operator\" [serving,client] validServingFor=[keda-operator,keda-operator,keda-operator.keda,keda-operator.keda.svc,keda-operator.keda.svc.cluster.local,keda-admission-webhooks,keda-admission-webhooks.keda,keda-admission-webhooks.keda.svc,keda-admission-webhooks.keda.svc.cluster.local,keda-operator-metrics-apiserver,keda-operator-metrics-apiserver.keda,keda-operator-metrics-apiserver.keda.svc,keda-operator-metrics-apiserver.keda.svc.cluster.local] issuer=\"KEDA\" (2025-02-24 08:22:42 +0000 UTC to 2026-02-24 09:22:42 +0000 UTC (now=2025-02-26 13:17:27.024423608 +0000 UTC))"
I0226 13:17:27.024631       1 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1740575846\" [serving] validServingFor=[apiserver-loopback-client] issuer=\"apiserver-loopback-client-ca@1740575846\" (2025-02-26 12:17:26 +0000 UTC to 2026-02-26 12:17:26 +0000 UTC (now=2025-02-26 13:17:27.024612436 +0000 UTC))"
I0226 13:17:31.227984       1 httplog.go:134] "HTTP" verb="GET" URI="/readyz" latency="389.849µs" userAgent="kube-probe/1.30" audit-ID="4fa161f9-be72-4e2a-8cb6-31ce93836d40" srcIP="10.31.0.19:44094" resp=200
I0226 13:17:34.227923       1 httplog.go:134] "HTTP" verb="GET" URI="/readyz" latency="263.557µs" userAgent="kube-probe/1.30" audit-ID="f70889c0-7805-49ce-8bc8-82bf0f6fc98b" srcIP="10.31.0.19:37150" resp=200
I0226 13:17:35.227801       1 httplog.go:134] "HTTP" verb="GET" URI="/healthz" latency="364.616µs" userAgent="kube-probe/1.30" audit-ID="6f8e70f4-2221-4291-8a64-626fa1487b0b" srcIP="10.31.0.19:37164" resp=200
I0226 13:17:37.227747       1 httplog.go:134] "HTTP" verb="GET" URI="/readyz" latency="255.326µs" userAgent="kube-probe/1.30" audit-ID="c3d1c53d-e907-42a1-b9ac-7bf17d32a46a" srcIP="10.31.0.19:37170" resp=200
I0226 13:17:40.227426       1 httplog.go:134] "HTTP" verb="GET" URI="/readyz" latency="277.61µs" userAgent="kube-probe/1.30" audit-ID="be76f7a1-576e-4f30-ad2f-35a67783c691" srcIP="10.31.0.19:37172" resp=200
I0226 13:17:43.227691       1 httplog.go:134] "HTTP" verb="GET" URI="/readyz" latency="306.675µs" userAgent="kube-probe/1.30" audit-ID="c2701650-a33c-44f4-9cd5-51e3d396e076" srcIP="10.31.0.19:33576" resp=200
I0226 13:17:45.228764       1 httplog.go:134] "HTTP" verb="GET" URI="/healthz" latency="335.982µs" userAgent="kube-probe/1.30" audit-ID="44b159d9-6f9b-431e-9835-a40cfcb78b04" srcIP="10.31.0.19:33588" resp=200
I0226 13:17:46.227260       1 httplog.go:134] "HTTP" verb="GET" URI="/readyz" latency="316.046µs" userAgent="kube-probe/1.30" audit-ID="58dc57b9-682e-48e7-8293-0ce4921720c8" srcIP="10.31.0.19:33594" resp=200
I0226 13:17:49.228854       1 httplog.go:134] "HTTP" verb="GET" URI="/readyz" latency="349.659µs" userAgent="kube-probe/1.30" audit-ID="5d5a5d8c-551a-4ab6-88d7-c6be6c30185f" srcIP="10.31.0.19:33606" resp=200
I0226 13:17:52.228383       1 httplog.go:134] "HTTP" verb="GET" URI="/readyz" latency="295.119µs" userAgent="kube-probe/1.30" audit-ID="66202f84-ddb5-46f7-b8d0-ba685804ef89" srcIP="10.31.0.19:33614" resp=200
I0226 13:17:55.230570       1 httplog.go:134] "HTTP" verb="GET" URI="/healthz" latency="320.293µs" userAgent="kube-probe/1.30" audit-ID="ed0c552b-5f8b-4d11-a54c-1a950ba3c5f5" srcIP="10.31.0.19:54900" resp=200
I0226 13:17:55.231728       1 httplog.go:134] "HTTP" verb="GET" URI="/readyz" latency="198.979µs" userAgent="kube-probe/1.30" audit-ID="e937bf08-5988-413c-bc04-f986fa3b6857" srcIP="10.31.0.19:54892" resp=200
I0226 13:17:58.227690       1 httplog.go:134] "HTTP" verb="GET" URI="/readyz" latency="247.854µs" userAgent="kube-probe/1.30" audit-ID="ba04dbaf-3727-4be7-94ca-b97a1f35d686" srcIP="10.31.0.19:54916" resp=200

KEDA Version

2.16.1

Kubernetes Version

1.30

Platform

Google Cloud

Scaler Details

Prometheus

Anything else?

No response
@patrostkowski patrostkowski added the bug Something isn't working label Feb 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
Roadmap - KEDA Core
Status: To Triage
Milestone
No milestone
Development

No branches or pull requests
1 participant
@patrostkowski