[operator] Operator misses RBAC permissions to create RBAC setup for k8sevents receiver

[ybasket]

Hello,
happy operator user here ❤️, just noticed one small problem: It looks like the operator helm chart doesn't grant the operator all permissions it tries to grant to collector instances that use the k8sevents receiver when createRbacPermissions is enabled. The RBAC creation support for k8sevents was added recently in 0.113.0 / open-telemetry/opentelemetry-operator#3420, but when I try to use it on an EKS 1.29 cluster, the operator fails to grant the necessary permissions as it doesn't have them itself:

{"level":"ERROR","timestamp":"2024-12-09T15:34:07Z","logger":"controllers.OpenTelemetryCollector","message":"failed to configure desired","opentelemetrycollector":{"name":"otel-standalone-k8sevents","namespace":"observability"},"object_name":"otel-standalone-k8sevents-observability-cluster-role","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"clusterroles.rbac.authorization.k8s.io \"otel-standalone-k8sevents-observability-cluster-role\" is forbidden: user \"system:serviceaccount:observability:opentelemetry-operator\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:observability\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"\"], Resources:[\"events\"], Verbs:[\"get\" \"list\" \"watch\"]}\n{APIGroups:[\"\"], Resources:[\"namespaces/status\"], Verbs:[\"get\" \"list\" \"watch\"]}\n{APIGroups:[\"\"], Resources:[\"nodes/spec\"], Verbs:[\"get\" \"list\" \"watch\"]}\n{APIGroups:[\"\"], Resources:[\"pods/status\"], Verbs:[\"get\" \"list\" \"watch\"]}\n{APIGroups:[\"\"], Resources:[\"replicationcontrollers\"], Verbs:[\"get\" \"list\" \"watch\"]}\n{APIGroups:[\"\"], Resources:[\"replicationcontrollers/status\"], Verbs:[\"get\" \"list\" \"watch\"]}\n{APIGroups:[\"\"], Resources:[\"resourcequotas\"], Verbs:[\"get\" \"list\" \"watch\"]}\n{APIGroups:[\"batch\"], Resources:[\"cronjobs\"], Verbs:[\"get\" \"list\" \"watch\"]}\n{APIGroups:[\"extensions\"], Resources:[\"daemonsets\"], Verbs:[\"get\" \"list\" \"watch\"]}\n{APIGroups:[\"extensions\"], Resources:[\"deployments\"], Verbs:[\"get\" \"list\" \"watch\"]}","stacktrace":"github.com/open-telemetry/opentelemetry-operator/controllers.reconcileDesiredObjects\n\t/home/runner/work/opentelemetry-operator/opentelemetry-operator/controllers/common.go:171\ngithub.com/open-telemetry/opentelemetry-operator/controllers.(*OpenTelemetryCollectorReconciler).Reconcile\n\t/home/runner/work/opentelemetry-operator/opentelemetry-operator/controllers/opentelemetrycollector_controller.go:307\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224"}
{"level":"ERROR","timestamp":"2024-12-09T15:34:07Z","logger":"controllers.OpenTelemetryCollector","message":"failed to configure desired","opentelemetrycollector":{"name":"otel-standalone-k8sevents","namespace":"observability"},"object_name":"otel-standalone-k8sevents-observability-collector","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"clusterroles.rbac.authorization.k8s.io \"otel-standalone-k8sevents-observability-cluster-role\" not found","stacktrace":"github.com/open-telemetry/opentelemetry-operator/controllers.reconcileDesiredObjects\n\t/home/runner/work/opentelemetry-operator/opentelemetry-operator/controllers/common.go:171\ngithub.com/open-telemetry/opentelemetry-operator/controllers.(*OpenTelemetryCollectorReconciler).Reconcile\n\t/home/runner/work/opentelemetry-operator/opentelemetry-operator/controllers/opentelemetrycollector_controller.go:307\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224"}

In essence, these are read permissions for a lot of different k8s objects the receiver uses to enrich its logs (docs). I believe the solution would probably be to add them to the operator role in the createRbacPermissions section, like the very similar #1436 tries to? If someone confirms this, I can try to put up a PR adding all these permissions.

[diranged]

I ultimately just created a custom ClusterRole/ClusterRoleBinding that grants the escalate privilege.. as that feels like the right answer, instead of trying to keep updating the Operator privs based on what its controlled pods need.

# https://github.com/open-telemetry/opentelemetry-helm-charts/issues/1459
{{- with .Subcharts.operator }}
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: {{ template "opentelemetry-operator.fullname" . }}-escalate-fix
  labels:
    {{- include "opentelemetry-operator.labels" . | nindent 4 }}
    app.kubernetes.io/component: controller-manager
rules:
  - apiGroups:
    - rbac.authorization.k8s.io
    resources:
    - clusterroles
    - clusterrolebindings
    - rolebindings
    - roles
    verbs:
    - create
    - get
    - list
    - update
    - delete
    - watch
    - bind
    - escalate
---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    {{- include "opentelemetry-operator.labels" . | nindent 4 }}
    app.kubernetes.io/component: controller-manager
  name: {{ template "opentelemetry-operator.fullname" . }}-escalate-fix
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: {{ template "opentelemetry-operator.fullname" . }}-escalate-fix
subjects:
  - kind: ServiceAccount
    name: {{ template "opentelemetry-operator.serviceAccountName" . }}
    namespace: {{ .Release.Namespace }}
{{- end }}