(eks): EKS ALB Controller cannot provision ALB with OIDC configuration

[markusl]

Describe the bug

AlbController cannot provision ALB with OIDC configuration: User "system:serviceaccount:kube-system:aws-load-balancer-controller" cannot get resource "secrets" in API group.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

ALB Controller can be used to provision ALB's

Current Behavior

    "log": "{\"level\":\"error\",\"ts\":\"2025-01-29T08:26:18Z\",\"msg\":\"Reconciler error\",\"controller\":\"ingress\",\"object\":{\"name\":\"app-name.domain-name-ingress\",\"namespace\":\"namespace-name\"},\"namespace\":\"namespace-name\",\"name\":\"app-name.domain-name-ingress\",\"reconcileID\":\"cba15a83-4c7c-4c6a-8827-010514fc27da\",\"error\":\"ingress: namespace-name/app-name.domain-name-ingress: secrets \\\"golinks-okta-secret\\\" is forbidden: User \\\"system:serviceaccount:kube-system:aws-load-balancer-controller\\\" cannot get resource \\\"secrets\\\" in API group \\\"\\\" in the namespace \\\"namespace-name\\\"\"}",

Reproduction Steps

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/auth-type: oidc
    alb.ingress.kubernetes.io/auth-idp-oidc: '{"issuer":"https://sso.supercell.com","authorizationEndpoint":"https://okta/oauth2/v1/authorize","tokenEndpoint":"https://okta/oauth2/v1/token","userInfoEndpoint":"https://okta/oauth2/v1/userinfo","secretName":"app-name-okta-secret"}'
    

Possible Solution

Please add the required permissions for it to work :) Is there any workaround for this?

Additional Information/Context

No response

CDK CLI Version

2.177.0 (build b396961)

Framework Version

No response

Node.js Version

v22.13.0

OS

Mac

Language

TypeScript

Language Version

No response

Other information

No response

[pahud]

Hi @markusl

Can you share your minimal CDK code? Looks like we'll need to grant secret read privileges and I am not sure if aws-eks L2 should handle that. Having the visibility of your CDK code snippet would help.

[markusl]

@pahud "normal" ALB can provisioned just fine with the following code:

    const cluster = new eks.Cluster(this, `${clusterName}-cluster`, {
      vpc: vpc,
      clusterName: clusterName,
      version: eks.KubernetesVersion.V1_31,
      albController: {
        version: eks.AlbControllerVersion.V2_8_2,
      },
    });

However, when using alb.ingress.kubernetes.io/auth-idp-oidc user needs to configure a secret as instructed in the documentation: https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/ingress/annotations/

apiVersion: v1
kind: Secret
metadata:
  namespace: testcase
  name: my-k8s-secret
data:
  clientID: base64 of your plain text clientId
  clientSecret: base64 of your plain text clientSecret

Reading the secret fails with the default settings of albController. Hope this helps to understand the issue!

[pahud]

Thank you.

aws-cdk/packages/aws-cdk-lib/aws-eks/lib/alb-controller.ts

Lines 315 to 330 in 6fa1d05

	const serviceAccount = new ServiceAccount(this, 'alb-sa', { namespace, name: 'aws-load-balancer-controller', cluster: props.cluster });
	if (props.version.custom && !props.policy) {
	throw new Error("'albControllerOptions.policy' is required when using a custom controller version");
	}
	// https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/deploy/installation/#iam-permissions
	const policy: any = props.policy ?? JSON.parse(fs.readFileSync(path.join(__dirname, 'addons', `alb-iam_policy-${props.version.version}.json`), 'utf8'));
	for (const statement of policy.Statement) {
	const rewrittenStatement = {
	...statement,
	Resource: this.rewritePolicyResources(statement.Resource),
	};
	serviceAccount.addToPrincipalPolicy(iam.PolicyStatement.fromJson(rewrittenStatement));
	}

Sounds like we need to allow the serviceAccount to access that secret but I am not pretty sure what is the best action we need to do from CDK. Can you find any document about how to install the alb controller with OIDC support without using CDK? I guess we need to first find out what's the correct steps before we could figure out what CDK can do here.

[markusl]

@pahud it turns something like this can be used as a workaround:

# secrets-access.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secrets-access
  namespace: app-namespace
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "watch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: secrets-access-binding
  namespace: app-namespace
subjects:
  - kind: ServiceAccount
    name: aws-load-balancer-controller
    namespace: kube-system
roleRef:
  kind: Role
  name: secrets-access
  apiGroup: rbac.authorization.k8s.io

If it turns out to be difficult to implement, at least the limitation should be documented in AlbController, and preferably a convenience method could be added to easily attach the required resources if user is using ALB + OIDC.

Thank you!

[pahud]

@markusl

Absolutely. Do you think this deserves a PR for aws-eks README update?

[markusl]

@pahud Yep, I think this should be documented as a corner case that if AlbController is used with ALBs that have OIDC configured, then the user must grant access to the secrets manually.