Add a controller namespace filter and/or provider filter

[paalkr]

Is your feature request related to a problem? Please describe.
In a multi-tenant cluster, we need a better way to silo access to the terraform state backend (S3 +DynamoDB). The Terranetes controller runs with a service account associated with an IAM role, using the AWS IRSA infrastructure. This role need access to all possible state store files. I'm aware that it's possible to configure a role to assume in a provider resource, and we are using this feature. But this does not limit access to the state store backend, only the cloud resources generated by the controller.

Describe the solution you'd like
We would like to run several copies of the Terranetes controller in a single cluster. Each controller should have a separate service account with a separate IAM role. All this it's possible as of today, just by deploying the helm chart several times in different namespaces. The problem is that all Terranetes controllers would react to all configurations. We would like a way to filter which controller process which configurations.

Describe alternatives you've considered
I see to possible solutions.

1. Add a namespace filter to a controller, equal to what you have for policies, that limits which namespaces the controller listens to. Only configurations in namespaces that corresponds with the filter will be processed by the controller.
2. Add a provider filter to a controller. A controller should only process configurations using a provider that corresponds with the filter.

Both solutions would work nicely in our use case, as we already deploy separate providers and policies pr tenant in a cluster. And all namespaces that belongs to a tenant have a certain set of labels that we could use in a namespace filter.

[gambol99]

Hi @paalkr … sounds reasonable! .. Off the top of my head i guess the tricky pieces will be how the webhooks (validating and mutating) are handler, and who’d be responsible for reconciling global resources such as policies, providers etc .. But i’m happy to take a mental stab at it ..

[paalkr]

Thx! Yeah, you are absolutely right. I did think about that, but I did not come to a good conclusion.

I guess one solution could be to have type of a "master" deployment that maintains global resources and react to all configurations that are not picked up by any filter (namespace or provider). And the additional controllers will be only "slave" controllers (with separate service accounts) which process whatever applies to the controller filter. I guess the webhook challenge isn't solved by this though.

[paalkr]

@gambol99 , have you given this feature request any future thoughts?

[gambol99]

@gambol99 , have you given this feature request any future thoughts?

Wow … has it been that long already!! sorry

In a multi-tenant cluster, we need a better way to silo access to the terraform state backend (S3 +DynamoDB). The Terranetes controller runs with a service account associated with an IAM role, using the AWS IRSA infrastructure. This role need access to all possible state store files. I'm aware that it's possible to configure a role to assume in a provider resource, and we are using this feature. But this does not limit access to the state store backend, only the cloud resources generated by the controller.

Out of interest would the following work …

a) Create a provider per tenant, using rbac on Providers to lock down to the tenant
b) On the per tenant Provider use a custom template (not sure it’s documented but essentially a kubernetes security defining the backend)
c) Each Provider has it’s own credentials secret or service accounts - the identity policy of which is locked to the specific backend attributes

I guess the annoying bit for the tenant would be knowing which Provider they should use - but they could be a naming convention, or a feature that defaults the Provider to one or more namespaces if not defined

[paalkr]

Hi,

What you have suggested in step a, b and c is exactly what we have implemented a while back. We do have a naming convention for the providers so it easy for the operators to select the correct one. We also have a kyverno validating policy that will reject configurations that tries to use a provider not allowed in the current environment. Currently we also consider adding a kyverno mutating policy that just injects the correct provider ref name based on the environment the configuration is deployed to.

Our main issue is still that all configurations and providers are evaluated by the same Terranetes controller, using a single service account. The service account uses IRSA to get access to AWS cloud resource operations. Sp the IAM role assigned to the IRSA and then used by the service account must have all privileges that in sum all the providers needs. We would like to limit that scope by running several Terranetes controllers with independent IRSA/IAM roles attached to it. This way we can make very fin grained IAM policies and policy boundaries for each environment.