Wordpress charm creates too many k8s secrets and never cleans up

[kot0dama]

Bug Description

The wordpress charm seems to create a lot of k8s secrets and never clean them up. All our wordpress deployments seem affected and hold more than 3000 secrets in their respective namespaces each.

This greatly increases the load on the k8s cluster and affects its stability.

Also, since we are collecting metrics through kube-state-metrics, it also yields 5 high-cardinality metrics in our Prometheus instance (kube_secret_metadata_resource_version, kube_secret_type, kube_secret_created, kube_secret_info, kube_secret_labels currently have 127180 cardinality which is way too much).

It seems some event triggers creating 3 tokens modeloperator-token-RANDOMBITS model-exec-token-RANDOMBITS wordpress-operator-token-RANDOMBITS.

I am unsure at this point what creates these k8s secrets, but if these are triggered through juju actions or other automated actions, I believe this is a bad idea to store forever such tokens in k8s secrets. Sadly, it doesn't look like k8s secrets can be set with an expiry time.

Could you please elaborate on what these secrets would be used for, and ideally come up with a solution to either not use k8s secrets as storage, or at least prune these secrets after a while?

Thank you!

To Reproduce

Deploy wordpress-k8s charm with listed versions and wait for it to create secrets.

Environment

Juju 2.9.49
Charm: wordpress-k8s
Channel: stable
Revs where this happens: 7, 25, 114

Relevant log output

This is the full list of 3k+ secrets on a single deployment over 3 years+ (for Canonical employees only): https://pastebin.canonical.com/p/fr6DGKKw6Q/

Additional context

No response

[amandahla]

@alithethird Can you have a look, please? Thanks.

[weiiwang01]

I believe this is a Juju/Kubernetes problem, not a charm issue, as the secrets that aren't cleaned up are all of type kubernetes.io/service-account-token. I think this is the same problem described in this issue, caused by juju updating the service account.

kubernetes/kubernetes#84642

[kot0dama]

@weiiwang01 Thanks for digging that up!

Although, iiu the last comment, it seems kubernetes would be unable to determine itself if a service account needs to be cleaned up, and it would fall onto the application / deployment process to handle these service accounts properly. This could be related to juju exec commands?

[kot0dama]

I've just reported a juju bug for this https://bugs.launchpad.net/juju/+bug/2093312

[weiiwang01]

@weiiwang01 Thanks for digging that up!

Although, iiu the last comment, it seems kubernetes would be unable to determine itself if a service account needs to be cleaned up, and it would fall onto the application / deployment process to handle these service accounts properly. This could be related to juju exec commands?

I am not exactly sure what causes this within Juju, but if I had to guess, it might be related to the high container restart rate in the deployment. Unfortunately, there's nothing we can do at the WordPress charm level, as listing and removing secrets may require elevated permissions, and outside Juju's internal system, it would be difficult to determine for sure which secrets are necessary and which are not. For now, the solution is to manually remove them and wait for Juju to address this issue.

Since this issue now appears to be unrelated to the WordPress charm, I will close this issue in the WordPress repository. I will continue to follow up on this issue on the Juju side. Let me know if there's any follow up that need to be done in the WordPress charm. Thank you!