GitHub - terraform-ibm-modules/terraform-ibm-external-secrets-operator: Synchronizes secrets between Secrets Manager and an IBM Cloud OpenShift cluster

Terraform IBM External Secrets Operator module

This module automates the installation and configuration of the External Secrets Operator in a cluster.

Overview

external-secrets-operator-module

External Secrets Operator synchronizes secrets in the Kubernetes cluster with secrets that are mapped in Secrets Manager.

The module provides the following features:

Install and configure External Secrets Operator (ESO).
Customise External Secret Operator deployment on specific cluster workers by configuration approriate NodeSelector and Tolerations in the ESO helm release More details below

The submodules automate the configuration of an operator, providing the following features:

Deploy and configure ClusterSecretStore resources for cluster scope secrets store eso-clusterstore
Deploy and configure SecretStore resources for namespace scope secrets store eso-secretstore
Leverage on two authentication methods to be configured on the single stores instances:
- IAM apikey standard authentication
- IAM Trusted profile: in conjunction with eso-trusted-profile submodule, which allows to create one or more trusted profiles to use with the ESO module for trusted profile authentication.
Configure the ExternalSecret resources to be bound to the expected secrets store (according to the visibility you need) and to configure the target secret details
- The following secret types of Kubernetes Secrets are currently supported:
  - Opaque (opaque in this module)
  - kubernetes.io/dockerconfigjson (dockerconfigjson in this module)

The current version of the module supports multitenants configuration by setting up "ESO as a service" (ref. https://cloud.redhat.com/blog/how-to-setup-external-secrets-operator-eso-as-a-service) for both authentication methods More details below

The following combinations of Kubernetes Secrets and Secrets Manager secrets are used with given External-Secret type.

es_kubernetes_secret_type¹	sm_secret_type²	external_secret_type³
dockerconfigjson	arbitrary	arbitrary
dockerconfigjson	iam_credentials	iam_credentials
dockerconfigjson	username_password	username_password
opaque	arbitrary	arbitrary
opaque	iam_credentials	iam_credentials
opaque	username_password	username_password
opaque	kv	kv
tls	imported_cert	imported_cert
tls	public_cert	public_cert
tls	private_cert	private_cert

Customise ESO deployment on specific cluster nodes

In order to customise the NodeSelector and the tolerations to make the External Secret Operator deployed on specific cluster nodes it is possible to configure the following input variable with the appropriate values:

variable "eso_cluster_nodes_configuration" {
  description = "Configuration to use to customise ESO deployment on specific cluster nodes. Setting appropriate values will result in customising ESO helm release. Default value is null to keep ESO standard deployment."
  type = object({
    nodeSelector = object({
      label = string
      value = string
    })
    tolerations = object({
      key = string
      operator = string
      value = string
      effect = string
    })
  })
  default = null
}

For example:

module "external_secrets_operator" {
  (...)
  eso_cluster_nodes_configuration = {
    nodeSelector = {
      label = "dedicated"
      value = "edge"
    }
    tolerations = {
      key = "dedicated"
      operator = "Equal"
      value = "edge"
      effect = "NoExecute"
    }
  }
  (...)

will make the External Secret Operator to run on clusters nodes labeled with dedicated: edge.

The resulting helm release configuration, according to the terraform plan output would be like

(...)
# module.external_secrets_operator.helm_release.external_secrets_operator[0] will be created
  + resource "helm_release" "external_secrets_operator" {
      + atomic                     = false
      + chart                      = "external-secrets"
      + cleanup_on_fail            = false
      + create_namespace           = false
      + dependency_update          = false
      + disable_crd_hooks          = false
      + disable_openapi_validation = false
      + disable_webhooks           = false
      + force_update               = false
      + id                         = (known after apply)
      + lint                       = false
      + manifest                   = (known after apply)
      + max_history                = 0
      + metadata                   = (known after apply)
      + name                       = "external-secrets"
      + namespace                  = "es-operator"
      + pass_credentials           = false
      + recreate_pods              = false
      + render_subchart_notes      = true
      + replace                    = false
      + repository                 = "https://charts.external-secrets.io"
      + reset_values               = false
      + reuse_values               = false
      + skip_crds                  = false
      + status                     = "deployed"
      + timeout                    = 300
      + values                     = [
          + <<-EOT
                installCRDs: true
                extraVolumes:
                - name: sa-token
                  projected:
                      defaultMode: 0644
                      sources:
                      - serviceAccountToken:
                          path: sa-token
                          expirationSeconds: 3600
                          audience: iam
                extraVolumeMounts:
                - mountPath: /var/run/secrets/tokens
                  name: sa-token
                webhook:
                  extraVolumes:
                  - name: sa-token
                    projected:
                      defaultMode: 0644
                      sources:
                      - serviceAccountToken:
                          path: sa-token
                          expirationSeconds: 3600
                          audience: iam
                  extraVolumeMounts:
                  - mountPath: /var/run/secrets/tokens
                    name: sa-token
            EOT,
          + <<-EOT
                nodeSelector: { dedicated: edge }
                tolerations:
                - key: dedicated
                  operator: Equal
                  value: edge
                  effect: NoExecute
                webhook:
                  nodeSelector: { dedicated: edge }
                  tolerations:
                  - key: dedicated
                    operator: Equal
                    value: edge
                    effect: NoExecute
                certController:
                  nodeSelector: { dedicated: edge }
                  tolerations:
                  - key: dedicated
                    operator: Equal
                    value: edge
                    effect: NoExecute
            EOT,
        ]
      + verify                     = false
      + version                    = "0.9.7"
      + wait                       = true
      + wait_for_jobs              = false
    }
(...)

Similarly exporting this environment variable

export TF_VAR_eso_cluster_nodes_configuration="{\"nodeSelector\": {\"label\": \"dedicated\", \"value\": \"transit\"}, \"tolerations\": {\"key\": \"dedicated\", \"operator\": \"Equal\", \"value\": \"transit\", \"effect\": \"NoExecute\"}}"

will make the External Secret Operator to run on clusters nodes labeled with dedicated: transit.

The default null value keeps the default ESO behaviour.

Example of Multitenancy configuration example in namespaced externalsecrets stores

To configure a set of tenants to be configured in their proper namespace (to achieve tenant isolation) you need simply to follow these steps:

deploy ESO in the cluster

module "external_secrets_operator" {
  source               = "terraform-ibm-modules/external-secrets-operator/ibm"
  version              = "1.0.0"
  eso_namespace     = var.eso_namespace # namespace to deploy ESO
  service_endpoints = var.service_endpoints # use public or private endpoints for IAM and Secrets Manager
  eso_cluster_nodes_configuration = <<the eso configuration for specific cluster nodes selection if needed - read above>>
}

create multiple SecretStore(s) in the proper namespaces

With api_key authentication mode

module "eso_namespace_secretstore_1" {
  depends_on = [
    module.external_secrets_operator
  ]
  source                      = "../modules/eso-secretstore"
  eso_authentication          = "api_key"
  region                      = local.sm_region # SM region
  sstore_namespace            = var.es_kubernetes_namespaces[2] # namespace to create the secret store
  sstore_secrets_manager_guid = local.sm_guid # the guid of the secrets manager instance to use
  sstore_store_name           = "${var.es_kubernetes_namespaces[2]}-store" # store name
  # to pull the secrets from SM
  sstore_secret_apikey        = data.ibm_sm_iam_credentials_secret.secret_puller_secret.api_key # pragma: allowlist secret
  service_endpoints           = var.service_endpoints
  sstore_helm_rls_name        = "es-store" # helm release name suffix to use for the store
  sstore_secret_name          = "generic-cluster-api-key" #checkov:skip=CKV_SECRET_6
}

With trusted_profile authentication mode

module "eso_namespace_secretstores" {
  depends_on = [
    module.external_secrets_operator
  ]
  source                      = "../modules/eso-secretstore"
  eso_authentication          = "trusted_profile"
  region                      = local.sm_region # SM region
  sstore_namespace            = kubernetes_namespace.examples[count.index].metadata[0].name # namespace to create the secret store
  sstore_secrets_manager_guid = local.sm_guid # the guid of the secrets manager instance to use
  sstore_store_name           = "${kubernetes_namespace.examples[count.index].metadata[0].name}-store" # store name
  sstore_trusted_profile_name = module.external_secrets_trusted_profiles[count.index].trusted_profile_name # trusted profile name to use into this secret store
  service_endpoints           = var.service_endpoints
  sstore_helm_rls_name        = "es-store-${count.index}" # helm release name suffix to use for the store
  sstore_secret_name          = "secretstore-api-key" #checkov:skip=CKV_SECRET_6
}

More details can be found in the examples linked below.

Important current limitation of ESO deployment

The current ESO version doesn't allow to customise the default IAM endpoint (https://iam.cloud.ibm.com) it uses when authenticating through apikey (api_key authentication) for both ClusterSecretStore and SecretStore APIs.

As a direct effect of this limitation, for a standard OCP cluster topology as defined by GoldenEye design (3 workers zones edge private and transit), an ESO deployment with api_key authentication configuration needs to be performed on the workers pool with access to the public network (dedicated: edge label in GE usual topology) to work fine. If the ESO deployment is performed on a workers pool without access to public network (i.e. to https://iam.cloud.ibm.com) the apikey authentication is expected to fail.

Pod Reloader

When secrets are updated, depending on you configuration pods may need to be restarted to pick up the new secrets. To do this you can use the Stakater Reloader. By default, the module deploys this to watch for changes in secrets and configmaps and trigger a rolling update of the related pods. To have Reloader watch a secret or configMap add the annotation reloader.stakater.com/auto: "true" to the secret or configMap, the same annotation can be added to deployments to have them restarted when the secret or configMap changes. When using the eso-external-secret submodule, use the reloader_watching variable to have the annotation added to the secret.

This can be further configured as needed, for more details see https://github.com/stakater/Reloader By default is watches all namespaces. If you do not need it please set reloader_deployed = false in the module call.

Troubleshooting

In the case of problems with secrets synchronization a good start point to the investigation is to list the externalsecrets resources in the cluster:

oc get externalsecrets -A
NAMESPACE       NAME                                                  AGE   STATUS   CAPABILITIES   READY
apikeynspace3   secretstore.external-secrets.io/apikeynspace3-store   32m   Valid    ReadOnly       True
apikeynspace4   secretstore.external-secrets.io/apikeynspace4-store   32m   Valid    ReadOnly       True
tpnspace1       secretstore.external-secrets.io/tpnspace1-store       32m   Valid    ReadOnly       True
tpnspace2       secretstore.external-secrets.io/tpnspace2-store       32m   Valid    ReadOnly       True

NAMESPACE   NAME                                                   AGE   STATUS   CAPABILITIES   READY
            clustersecretstore.external-secrets.io/cluster-store   32m   Valid    ReadOnly       True

NAMESPACE       NAME                                                                     STORE                 REFRESH INTERVAL   STATUS              READY
apikeynspace1   externalsecret.external-secrets.io/dockerconfigjson-uc                   cluster-store         1m                 SecretSyncedError   False
apikeynspace2   externalsecret.external-secrets.io/cloudant-opaque-arb                   cluster-store         5m                 SecretSyncedError   False
apikeynspace3   externalsecret.external-secrets.io/dockerconfigjson-arb                  apikeynspace3-store   1h                 SecretSyncedError   False
apikeynspace4   externalsecret.external-secrets.io/dockerconfigjson-iam                  apikeynspace4-store   1h                 SecretSyncedError   False
tpnspace1       externalsecret.external-secrets.io/geretain-tesoall-arbitrary-arb-tp-0   tpnspace1-store       5m                 SecretSynced        True
tpnspace2       externalsecret.external-secrets.io/geretain-tesoall-arbitrary-arb-tp-1   tpnspace2-store       5m                 SecretSynced        True

In the example above some of the externalsecrets are experiencing secrets synchronization errors. By describing them you should be able to identify the error:

oc describe externalsecret dockerconfigjson-uc -n apikeynspace1
Name:         dockerconfigjson-uc
Namespace:    apikeynspace1
Labels:       app=raw
              app.kubernetes.io/managed-by=Helm
              chart=raw-v0.2.5
              heritage=Helm
              release=apikeynspace1-es-docker-uc
Annotations:  meta.helm.sh/release-name: apikeynspace1-es-docker-uc
              meta.helm.sh/release-namespace: apikeynspace1
API Version:  external-secrets.io/v1beta1
Kind:         ExternalSecret
Metadata:
  (...)
Status:
  Conditions:
    Last Transition Time:  2023-06-27T15:18:31Z
    Message:               could not get secret data from provider
    Reason:                SecretSyncedError
    Status:                False
    Type:                  Ready
  Refresh Time:            <nil>
Events:
  Type     Reason        Age                  From              Message
  ----     ------        ----                 ----              -------
  (...)
  Warning  UpdateFailed  119s (x13 over 28m)  external-secrets  An error occurred while performing the 'authenticate' step: Post "https://iam.cloud.ibm.com/identity/token": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

In the output above there is a problem with reaching IAM endpoint (verify that the pods where ESO is running are able to reach that endpoint)

Important note

If you taint or destroy or simply make a change that needs the helm_release resource of the ESO operator to be deleted and recreated, this would make terraform to destroy the operator itself, including all the CRDs, which would destroy all the secrets synched through ESO, even if the helm_release resource of these CRDs aren't directly touched and terraform wouldn't be able to identify such a change. So in the case you plan to make changes to the operator helm_release once deployed, run preliminary a terraform plan to be sure that the release isn't destroyed and recreated.

Examples of the secrets format and layout

dockerconfigjson from arbitrary or iam_credentials

Secret Body >

apiVersion: v1
kind: Secret
metadata:
  name: dockerconfigjson-iam
  namespace: test-ns
data:
  .dockerconfigjson: [BASE64ENCODED]
type: kubernetes.io/dockerconfigjson

Base64 Decoded .dockerconfigson >

{
  "auths": {
    "us.icr.io": {
      "username": "iamapikey",
      "password": "APIKEYVALUE", # pragma: allowlist secret
      "email": "[email protected]"
    }
  }
}

dockerconfigjson from username_password

Secret Body >

apiVersion: v1
kind: Secret
metadata:
  name: dockerconfigjson-uc
  namespace: test-ns
data:
  .dockerconfigjson: [BASE64ENCODED]
type: kubernetes.io/dockerconfigjson

Base64 Decoded .dockerconfigson >

{
  "auths": {
    "xx.artifactory.xyz-devops.com": {
      "username": "[email protected]",
      "password": "APIKEYVALUE" # pragma: allowlist secret
    }
  }
}

opaque from arbitrary or iam_credentials

Secret Body >

apiVersion: v1
kind: Secret
metadata:
  name: opaque-arb
  namespace: test-ns
type: Opaque
data:
  apikey: APIKEYVALUE # pragma: allowlist secret

opaque from username_password

Secret Body >

apiVersion: v1
kind: Secret
metadata:
  name: opaque-uc
  namespace: test-ns
type: Opaque
data:
  username: test-user
  password: PASSWORDVALUE # pragma: allowlist secret

Usage

module "es_kubernetes_secret" {
  source                     = "../modules/eso-external-secret"
  es_kubernetes_secret_type = "dockerconfigjson"
  sm_secret_type = "iam_credentials"
  sm_secret_id = module.docker_config.serviceid_apikey_secret_id
  eso_setup = true
  es_kubernetes_namespaces = var.es_kubernetes_namespaces
  es_docker_email = "[email protected]"
  eso_generic_secret_apikey = data.ibm_secrets_manager_secret.secret_puller_secret.api_key # pragma: allowlist secret
  secrets_manager_guid = module.secrets_manager_iam_configuration.secrets_manager_guid
  region = "us-south"
  es_kubernetes_secret_name = "dockerconfigjson-iam"
  depends_on = [
    kubernetes_namespace.cluster_namespaces
  ]
  es_kubernetes_secret_data_key = "apiKey"
  es_helm_rls_name = "es-docker-iam"
}

Requirements

Name	Version
terraform	>= 1.0.0
helm	>= 2.11.0, < 3.0.0
kubernetes	>= 2.16.1, < 3.0.0

Modules

Name	Source	Version
eso_namespace	terraform-ibm-modules/namespace/ibm	1.0.2

Resources

Name	Type
helm_release.external_secrets_operator	resource
helm_release.pod_reloader	resource
kubernetes_namespace.existing_eso_namespace	data source

Inputs

Name	Description	Type	Default	Required
eso_chart_location	The location of the External Secrets Operator Helm chart.	`string`	`"https://charts.external-secrets.io"`	no
eso_chart_version	The version of the External Secrets Operator Helm chart. Ensure that the chart version is compatible with the image version specified in eso_image_version.	`string`	`"0.12.1"`	no
eso_cluster_nodes_configuration	Configuration to use to customise ESO deployment on specific cluster nodes. Setting appropriate values will result in customising ESO helm release. Default value is null to keep ESO standard deployment.	object({ nodeSelector = object({ label = string value = string }) tolerations = object({ key = string operator = string value = string effect = string }) })	`null`	no
eso_enroll_in_servicemesh	Flag to enroll ESO into istio servicemesh	`bool`	`false`	no
eso_image	The External Secrets Operator image in the format of `[registry-url]/[namespace]/[image]`.	`string`	`"ghcr.io/external-secrets/external-secrets"`	no
eso_image_version	The version or digest for the external secrets image to deploy. If changing the value, ensure it is compatible with the chart version set in eso_chart_version.	`string`	`"v0.12.1-ubi@sha256:d38834043de0a4e4feeac8a08d0bc96b71ddd7fe1d4c8583ee3751badeaeb01d"`	no
eso_namespace	Namespace to create and be used to install ESO components including helm releases. If eso_store_scope == cluster, this will also be used to deploy ClusterSecretStore/cluster_store in it	`string`	`null`	no
eso_pod_configuration	Configuration to use to customise ESO deployment on specific pods. Setting appropriate values will result in customising ESO helm release. Default value is {} to keep ESO standard deployment. Ignore the key if not required.	object({ annotations = optional(object({ # The annotations for external secret controller pods. external_secrets = optional(map(string), {}) # The annotations for external secret cert controller pods. external_secrets_cert_controller = optional(map(string), {}) # The annotations for external secret controller pods. external_secrets_webhook = optional(map(string), {}) }), {}) labels = optional(object({ # The labels for external secret controller pods. external_secrets = optional(map(string), {}) # The labels for external secret cert controller pods. external_secrets_cert_controller = optional(map(string), {}) # The labels for external secret controller pods. external_secrets_webhook = optional(map(string), {}) }), {}) })	`{}`	no
existing_eso_namespace	Existing Namespace to be used to install ESO components including helm releases. If eso_store_scope == cluster, this will also be used to deploy ClusterSecretStore/cluster_store in it	`string`	`null`	no
reloader_chart_location	The location of the Reloader Helm chart.	`string`	`"https://stakater.github.io/stakater-charts"`	no
reloader_chart_version	The version of the Reloader Helm chart. Ensure that the chart version is compatible with the image version specified in reloader_image_version.	`string`	`"1.2.1"`	no
reloader_custom_values	String containing custom values to be used for reloader helm chart. See https://github.com/stakater/Reloader/blob/master/deployments/kubernetes/chart/reloader/values.yaml	`string`	`null`	no
reloader_deployed	Whether to deploy reloader or not https://github.com/stakater/Reloader	`bool`	`true`	no
reloader_ignore_configmaps	Whether to ignore configmap changes or not	`bool`	`false`	no
reloader_ignore_secrets	Whether to ignore secret changes or not	`bool`	`false`	no
reloader_image	The reloader image in the format of `[registry-url]/[namespace]/[image]`.	`string`	`"ghcr.io/stakater/reloader"`	no
reloader_image_version	The version or digest for the reloader image to deploy. If changing the value, ensure it is compatible with the chart version set in reloader_chart_version.	`string`	`"v1.2.1-ubi@sha256:80a557100c6835c7e3c9842194250c9c4ca78f43200bc3a93a32e5b105ad11bb"`	no
reloader_is_argo_rollouts	Enable Argo Rollouts	`bool`	`false`	no
reloader_is_openshift	Enable OpenShift DeploymentConfigs	`bool`	`true`	no
reloader_log_format	The log format to use for reloader. Possible values are `json` or `text`. Default value is `json`	`string`	`"text"`	no
reloader_namespaces_selector	List of comma separated label selectors, if multiple are provided they are combined with the AND operator	`string`	`null`	no
reloader_namespaces_to_ignore	List of comma separated namespaces to ignore for reloader. If multiple are provided they are combined with the AND operator	`string`	`null`	no
reloader_pod_monitor_metrics	Enable to scrape Reloader's Prometheus metrics	`bool`	`false`	no
reloader_reload_on_create	Enable reload on create events	`bool`	`true`	no
reloader_reload_strategy	The reload strategy to use for reloader. Possible values are `env-vars` or `annotations`. Default value is `annotations`	`string`	`"annotations"`	no
reloader_resource_label_selector	List of comma separated label selectors, if multiple are provided they are combined with the AND operator	`string`	`null`	no
reloader_resources_to_ignore	List of comma separated resources to ignore for reloader. If multiple are provided they are combined with the AND operator	`string`	`null`	no
reloader_sync_after_restart	Enable sync after Reloader restarts for Add events, works only when reloadOnCreate is true	`bool`	`true`	no

Outputs

No outputs.

Contributing

You can report issues and request features for this module in GitHub issues in the module repo. See Report an issue or request a feature.

To set up your local development environment, see Local development setup in the project documentation.

es_kubernetes_secret_type: The Kubernetes Secrets type or format that ESO installs in the cluster. ↩
sm_secret_type: IBM Cloud Secrets Manager secret type that is used as source data by ESO. ↩
external_secret_type: The secret type that is used by ESO. ↩

Name		Name	Last commit message	Last commit date
Latest commit History 31 Commits
.github		.github
chart/raw		chart/raw
common-dev-assets @ a82b82a		common-dev-assets @ a82b82a
examples		examples
modules		modules
tests		tests
.gitignore		.gitignore
.gitmodules		.gitmodules
.pre-commit-config.yaml		.pre-commit-config.yaml
.releaserc		.releaserc
.secrets.baseline		.secrets.baseline
CODEOWNERS		CODEOWNERS
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
ci		ci
cra-config.yaml		cra-config.yaml
cra-tf-validate-ignore-rules.json		cra-tf-validate-ignore-rules.json
main.tf		main.tf
outputs.tf		outputs.tf
renovate.json		renovate.json
variables.tf		variables.tf
version.tf		version.tf

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Terraform IBM External Secrets Operator module

Overview

external-secrets-operator-module

Customise ESO deployment on specific cluster nodes

Example of Multitenancy configuration example in namespaced externalsecrets stores

More information links

Important current limitation of ESO deployment

Pod Reloader

Troubleshooting

Important note

Examples of the secrets format and layout

Usage

Requirements

Modules

Resources

Inputs

Outputs

Contributing

About

Releases 1

Packages

Contributors 3

Languages

License

terraform-ibm-modules/terraform-ibm-external-secrets-operator

Folders and files

Latest commit

History

Repository files navigation

Terraform IBM External Secrets Operator module

Overview

external-secrets-operator-module

Customise ESO deployment on specific cluster nodes

Example of Multitenancy configuration example in namespaced externalsecrets stores

More information links

Important current limitation of ESO deployment

Pod Reloader

Troubleshooting

Important note

Examples of the secrets format and layout

Usage

Requirements

Modules

Resources

Inputs

Outputs

Contributing

Footnotes

About

Topics

Resources

License

Code of conduct

Stars

Watchers

Forks

Releases 1

Packages 0

Contributors 3

Languages

Packages