Provision S3 buckets on cloudscale with cloudscale-SDK

[ccremer]

Summary

As user
I want to setup backups with S3 buckets on cloudscale.ch infrastructure
So that I can backup my DB instances

Context

In #74 we discussed how provider-jet-cloudscale should be integrated to make use of automatic S3 bucket provisioning.
In an internal discussion though, concerns were raised that this Crossplane integration, as well as provider-jet-cloudscale are not worth the added complexity vs directly provision buckets through cloud APIs and SDKs.

Thus we decided to implement bucket provisioning through code in the operator.

There are various terms that need to be defined:

• cloudscale.ch project: a project is the scope where the cloudscale.ch API tokens operate at. It's where all Object Users for a given cloudscale.ch token will be provisioned. There's no higher abstraction available via API, so we can't provision a project per tenant unfortunately. A cloudscale.ch account can have multiple projects.
• cloudscale.ch Object User: this represents an ACL and a set of credentials that can connect to cloudscale.ch's S3 API. By default the credentials associated with an Objects User can only create and access buckets for itself. It can't access buckets of other Object Users.
• postgres instance: a single instance provisioned by our operator.
• APPUiO Customer: a customer of APPUiO. Each customer can have multiple organizations.
• APPUiO Organization: a unit that groups together Users, Teams and Namespaces. It’s the entity that owns the associated resources and the business connection. Billing is done on this level.

Each instance should get its own bucket and Objects User. This avoids locking issues with K8up/restic.

Also, according to cloudscale.ch, if we announce it early enough they can increase limits on the S3 servics for us. So sharding should not be necessary on our end.

Out of Scope

• Crossplane integration
• Any other S3 provider other than cloudscale.ch
• Making bucket provisioning work in kind with Minio

Further links

• Cloudscale Go SDK: https://github.com/cloudscale-ch/cloudscale-go-sdk
• Cloudscale API docs: https://www.cloudscale.ch/en/api/v1
• POC: Create S3 bucket in Minio using API #90

Acceptance Criteria

Given an APPUiO Organisation
When they provision their first AppCat instance
Then provision an Objects Users with the name equal to the APPUiO Organisation ID

Given a PostgresqlStandalone instance CRD
When spec.backup.enabled is true
And create an S3 bucket on cloudscale.ch through API
And store the access credentials in a Kubernetes Secret in the service namespace
And configure K8up backup schedule to use this bucket

Given a PostgresqlStandalone instance CR
And the S3 bucket credentials exist
When the instance gets deleted
Then delete all objects in the backup bucket through S3 client *
And delete the backup bucket on cloudscale.ch through API

Given an instance with enabled backup
When the user disables the backup
Then simply remove the K8up schedule
And DON'T delete the bucket
And DON'T delete the bucket secret

*If deleting a bucket isn't possible without deleting all data first.

Implementation Ideas

No response

[ccremer]

I created a PoC that provisions buckets with Minio, for local testing.
It shows how relatively easy it is to create object users and buckets, and I expect that the cloudscale SDK should be similarly easy to use.
#90

[Kidswiss]

BTW: it doesn't look like cloudscale's SDK supports creating buckets. So for that we could re-use the logic to create the bucket from your POC.

[ccremer]

It's possible that we'll have to use a generic S3 client SDK to make buckets once an object user has been provisioned with a specific SDK.
Maybe the Minio client does it as well, maybe there are others. What's missing in the PoC is potentially adding the ACL so that the user can actually read-write the bucket

[Kidswiss]

The minio-sdk can be used as a general purpose S3 client. It's what we use in K8up (https://github.com/k8up-io/k8up/blob/master/restic/s3/client.go) and restic (https://github.com/restic/restic/blob/master/internal/backend/s3/s3.go)

As for the ACL: do you talk about the ACLs for a minio instance, or for cloudscale's S3? Because the latter sets up some ACLs already that give the object-users permissions to their own buckets.

[ccremer]

As for the ACL: do you talk about the ACLs for a minio instance, or for cloudscale's S3? Because the latter sets up some ACLs already that give the object-users permissions to their own buckets.

ACLs for cloudscale's S3. But you just answered my question :)

[Kidswiss]

I've also added the deletion of the bucket. For the first implementation we just remove the bucket as soon as the instance is deleted. We can add delays and deletion protection in further iterations.