Proposal: Operator-bootstrapped schema

[ecordell]

Ref: #115

The goal is to allow a user to supply a bootstrap file with schema and relationships to SpiceDB on start, via the kube apis.

Today

This is possible today via use of patches:

apiVersion: authzed.com/v1alpha1
kind: SpiceDBCluster
metadata:
  name: dev
spec:
  config:
    datastoreEngine: memory
    datastoreBootstrapFiles: /etc/bootstrap/init.yaml
  secretName: dev-spicedb-config
  patches:
  - kind: Deployment
    patch:
      spec:
        template:
          spec:
            volumes:
            - name:  bootstrap
              configMap:
                name: spicedb-bootstrap
            containers:
            - name: spicedb
              volumeMounts:
              - name: bootstrap
                mountPath: /etc/bootstrap
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: spicedb-bootstrap
data:
  init.yaml: |
    schema: |-
      definition user {}
    
      definition document {
          relation writer: user
          relation reader: user

          permission edit = writer
          permission view = reader + edit
      }
    relationships: |-
      document:firstdoc#writer@user:tom
      document:firstdoc#reader@user:fred
      document:seconddoc#reader@user:tom

But this will bootstrap every time a spicedb starts or is re-scheduled, which is not ideal.

Proposal: Managed schema

A new field, bootstrapConfigMapName will be added:

apiVersion: authzed.com/v1alpha1
kind: SpiceDBCluster
metadata:
  name: dev
spec:
  config:
    datastoreEngine: memory
  secretName: dev-spicedb-config
  bootstrapConfigMapName: spicedb-bootstrap
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: spicedb-bootstrap
data:
  init.yaml: |
    schema: |-
      definition user {}
    
      definition document {
          relation writer: user
          relation reader: user

          permission edit = writer
          permission view = reader + edit
      }
    relationships: |-
      document:firstdoc#writer@user:tom
      document:firstdoc#reader@user:fred
      document:seconddoc#reader@user:tom

Similarly to how migrations are handled, when bootstrapConfigMap is set, the operator will:

• Wait for the SpiceDB cluster to be up and running.
• Create a bootstrapping Job:
  • Run zed import, with the referenced configmap mounted as source data.
• Record a hash of the bootstrap file in the status after a successful Job run.

Any changes to the configmap will re-run the job (via comparison to the hash in the status).

We will assume that anyone using this feature wants the bootstrap to be continually run on changes; we can document a Job spec that will run once for anyone that doesn't want continual reconciliation.

Alternatives Considered

New APIs

A Schema or Bootstrap API might be nice, but the boostrap yaml file format is already well-defined and is consumable by SpiceDB and zed, so a ConfigMap is somewhat more straightforward (i.e. users can can take a file from playground and directly create it as a configmap with kubectl). There are also some in-flight discussions for changes to the local schema filesystem representation that we don't want to box ourselves out of.

SpiceDB --datastore-bootstrap-files instead of zed

The same general effect could be achieved by selectively turning the --datastore-bootstrap-files flag on and off on a SpiceDB pod, but it means that the pods will need to be rolled again after a successful bootstrap (to disable the boostrap). The Job approach avoids this.

[adel-chouchane]

hello, is this feature still far away / not considered ?

[ecordell]

It's not on the radar, especially since it can be done with existing features, but we'd welcome a pull request (or an explanation of why it should be higher priority?)

[adel-chouchane]

Hello, the existing feature support only "initiating" the datastore, and to use the patches approach you provided at the beginning of your proposal you need to also set --datastore-bootstrap-overwrite=true" and this will wipe your datastore (schema + data) on each reload/redeploy. The requested approach will make the SpiceDB Operator handles the updates of the associated schema not only the SpiceDB instances. It means, the deployment of the instance and the schema will be more GitOps friendly for people who want to keep their schemas always stored in git and if they want to update their schema they only do a simple commit and the operator will take in charge the update of the SpiceDB instance with the new schema without overwriting.
To conclude the operator will watch the configmap used to store the schema and it will launch a Job to execute a zed import each time there is a change. This will give a huge flexibility in maintaining/updating the schema in a GitOps way.

[jackwellsxyz]

I'd agree with @adel-chouchane. It would be great to have a way to only initialize or update the schema and not touch any of the relationships and other data. Also, I suppose the operator should flag a ConfigurationWarning if it's unable to update the schema because the schema is invalid? And we should probably run migrations first before attempting to write the schema.

[adel-chouchane]

Proposal: SpiceDB Schema Watcher

Purpose

This application is designed to continuously monitor a specified schema file and automatically update the schema in your SpiceDB instance whenever changes are detected, within a GitOps workflow. This ensures that your SpiceDB schema remains synchronized with your application's requirements while maintaining version control and automation.

NOTE: This application can be implemented with languages that has client libraries developed by the Authzed Team.

Sidecar Approach

Why ?

• Sidecars can communicate with the main application directly within the same pod, bypassing the Kubernetes network. This can potentially reduce the attack surface, as fewer network hops are involved.
• Real-time monitoring and updates based on schema changes.

Deployment Diagram

Patches

apiVersion: authzed.com/v1alpha1
kind: SpiceDBCluster
metadata:
  name: spicedb
  namespace: default
spec:
  config:
    image: ghcr.io/authzed/spicedb:v1.37.1
    replicas: 1
    datastoreEngine: postgres
  secretName: dev-spicedb-config
  patches:
  - kind: Deployment
    patch:
      spec:
        template:
          spec:
            volumes:
            - name:  bootstrap
              configMap:
                name: spicedb-bootstrap
            containers:
            - name: spicedb-schema-watcher
              image: # image name here
              securityContext:
                allowPrivilegeEscalation: false
                capabilities:
                  drop:
                  - ALL
                readOnlyRootFilesystem: true
                runAsGroup: 65532
                runAsNonRoot: true
                runAsUser: 65532
                seccompProfile:
                  type: RuntimeDefault
              volumeMounts:
              - name: bootstrap
                mountPath: /app/data
                readOnly: true
              env:
              - name: SCHEMA_FILE_PATH
                value: /app/data/init.yaml
              - name: SPICEDB_KEY
                valueFrom:
                  secretKeyRef:
                    name: dev-spicedb-config
                    key: preshared_key

Conclusion

This solution might add another layer to the management of SpiceDB ecosystem, but unless there is an upgrade in the SpiceDB Operator to watch the used schemas and apply changes to the respective SpiceDB clusters, it's a solid approach to handle the schema with GitOps

[jackwellsxyz]

Hey @adel-chouchane that sounds like a reasonable approach, but I'm wondering if it might be more elegant to implement this logic inside the operator itself instead of creating a new sidecar. It seems like the solution would look similar to the secret adopter inside the operator.

[adel-chouchane]

Hey @jackwellsxyz , indeed it's just a small workaround to patch things up until it's officially supported by the Operator.