feat: docker & k8s

[tchataigner]

This PR introduces container and docker publishing for the Light Clients.

1. I revamped our binaries to only encompass one binary for the proof server. It has two modes, single and split. This allows us to keep the flow we had previously or up only one server that handles everything. This is particularly useful when it comes to k8s configuration as it allows us to use the HorizontalPodAutoscaler.
2. Dockerfile: There is a dedicated Dockerfile in the docker folder that is used to build an image containing the two binaries for client and proof_server.
3. docker-compose.yml: This is a configuration file to create a local cluster of containers that emulates the Light Client running. This will be taxing on the machine running the compose configuration.
4. k8s configuration: I have created 3 sets of files, one for the proof_server and one for each client. This is only a base to what would be used in the future to deploy in an automated fashion our LC with terraform.
5. Minor proof-server update: We now generate the proving and verifying keys before the proof server starts to listen to connection. I also added a small method in the client to test connection to the proof server before actually starting the initialization of the client.
6. A docker publish workflow to be called on a branch to publish the docker image of the desired LC.

[huitseeker]

I'm not sure I understand why the clients need a deployment?

[huitseeker]

Here I would recommend using let-else syntax to deal with the error case close to the point of deserialization (rather than looking 100 lines below).
Then we can pattern-match on the request.

[huitseeker]

Consider using backoff with its tokio feature here:
https://docs.rs/backoff/latest/backoff/#async

[huitseeker]

I think we want to ensure at least 2 replicas (1 active + 1 spare)

  minReplicas: 2

The max can be larger (5? 10?): My point is if the customer wants to stagger requests, that's ok.

[huitseeker]

This one is worth insisting on.

[huitseeker]

That's good, but we probably want to set up a health, and more importantly a readiness probe on each server, so that we can route queries only to available healthy provers? Once we have that (which we might want to open an issue for?), this would look like:

readinessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5
livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 15
  periodSeconds: 10

[tchataigner]

From what I understood of how HPA works, it seems that the scaling and the routing are both handled by it. We can pass some custom parameters to the configuration to set a "readiness" timer as per the documentation:

Due to technical constraints, the HorizontalPodAutoscaler controller cannot exactly determine the first time a pod becomes ready when determining whether to set aside certain CPU metrics. Instead, it considers a Pod "not yet ready" if it's unready and transitioned to ready within a short, configurable window of time since it started. This value is configured with the --horizontal-pod-autoscaler-initial-readiness-delay flag, and its default is 30 seconds.

Then the HPA will know when the Pod is ready to be used as a valid replica.

[huitseeker]

Right. I'm not trying to drive the HPA by suggesting to add these readiness and health endpoints.

I'm trying to drive the "available replica count", which will drive the service's routing decisions (you're right that the HPA will make scaling decisions considering metrics from all pods, including those that are "not ready." This can lead to a situation where the HPA scales up or down based on metrics from pods that are not actively serving traffic).

But driving the service, since it is user-facing, is still important.

Readiness and health are K8s-wide concepts (hence more general than the HPA):

• Liveness Probes: These are used to indicate if a pod is running. If a liveness probe fails, Kubernetes will kill the pod and attempt to restart it.
• Readiness Probes: These are used to indicate when a pod is ready to start serving traffic. If a readiness probe fails, the pod is removed from the Service's load balancer, meaning it will not receive any traffic until it is ready again.

See https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes/

[tchataigner]

Got it thank you! I wasn't familiar with those concepts. I'll try a first implementation.

[huitseeker]

Same remark as above on the backoff.

[huitseeker]

Same as above on the backoff, and since this is the 3d snippet, we may want to generalize to a utility function.

[tchataigner]

I'm not sure I understand why the clients need a deployment?

I thought that it would be useful to have k8s configuration for it. However, it is true that users should have their own logic to interact with the proof servers. I'll remove it.

[huitseeker]

This looks good, but I opened #197 for my OCD

[huitseeker]

This one is worth insisting on.

[wwared]

I've skimmed things and this looks good and looks like the feedback has been addressed, approving to unblock

[huitseeker]

This looks great, thanks!!