[QUESTION/HELP] How to use multiple caching proxies at once?

[cowwoc]

Question / Where do you need Help?

Given this warning when creating a new cluster: WARN[0000] More than one registry specified, but the LocalRegistryHostingV1 spec only supports one -> Selecting the first one: k3d-docker-io.localhost how are we supposed to use multiple cache-on-pull docker registries against a single cluster?

My understanding is that I need to specify one local registry per remote registry I am caching (e.g. one for docker hub, another for github repository, etc). Next, I provide k3d with repositories.yaml that indicates which proxy is caching which remote registry.

But what's the point of doing all of this if k3d can only make use of one caching proxy at a time. Am I missing something?

[cowwoc]

To clarify, if I spin up multiple registries using k3d registry create and pass them into a cluster using k3d cluster create "--disable-default-registry-endpoint@all:*" --registry-use X --registry-use Y --registry-use Z then I see pulls working for X but failing for Y and `Z.

The error I get is:

Failed to pull image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.5.1": failed to pull and unpack image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.5.1": failed to resolve reference "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.5.1": failed to do request: Head "http://k3d-registry-k8s-io.localhost:5003/v2/ingress-nginx/kube-webhook-certgen/manifests/v1.5.1?ns=registry.k8s.io": dial tcp 172.19.0.5:5003: connect: connection refused

I am only aware of two ways to have k3d pull from multiple cached repositories:

1. https://github.com/ligfx/k3d-registry-dockerd
2. Install the open-source version of https://www.sonatype.com/products/sonatype-nexus-oss-download

• Create a "docker (proxy)" for each repository
• Create a "docker (group)" that delegates to the aforementioned repositories
• Point k3d at the "docker (group)" repository. So from k3d's perspective, it only sees a single local repository.

[cowwoc]

In retrospect, option 1 is pretty damn great. It is far easier to setup than the alternatives, so I started using it. I guess the only remaining question is: does the option recommended by the documentation support multiple repositories? And, if not, then maybe we should recommend option 1 to people instead.