[BUG] k3d-managed registry with non-5000 port not accessible from within the cluster

[yashgorana]

What did you do

How was the cluster created?

$ k3d registry create registry.localhost --port 12345
$ k3d cluster create test-cluster --registry-use k3d-registry.localhost:12345

What did you do afterwards?

Tried to access the registry k3d-registry.localhost:12345 from a pod in the k3d cluster

$ kubectl run -i --rm --tty busybox --image=busybox --restart=Never -- sh
/ # wget -qO- http://k3d-registry.localhost:12345/v2/_catalog
wget: can't connect to remote host (172.19.0.2): Connection refused

What did you expect to happen

Since registry is exposed on :12345, and k3d injects this domain in coredns' NodeHosts, http://k3d-registry.localhost:12345/ should be accessible from within the cluster. Instead k3d-registry.localhost:5000 is accessible. I don't want to use host.k3d.internal:12345 as it requires us to tweak the domains at application layer, which isn't ideal.

If we map k3d-registry.localhost to host system's IP (like we do for host.k3d.internal), instead of the registry IP, then the above works as expected.

Screenshots or terminal output

Which OS & Architecture

Debian on WSL2

arch: x86_64
cgroupdriver: cgroupfs
cgroupversion: "1"
endpoint: /var/run/docker.sock
filesystem: extfs
infoname: docker-desktop
name: docker
os: Docker Desktop
ostype: linux
version: 25.0.1

Which version of k3d

k3d version v5.6.0
k3s version v1.27.4-k3s1 (default)

Which version of docker

Client:
 Cloud integration: v1.0.35+desktop.10
 Version:           25.0.1
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        29cf629
 Built:             Tue Jan 23 23:08:30 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Desktop
 Engine:
  Version:          25.0.1
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       71fa3ab
  Built:            Tue Jan 23 23:09:46 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.27
  GitCommit:        a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc:
  Version:          1.1.11
  GitCommit:        v1.1.11-0-g4bccb38
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Client:
 Version:    25.0.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1-desktop.4
    Path:     /usr/local/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.3-desktop.1
    Path:     /usr/local/lib/docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.22
    Path:     /usr/local/lib/docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.21
    Path:     /usr/local/lib/docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     /usr/local/lib/docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.0.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.3.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-scout
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-scan: no such file or directory

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 13
 Server Version: 25.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc version: v1.1.11-0-g4bccb38
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.133.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 28
 Total Memory: 15.5GiB
 Name: docker-desktop
 ID: a849da8e-3fd3-4c5f-9422-bdce4b1a7bae
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

[yakom]

i encountered this bug while using k3d v5.8.1.

the registry container that k3d registry create spawns is configured to listen on the port 5000 regardless of what's passed via the --port flag.

[iwilltry42]

The --port flag on the registry create command only changes the exposed port (host port), as is described in the command help text.
So I'd rather take this as a feature request to enhance --port to also tweak the internal container port the registry listens on?

I'm happy that @yakom offered to take a stab at this - that would be great!

[yakom]

this is a new codebase to me so it might take me a while, but i'll try.

So I'd rather take this as a feature request to enhance --port to also tweak the internal container port the registry listens on?

@iwilltry42 is it indeed a feature request? these port numbers have to match, otherwise the registry is not usable - at least with what i think is the default usage, one even displayed by the k3d registry create post-completion help message?

[iwilltry42]

@yakom , yes it is indeed a feature request.
The original intention of the registry create command (and siblings) is to have an image registry that can be used by K3s, i.e. that enables you to create pods using images that are stored in that registry. Also, it should be accessible from your host machine such that you can push images to it.
Both are possible. In addition to that, we provide a configmap that makes it work with Tilt/Skaffold.

What is not possible as per this comment is to reach the registry from within a pod inside the cluster on the exposed port. That is because the exposed port that is set via the --port flag is exposed to the host and is not the port the registry listens on in its container. When querying the registry from within a pod in the k3d cluster, the traffic never leaves the docker network and as such doesn't find that exposed port.

these port numbers have to match

Just what I meant with "enhance --port to also tweak the internal container port the registry listens on" - might be badly phrased from my side. It was just convenience that we can safely assume 5000 for everything regarding "internal" traffic.
AFAICT, this should be as "simple" as setting the REGISTRY_HTTP_ADDR env var in the registry container... and changing all the occurrences of its usage in the k3d code 😬