Can't Pull Image From Local Registry

[enchant3dmango]

Hi guys, I just deployed a Kind cluster in my local (M1 Pro 2021). I followed the guide at https://kind.sigs.k8s.io/docs/user/local-registry/, but somehow, I can't pull any local image to my Kind registry (I can't connect to localhost:5001). Kindly help me, I'm stuck.

Here is the log when I try to pull image from my local registry:

# cat /etc/containerd/certs.d/localhost:5001/hosts.toml
[host."http://kind-registry:5000"]

# ctr --namespace=k8s.io images pull localhost:5001/airflow:2.9.3-python-3.11
INFO[0000] host will try HTTPS first since it is configured for HTTP with a TLS configuration, consider changing host to HTTPS or removing unused TLS configuration  host="localhost:5001"
INFO[0000] trying next host                              error="failed to do request: Head \"https://localhost:5001/v2/airflow/manifests/2.9.3-python-3.11\": dial tcp [::1]:5001: connect: connection refused" host="localhost:5001"
ctr: failed to resolve reference "localhost:5001/airflow:2.9.3-python-3.11": failed to do request: Head "https://localhost:5001/v2/airflow/manifests/2.9.3-python-3.11": dial tcp [::1]:5001: connect: connection refused

This is my provision script and my kind cluster config:

• k8s/provision.sh

#!/bin/sh
set -o errexit

# 1. Create registry container unless it already exists
reg_name='kind-registry'
reg_port='5001'
cluster_name=$1

if [ "$(docker inspect -f '{{.State.Running}}' "${reg_name}" 2>/dev/null || true)" != 'true' ]; then
  docker run \
    -d --restart=always -p "127.0.0.1:${reg_port}:5000" --network bridge --name "${reg_name}" \
    registry:2
fi

# 2. Create kind cluster with containerd registry config dir enabled
# TODO: kind will eventually enable this by default and this patch will
# be unnecessary.
#
# See:
# https://github.com/kubernetes-sigs/kind/issues/2875
# https://github.com/containerd/containerd/blob/main/docs/cri/config.md#registry-configuration
# See: https://github.com/containerd/containerd/blob/main/docs/hosts.md
kind create cluster --name ${cluster_name} --config k8s/kind-cluster.yaml

# 3. Add the registry config to the nodes
#
# This is necessary because localhost resolves to loopback addresses that are
# network-namespace local.
# In other words: localhost in the container is not localhost on the host.
#
# We want a consistent name that works from both ends, so we tell containerd to
# alias localhost:${reg_port} to the registry container when pulling images
REGISTRY_DIR="/etc/containerd/certs.d/localhost:${reg_port}"
for node in $(kind get nodes --name ${cluster_name}); do
  docker exec "${node}" mkdir -p "${REGISTRY_DIR}"
  cat <<EOF | docker exec -i "${node}" cp /dev/stdin "${REGISTRY_DIR}/hosts.toml"
[host."http://${reg_name}:5000"]
EOF
done

# 4. Connect the registry to the cluster network if not already connected
# This allows kind to bootstrap the network but ensures they're on the same network
if [ "$(docker inspect -f='{{json .NetworkSettings.Networks.kind}}' "${reg_name}")" = 'null' ]; then
  docker network connect "kind" "${reg_name}"
fi

# 5. Document the local registry
# https://github.com/kubernetes/enhancements/tree/master/keps/sig-cluster-lifecycle/generic/1755-communicating-a-local-registry
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: local-registry-hosting
  namespace: kube-public
data:
  localRegistryHosting.v1: |
    host: "localhost:${reg_port}"
    help: "https://kind.sigs.k8s.io/docs/user/local-registry/"
EOF

# 6. Create GCP service account
# Create a file named serviceaccount.json inside files/ containing your GCP service account json
sa_file_path='files/serviceaccount.json'
kubectl create ns airflow
kubectl create secret generic airflow-gcp-sa --from-file=${sa_file_path} -n airflow

# Extra step: uncomment this step and create a file named gitSshKey inside ./files/ containing the git ssh key
# https://airflow.apache.org/docs/helm-chart/stable/manage-dags-files.html#mounting-dags-from-a-private-github-repo-using-git-sync-sidecar
# 7. Create Git SSH key
# ssh_key_file_path=./files/gitSshKey
# kubectl create secret generic airflow-ssh-secret --from-file=${ssh_key_file_path} -n airflow

• k8s/kind-cluster.yaml

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry]
    config_path = "/etc/containerd/certs.d"
nodes:
  - role: control-plane
  - role: worker
    kubeadmConfigPatches:
      - |
        kind: JoinConfiguration
        nodeRegistration:
          kubeletExtraArgs:
            node-labels: "has-cpu=true"
  - role: worker
    kubeadmConfigPatches:
      - |
        kind: JoinConfiguration
        nodeRegistration:
          kubeletExtraArgs:
            node-labels: "has-gpu=true"

Thanks in advance, guys!

[stmcginnis]

I don't have time to dig in right now, but my hunch is the local registry docs need to be updated now that #3828 has changed the version of containerd.

[BenTheElder]

I don't have time to dig in right now, but my hunch is the local registry docs need to be updated now that #3828 has changed the version of containerd.

They shouldn't need to as we are just explicitly enabling a default that should be default in containerd 2.0

But also that PR should only affect things if you run kind installed at HEAD instead of a tagged release, or explicitly use the one new image we've pushed.

[BenTheElder]

Can you try to find a minimum reproducer for your issue? There's a lot going on in this setup and some of it probably isn't related?

What does docker info show (or podman info or nerdctl or)? What container runtime etc do you have installed? What version / settings / OS / install tooling?

What kind version?

[enchant3dmango]

@BenTheElder, I'm sorry. Let me add the details.

Step to reproduce:

• Run ./k8s/provision.sh kind.
• Run docker exec -it kind-worker sh.
• Check the registry config in the node (kind-worker for example), run docker exec -it kind-worker cat /etc/containerd/certs.d/localhost:5001/hosts.toml.
• Test connection to local registry, run docker exec -it kind-worker curl http://localhost:5001/v2/_catalog.

I'm using kind version 0.26.0. And here is my docker info:

docker info
Client:
 Version:    27.4.0
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  ai: Ask Gordon - Docker Agent (Docker Inc.)
    Version:  v0.5.1
    Path:     /Users/enchant3dmango/.docker/cli-plugins/docker-ai
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.19.2-desktop.1
    Path:     /Users/enchant3dmango/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.31.0-desktop.2
    Path:     /Users/enchant3dmango/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.37
    Path:     /Users/enchant3dmango/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Beta) (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/enchant3dmango/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/enchant3dmango/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.27
    Path:     /Users/enchant3dmango/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/enchant3dmango/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /Users/enchant3dmango/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/enchant3dmango/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.15.1
    Path:     /Users/enchant3dmango/.docker/cli-plugins/docker-scout

Server:
 Containers: 5
  Running: 4
  Paused: 0
  Stopped: 1
 Images: 4
 Server Version: 27.4.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.14-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 7.654GiB
 Name: docker-desktop
 ID: a6ad3bb4-4619-42ca-857c-2dc31fdad94c
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/enchant3dmango/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

[BenTheElder]

Step to reproduce:

I meant: Can you identify the simplest setup that presents this issue in your environment? So we can ignore the non-essential options and identify the root cause more quickly.

I can't repro this directly due to docker desktop's licensing changes, it is not permitted at my employer.

https://www.theregister.com/2021/08/31/docker_desktop_no_longer_free/

I have docker installed on linux directly, but not docker desktop on a mac.

[BenTheElder]

Test connection to local registry, run docker exec -it kind-worker curl http://localhost:5001/v2/_catalog.

That's expected to not work, we tell containerd to actually pull localhost:5001 from kind-registry:5000

This is so we can present the same image name on the host and in the cluster.
localhost IP addresses are local to containers (their network namespace) and the host and is not actually shared between them, but we ask docker to port forward from the host to the registry container so it works on the host, and we tell containerd to actually pull from the container's name and make sure the container is on the kind network.

[enchant3dmango]

The simplest setup referring to this guide (https://kind.sigs.k8s.io/docs/user/local-registry/) is to remove the step 6 from the k8s/provision.sh. Sorry if I misunderstood again.

# 6. Create GCP service account
# Create a file named serviceaccount.json inside files/ containing your GCP service account json
sa_file_path='files/serviceaccount.json'
kubectl create ns airflow
kubectl create secret generic airflow-gcp-sa --from-file=${sa_file_path} -n airflow

I'm trying to understand the last comment, I'll get back later on when I have something to say. Thanks for the insight, sir!

[enchant3dmango]

@BenTheElder thanks for the clue sir, even though I'm not sure if I understand it well enough. I just realized how silly I am, I forgot to push the tagged image to localhost:5001. 😅

Closing this issue now. Thanks for the help sir @BenTheElder and sir @stmcginnis. Most appreciated it!

[BenTheElder]

Glad it's working now :-)