Expose / pod never starts fails on readinessProbe

[echo-devnull]

I created a kubernetes service that listens on port 222 which I want to "reverse tunnel" to my local machine.

kubectl create svc clusterip gitlab --tcp=222:222

Then I expose it via:

localizer expose default/gitlab

But the pod that gets created, never actually gets to "running" state:

Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  2m45s                default-scheduler  Successfully assigned default/localizer-gitlab-kjqpt to lgs-lnx7624
  Normal   Pulled     2m44s                kubelet            Container image "linuxserver/openssh-server" already present on machine
  Normal   Created    2m44s                kubelet            Created container default
  Normal   Started    2m44s                kubelet            Started container default
  Warning  Unhealthy  5s (x20 over 2m44s)  kubelet            Readiness probe failed: dial tcp 172.29.4.101:2222: connect: connection refused

The yaml:

apiVersion: v1
kind: Pod
metadata:
annotations:
  cni.projectcalico.org/containerID: 58d1352a3f8739d41f8589dd1e1dd31a15517060b79687e1cee386f9a45e7454
  cni.projectcalico.org/podIP: 172.29.4.101/32
  cni.projectcalico.org/podIPs: 172.29.4.101/32
  kubernetes.io/psp: default-psp
  localizer.jaredallard.github.com/objects: '[]'
labels:
  app: gitlab
  localizer.jaredallard.github.com/exposed: "true"
name: localizer-gitlab-kjqpt
namespace: default
spec:
containers:
- env:
  - name: PASSWORD_ACCESS
    value: "true"
  - name: USER_PASSWORD
    value: supersecretpassword
  - name: USER_NAME
    value: outreach
  - name: DOCKER_MODS
    value: linuxserver/mods:openssh-server-ssh-tunnel
  image: linuxserver/openssh-server
  imagePullPolicy: IfNotPresent
  name: default
  ports:
  - containerPort: 222
  readinessProbe:
    tcpSocket:
      port: 2222
  resources:
    limits:
      cpu: 100m
      memory: 100Mi
  volumeMounts:
  - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
    name: kube-api-access-9kgvq
    readOnly: true
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: OnFailure
serviceAccountName: default
tolerations:
- effect: NoExecute
  key: node.kubernetes.io/not-ready
  operator: Exists
  tolerationSeconds: 300
- effect: NoExecute
  key: node.kubernetes.io/unreachable
  operator: Exists
  tolerationSeconds: 300
volumes:
- name: kube-api-access-9kgvq
  projected:
    sources:
    - serviceAccountToken:
        expirationSeconds: 3607
        path: token
    - configMap:
        items:
        - key: ca.crt
          path: ca.crt
        name: kube-root-ca.crt
    - downwardAPI:
        items:
        - fieldRef:
            fieldPath: metadata.namespace
          path: namespace

And the logging from the running localizer:

INFO[0023] created pod localizer-gitlab-kjqpt            component=exposer service=default/gitlab
INFO[0023] waiting for remote pod to be ready ...        component=exposer service=default/gitlab
INFO[0360] pkg/mod/github.com/jaredallard/[email protected]/tools/cache/reflector.go:169: watch of *v1.Deployment ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 11; INTERNAL_ERROR; received from peer") has prevented the request from succeeding  caller="reflector.go:347"
INFO[0361] pkg/mod/github.com/jaredallard/[email protected]/tools/cache/reflector.go:169: watch of *v1.StatefulSet ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 17; INTERNAL_ERROR; received from peer") has prevented the request from succeeding  caller="reflector.go:347"

[jaredallard]

Does the pod have any notable logs? It sounds like the SSH server inside of the pod never became happy, for whatever reason.

[echo-devnull]

No all the logging is in the original post I'm afraid. The pod itself did not create any logging.

[jaredallard]

That's... bizarre. Makes me think something went wrong with the image being downloaded. Can you try again but ensure that your nodes don't have an existing linuxserver/openssh-server image present? That should download the latest image and hopefully work 😓

[echo-devnull]

I've got the time to look into this again ;-)

So, the expose (reverse tunnel) pod that get's created does have a single log entry:

[mod-init] Attempting to run Docker Modification Logic

But does not move beyond that, or ever get to the ready status.
According to describe the pod itself is running, but the Readiness probe failed. It is trying to connect to "2222" ? That seems wrong to me. Should it not be "22" ?

Nevermind, it seems to indeed be the default number: https://docs.linuxserver.io/images/docker-openssh-server

[jaredallard]

@echo-devnull Is there some sort of firewall being used? I believe that part of the linuxserver openssh image downloads a git repo at runtime, so I wonder if it's something related to that? Or, DNS lookup issues. Spitballing ideas. I haven't been able to repro this so I don't know what else to suggest 😭