Kubedns returns NXDOMAIN from inside PODs but works from outside

[geosword]

image: gcr.io/google_containers/k8s-dns-(kube-dns|dnsmasq-nanny|sidecar)-amd64:1.14.8

I've run through: https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/
Everything is correct (apart from performing nslookups from the busybox pod).

bddcbpkbn1:~ # kubectl exec -ti busybox-5ccc978d8d-x87kb -- nslookup kubernetes
Server:         172.17.0.10
Address:        172.17.0.10:53

** server can't find kubernetes: NXDOMAIN

*** Can't find kubernetes: No answer

However:

bddcbpkbn1:~ # kubectl exec -ti busybox-5ccc978d8d-x87kb -- nslookup kubernetes.default.svc.cluster.local
Server:         172.17.0.10
Address:        172.17.0.10:53

Non-authoritative answer:
Name:   kubernetes.default.svc.cluster.local
Address: 172.17.0.1

*** Can't find kubernetes.default.svc.cluster.local: No answer

The resolve.conf of busybox:

bddcbpkbn1:~ # kubectl exec -ti busybox-5ccc978d8d-x87kb -- cat /etc/resolv.conf 
nameserver 172.17.0.10
search default.svc.cluster.local svc.cluster.local cluster.local my.local.domain
options ndots:5

Aside from busybox not being able to resolve "Kubernetes" on its own. "Non-authoritative answer:" Seems incorrect as well.

Just for the removal of doubt. Im expecting

bddcbpkbn1:~ # kubectl exec -ti busybox-5ccc978d8d-x87kb -- nslookup kubernetes

To return "172.17.0.1"

[fturib]

Can you provide the version of busybox you are using ?

We found several similar use case lately, and it was related to busybox itself.

check this comment : kubernetes/kubernetes#66924 (comment)

It looks like DNS inside busybox does not work properly.
At least it works for me with busybox images <= 1.28.4

and this issue related to busybox : docker-library/busybox#48

[geosword]

That was exactly it. Would never have thought of that. Thank you so much.

[fturib]

@geosword : I guess we can close that issue now ?

[AATHITH]

Can you provide the version of busybox you are using ?

We found several similar use case lately, and it was related to busybox itself.

check this comment : kubernetes/kubernetes#66924 (comment)

It looks like DNS inside busybox does not work properly.
At least it works for me with busybox images <= 1.28.4

and this issue related to busybox : docker-library/busybox#48

im not using any busybox but still getting this same error! @fturib
Server: 67.207.67.3
Address: 67.207.67.3#53

** server can't find consul-0: NXDOMAIN

[chrisohaver]

@AATHITH, What did the command look like as you wrote it? From where did you execute the command?

[AATHITH]

What did the command look like as you wrote it? From where did you execute the command?

@chrisohaver
cmd: nslookup podname
result:
Server: 67.207.67.3
Address: 67.207.67.3#53

** server can't find podname: NXDOMAIN

I run the command in kubernetes master
k8's version:1.14
ubuntu:16.04

[chrisohaver]

I run the command in kubernetes master

This will not work from a node, it will only work from a Pod. From a node, you are just querying the upstream server (67.207.67.3), and of course it has no knowledge of the Pods in your cluster.