Kratos container sends outbound DNS lookup requests for localhost.<domain>

[jertel]

This discussion is being posted for long-term preservation. It was originally discussed in the Ory Kratos Slack chat:

I'm seeing DNS lookups for localhost.. occuring roughly on 30 second intervals. I'm exec'd into the kratos container and it has localhost defined in /etc/hosts. Can anyone point me to why this is occurring or kratos scheduled jobs/cache timeouts that could explain this?

More details:

If I modify the dockerfile for Kratos to have an entrypoint of /bin/sh, and then apk add tcpdump there are no DNS lookups for "localhost..." If I then docker exec into the container in another terminal and startup kratos, immediately there's an outbound DNS lookup for that localhost.xxx.yyy name. It then repeats every 30 seconds. It's consistently reproducible.

The config file settings appear to not have an impact on this, as any URLs, hostnames I define in the config does not stop the "localhost." lookups.

Below is a tcpdump output (from within the container) starting right a couple minutes before I launched the kratos process inside the container. At 19:51:16 I started kratos in another terminal window, via docker exec. Immediately after that PTR request, there's a localhost.home.mypersonaldomain.com lookup. That repeats roughly every 30 seconds. If I then kill kratos in the second terminal window, the localhost lookups cease.

19:51:03.811409 IP 172.17.0.1.48334 > 239.255.255.250.1900: UDP, length 167
19:51:04.812416 IP 172.17.0.1.48334 > 239.255.255.250.1900: UDP, length 167
19:51:05.813520 IP 172.17.0.1.48334 > 239.255.255.250.1900: UDP, length 167
19:51:06.814319 IP 172.17.0.1.48334 > 239.255.255.250.1900: UDP, length 167
19:51:10.913214 IP 172.17.0.2.33716 > 239.255.255.250.1900: UDP, length 146
19:51:10.913291 IP 172.17.0.2.41620 > 239.255.255.250.1900: UDP, length 146
19:51:11.013715 IP 172.17.0.2.33716 > 239.255.255.250.1900: UDP, length 146
19:51:11.013767 IP 172.17.0.2.41620 > 239.255.255.250.1900: UDP, length 146
19:51:11.114162 IP 172.17.0.2.33716 > 239.255.255.250.1900: UDP, length 146
19:51:11.114217 IP 172.17.0.2.41620 > 239.255.255.250.1900: UDP, length 146
19:51:16.939246 IP6 fe80::e4b4:12ff:aabb:ccdd.5353 > ff02::fb.5353: 0 [9q] PTR (QM)? _ipps._tcp.local. PTR (QM)? _ftp._tcp.local. PTR (QM)? _webdav._tcp.local. PTR (QM)? _webdavs._tcp.local. PTR (QM)? _sftp-ssh._tcp.local. PTR (QM)? _smb._tcp.local. PTR (QM)? _afpovertcp._tcp.local. PTR (QM)? _nfs._tcp.local. PTR (QM)? _ipp._tcp.local. (141)
19:51:17.049277 IP 02f23d91b430.37260 > firewall.home.mypersonaldomain.com.53: 47636+ AAAA? localhost.home.mypersonaldomain.com. (47)
19:51:17.049277 IP 02f23d91b430.46393 > firewall.home.mypersonaldomain.com.53: 49474+ A? localhost.home.mypersonaldomain.com. (47)
19:51:17.049496 IP firewall.home.mypersonaldomain.com.53 > 02f23d91b430.46393: 49474* 1/0/0 A 127.0.0.1 (63)
19:51:17.049508 IP firewall.home.mypersonaldomain.com.53 > 02f23d91b430.37260: 47636* 1/0/0 AAAA ::1 (75)
19:51:22.189964 ARP, Request who-has 02f23d91b430 tell 172.17.0.1, length 28
19:51:22.189972 ARP, Request who-has 172.17.0.1 tell 02f23d91b430, length 28
19:51:22.189979 ARP, Reply 02f23d91b430 is-at 02:42:ac:11:00:0a (oui Unknown), length 28
19:51:22.189990 ARP, Reply 172.17.0.1 is-at 02:42:ee:78:b9:ab (oui Unknown), length 28
19:51:34.221997 IP6 fe80::e4b4:12ff:aabb:ccdd > ip6-allrouters: ICMP6, router solicitation, length 16
19:51:46.051222 IP 02f23d91b430.45202 > firewall.home.mypersonaldomain.com.53: 56519+ AAAA? localhost.home.mypersonaldomain.com. (47)
19:51:46.051222 IP 02f23d91b430.51812 > firewall.home.mypersonaldomain.com.53: 52197+ A? localhost.home.mypersonaldomain.com. (47)
19:51:46.051495 IP firewall.home.mypersonaldomain.com.53 > 02f23d91b430.51812: 52197* 1/0/0 A 127.0.0.1 (63)
19:51:46.051537 IP firewall.home.mypersonaldomain.com.53 > 02f23d91b430.45202: 56519* 1/0/0 AAAA ::1 (75)
19:52:01.266030 IP6 fe80::42:eeff:1122.3344.5353 > ff02::fb.5353: 0 [7q] PTR (QM)? _ftp._tcp.local. PTR (QM)? _nfs._tcp.local. PTR (QM)? _afpovertcp._tcp.local. PTR (QM)? _smb._tcp.local. PTR (QM)? _sftp-ssh._tcp.local. PTR (QM)? _webdavs._tcp.local. PTR (QM)? _webdav._tcp.local. (118)
19:52:01.266128 IP 172.17.0.1.5353 > 224.0.0.251.5353: 0 [7q] PTR (QM)? _ftp._tcp.local. PTR (QM)? _nfs._tcp.local. PTR (QM)? _afpovertcp._tcp.local. PTR (QM)? _smb._tcp.local. PTR (QM)? _sftp-ssh._tcp.local. PTR (QM)? _webdavs._tcp.local. PTR (QM)? _webdav._tcp.local. (118)
19:52:01.266216 IP 02f23d91b430.39772 > firewall.home.mypersonaldomain.com.53: 13346+ PTR? b.a.9.b.8.7.e.f.f.f.e.e.2.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa. (90)
19:52:01.266430 IP firewall.home.mypersonaldomain.com.53 > 02f23d91b430.39772: 13346 NXDomain* 0/1/0 (149)
19:52:01.266568 IP 02f23d91b430.41674 > firewall.home.mypersonaldomain.com.53: 15243+ PTR? 251.0.0.224.in-addr.arpa. (42)
19:52:01.294839 IP firewall.home.mypersonaldomain.com.53 > 02f23d91b430.41674: 15243 NXDomain 0/1/0 (99)
19:52:06.477980 ARP, Request who-has 172.17.0.1 tell 02f23d91b430, length 28
19:52:06.477989 ARP, Request who-has 02f23d91b430 tell 172.17.0.1, length 28
19:52:06.477998 ARP, Reply 02f23d91b430 is-at 02:42:ac:11:00:0a (oui Unknown), length 28
19:52:06.478002 ARP, Reply 172.17.0.1 is-at 02:42:ee:78:b9:ab (oui Unknown), length 28
19:52:10.910901 IP 172.17.0.2.33716 > 239.255.255.250.1900: UDP, length 146
19:52:10.911004 IP 172.17.0.2.41620 > 239.255.255.250.1900: UDP, length 146
19:52:11.011318 IP 172.17.0.2.33716 > 239.255.255.250.1900: UDP, length 146
19:52:11.011383 IP 172.17.0.2.41620 > 239.255.255.250.1900: UDP, length 146
19:52:11.113182 IP 172.17.0.2.33716 > 239.255.255.250.1900: UDP, length 146
19:52:11.113235 IP 172.17.0.2.41620 > 239.255.255.250.1900: UDP, length 146
19:52:16.053406 IP 02f23d91b430.57210 > firewall.home.mypersonaldomain.com.53: 38941+ A? localhost.home.mypersonaldomain.com. (47)
19:52:16.053406 IP 02f23d91b430.52305 > firewall.home.mypersonaldomain.com.53: 56150+ AAAA? localhost.home.mypersonaldomain.com. (47)
19:52:16.053671 IP firewall.home.mypersonaldomain.com.53 > 02f23d91b430.52305: 56150* 1/0/0 AAAA ::1 (75)
19:52:16.053700 IP firewall.home.mypersonaldomain.com.53 > 02f23d91b430.57210: 38941* 1/0/0 A 127.0.0.1 (63)

[jertel]

It looks like the localhost FQDN part of the problem is a known issue with golang:

• Alpine linux doesn't have nsswitch configuration file gliderlabs/docker-alpine#367
• net: pure go version of dnsclient does a DNS lookup of "localhost" qualified with the domain name. golang/go#32017.

I can work around this with an additional line in my Dockerfile:

RUN [ ! -e /etc/nsswitch.conf ] && echo 'hosts: files dns' > /etc/nsswitch.conf

Though I'm still curious what is causing Kratos to attempt a localhost lookup every 30 seconds, in case anyone knows the answer to that.

[aeneasr]

Without logs / reproducible repo not possible to say. Could be anything from DB to DNS to ...

…

On 15. Feb 2021, at 12:29, Jason Ertel ***@***.***> wrote:  It looks like the localhost FQDN part of the problem is a known issue with golang: gliderlabs/docker-alpine#367 golang/go#32017. I can work around this with an additional line in my Dockerfile: RUN [ ! -e /etc/nsswitch.conf ] && echo 'hosts: files dns' > /etc/nsswitch.conf Though I'm still curious what is causing Kratos to attempt a localhost lookup every 30 seconds, in case anyone knows the answer to that. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[aeneasr]

Ok cool! I'm honestly intrigued to learn what is going on there. I think the explanation is very simple. I also made sure to check if we're actually doing any tcp traffic but it does indeed look like a DNS lookup only. I still think it's caused by SMTP.

[aeneasr]

This will probably be auto-fixed with Go 1.16:

On Linux, host name lookups no longer use DNS before checking /etc/hosts when /etc/nsswitch.conf is missing; this is common on musl-based systems and makes Go programs match the behavior of C programs on those systems.

[jertel]

Good find. Thanks!

[jertel]

I've confirmed that the DNS lookup noise is caused by the import of the github.com/ory/x/tracing package.

Unfortunately, the mere import alone is all that's necessary to cause the repeating localhost lookups. Choosing not to activate the tracing is irrelevant. I do not see any code in the ory/x/tracing package that would cause this, so that would likely suggest that one of the third-party imports, such as open-tracing, jaeger, datadog, elastic, etc. has some code, such as a package init() function, that is spawning an async thread in the background.

I've reached my time limit on this research, perhaps someone else could take the next step of isolating which dependent package inside of the ory/x/tracing package is causing this. Or kratos could offer a build switch that prevented the tracing package from being imported.

[aeneasr]

Wow, nice find! That's definitely already worth a lot. I'm pretty sure that it's one of the connectors. Maybe they are trying to find an agent on the local host to set up tracing - not sure.

Thank you for spending time on this!