After updating to latest build (not sure if that triggered the issue), I am getting this:

[sergi0g]

After updating to latest build (not sure if that triggered the issue), I am getting this:

WARN  Failed to authenticate to registry registry-1.docker.io with given token!

Is there a way to remove the cached token? Might be nice to include that in the message.

Originally posted by @johnnyggalt in #18 (comment)

[sergi0g]

Hello @johnnyggalt!

The problem is most likely that you have private images pulled from Docker Hub and you haven't configured authentication for Cup. Please also read the docs for the configuration.

Images which couldn't be checked are labelled as unknown in the output, so you should easily be able to tell if that's the problem.

[johnnyggalt]

Thanks for your response, @sergi0g.

Sorry, I really didn't provide enough context. I am using this script to periodically check for updates:

#!/bin/bash

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v ${PWD}/cup.json:/config/cup.json -t ghcr.io/sergi0g/cup -c /config/cup.json check -i

Here is my cup.json:

{
    "insecure_registries": ["myregistry:5000"]
}

I have been running this for a while without issue, but then cup itself was listed as having an update, so I pulled latest. After that I started getting token failures in the output for all Docker Hub images, which means they display as "unknown" in the results.

But the thing is, none of the Docker Hub images are private. And they were all checking just fine before I updated cup.

[sergi0g]

Thanks for clarifying! In this case, it would be helpful if you could give me a list of images labelled as unknown in the output and the token from the logs (don't worry, this isn't dangerous).

[johnnyggalt]

OK, I just ran it again and checked the contents of the JWT once I saw the errors pop up (the process hadn't even finished yet). The token had already expired because it's taking significantly longer than 5 minutes to check the images, so perhaps the issue is that the token is assumed to be valid indefinitely, rather than requesting another token once it has expired?

I don't know if that explains why it worked for me previously because I would always just start it running and then come back to it later, paying no attention to how long it was taking to execute. So perhaps it was executing faster in the previous version so as not to trigger this problem.

In any case, here is the relevant info...

Output

✓ Done!                                                                                                                  ghcr.io/immich-app/immich-machine-learning:v1.112.1                                                       Up to date
 ghcr.io/immich-app/immich-server:v1.112.1                                                                 Up to date
 ghcr.io/linkwarden/linkwarden:v2.7.1                                                                      Up to date
 ghcr.io/sergi0g/cup:latest                                                                                Up to date
 quay.io/vouch/vouch-proxy:alpine-0.40.0                                                                   Up to date
 cloudflare/cloudflared:latest                                                                                Unknown
 crazymax/diun:4                                                                                              Unknown
 deluan/navidrome:0.52.5                                                                                      Unknown
 frooodle/s-pdf:0.29.0                                                                                        Unknown
 frooodle/s-pdf:0.29.0-fat                                                                                    Unknown
 gitea/gitea:1.22.1                                                                                           Unknown
 joxit/docker-registry-ui:2                                                                                   Unknown
 louislam/dockge:1                                                                                            Unknown
 louislam/uptime-kuma:1                                                                                       Unknown
 nginx:1.25.4                                                                                                 Unknown
 portainer/portainer-ce:2.20.3-alpine                                                                         Unknown
 postgres:14                                                                                                  Unknown
 postgres:16                                                                                                  Unknown
 postgres:16-alpine                                                                                           Unknown
 registry:2.8.3                                                                                               Unknown
 myregistry:5000/keycloak:latest                                                                              Unknown
 myregistry:5000/mythtv:mythbackend                                                                           Unknown
 myregistry:5000/mythtv:mythdatabase                                                                          Unknown
 stonith404/pingvin-share:v0.29.0                                                                             Unknown
INFO  ✨ Checked 24 images in 1031125ms

Token

eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsIng1YyI6WyJNSUlFRmpDQ0F2NmdBd0lCQWdJVUlvQW42a0k2MUs3bTAwNVhqcXVpKzRDTzVUb3dEUVlKS29aSWh2Y05BUUVMQlFBd2dZWXhDekFKQmdOVkJBWVRBbFZUTVJNd0VRWURWUVFJRXdwRFlXeHBabTl5Ym1saE1SSXdFQVlEVlFRSEV3bFFZV3h2SUVGc2RHOHhGVEFUQmdOVkJBb1RERVJ2WTJ0bGNpd2dTVzVqTGpFVU1CSUdBMVVFQ3hNTFJXNW5hVzVsWlhKcGJtY3hJVEFmQmdOVkJBTVRHRVJ2WTJ0bGNpd2dTVzVqTGlCRmJtY2dVbTl2ZENCRFFUQWVGdzB5TkRBNU1qUXlNalUxTURCYUZ3MHlOVEE1TWpReU1qVTFNREJhTUlHRk1Rc3dDUVlEVlFRR0V3SlZVekVUTUJFR0ExVUVDQk1LUTJGc2FXWnZjbTVwWVRFU01CQUdBMVVFQnhNSlVHRnNieUJCYkhSdk1SVXdFd1lEVlFRS0V3eEViMk5yWlhJc0lFbHVZeTR4RkRBU0JnTlZCQXNUQzBWdVoybHVaV1Z5YVc1bk1TQXdIZ1lEVlFRREV4ZEViMk5yWlhJc0lFbHVZeTRnUlc1bklFcFhWQ0JEUVRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTGRWRDVxNlJudkdETUxPVysrR1MxWENwR2FRRHd0V3FIS2tLYlM5cVlJMXdCallKWEJ6U2MweTBJK0swU0lVd2pqNGJJT3ZpNXNyOGhJajdReGhrY1ppTlU1OEE5NW5BeGVFS3lMaU9QU0tZK3Y5VnZadmNNT2NwVW1xZ1BxWkhoeTVuMW8xbGxmek92dTd5SDc4a1FyT0lTMTZ3RFVVZm8yRkxPaERDaElsbCtYa2VlbFB6c0tiRWo3ZGJqdXV6RGxIODlWaE4yenNWNFV3c244UVpGVTB4V00wb3R2d0lQN3k0UDZGWDBuUFJuTVQyMlRIajVIWVJ3SUFVdm1FN0l3YlZVQ2wvM1hPaGhwbGNJZFQxREZGOUJUMHJOUC93ZTBWMklId1RHdVdTVENWb3M2b3R5ekk3a1hEdGZzeWRjU2Q5TklpSXZITHFYamJPVGtidWVjQ0F3RUFBYU43TUhrd0RnWURWUjBQQVFIL0JBUURBZ0dtTUJNR0ExVWRKUVFNTUFvR0NDc0dBUVVGQndNQk1CSUdBMVVkRXdFQi93UUlNQVlCQWY4Q0FRQXdIUVlEVlIwT0JCWUVGSmZWdXV4Vko3UXh1amlMNExZajFjQjEzbWhjTUI4R0ExVWRJd1FZTUJhQUZDNjBVUE5lQmtvZ1kyMnRYUGNCTUhGdkczQ3NNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUFiQkdlZHZVVzhOVWp1VXJWbDlrWmMybDRDbjhJbDFzeVBVTDNYVXdQSHprcy9iUFJ4S1loeFlIODdOb1NwdDZJT3ZPS0k3ZCthQmoyM1lldTdDWGltTWxMUWl4UGhwQ0J0dC92Vmx1UXNJbVZXZXBJWCtraENienNGemtNbUNpbHo1OXVxOURiaGg3VUw1NjQxUjdFQ2pZc0h0Y2RKeURXRWFqMXFEVFoyOUUwY1UvWmhmbmsrVFVOTExkNjYxNldCREQ3TDlSNkgzK3VkRXBRRDFlcXYzU1YwczY3R2ZVT3l0RXhzMVRja3U4aUJCdnJLbnhZa3BZMVZDbW5UMUxSaFo4K283YU94YjR4ZHByMFR6ZnBqN3BidEhWQnQwSGNUUlpIdG54MkhCN3pzWXRFZUl3eVE3bGhhMVJ4eDJNQmR0R2tBREFLUk9RRnpmMEJubm91TSJdfQ.eyJhY2Nlc3MiOlt7ImFjdGlvbnMiOlsicHVsbCJdLCJuYW1lIjoiY2xvdWRmbGFyZS9jbG91ZGZsYXJlZCIsInBhcmFtZXRlcnMiOnsicHVsbF9saW1pdCI6IjEwMCIsInB1bGxfbGltaXRfaW50ZXJ2YWwiOiIyMTYwMCJ9LCJ0eXBlIjoicmVwb3NpdG9yeSJ9LHsiYWN0aW9ucyI6WyJwdWxsIl0sIm5hbWUiOiJjcmF6eW1heC9kaXVuIiwicGFyYW1ldGVycyI6eyJwdWxsX2xpbWl0IjoiMTAwIiwicHVsbF9saW1pdF9pbnRlcnZhbCI6IjIxNjAwIn0sInR5cGUiOiJyZXBvc2l0b3J5In0seyJhY3Rpb25zIjpbInB1bGwiXSwibmFtZSI6ImRlbHVhbi9uYXZpZHJvbWUiLCJwYXJhbWV0ZXJzIjp7InB1bGxfbGltaXQiOiIxMDAiLCJwdWxsX2xpbWl0X2ludGVydmFsIjoiMjE2MDAifSwidHlwZSI6InJlcG9zaXRvcnkifSx7ImFjdGlvbnMiOlsicHVsbCJdLCJuYW1lIjoiZnJvb29kbGUvcy1wZGYiLCJwYXJhbWV0ZXJzIjp7InB1bGxfbGltaXQiOiIxMDAiLCJwdWxsX2xpbWl0X2ludGVydmFsIjoiMjE2MDAifSwidHlwZSI6InJlcG9zaXRvcnkifSx7ImFjdGlvbnMiOlsicHVsbCJdLCJuYW1lIjoiZ2l0ZWEvZ2l0ZWEiLCJwYXJhbWV0ZXJzIjp7InB1bGxfbGltaXQiOiIxMDAiLCJwdWxsX2xpbWl0X2ludGVydmFsIjoiMjE2MDAifSwidHlwZSI6InJlcG9zaXRvcnkifSx7ImFjdGlvbnMiOlsicHVsbCJdLCJuYW1lIjoiam94aXQvZG9ja2VyLXJlZ2lzdHJ5LXVpIiwicGFyYW1ldGVycyI6eyJwdWxsX2xpbWl0IjoiMTAwIiwicHVsbF9saW1pdF9pbnRlcnZhbCI6IjIxNjAwIn0sInR5cGUiOiJyZXBvc2l0b3J5In0seyJhY3Rpb25zIjpbInB1bGwiXSwibmFtZSI6ImxpYnJhcnkvbmdpbngiLCJwYXJhbWV0ZXJzIjp7InB1bGxfbGltaXQiOiIxMDAiLCJwdWxsX2xpbWl0X2ludGVydmFsIjoiMjE2MDAifSwidHlwZSI6InJlcG9zaXRvcnkifSx7ImFjdGlvbnMiOlsicHVsbCJdLCJuYW1lIjoibGlicmFyeS9wb3N0Z3JlcyIsInBhcmFtZXRlcnMiOnsicHVsbF9saW1pdCI6IjEwMCIsInB1bGxfbGltaXRfaW50ZXJ2YWwiOiIyMTYwMCJ9LCJ0eXBlIjoicmVwb3NpdG9yeSJ9LHsiYWN0aW9ucyI6WyJwdWxsIl0sIm5hbWUiOiJsaWJyYXJ5L3JlZ2lzdHJ5IiwicGFyYW1ldGVycyI6eyJwdWxsX2xpbWl0IjoiMTAwIiwicHVsbF9saW1pdF9pbnRlcnZhbCI6IjIxNjAwIn0sInR5cGUiOiJyZXBvc2l0b3J5In0seyJhY3Rpb25zIjpbInB1bGwiXSwibmFtZSI6ImxvdWlzbGFtL2RvY2tnZSIsInBhcmFtZXRlcnMiOnsicHVsbF9saW1pdCI6IjEwMCIsInB1bGxfbGltaXRfaW50ZXJ2YWwiOiIyMTYwMCJ9LCJ0eXBlIjoicmVwb3NpdG9yeSJ9LHsiYWN0aW9ucyI6WyJwdWxsIl0sIm5hbWUiOiJsb3Vpc2xhbS91cHRpbWUta3VtYSIsInBhcmFtZXRlcnMiOnsicHVsbF9saW1pdCI6IjEwMCIsInB1bGxfbGltaXRfaW50ZXJ2YWwiOiIyMTYwMCJ9LCJ0eXBlIjoicmVwb3NpdG9yeSJ9LHsiYWN0aW9ucyI6WyJwdWxsIl0sIm5hbWUiOiJwb3J0YWluZXIvcG9ydGFpbmVyLWNlIiwicGFyYW1ldGVycyI6eyJwdWxsX2xpbWl0IjoiMTAwIiwicHVsbF9saW1pdF9pbnRlcnZhbCI6IjIxNjAwIn0sInR5cGUiOiJyZXBvc2l0b3J5In0seyJhY3Rpb25zIjpbInB1bGwiXSwibmFtZSI6InN0b25pdGg0MDQvcGluZ3Zpbi1zaGFyZSIsInBhcmFtZXRlcnMiOnsicHVsbF9saW1pdCI6IjEwMCIsInB1bGxfbGltaXRfaW50ZXJ2YWwiOiIyMTYwMCJ9LCJ0eXBlIjoicmVwb3NpdG9yeSJ9XSwiYXVkIjoicmVnaXN0cnkuZG9ja2VyLmlvIiwiZXhwIjoxNzI5MjAzMjMyLCJpYXQiOjE3MjkyMDI5MzIsImlzcyI6ImF1dGguZG9ja2VyLmlvIiwianRpIjoiZGNrcl9qdGlfdUJ1ZnlvSW8wMy1FWHkzTFZBd0d4YVNURHBFPSIsIm5iZiI6MTcyOTIwMjYzMiwic3ViIjoiIn0.W8lDRMYzvAJ_6UAKOcmcfB3UAI-fimELtkwmnvWH02TGQ4hs7cmzTWNYSs3On5jwRAFJmBMCJ3Wxz7pBsaFe_6dUtUTWclCIPfh7x0Du4lof1hX-iCHAJV8BPQIn9i_nhtfrZ1X3UlSdJfHfJPW6MgSs7PBG7nZjv6W3YaO8SmwOs8eCld90DIXvKnJGxNxwcfII109zpbbDqO0rT-rrAhJEB1O9FNM535KewmvNbPh4KYiyrobaaTwn8ehw-aA-WMMk3iLyT3UpCtZgIOMHY6ByPEUIax-hDwvFTdGbxBMt2sqqVphI65YsoQAtPTSwCbA8pPfeHG13BIydG0BYIg

[sergi0g]

The token had already expired because it's taking significantly longer than 5 minutes to check the images

Thanks for debugging! Maybe handling timeouts would be a good idea, but the real issue here is the time it took.

Cup should be finishing in seconds, not minutes! I'd suggest investigating both the network and the hardware (maybe it's too overloaded?).

If you're getting errors like WARN Connection to registry registry-1.docker.io failed., I'd also suggest changing DNS and checking if that makes a difference.

Finally, you can try checking with the old image (ghcr.io/sergi0g/cup:v2.3.1) and see if there's a timing difference.

[johnnyggalt]

I see, thanks. Running on my local machine is indeed very fast, though I do have different containers running.

The problem occurs when running on my Synology server. It is not under-powered or overworked. Other network traffic is super fast (including torrents, for example). I do have network traffic routed through VPN, but I just tried disabling that and the problem persists.

I tried running with v2.3.1 with no change. I also tried running without configuration too (just in case) but the same thing occurs. Running a traceroute registry-1.docker.io looks fine (though I'm no networking expert).

I then tried going through each Docker Hub image individually and running a check against only that image. All such checks completed in under 2 seconds. Here's one example:

sudo docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -t ghcr.io/sergi0g/cup check nginx:1.25.4
✓ Done!
nginx:1.25.4                                                                                                                                                                                                                                                Up to date
INFO  ✨ Checked 1 images in 1955ms

However, things got interesting when I tried the same against one of the images hosted in my own registry:

sudo docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v ${PWD}/cup.json:/config/cup.json -t ghcr.io/sergi0g/cup -c /config/cup.json check myregistry:5000/keycloak:latest
⠇ Checking...
WARN  Connection to registry myregistry:5000 failed.
⠇ Checking...
WARN  Connection to registry failed.
✓ Done!
myregistry:5000/keycloak:latest                                                                                                                                                                                                                                  Unknown
INFO  ✨ Checked 1 images in 1024777ms

I don't quite understand why such a long check time against the private registry would impact Docker Hub checks unless the auth token is obtained first, then the private registry is checked, then Docker Hub images are checked. However, it's the only thing I can find that makes some kind of sense. Unfortunately, it's not easy for me to remove those containers from my setup to try running cups without them in the picture, nor can I find a way to configure cups to straight-up ignore them.

Is there a way to run cups with verbose output, so I can more easily see what's happening? Or anything else you'd suggest to move forward?

[sergi0g]

I don't quite understand why such a long check time against the private registry would impact Docker Hub checks unless the auth token is obtained first, then the private registry is checked, then Docker Hub images are checked.

This is exactly what's happening. It's necessary for all images to be checked concurrently, but I never thought about timeouts, since all registries I've tried are very fast. If I revert the code, Cup will be slower overall, so I'd suggest investigating why your local registry is taking so long.

What I could do is implement a config option to ignore specific images. What do you think?

[johnnyggalt]

Believe it or not, this came down to the difference between checking myregistry:5000/whatever and Myregistry:5000/whatever. My actual host name is capitalized, but somewhere along the way - including in image tags - I used lowercase and cup is unable to resolve from that. Interestingly, I can ping either form, so I don't really understand where things are going awry. Do you think cup could/should be lenient here and work with either form?

What I could do is implement a config option to ignore specific images. What do you think?

I think this could well be useful. To be honest, I think the most useful thing would have been some form of verbose output though.

[sergi0g]

Hey! I've never seen an issue like this before! Cup requests the image in the exact same way Docker does, so if a delay is being caused, it's either a weird network bug or an issue in one of the libraries Cup uses. I'll have to try to reproduce this and I'll let you know as soon as I can.

[johnnyggalt]

After going to the trouble of switching my self-hosted images over to being tagged with Myregistry:5000, cup still stalled on those particular images (although I see you've fixed it so that it doesn't impact Docker Hub images, which is great). What's more, my registry UI broke, insistently redirecting from capitalized to lowercase and displaying an error message that the lowercase name must be allowed in CORS header.

And when I tried updating another part of my code stack, I got an error message that really drove it home:

'repository' cannot contain uppercase characters. (Parameter 'repository')

That caused me to do a quick google and find that, indeed, Docker requires the repository be all lowercase! So I feel a bit stuck here, though I don't think it's a cup problem. On the one hand, I am forced to tag with lowercase, but on the other my actual server name is capitalized, which causes something of a mismatch somewhere in the cup/Docker stack.

At this point it feels like my only recourse is to either rename my server, or use DNS (I guess - I have no idea where resolution is actually going awry) so that it still resolves to my server. Pretty crazy!

[parasiteoflife]

I'm getting the same messages:

:5:192mWARN  Failed to authenticate to registry registry-1.docker.io with given token!
null

Using a Read-Only token, similar token works with Portainer

[sergi0g]

@johnnyggalt I'm terribly sorry, I haven't had any time to reproduce your issue! Please don't change your server configuration or anything until I've done that.

although I see you've fixed it so that it doesn't impact Docker Hub images, which is great

I haven't made any code changes since your last comment. It just happened that the async tasks were scheduled in a particular order that worked.

Update: I just noticed that in a previous comment you posted your config file and the hostname isn't capitalized there. Could that be the issue?

Here is my cup.json:

{
    "insecure_registries": ["myregistry:5000"]
}

[sergi0g]

I'm getting the same messages:

:5:192mWARN  Failed to authenticate to registry registry-1.docker.io with given token!
null

Using a Read-Only token, similar token works with Portainer

Hi @parasiteoflife! Your issue is unrelated to this one. It seems that Cup is trying to use a token that is null which will obviously not work. Please open a new issue that also includes your config file with sensitive info redacted (if you have one) and any additional info that could help with troubleshooting. Thanks!

[johnnyggalt]

I haven't made any code changes since your last comment. It just happened that the async tasks were scheduled in a particular order that worked.

Ah, I see, luck of the draw.

I just noticed that in a previous comment you posted your config file and the hostname isn't capitalized there

Correct, it wasn't capitalized there because the image's registry name wasn't, which I had initially done without too much thought and presumably because something pushed me in that direction. In any case, it doesn't make a difference how I specify it in cup.json - I even tried including both variants:

{
    "insecure_registries": ["myregistry:5000","Myregistry:5000"]
}

I'm terribly sorry, I haven't had any time to reproduce your issue

No problem. OK, so to summarize from my perspective:

1. Docker requires the registry name be in lowercase
2. My host name is in upper case
3. The mismatch causes resolution to go wrong somewhere in the stack, resulting in an inability to check those images after a lengthy timeout
4. The only feasible workarounds I can imagine at this point are changing the host name or setting up some DNS magic, neither of which I particularly want to pursue

I will wait for your investigation. If it came down to it, being able to ignore those images (or that registry altogether) would be a decent workaround for me. Since they're self-hosted images, I'm probably not going to forget to pull and apply updates that I've just pushed to my registry.

Probably 🙂

[sergi0g]

Hi @johnnyggalt!

I've set up 2 VMs on the same network and they are discoverable through mDNS (I'm guessing that's your setup). One runs a registry to which I've pushed a test image and the other is for running Cup and trying to connect to it. They both have capitalized hostnames (Myregistry1 and Myregistry2 (don't ask why even though I have 1 registry)).

I've configured the local registry as insecure for both Docker and Cup and I've pulled the image. When running Cup, I get an error that the connection failed (which shouldn't happen), but the time taken is completely normal. The connection times out in less than 10 seconds.

My next steps from here are debugging why the connection fails. I can't understand why there is such a huge delay on your end. I wish I had access to your system, but that can't happen. If this was my server, I would have installed Rust in a VM, created a development build of Cup and then tried to reproduce the issue. If it did occur, I'd run with samply and then analyze the output to see where the delay is caused. Of course, I completely understand that you may want to keep your server installation minimal and that you may not have the technical expertise to do this. I will try to debug this on my own, but your help would be valuable.

[sergi0g]

I finally found why the local domain isn't resolving. Musl libc (with which Cup is compiled) doesn't support resolving local domains with mDNS and needs a sort of bridge to be running. Even if we compile with glibc, an mDNS service still needs to be running in the container for local domain resolution to work.

To be honest, I don't really think implementing local domain resolution in a container is a good investment of time. It will probably increase the container size and complexity significantly.

My conclusion is that whether your hostname is capitalized or not plays no role in whether Cup will be able to connect to it. If you're running a local DNS server, I'd suggest pointing an A record at your registry's IP and using that domain name. I think the best solution is to just filter out domain names that rely on mDNS.

[johnnyggalt]

@sergi0g Thanks for looking into it. Would it be possible to add a config option to ignore certain registries?

[sergi0g]

Sure! I'll try to add this in version 3 (hopefully coming soon).