Enormous traffic in case of update errors.

[senia-psm]

Watchtower generates enormous traffic (~1TB/day in my case) in case of some issues with update (no free space in my case).
I'm not sure if it's a bug or a known behaviour.
One possible solution would be to limit a number of retries of failed update or to add some kind if increasing timeout between retries.
Examples from documentation (without --cleanup) can lead to this issue.
Should I create an issue for this?

[piksel]

What interval are you using? Could you provide some logs of what is happening?
There have been discussions earlier about having a type of "back off" system, but no PR to implement it has been done for it since.

[senia-psm]

It was --interval 30. I have no logs - I'm not the owner of registry. All I have is some traffic details from billing information. All the traffic is on docker registry, there is only 1 image in that registry and the only thing that has access to that registry is watchtower.

[simskij]

Uhm, yeah, we should probably look into if anything can be done to improve this.

The issue is, we don't really know what we're running on, so it might be hard to poll disk space for instance. And as in most cases, errors are intermittent, most users likely want watchtower to try again after a while. But you're more than welcome to raise a ticket! We should at least investigate if there are any viable ways forward here.

[piksel]

Yeah, if it's set to check every 30 seconds if there is an update and try to perform said update it would certainly cause a huge amount of bandwidth used. That is, if the error is that docker fails to download the image, otherwise they would just be pulled once, so this is kind of specific.
The back of, by remembering the hashes of images that failed to be updated to would of course not keep trying to update the same image, but if there are frequent updates it would still try any new images. I assume that is why the interval is set that high (or low, depending on perspective 😁)?

[simskij]

I think the preferred way to solve this is by monitoring disk space and making sure the host never runs out. With that said, my stance above on wanting to explore this a bit further stands. 😅