How to limit GeoIP database refresh frequency?

[ajmas]

Shlink version

4.1.0

PHP version

0.0.0

How do you serve Shlink

Self-hosted nginx

Database engine

PostgreSQL

Database version

16

Summary

Since the past few days we are getting "Daily GeoIP Database Download Limit Reached" emails from MaxMind.

Looking at the shlink logs does show that an attempt is being made to fetch an updated database and it is failing for this reason.

I did see a closed issue for something similar, from the start of the year, but we are using a more recent version and this.

Is there any way to throttle or pause the GeoIP data refresh?

BTW a partial log output:

2024-11-28T19:21:06+0000	INFO	server      	[2024-11-28T19:21:06.290371+00:00] [4c8dbf76-afbb-46e4-b38b-3db16fb2e502] Shlink.ERROR - GeoLite2 database download failed.
GuzzleHttp\Exception\ClientException: Client error: `GET https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-City&license_key=xxxxxxxxx&suffix=tar.gz` resulted in a `429 Too Many Requests` response:
Daily GeoIP database download limit reached
 in /etc/shlink/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113
Stack trace:
#0 /etc/shlink/vendor/guzzlehttp/guzzle/src/Middleware.php(72): GuzzleHttp\Exception\RequestException::create(Object(GuzzleHttp\Psr7\Request), Object(GuzzleHttp\Psr7\Response), NULL, Array, NULL)
#1 /etc/shlink/vendor/guzzlehttp/promises/src/Promise.php(209): GuzzleHttp\Middleware::GuzzleHttp\{closure}(Object(GuzzleHttp\Psr7\Response))
#2 /etc/shlink/vendor/guzzlehttp/promises/src/Promise.php(158): GuzzleHttp\Promise\Promise::callHandler(1, Object(GuzzleHttp\Psr7\Response), NULL)
#3 /etc/shlink/vendor/guzzlehttp/promises/src/TaskQueue.php(52): GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}()
#4 /etc/shlink/vendor/guzzlehttp/promises/src/Promise.php(251): GuzzleHttp\Promise\TaskQueue->run(true)
#5 /etc/shlink/vendor/guzzlehttp/promises/src/Promise.php(227): GuzzleHttp\Promise\Promise->invokeWaitFn()
#6 /etc/shlink/vendor/guzzlehttp/promises/src/Promise.php(272): GuzzleHttp\Promise\Promise->waitIfPending()
#7 /etc/shlink/vendor/guzzlehttp/promises/src/Promise.php(229): GuzzleHttp\Promise\Promise->invokeWaitList()
#8 /etc/shlink/vendor/guzzlehttp/promises/src/Promise.php(69): GuzzleHttp\Promise\Promise->waitIfPending()
#9 /etc/shlink/vendor/guzzlehttp/guzzle/src/Client.php(189): GuzzleHttp\Promise\Promise->wait()
#10 /etc/shlink/vendor/shlinkio/shlink-ip-geolocation/src/GeoLite2/DbUpdater.php(57): GuzzleHttp\Client->request('GET', 'https://downloa...', Array)
#11 /etc/shlink/vendor/shlinkio/shlink-ip-geolocation/src/GeoLite2/DbUpdater.php(45): Shlinkio\Shlink\IpGeolocation\GeoLite2\DbUpdater->downloadDbFile('/etc/shlink/con...', Object(Closure))
#12 /etc/shlink/module/CLI/src/GeoLite/GeolocationDbUpdater.php(116): Shlinkio\Shlink\IpGeolocation\GeoLite2\DbUpdater->downloadFreshCopy(Object(Closure))
#13 /etc/shlink/module/CLI/src/GeoLite/GeolocationDbUpdater.php(70): Shlinkio\Shlink\CLI\GeoLite\GeolocationDbUpdater->downloadNewDb(true, Object(Closure), Object(Closure))
#14 /etc/shlink/module/CLI/src/GeoLite/GeolocationDbUpdater.php(53): Shlinkio\Shlink\CLI\GeoLite\GeolocationDbUpdater->downloadIfNeeded(Object(Closure), Object(Closure))
#15 /etc/shlink/module/Core/src/EventDispatcher/UpdateGeoLiteDb.php(41): Shlinkio\Shlink\CLI\GeoLite\GeolocationDbUpdater->checkDbUpdate(Object(Closure), Object(Closure))
#16 /etc/shlink/vendor/shlinkio/shlink-event-dispatcher/src/RoadRunner/RoadRunnerTaskConsumerToListener.php(52): Shlinkio\Shlink\Core\EventDispatcher\UpdateGeoLiteDb->__invoke(Object(Shlinkio\Shlink\Core\EventDispatcher\Event\VisitLocated))
#17 /etc/shlink/bin/roadrunner-worker.php(32): Shlinkio\Shlink\EventDispatcher\RoadRunner\RoadRunnerTaskConsumerToListener->listenForTasks(Object(Closure))
#18 /etc/shlink/bin/roadrunner-worker.php(36): {closure}()
#19 {main}

No evidence of pod restart, since 27 August 2024.

And... We are using the 'shlinkio/shlink:stable' docker image, in a Kubernetes cluster, and a hosted database at Digital Ocean.

[acelaya]

It is a bit unfortunate, but there's sadly something in Shlink that makes this happen from time to time, and no one has been able to reproduce it consistently or find the root cause.

The issue you mention there is an old bug that was fixed, but the same happened later and was reported here #2114

With no clear cause or way to reproduce it, my plan is to change how Shlink downloads the GeoLite database, in an attempt to reduce the chances for this to happen. You can see more details here #2124

At the moment, the only thing I can suggest is creating a new license key and trying to restart the service. Hopefully the containers will download new GeoLite instances and this issue will be silenced.

[acelaya]

Something else you could try is restarting the service without providing the license key. Shlink will not try to download the GeoLite DB file in that case.

Leave it for 24h (to clear the download limit) and then restart it with the key set again.

The only side effect is that all visits happening during that period will appear with unknown location, but you can then run shlink visit:locate --retry to get them located once you have a GeoLite db again.

[ajmas]

I wonder whether adding some logging code to indicate the triggering condition could help?

[acelaya]

FYI, Shlink 4.4.0 introduced a new system to determine when to update the geolite db file, which should avoid, or at least reduce a lot, API limit errors. I would recommend trying this version.

You can find more details here #2124

[ajmas]

The restart seemed to have resolved the issue, but we will look into the upgrade.