Bug: cycle to next public ip API if 403 is received

[samip5]

Is this urgent?

No

Host OS

Debian Bookworm

CPU arch

x86_64

VPN service provider

Privado

What are you using to run the container

Kubernetes

What is the version of Gluetun

Running version latest built on 2024-10-06T09:52:17.759Z (commit 99e9bc8)

What's the problem 🤔

2024-10-06T10:45:03Z ERROR [vpn] getting public IP address information: fetching information: too many requests sent for this month from https://ipinfo.io/: 403 403 Forbidden

Share your logs (at least 10 lines)

2024-10-06T10:45:01Z INFO [vpn] starting
2024-10-06T10:45:01Z INFO [firewall] allowing VPN connection...
2024-10-06T10:45:01Z DEBUG [firewall] /sbin/iptables -t filter -L OUTPUT --line-numbers -n -v
2024-10-06T10:45:01Z DEBUG [firewall] found iptables chain rule matching "--delete OUTPUT -d 91.148.241.53 -o eth0 -p udp -m udp --dport 1194 -j ACCEPT" at line number 7
2024-10-06T10:45:01Z DEBUG [firewall] /sbin/iptables -t filter -D OUTPUT 7
2024-10-06T10:45:01Z DEBUG [firewall] /sbin/iptables -t filter -L OUTPUT --line-numbers -n -v
2024-10-06T10:45:01Z DEBUG [firewall] found iptables chain rule matching "--delete OUTPUT -d 91.148.241.53 -o eth0 -p udp -m udp --dport 1194 -j ACCEPT" at line number 7
2024-10-06T10:45:01Z DEBUG [firewall] /sbin/iptables -t filter -D OUTPUT 7
2024-10-06T10:45:01Z DEBUG [firewall] /sbin/iptables -t filter -L OUTPUT --line-numbers -n -v
2024-10-06T10:45:01Z DEBUG [firewall] found iptables chain rule matching "--delete OUTPUT -o tun0 -j ACCEPT" at line number 7
2024-10-06T10:45:01Z DEBUG [firewall] /sbin/iptables -t filter -D OUTPUT 7
2024-10-06T10:45:01Z DEBUG [firewall] /sbin/ip6tables -t filter -L OUTPUT --line-numbers -n -v
2024-10-06T10:45:01Z DEBUG [firewall] found iptables chain rule matching "--delete OUTPUT -o tun0 -j ACCEPT" at line number 8
2024-10-06T10:45:01Z DEBUG [firewall] /sbin/ip6tables -t filter -D OUTPUT 8
2024-10-06T10:45:01Z DEBUG [firewall] /sbin/iptables --append OUTPUT -d 91.148.247.216 -o eth0 -p udp -m udp --dport 1194 -j ACCEPT
2024-10-06T10:45:01Z DEBUG [firewall] /sbin/iptables --append OUTPUT -d 91.148.247.216 -o eth0 -p udp -m udp --dport 1194 -j ACCEPT
2024-10-06T10:45:01Z DEBUG [firewall] /sbin/iptables --append OUTPUT -o tun0 -j ACCEPT
2024-10-06T10:45:01Z DEBUG [firewall] /sbin/ip6tables --append OUTPUT -o tun0 -j ACCEPT
2024-10-06T10:45:01Z INFO [openvpn] OpenVPN 2.6.11 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD]
2024-10-06T10:45:01Z INFO [openvpn] library versions: OpenSSL 3.3.2 3 Sep 2024, LZO 2.10
2024-10-06T10:45:01Z INFO [openvpn] TCP/UDP: Preserving recently used remote address: [AF_INET]91.148.247.216:1194
2024-10-06T10:45:01Z INFO [openvpn] UDPv4 link local: (not bound)
2024-10-06T10:45:01Z INFO [openvpn] UDPv4 link remote: [AF_INET]91.148.247.216:1194
2024-10-06T10:45:02Z INFO [openvpn] [mad-006.vpn.privado.io] Peer Connection Initiated with [AF_INET]91.148.247.216:1194
2024-10-06T10:45:03Z INFO [openvpn] TUN/TAP device tun0 opened
2024-10-06T10:45:03Z INFO [openvpn] /sbin/ip link set dev tun0 up mtu 1500
2024-10-06T10:45:03Z INFO [openvpn] /sbin/ip link set dev tun0 up
2024-10-06T10:45:03Z INFO [openvpn] /sbin/ip addr add dev tun0 172.21.24.47/23
2024-10-06T10:45:03Z INFO [openvpn] UID set to nonrootuser
2024-10-06T10:45:03Z INFO [openvpn] Initialization Sequence Completed
2024-10-06T10:45:03Z INFO [healthcheck] healthy!
2024-10-06T10:45:03Z ERROR [vpn] getting public IP address information: fetching information: too many requests sent for this month from https://ipinfo.io/: 403 403 Forbidden

Share your configuration

env:
              VPN_SERVICE_PROVIDER: "privado"
              SERVER_COUNTRIES: "Netherlands,Switzerland"
              FIREWALL_INPUT_PORTS: *port
              FIREWALL_DEBUG: "on"
              FIREWALL_OUTBOUND_SUBNETS: "10.40.0.0/16,10.41.0.0/16,192.168.10.0/24,fd94:9bde:1ebb::/48" # Allow access to k8s subnets
              LOG_LEVEL: "debug"
              DOT: "off"

[github-actions]

@qdm12 is more or less the only maintainer of this project and works on it in his free time.
Please:

• do not ask for updates, be patient
• 👍 the issue to show your support instead of commenting
  @qdm12 usually checks issues at least once a week, if this is a new urgent bug,
  revert to an older tagged container image

[qdm12]

I think cycling to the next API (i.e. ip2location) would do, in case we receive a 403 / too many requests sent, right? Thanks!

[samip5]

I think cycling to the next API (i.e. ip2location) would do, in case we receive a 403 / too many requests sent, right? Thanks!

Most services like these are very simple to use and deploy, like ifconfig.co, and I would prefer if the fix for this would be two fold.

1. Cycle the service used when 403 is received
2. let us change the endpoint on runtime, to eg ifconfig.co or own self-hosted version of that service as it's open source.

[qdm12]

1. Public IP information is only fetched on VPN connection from commit 03deb9a I could not find anywhere why we had a public ip period to fetch the data, this doesn't make much sense really imo, so it got removed.
2. With cycling the service, you'll be able to specify multiple APIs in PUBLICIP_APIS (previously PUBLICIP_API). APIs specified after the first one (if any) are cycled through if the first one fails with a "too many requests sent" (429 or 403 status code for ipinfo.io) error. This is still WIP I'm programming right now, so feel free to criticize and propose a different approach!
3. let us change the endpoint on runtime: I don't think there is much point in this, as long as cycling to the next service acts when a too-many-requests error is encountered?
4. ifconfig.co: added support for it in 5b2923c
5. own self-hosted version of that service as it's open source: what do you mean? Can you point out how you would self host that? Using the maxmind database? 🤔

[samip5]

1. Public IP information is only fetched on VPN connection from commit 03deb9a I could not find anywhere why we had a public ip period to fetch the data, this doesn't make much sense really imo, so it got removed.

   2. With cycling the service, you'll be able to specify multiple APIs in PUBLICIP_APIS (previously PUBLICIP_API). APIs specified after the first one (if any) are cycled through if the first one fails with a "too many requests sent" (429 or 403 status code for ipinfo.io) error. This is still WIP I'm programming right now, so feel free to criticize and propose a different approach!

   3. let us change the endpoint on runtime: I don't think there is much point in this, as long as cycling to the next service acts when a too-many-requests error is encountered?

   4. ifconfig.co: added support for it in 5b2923c

   5. own self-hosted version of that service as it's open source: what do you mean? Can you point out how you would self host that? Using the maxmind database? 🤔

3. It would make sense if you're self-hosting the one ifconfig.co uses, and wish to use that instead of 3rd party one.
4. That's very welcomed. :)
5. The source code is available at: https://github.com/mpolden/echoip and I have an container for it: https://github.com/skyssolutions/containers/pkgs/container/echoip (source: https://github.com/skyssolutions/containers/tree/main/apps/echoip) It does use maxmind.

[qdm12]

3. and 5.

Oh ok, but not at runtime, you mean as an environment variable i.e. PUBLICIP_APIS=echoip//https://someip:someport for example right? The echoip// prefix would indicate which API format to use for the custom address. I also thought to bundle the maxmind database within Gluetun, but the database is rather too big I would say, given the obsession to have a small-ish Docker image.

[samip5]

3. and 5.

Oh ok, but not at runtime, you mean as an environment variable i.e. PUBLICIP_APIS=echoip//https://someip:someport for example right? The echoip// prefix would indicate which API format to use for the custom address. I also thought to bundle the maxmind database within Gluetun, but the database is rather too big I would say, given the obsession to have a small-ish Docker image.

Yes, on runtime I meant as I can tell it to use my own API for it instead. I think the prefix should like echoip:// so it would look more like a url.

[qdm12]

See #2518 can you try image tag :pr-2518 see how it does? This is for falling back on backup IP data services if we get banned, and to keep track of bans. Not the custom echo service yet, I'll do that in another PR after.

[qdm12]

FYI I fixed the build just now, sorry about that, so image tag :pr-2518 should appear soon.

[samip5]

Hmm, it does seem to work at least?

[qdm12]

Hmm, it does seem to work at least?

Indeed, you probably cannot try it unless you get rate limited 😄 I'll merge it as-is then.

Now, back to that custom echoip: I'm thinking having two // makes it an invalid URL, perhaps a better prefix would be echoip# followed by the URL? I'm opened to any other suggestion for a prefix too.

[qdm12]

Please see #2529 and try image tag :pr-2529 using PUBLICIP_API=echoip#http://yourhost to see if it works fine.
I basically just moved over ifconfig.co implementation to an echoip implementation with a customizable url.
Let me know if it works or not, and then I'll merge it. Thanks!