Networks created by docker-compose conflict with private ips used @dcs

[lloyd-uot-cs]

We configured the docker daemon to have the docker0 interface use 192.168.152.0
instead of the default 172.17.0.0 (which collides with red.sandbox on dcs machines).

This works when just the docker daemon is running:

tapp@doc:~/tapp$ netstat -ar
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
default router.pub.sand 0.0.0.0 UG 0 0 0 ens160
localnet * 255.255.255.0 U 0 0 0 ens160
192.168.152.0 * 255.255.255.0 U 0 0 0 docker0

Once we launch the tapp app, docker-compose spins up two more
network interfaces, (tapp_internal and tapp_external) and they are
allocated 172.17.0.0 and 172.18.0.0, which again collide:

tapp@doc:~/tapp$ docker-compose up -d
Creating network "tapp_internal" with driver "bridge"
Creating network "tapp_external" with driver "bridge"
Creating tapp_postgres_1
Creating tapp_rails-app

tapp@doc:~/tapp$ netstat -ar
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
default router.pub.sand 0.0.0.0 UG 0 0 0 ens160
172.17.0.0 * 255.255.0.0 U 0 0 0 br-c489d67e84d7
172.18.0.0 * 255.255.0.0 U 0 0 0 br-3824b523e9aa
localnet * 255.255.255.0 U 0 0 0 ens160
192.168.152.0 * 255.255.255.0 U 0 0 0 docker0

Unlike docker0, this isn't a docker install issue, it is specific to tapp because
if I bring up a generic docker container that serves web pages it doesn't
add additional interfaces and I can access the host from a red net machine.

pocadmin@doc:~/mytest$ docker run -it --rm --name my-apache-app -p80:80 -v "$(pwd)":/usr/local/apache2/htdocs/ httpd:2.4

tapp@doc:~/tapp$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ae8beb922348 httpd:2.4 "httpd-foreground" 3 seconds ago Up 3 seconds 0.0.0.0:80->80/tcp my-apache-app
tapp@doc:~/tapp$ netstat -ar
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
default router.pub.sand 0.0.0.0 UG 0 0 0 ens160
localnet * 255.255.255.0 U 0 0 0 ens160
192.168.152.0 * 255.255.255.0 U 0 0 0 docker0

I assume this explains something about a solution
https://docs.docker.com/compose/compose-file/#external-1

external

If set to true, specifies that this network has been created outside of Compose.
docker-compose up will not attempt to create it, and will raise an error if it doesn’t exist.

lloyd

[jmzaleski]

I have an idea (Peter Marbach made me think this).
Clue: we are trying to route non-routable IPs
Hence: don't

Instead, put the reverse proxy onto a separate VM -- hence it's routing won't be polluted by all of dockers fancy bridge network adapters.

Like so:

[lloyd-uot-cs]

Fixed on doc.pub.sandbox with:

# docker network create --driver=bridge --subnet=192.168.153.0/24 --ip-range=192.168.153.0/24 --gateway=192.168.153.254 frontend
# docker network create --driver=bridge --subnet=192.168.154.0/24 --ip-range=192.168.154.0/24 --gateway=192.168.154.254 backend

Changed ~tapp/tapp/docker-compose.yml to have :

networks:
# Names of the networks will be prefixed with project name by docker
backend:
external: true

frontend:
external: true

Rebuilt app, populated database from backup, now can reach
http://doc.pub.sandbox:3000/index.html/summary from a red subnet machine.

I have a /etc/daemon/daemon.json file that I think does the docker network create
at boot time... still testing.

[lloyd-uot-cs]

I don't have the daemon.json file syntax right, but will fiddle with the network(s) to remove their ability to route for