[ECS] [request] User Defined Networks

[ktersius]

I'm trying to see if user defined networks is supported?

I've looked at the task definition options and could not find any place to set the network the container should connect to?

Is it supported yet?

[samuelkarp]

User-defined networks are not yet supported on the task definition. Can you help us understand what you'd intend to use them for? Are you looking for something like service discovery, security isolation, or something else?

[ktersius]

Mostly in interested in the automated service discovery part where I can setup predefined domain names for containers and connect my services via them.

Unfortunately this only works using user defined networks.

Currently I'm setting up a host DNS server which then scans the running containers and updates the DNS entries manually which is not ideal.

[stefansedich]

I am running into wanting this too for the service discovery aspect, I see it supports container links but I was of the understand that it was deprecated now in favor of using networks, is this something that will be implemented soon?

[mkleint]

my usecase is basically a bidirectional linking (see http://stackoverflow.com/questions/25324860/how-to-create-a-bidirectional-link-between-containers)

1. start selenium container
2. run an application build in another container and run the webdriver tests against the 'remote selenium' from the other container. --link doesn't seem to cut it as I not only need visibility from 2. to 1. but also from 1. to 2. (as the browser run in the selenium container needs to visit the app running in the other container.

[bheusinkveld]

my use case is I would like to be able to scale containers in services separately, but have communication to other containers in a different service/task definition. If multiple different task definitions were able to connect to a user defined network, all containers across different task definitions would have network connectivity on that user defined network by hostnames.

[voltechs]

Would really love to see this. Currently Service Discovery is a huge pain requiring yet another service (which itself is usually cluster-based and self-discovers and then listens for other services). It's a messy solution, not to mention the Lambda "solutions" that are even more obnoxious to implement and maintain.

ECS needs native service discovery support out of the box. Specific example would be clustering services such as RabbitMQ or similar services.

[CpuID]

+1 to seeing this in place.

At a minimum passing through the equivalent of the --network docker run arg would be useful I think, defined in the container definition most likely.

[ghost]

I believe this needs to be looked into with a higher priority. The legacy links feature is currently deprecated and may be removed. This warning is in place on the documentation for the feature.

https://docs.docker.com/engine/userguide/networking/default_network/dockerlinks/

[techcto]

+1 - Really need this feature to create mysql replicas without putting on same host/task.

[eedwardsdisco]

+1 Linking is going away, and there are many services which require knowing their 'externally reachable host address (host ip + external port)' at runtime, which theoretically could be solved with user-defined networks

[lareeth]

+1 I would very much like to be able to define my own network instead of being forced to use either 'Host', 'Bridge' or 'None'. The agent doesn't even need to create the network, just allow me to put in a network name that's custom and then at runtime see if it fails to start because the network doesn't exist.

[thovden]

I need to route traffic through a container that is running a VPN client. That way the actual containers can be used without modification when they need to use a VPN. Similar to the --net=container:network option that has removed from Docker.

[prog893]

+1 In docker links are indeed already deprecated

[alrooney]

👍 need this for elasticsearch nodes

[dzalbo]

👍 need this for hazelcast

[dejonghe]

👍 Would be useful for ZooKeeper.

[andreloc]

👍 Consul ... service discovery

[bploetz]

👍 Use case for us is an nginx reverse proxy container which sits in front of an upstream API service running in another container on the same host. Currently our only option is using the deprecated link feature over the bridge network, or using something like DNS/ELB/Consul. But obviously we'd like to avoid making a network hop to call something that's running on the same host.

[adamrbennett]

A major disappointment I have with most (all?) orchestration tools is the assumption that all containers will be mapped to ports on the host. With overlay networks, this is not necessary. Containers can communicate within the network on ports that are not exposed or mapped to the host. This is clearly preferable as it almost completely eliminates any sort of port management and the possibility for port conflicts.

Start your containers in an overlay network, listen on standard ports (i.e.: 80/443) without worrying about conflicts, and setup a proxy to forward requests to your containers by name. Map your proxy to host port 80/443 and point your ELB at it. Manage it all using your service discovery DNS. This is the most elegant and maintainable solution, yet most orchestration tools will not support it. It's a crying shame. Literally, I am crying over it.

I shudder to think about managing 10,000 containers with port mapping. If each container exposes two ports, that's 20,000 ports I have to manage! Oh, I can make them map to random host ports, but now my proxy logic is so much more complicated, and someday I'll simply run out of ports. The bottom line is that a "scalable" solution that's built on port mapping is not scalable -- because mapping ports is not scalable.

I have modified the ECS agent to support this, and it works perfectly for my needs. However, it's less than ideal, because I lose the regular updates to the agent, unless I continually merge them in, and I have little to no visibility or control into the networks from the console or the CLI.

Guys, let's ditch the port mapping nonsense. It's not necessary with overlay networks.

[elasticsearcher]

@samuelkarp Is this currently in the works?

For anyone trying to do service discovery, take a look at the following article:
https://aws.amazon.com/blogs/compute/microservice-delivery-with-amazon-ecs-and-application-load-balancers/

From what I understand, you can use a single application load balancer to load balance up to 75 services by assigning a unique path prefix for each service, which you can then use to address your services. This doesn't cover all use cases, but should be enough for many applications.

[samuelkarp]

@elasticsearcher We're currently working on the ability to attach an ENI to a task and use native VPC networking. We believe that this will address many of the use-cases described in this issue, as well as provide integration with existing VPC network topology and features that people are using today.

If you're interested in details, check out aws/amazon-ecs-agent#701 (description of how we're planning to do this), the dev branch of amazon-ecs-cni-pugins (where we're working on some of the software behind this), as well as aws/amazon-ecs-agent#777 and aws/amazon-ecs-agent#779 (some of the changes necessary in the agent for this feature).

[ghost]

👍
simply exposing --net=my-user-defined-network in container definition, and adding user defined network in task definition is most appropriate.

My use case assumes certain containers will join user defined networks and call each other by host. This setup is meant to run both outside of and inside AWS (through various development phases). No need to reinvent the wheel. Please support whats already there.

[mrt123]

👍
simply exposing --net=my-user-defined-network in container definition, and adding user defined network in task definition is most appropriate.

My use case assumes certain containers will join user defined networks and call each other by host. This setup is meant to run both outside of and inside AWS (through various development phases). No need to reinvent the wheel. Please support whats already there.

[johnbrandborg]

We require a number of containers to be bundled together with open communication, only exposing what needs to be consumed by the outside world. Link is ugly and not scalable, and we need to be able to set the networks within our task definitions. No need to over engineer whats already available.

[jamessewell]

Any updates here? - this is a really needed feature

[drzhouq]

This is much needed feature. I don't understand why AWS does not agree with the users. The use case is fairly common. Let's say you have a database container (serviceDB) that needs to be connected by multiple app containers (serviceApp). Put the database container and app container in one task definition and link them is not going to work.

[elasticsearcher]

Surprised no one's mentioned Weaveworks' integration with ECS, because it does pretty much what everyone here is asking for:
https://www.weave.works/docs/tutorials/old-guides/ecs/

Basically, Weave assigns an IP address to each container and runs an auto-managed DNS service, which lets any container in the same cluster address any other container by its name. The DNS service also automatically load-balances all containers.

I just tried it out and haven't encountered any issues so far. Just had to examine the ECS cluster setup script that they provide in the example to figure out the required SG and IAM configs.

Does anyone have experience with Weave and ECS? Any feedback would be super helpful.

[elasticsearcher]

@errordeveloper or @2opremio, would you mind chiming in please? I thought I'd loop you in since Weaveworks' solution seems to perfectly address this long-standing ECS feature request. Are there any limitations/concerns that we should be aware of or it's stable enough to use in production? :)

[2opremio]

Yes, Weave Net should be able to solve most (if not all) the use cases presented above. It's production ready and we provide AMIs and Cloud Formation templates to run it.

See

https://www.weave.works/docs/scope/latest/ami/
https://www.weave.works/docs/tutorials/old-guides/ecs/
https://www.slideshare.net/mobile/weaveworks/weaveworks-at-aws-reinvent-2016-operations-management-with-amazon-ecs

[elasticsearcher]

Thanks, @2opremio, that's great to hear! Weave Net makes connecting containerized apps so much easier.

[sGeeK44]

If someone is interrested, i have found a workaround....

Create a script setup-server.py:

import os
import docker

docker_client = docker.from_env()
networks_client = docker_client.networks
network_name = os.environ['CONTAINER_NETWORK']
network_list = networks_client.list(network_name)
if network_list:
    network = network_list[0]
else:
    network = networks_client.create(network_name, driver="bridge")

container_id = os.environ['HOSTNAME']
network.connect(container_id)

Create a script run-server.sh:

#!/bin/bash

python3 ./setup-server.py

# Add your usaul entry point there
.....

In dockerfile add scripts and python tools

RUN apt-get update \
    && apt-get --yes install \
        python3-pip

RUN pip3 install docker

COPY setup-server.py setup-server.py
RUN chmod +x setup-server.py

COPY runserver.sh runserver.sh
RUN chmod +x runserver.sh

In ECS task:

• add env var for user defiine network name
• add shell script in entry point ["/bin/bash","/your/path/runserver.sh"]
• Shared a volume to access at host docker deamon. /var/run/docker.sock:/var/run/docker.sock

It works nicely for me. But I hope ECS Team will make something to get user-defined network functionnality.

[yhlee-aws]

Hi Everyone,

Thank you for your feedback regarding using Docker's user-defined networks.

1. We have heard your feedback on awsvpc task networking limitations, in particular with regards to ENI density. Our teams are working on improving this for all AWS users. You should consider using AWS Fargate which automatically handles the provisioning and lifecycle for the compute needed to run your containers. Fargate automatically assigns an ENI to every task, solving the ENI density issue.
2. We have launched a native experience for ECS service discovery for both EC2 mode and Fargate. This works with all networking modes and will be available in 10 regions by end of day today. Please reach out to us if you have any feedback on usage or documentation related to this.
3. We are continuing to invest heavily in new capabilities and features for ECS. Our announcements page has the latest information on recent feature additions, including Daemon Task Scheduling, Service Discovery, support for Docker Container Healthchecks, and support for Docker volumes.
4. Whether you should use ECS or EKS for your app is a great question. The way to think about it is this. It's all about which APIs with which you want to interact. If you are super familiar with the K8s APIs, then EKS is probably a great way to go. If you like the AWS APIs and deep integrations with AWS services, then ECS is probably the service for you. That isn't to say EKS won't have deep integrations with other AWS services over time. We intend to always run the latest upstream release of K8s, so we need to make sure any changes are approved by the community.

[psyhomb]

You should consider using AWS Fargate

1. In other words we have to pay for this feature

We have launched a native experience for ECS service discovery for EC2 mode

2. Will this feature allow communication between containers by using container name (possible with user-defined networks on Docker layer), defined under same task definition?
   Could you please provide some documentation link for this new service discovery feature, thanks?

The way to think about it is this. It's all about which APIs with which you want to interact.

4. From this statement I get an impression that ECS and Kubernetes are basically the same kind of orchestrator, but it's only matter of preference, which is IMHO, absolutely incorrect.
   The only reason why Amazon decided to provide K8S as managed service is beacause they realized this is the horse which is going to win the race.
   https://youtu.be/shV2sokuF5k

[dzalbo]

@psyhomb https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-discovery.html

[psyhomb]

I tought @yunhee-l was talking about some new features regarding service discovery, but obviously I was wrong, I'm very familiar with service discovery (and doc) they have launched some time ago but they're forcing us to use Route53 so that's why I don't what to use it.
This is where the answer to the question, regarding user-defined networks and why they're still "working" on the implementation, is hiding.

[ruslantalpa]

my two cents
the problem is a simple one (when using docker on local host) - have two containers in the same task talk to each other

Whatever direction you go you hit major problems with the simple task

• host mode - only one container per port
• bridge mode
  • links being deprecated and can't have bidirectional coms
  • user defined networks not supported (this specific issue)
  • service discovery can't be used if you don't want to expose the ports to outside world (as in current VPC) + you'd have to write special entrypoints to look at the SRV records (so no off-the-shelf images ) and if the other service dies ... the info is stale (since those scripts run only on startup)
• awsvpc
  • ENI limitation (that will go away soon it seems)
  • private vpc need to define gateway
  • public vpc, network just flat does not work (though it does for fargate).

the best option is to enable user defined networks (as requested in this issue) but also removing the "outside network communication" limitation for ec2 launch time would be a welcomed addition

[andres99x]

+1

[cloudwitch]

+1. I would like to preach the evils of linking containers, and you can't link in fargate which means no sidecars in fargate.

[coultn]

+1. I would like to preach the evils of linking containers, and you can't link in fargate which means no sidecars in fargate.

Can you explain more about the connection between linking and sidecars in Fargate? We definitely support sidecars in Fargate; if you need containers in the same Fargate task to communicate, they can do so via localhost.

[cloudwitch]

I was enlightened this morning about this article:
https://aws.amazon.com/blogs/compute/migrating-your-amazon-ecs-containers-to-aws-fargate/

localhost/127.0.0.1:<some_port_number> solves my concerns. Thanks for the quick followup.

[normangilmore]

Looks like the ENI constraints have finally been addressed.

Amazon ECS now supports increased elastic network interface (ENI) limits for tasks in awsvpc Networking Mode
Posted On: Jun 6, 2019

Amazon Elastic Container Service (ECS) now supports increased ECS task limits for select Amazon EC2 instances when using awsvpc task networking mode. When you use these instance types and opt in to the awsvpcTrunking account setting, additional Elastic Network Interfaces (ENIs) are available for tasks using awsvpc networking mode on newly launched container instances.

Previously, the number of tasks in awsvpc network mode that could be run on an instance was limited by the number of available Elastic Network Interfaces (ENIs) on the instance; those ENIs could be used by ECS tasks or by other processes outside of ECS. As a result, the number of tasks that could be placed on EC2 instances often was constrained despite there being ample vCPU and memory available for additional containers to utilize. Now, you have access to an increased number of ENIs for use exclusively by tasks in awsvpc networking mode for select instance types. The increase is anywhere from 3 to 8 times the previous limits, depending on the instance type.

The improved ENI limits are available in all regions where ECS is available. Please visit the AWS region table to see where Amazon ECS is available.

To learn more on how to opt in, see Account Settings. To get started with increased ENI limits, read our documentation.
https://aws.amazon.com/about-aws/whats-new/2019/06/Amazon-ECS-Improves-ENI-Density-Limits-for-awsvpc-Networking-Mode/

[pauldraper]

Unfortunately, awsvpc (in ECS, not Fargate) has half-baked ENIs that don't allow public addresses.

This necessitates setting up single-point-of-failure/costly NATs just for the privilege of accessing the internet.

#423

[matschaffer]

I ended up mixing links, extraHosts, and mapped ports to work around the lack of bidirectional linking imposed by lack of user defined networks.

Here's the part of the config that does it incase it helps anyone else.

    {
      "name": "grafana",
      "image": "docker.pkg.github.com/safecast/reporting2/grafana:latest",
      "memoryReservation": 128,
      "essential": true,
      "portMappings": [
        {
          "hostPort": 3000,
          "containerPort": 3000
        }
      ],
      "links": [
        "renderer"
      ],
      "mountPoints": [
        {
          "sourceVolume": "grafana",
          "containerPath": "/etc/grafana",
          "readOnly": true
        }
      ]
    },
    {
      "name": "renderer",
      "image": "grafana/grafana-image-renderer:2.0.0",
      "memoryReservation": 128,
      "essential": true,
      "portMappings": [
        {
          "hostPort": 8081,
          "containerPort": 8081
        }
      ],
      "mountPoints": [],
      "extraHosts": [
        {
          "hostname": "grafana",
          "ipAddress": "172.17.0.1"
        }
      ]
    }

User defined networks would still be appreciated since it'd make this way simpler.

[maiconbaumx]

Still a required feature of ECS. I do understand that we can use awsvpc as an alternative, but for my use case this is not an option, since I have around 3k of services that could scale to something around 10k replicas and I don't have an infinite network mask within this environment to allocate. It would be great to have an option like user defined networks.

[jayfry1077]

I ended up mixing mapped links and mapped ports to work around the lack of bidirectional linking imposed by lack of user defined networks.

Here's the part of the config that does it incase it helps anyone else.

    {
      "name": "grafana",
      "image": "docker.pkg.github.com/safecast/reporting2/grafana:latest",
      "memoryReservation": 128,
      "essential": true,
      "portMappings": [
        {
          "hostPort": 3000,
          "containerPort": 3000
        }
      ],
      "links": [
        "renderer"
      ],
      "mountPoints": [
        {
          "sourceVolume": "grafana",
          "containerPath": "/etc/grafana",
          "readOnly": true
        }
      ]
    },
    {
      "name": "renderer",
      "image": "grafana/grafana-image-renderer:2.0.0",
      "memoryReservation": 128,
      "essential": true,
      "portMappings": [
        {
          "hostPort": 8081,
          "containerPort": 8081
        }
      ],
      "mountPoints": [],
      "extraHosts": [
        {
          "hostname": "grafana",
          "ipAddress": "172.17.0.1"
        }
      ]
    }

User defined networks would still be appreciated since it'd make this way simpler.

Thank you so much for this.

[AlonAvrahami]

Would be helpful to get some timeline/info on this request, this is a real blocker for us until we migrate our workload to EKS which does not have this issue.

[sparrc]

It seems like a lot of the desire here is to have DNS resolution between containers on the docker bridge.

One problem with this is currently the actual container name that you define in ECS is not final name that docker creates the container as, and furthermore we append a random string to the actual name (code reference here) so that the name of your container on the bridge network is in the format ecs-$TASK_FAMILY-$TASK_VERSION-$NAME-$RANDOM_STRING, like ecs-myBusyboxTask-8-myContainerName-dab9f0dfb993988fb701. From what I understand this would then still present a discoverability issue since one container does not know the random string that was assigned to another container in the task (or in another task).

So I believe to usefully support a user-defined network we would also need to change this naming convention to make it more easily discoverable, perhaps to something like ecs-$TASK_FAMILY-$NAME-$TASK_ID ?

[fierlion]

aws/amazon-ecs-agent#3793 <- this has some overlap with the feature request here. (Looking into the pr now)

[jsuchenia]

Thanks, @fierlion - in my PR we implemented a custom bridge and Network Alias name - so we don't change the task container name
So it can give us the ability to schedule in a custom docker network and (if needed) register under some name - taken from a container name

[BenDenney]

Thanks, @fierlion - in my PR we implemented a custom bridge and Network Alias name - so we don't change the task container name So it can give us the ability to schedule in a custom docker network and (if needed) register under some name - taken from a container name

Well, this really should be lined up to be released?
It would solve so many issues for us trying to use custom networks for multi-container tasks (especially useful for running tests in dev pipelines).
Do we have at least a proposed date for this?