Introduce a new environment variable ECS_EXCLUDE_IPV6_PORTBINDING #3025

chienhanlin · 2021-09-13T19:38:33Z

Summary

A behavioral change in docker container port binding, where both IPv4 and IPv6 bindings will be exposed starting from Docker version 20.10.6, is reported in following issues and PR:

Docker 20.10.6 IPv6 bindings shouldn't be mapped as network bindings for tasks for non IPv6 networks. #2870
Filter out ipv6 port forwards when an ipv6 default route doesn't exist #2855
[ECS] [request]: Support PortMappings in Bridge mode on Docker 20.10.6 containers-roadmap#1377
Different port number are getting mapped on ipv4 and ipv6 for same expose port moby/moby#42442
Regression in 20.10.6: Docker report a published port on ipv6 when ipv6 is disabled moby/moby#42375

To mitigate this issue, this PR introduces a new agent environment variable ECS_EXCLUDE_IPV6_PORTBINDING to filter out IPv6 port bindings for default network mode tasks. This config variable is true by default, which means the IPv6 port bindings will be filtered out in the Container State Change Payload. To include IPv6 port bindings, set ECS_EXCLUDE_IPV6_PORTBINDING=false in the /etc/ecs/ecs.config file and then restart the agent.

No change is made on the metadata. Customers can still find both IPv4 and IPv6 port bindings from the task metadata endpoint even when ECS_EXCLUDE_IPV6_PORTBINDING=true. This raw data, which is retrieved from docker inspect, is kept for further debug operation.

Implementation details

Add a new type ShouldExcludeIPv6PortBinding and a config variable ECS_EXCLUDE_IPV6_PORTBINDING to agent and is true by default.
Update README with the new config variable ECS_EXCLUDE_IPV6_PORTBINDING.
Filter out IPv6 port bindings while building the container state change payload if ShouldExcludeIPv6PortBinding is true.
Update unit tests with the new config.

Testing

Following two docker versions are used for testing:

Docker version 20.10.4, which dose not have the behavioral change.
Docker version 20.10.7, which exposes both IPv4 and IPv6 port bindings.

Manual test cases:
Steps:

Run a Task to launch a docker container using the nginx docker image under the bridge network mode, and define the container port mappings in the task definition.

      "portMappings": [
        {
          "hostPort": 0,
          "protocol": "tcp",
          "containerPort": 80
        }

Run docker ps and docker inspect to get the docker container port mappings.
Get metadata for the task by curl ${ECS_CONTAINER_METADATA_URI_V4}/task.
Run AWS CLI describe-tasks to verify the port binding.

Results of each test case are listed as follows:

Docker version 20.10.4 and ECS_EXCLUDE_IPV6_PORTBINDING is not set.
Docker version 20.10.4 and ECS_EXCLUDE_IPV6_PORTBINDING is set to true.
Docker version 20.10.4 and ECS_EXCLUDE_IPV6_PORTBINDING is set to false.

Test cases 1-3 have the same responses.

No IPv6 port binding is found in the nginx docker container.
The response of AWS CLI describe-tasks command shows only IPv4 port binding in the container.

 "networkBindings": [
    {
        "protocol": "tcp", 
         "bindIP": "0.0.0.0", 
         "containerPort": 80, 
         "hostPort": 49155
    }
],

The response of {ECS_CONTAINER_METADATA_URI_V4}/task does not contains IPv6 port binding info.

Docker version 20.10.7 and ECS_EXCLUDE_IPV6_PORTBINDING is not set.
Docker version 20.10.7 and ECS_EXCLUDE_IPV6_PORTBINDING is set to true.

Test cases 4-5 have the same responses.

No IPv6 port binding in the nginx docker container

        "NetworkSettings": {
            "Bridge": "",
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "80/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "49154"
                    }
                ]
            },
          ...
        }

The response of AWS CLI describe-tasks command shows only IPv4 port binding in the container

"networkBindings": [
    {
        "protocol": "tcp", 
        "bindIP": "0.0.0.0", 
        "containerPort": 80, 
         "hostPort": 49154
    }
],

The response of {ECS_CONTAINER_METADATA_URI_V4}/task contains both IPv4 and IPv6 port bindings info.

Docker version 20.10.7 and ECS_EXCLUDE_IPV6_PORTBINDING is set to false.

An IPv6 port binding is found in the nginx docker container

        "NetworkSettings": {
            "Bridge": "",
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "80/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "49155"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "49155"
                    }
                ]
            },
          ...
        }

The response of AWS CLI describe-tasks command shows both IPv4 and IPv6 port bindings in the container

"networkBindings": [
    {
        "protocol": "tcp", 
        "bindIP": "0.0.0.0", 
        "containerPort": 80, 
         "hostPort": 49155
    }, 
    {
        "protocol": "tcp", 
        "bindIP": "::", 
        "containerPort": 80, 
        "hostPort": 49155
    }
],

The response of {ECS_CONTAINER_METADATA_URI_V4}/task contains both IPv4 and IPv6 port bindings info.

awsvpc and host network mode

No behavioral change, comparing with agent version 1.55.2, is found while running a task with awsvpc and host mode

New tests cover the changes: no, but existing config unit tests are updated.

Description for the changelog

Enhancement - Introduce a new environment variable ECS_EXCLUDE_IPV6_PORTBINDING, which is true by default. When this filter is enabled, the response of DescribeTasks API call will not show the IPv6 port bindings for default network mode tasks, but it is still included in the Task metadata endpoint. This is a workaround for docker's bug as detailed in #2870.

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

fenxiong · 2021-09-14T17:20:24Z

just wondering - does this filtering change anything for container in host and awsvpc network mode? i'm assuming no, but would be good to do a manual test to verify

chienhanlin · 2021-09-14T18:12:33Z

just wondering - does this filtering change anything for container in host and awsvpc network mode? i'm assuming no, but would be good to do a manual test to verify

This filter does not change anything for containers in host and awsvpc network mode. Will add the manual test results to the PR, and also add following info as a reference. Thanks!

For Task definition:

Network Mode : Host
Your containers will share the container instance network stack. Port mappings can only specify container ports (any existing host port specifications will be removed).
Network Mode : awsvpc
Your containers in the task will share an ENI using a common network stack. Port mappings can only specify container ports (any existing host port specifications will be removed).
Network Mode : None
Your containers will not have any network access to external routes. Port mappings are not valid with this network mode (any existing port mappings will be removed).

For Amazon ECS IPv6 support
See Amazon ECS now supports Internet Protocol Version 6 (IPv6) in awsvpc networking mode announcement for more details.

yinyic · 2021-09-14T18:27:25Z

agent/api/ecsclient/client_test.go

+			Cluster:                 configuredCluster,
+			AWSRegion:               "us-east-1",
+			InstanceAttributes:      additionalAttributes,
+			IPv6PortBindingExcluded: config.BooleanDefaultTrue{Value: config.ExplicitlyEnabled},


nit - maybe a more appropriate name would be ShouldExcludeIPv6PortBinding? Technically at the moment that this variable is checked it's not excluded yet.

Update the config variable to ShouldExcludeIPv6PortBinding as recommended. Thanks!

yinyic · 2021-09-14T18:28:44Z

agent/api/ecsclient/client.go

+	networkBindings := []*ecs.NetworkBinding{}
+	for _, binding := range change.PortBindings {
+		if binding.BindIP == "::" && iPv6PortBindingExcluded {
+			seelog.Debugf("To exclude IPv6 port bindings: %v as IPv6 port bindings is requested to be excluded", binding)


nit - we can probably make it shorter
"Excluded IPv6 port binding: %v"

Update the debug message as recommended. Thanks!

chienhanlin · 2021-09-14T21:43:04Z

Note: TestHandlerDoesntLeakGoroutines failed in the windows unit tests check, which is not introduced by code changes in this PR.

vsiddharth · 2021-09-14T22:02:28Z

Note: TestHandlerDoesntLeakGoroutines failed in the windows unit tests check, which is not introduced by code changes in this PR.

That test has been flaky for a while on Windows. Can we update or skip that test for a more consistent signal?

sharanyad · 2021-09-14T22:30:03Z

agent/api/ecsclient/client.go

@@ -454,7 +454,7 @@ func (client *APIECSClient) buildManagedAgentStateChangePayload(change api.Manag
 	}
 }

-func (client *APIECSClient) buildContainerStateChangePayload(change api.ContainerStateChange) *ecs.ContainerStateChange {
+func (client *APIECSClient) buildContainerStateChangePayload(change api.ContainerStateChange, shouldExcludeIPv6PortBinding bool) *ecs.ContainerStateChange {


is there any reason why ipv6 ports are excluded only in state change calls?
Wouldn't the ipv6 ports show up in task metadata endpoint otherwise?
its better to filter it here to avoid any discrepancies:

amazon-ecs-agent/agent/engine/docker_task_engine.go

Line 494 in 225bc3a

container.SetKnownPortBindings(metadata.PortBindings)

Update README file and the description for the changelog to make the purpose/impact of this PR more clear. Thanks!

sharanyad · 2021-09-15T00:51:06Z

README.md

@@ -194,7 +194,8 @@ additional details on each available environment variable.
 | `ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE` | `true` | Whether to enable awslogs log driver to authenticate via credentials of task execution IAM role. Needs to be true if you want to use awslogs log driver in a task that has task execution IAM role specified. When using the ecs-init RPM with version equal or later than V1.16.0-1, this env is set to true by default. | `false` | `false` |
 | `ECS_FSX_WINDOWS_FILE_SERVER_SUPPORTED` | `true` | Whether FSx for Windows File Server volume type is supported on the container instance. This variable is only supported on agent versions 1.47.0 and later. | `false` | `true` |
 | `ECS_ENABLE_RUNTIME_STATS` | `true` | Determines if [pprof](https://pkg.go.dev/net/http/pprof) is enabled for the agent. If enabled, the different profiles can be accessed through the agent's introspection port (e.g. `curl http://localhost:51678/debug/pprof/heap > heap.pprof`). In addition, agent's [runtime stats](https://pkg.go.dev/runtime#ReadMemStats) are logged to `/var/log/ecs/runtime-stats.log` file. | `false` | `false` |
-
+| `ECS_EXCLUDE_IPV6_PORTBINDING` | `true` | Determines if agent should exclude IPv6 port binding. If enabled, IPv6 port binding will be filtered out, and the response of DescribeTasks API call will not show tasks' IPv6 port bindings, but it is still included in Task metadata endpoint. | `true` | `true` |


we may want to indicate this is only for bridge mode tasks

Update the description with the network mode as follows,
"Determines if agent should exclude IPv6 port binding using default network mode. If enabled, IPv6 port binding will be filtered out, and the response of DescribeTasks API call will not show tasks' IPv6 port bindings, but it is still included in Task metadata endpoint."
Thanks!

sharanyad · 2021-09-15T00:51:42Z

agent/api/ecsclient/client.go

+	networkBindings := []*ecs.NetworkBinding{}
+	for _, binding := range change.PortBindings {
+		if binding.BindIP == "::" && shouldExcludeIPv6PortBinding {
+			seelog.Debugf("Excluded IPv6 port binding: %v", binding)


can we also log the container name/task ID if possible, so this is more useful?

Add the container name and the task ARN to the debug log, as

level=debug time=xxx msg="Exclude IPv6 port binding {80 49155 :: 0} for container web in task xxxxx" module=client.go

mythri-garaga · 2021-09-15T20:31:47Z

Skipped Windows unit tests in favor of #3030

zen0wu · 2021-12-23T07:54:41Z

Just noticed this behavior today - has this recently regressed?

I'm on Docker 20.10.7 and ecs-agent 1.57.1 and am still noticing IPV6 binding being created, even after setting ECS_EXCLUDE_IPV6_PORTBINDING=true explicitly (and restart the ecs-agent), it still doesn't prevent the issue.

vekien · 2022-02-01T13:47:07Z

Is this working? I have also added echo ECS_EXCLUDE_IPV6_PORTBINDING=true >> /etc/ecs/ecs.config; but my docker tasks are still binding to ipv6.

chienhanlin force-pushed the ipv6PortBindingVer3 branch from f25098d to 113c257 Compare September 13, 2021 20:54

chienhanlin added the bot/test label Sep 13, 2021

amazon-ecs-bot removed the bot/test label Sep 13, 2021

chienhanlin force-pushed the ipv6PortBindingVer3 branch from 113c257 to 762dca1 Compare September 14, 2021 17:16

chienhanlin force-pushed the ipv6PortBindingVer3 branch 3 times, most recently from 841a440 to 9ea0155 Compare September 14, 2021 18:02

chienhanlin force-pushed the ipv6PortBindingVer3 branch from 9ea0155 to c9504d2 Compare September 14, 2021 18:19

yinyic reviewed Sep 14, 2021

View reviewed changes

chienhanlin force-pushed the ipv6PortBindingVer3 branch 3 times, most recently from aa70721 to 2c3e4c0 Compare September 14, 2021 20:12

chienhanlin added the bot/test label Sep 14, 2021

amazon-ecs-bot removed the bot/test label Sep 14, 2021

sharanyad reviewed Sep 14, 2021

View reviewed changes

yinyic previously approved these changes Sep 14, 2021

View reviewed changes

chienhanlin dismissed yinyic’s stale review via 0a92d7a September 15, 2021 00:19

chienhanlin force-pushed the ipv6PortBindingVer3 branch from 2c3e4c0 to 0a92d7a Compare September 15, 2021 00:19

chienhanlin added the bot/test label Sep 15, 2021

amazon-ecs-bot removed the bot/test label Sep 15, 2021

sharanyad reviewed Sep 15, 2021

View reviewed changes

Introduce a new environment variable ECS_EXCLUDE_IPV6_PORTBINDING

7ec9685

chienhanlin force-pushed the ipv6PortBindingVer3 branch from 0a92d7a to 7ec9685 Compare September 15, 2021 01:12

chienhanlin added the bot/test label Sep 15, 2021

amazon-ecs-bot removed the bot/test label Sep 15, 2021

sharanyad approved these changes Sep 15, 2021

View reviewed changes

yinyic approved these changes Sep 15, 2021

View reviewed changes

mythri-garaga merged commit c003094 into aws:dev Sep 15, 2021

chienhanlin mentioned this pull request Sep 27, 2021

Filter out invalid IPv6 port bindings from TestDynamicPortForward and TestMultipleDynamicPortForward #3043

Merged

vincentjames501 mentioned this pull request Nov 23, 2021

No way to differentiate between IPV4/IPV6 Port Mapptings #3096

Closed

chienhanlin deleted the ipv6PortBindingVer3 branch June 2, 2022 02:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce a new environment variable ECS_EXCLUDE_IPV6_PORTBINDING #3025

Introduce a new environment variable ECS_EXCLUDE_IPV6_PORTBINDING #3025

chienhanlin commented Sep 13, 2021 •

edited

Loading

fenxiong commented Sep 14, 2021

chienhanlin commented Sep 14, 2021

yinyic Sep 14, 2021

chienhanlin Sep 14, 2021

yinyic Sep 14, 2021 •

edited

Loading

chienhanlin Sep 14, 2021

chienhanlin commented Sep 14, 2021

vsiddharth commented Sep 14, 2021

sharanyad Sep 14, 2021

chienhanlin Sep 15, 2021

sharanyad Sep 15, 2021

chienhanlin Sep 15, 2021

sharanyad Sep 15, 2021

chienhanlin Sep 15, 2021

mythri-garaga commented Sep 15, 2021

zen0wu commented Dec 23, 2021

vekien commented Feb 1, 2022

Introduce a new environment variable ECS_EXCLUDE_IPV6_PORTBINDING #3025

Introduce a new environment variable ECS_EXCLUDE_IPV6_PORTBINDING #3025

Conversation

chienhanlin commented Sep 13, 2021 • edited Loading

Summary

Implementation details

Testing

Description for the changelog

Licensing

fenxiong commented Sep 14, 2021

chienhanlin commented Sep 14, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yinyic Sep 14, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chienhanlin commented Sep 14, 2021

vsiddharth commented Sep 14, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mythri-garaga commented Sep 15, 2021

zen0wu commented Dec 23, 2021

vekien commented Feb 1, 2022

chienhanlin commented Sep 13, 2021 •

edited

Loading

yinyic Sep 14, 2021 •

edited

Loading