NAT Gateways not recreated #16876

zaneclaes · 2024-10-04T21:58:17Z

/kind bug

1. What kops version are you running? The command kops version, will display
this information.

1.30.1

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

1.30.2

3. What cloud provider are you using?

AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

Manually deleted the NAT gateways and EIPs on AWS (whoops).
Tried kops update cluster but it does not detect the deletion; instead it spits out NAT gateway errors:

W1004 15:50:51.222631 35897 executor.go:141] error running task "ElasticIP/us-east" (7m29s remaining to succeed): error finding AssociatedNatGatewayRouteTable: error listing NatGateway "nat-0460de55eeb540794": operation error EC2: DescribeNatGateways, https response error StatusCode: 400, RequestID: 6af7f0d1-1b02-461f-be25-b83f6b4330c9, api error NatGatewayNotFound: The Nat gateway nat-0460de55eeb540794 was not found

5. What happened after the commands executed?

Cluster is no longer working, and no kops commands seem to fix it.

6. What did you expect to happen?

According to #6830 and #6518 this should be fixed.

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: "2024-10-03T13:53:21Z"
  generation: 12
  name: k8s.mysite.com
spec:
  additionalPolicies:
    node: |
      [
        {
          "Effect": "Allow",
          "Action": [
            "sts:AssumeRole"
          ],
          "Resource": [
            "arn:aws:iam::AWS_ID:role/k8s-moongate"
          ]
        }
      ]
  api:
    loadBalancer:
      class: Network
      type: Public
  authorization:
    rbac: {}
  channel: stable
  cloudControllerManager: {}
  cloudProvider: aws
  configBase: s3://com-state-store/k8s.mysite.com
  etcdClusters:
  - cpuRequest: 200m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: control-plane-us-east-2a
      name: a
    - encryptedVolume: true
      instanceGroup: control-plane-us-east-2b
      name: b
    - encryptedVolume: true
      instanceGroup: control-plane-us-east-2c
      name: c
    manager:
      backupRetentionDays: 90
    memoryRequest: 100Mi
    name: main
  - cpuRequest: 100m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: control-plane-us-east-2a
      name: a
    - encryptedVolume: true
      instanceGroup: control-plane-us-east-2b
      name: b
    - encryptedVolume: true
      instanceGroup: control-plane-us-east-2c
      name: c
    manager:
      backupRetentionDays: 90
    memoryRequest: 100Mi
    name: events
  externalPolicies:
    node:
    - arn:aws:iam::AWS_ID:policy/AWSLoadBalancerControllerIAMPolicy2
  iam:
    allowContainerRegistry: true
    legacy: false
  kubeProxy:
    enabled: false
  kubelet:
    anonymousAuth: false
  kubernetesApiAccess:
  - 0.0.0.0/0
  - ::/0
  kubernetesVersion: 1.30.2
  masterPublicName: api.k8s.mysite.com
  networkCIDR: 172.20.0.0/16
  networking:
    cilium:
      enableNodePort: true
  nonMasqueradeCIDR: ::/0
  sshAccess:
  - 0.0.0.0/0
  - ::/0
  subnets:
  - cidr: 172.20.0.0/18
    ipv6CIDR: /64#0
    name: us-east-2a
    type: Public
    zone: us-east-2a
  - cidr: 172.20.64.0/18
    ipv6CIDR: /64#1
    name: us-east-2b
    type: Public
    zone: us-east-2b
  - cidr: 172.20.128.0/18
    ipv6CIDR: /64#2
    name: us-east-2c
    type: Public
    zone: us-east-2c
  topology:
    dns:
      type: None

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-10-03T13:53:22Z"
  labels:
    kops.k8s.io/cluster: k8s.mysite.com
  name: control-plane-us-east-2a
spec:
  image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20240607
  machineType: t3.medium
  maxSize: 1
  minSize: 1
  role: Master
  subnets:
  - us-east-2a

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-10-03T13:53:22Z"
  labels:
    kops.k8s.io/cluster: k8s.mysite.com
  name: control-plane-us-east-2b
spec:
  image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20240607
  machineType: t3.medium
  maxSize: 1
  minSize: 1
  role: Master
  subnets:
  - us-east-2b

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-10-03T13:53:22Z"
  labels:
    kops.k8s.io/cluster: k8s.mysite.com
  name: control-plane-us-east-2c
spec:
  image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20240607
  machineType: t3.medium
  maxSize: 1
  minSize: 1
  role: Master
  subnets:
  - us-east-2c

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-10-03T13:53:22Z"
  generation: 5
  labels:
    kops.k8s.io/cluster: k8s.mysite.com
  name: nodes-us-east-2a
spec:
  image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20240607
  instanceMetadata:
    httpPutResponseHopLimit: 2
    httpTokens: required
  machineType: t3.medium
  maxSize: 1
  minSize: 1
  role: Node
  subnets:
  - us-east-2a

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-10-03T13:53:22Z"
  generation: 5
  labels:
    kops.k8s.io/cluster: k8s.mysite.com
  name: nodes-us-east-2b
spec:
  image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20240607
  instanceMetadata:
    httpPutResponseHopLimit: 2
    httpTokens: required
  machineType: t3.medium
  maxSize: 1
  minSize: 1
  role: Node
  subnets:
  - us-east-2b

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-10-03T13:53:23Z"
  generation: 5
  labels:
    kops.k8s.io/cluster: k8s.mysite.com
  name: nodes-us-east-2c
spec:
  image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20240607
  instanceMetadata:
    httpPutResponseHopLimit: 2
    httpTokens: required
  machineType: t3.medium
  maxSize: 1
  minSize: 1
  role: Node
  subnets:
  - us-east-2c

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

9. Anything else do we need to know?

Fat-fingered NAT deletion... but I really don't want to rebuild the whole cluster 😢 🙏

The text was updated successfully, but these errors were encountered:

hakman · 2024-10-05T05:38:35Z

@zaneclaes It seems that there are some resources (EIPs) that stil reference the deleted NATGW. Could you delete those also and retry?

zaneclaes · 2024-10-05T11:43:38Z

There are zero EIPs in the AWS account. They were deleted at the same time as the NAT gateway. What gave that impression?

…

On Fri, Oct 4, 2024 at 11:38 PM Ciprian Hacman ***@***.***> wrote: @zaneclaes <https://github.com/zaneclaes> It seems that there are some resources (EIPs) that stil reference the deleted NATGW. Could you delete those also and retry? — Reply to this email directly, view it on GitHub <#16876 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAP47VMN34WXOCKSGN6VQQLZZ53PDAVCNFSM6AAAAABPMVJS7GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJUHEZTMOBUG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

hakman · 2024-10-05T11:50:04Z

There is something that still mentions "nat-0460de55eeb540794" in logs, maybe routing table, but such references should not exist anymore.
See the error message that you pasted. There is nothing in kOps config that references the NATGW ID.

zaneclaes · 2024-10-05T12:11:29Z

I have verified that the routing table and any resources I’m aware of that are tied to the NAT were also deleted. The way I read the log statement, kops thinks the EIP exists and is trying to describe the NAT gateway which should be attached to it. It sounds like the Describe check is happening as a child of the EIP provisioning, but a Describe is invalid on a deleted gateway…

…

On Sat, Oct 5, 2024 at 5:50 AM Ciprian Hacman ***@***.***> wrote: There is something that still mentions "nat-0460de55eeb540794" in logs, maybe routing table, but such references should not exist anymore. — Reply to this email directly, view it on GitHub <#16876 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAP47VNLMEBSNF4VMRCKKTDZZ7HAFAVCNFSM6AAAAABPMVJS7GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJVGAZTANJWGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

hakman · 2024-10-05T12:51:01Z

Yes, but kOps does not keep track of the resources. The NGW ID you see there comes from some other resource that used to reference it.

rifelpet · 2024-10-05T13:09:02Z

According to the Kops source code, one of the AWS route tables for your cluster contains a route to the deleted NGW ID. The route likely has a state of blackhole because the NGW no longer exists. Deleting that route should fix the problem.

zaneclaes · 2024-10-05T13:22:40Z

Thanks for the clarifications; that's very helpful. I've cleared out all the routes in the account (by first removing them from the associated subnets) except the default Route Table for the cluster (which cannot be deleted as the default for the VPC). However kops update still gives the same error, leading me to assume there's something else referencing that NAT gateway somewhere in the AWS account (and the solution is not to delete the default route table somehow)...

rifelpet · 2024-10-05T13:39:36Z

Can you find any Elastic IPs for the cluster? They'll be tagged with your cluster name but i'm not sure if they'll have an association, given their NGW was deleted. Deleting those EIPs may help.

zaneclaes · 2024-10-06T13:00:11Z

@rifelpet every time I delete all the Elastic IPs and then run kops update, it recreates the EIPs but then shows the same exact NAT gateway error, without creating any new NAT gateway in the process.

Just to be clear:

The only route table in my account is the default for the VPC
There are no Elastic IPs at all
There are no NAT gateways at all

When I run a kops update, the route tables and EIPs are recreated, but then the same NAT gateway error appears:

W1006 06:59:55.210178   57754 executor.go:141] error running task "NatGateway/us-east-" (9m13s remaining to succeed): error listing NatGateway "nat-0460de55eeb540794": operation error EC2: DescribeNatGateways, https response error StatusCode: 400, RequestID: 2495c451-8c60-42a7-81a3-e34dddcad6e8, api error NatGatewayNotFound: The Nat gateway nat-0460de55eeb540794 was not found

k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Oct 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NAT Gateways not recreated #16876

NAT Gateways not recreated #16876

zaneclaes commented Oct 4, 2024 •

edited

Loading

hakman commented Oct 5, 2024

zaneclaes commented Oct 5, 2024 via email

hakman commented Oct 5, 2024 •

edited

Loading

zaneclaes commented Oct 5, 2024 via email

hakman commented Oct 5, 2024

rifelpet commented Oct 5, 2024

zaneclaes commented Oct 5, 2024

rifelpet commented Oct 5, 2024

zaneclaes commented Oct 6, 2024 •

edited

Loading

NAT Gateways not recreated #16876

NAT Gateways not recreated #16876

Comments

zaneclaes commented Oct 4, 2024 • edited Loading

hakman commented Oct 5, 2024

zaneclaes commented Oct 5, 2024 via email

hakman commented Oct 5, 2024 • edited Loading

zaneclaes commented Oct 5, 2024 via email

hakman commented Oct 5, 2024

rifelpet commented Oct 5, 2024

zaneclaes commented Oct 5, 2024

rifelpet commented Oct 5, 2024

zaneclaes commented Oct 6, 2024 • edited Loading

zaneclaes commented Oct 4, 2024 •

edited

Loading

hakman commented Oct 5, 2024 •

edited

Loading

zaneclaes commented Oct 6, 2024 •

edited

Loading