Add e2e test case for SecondaryNetwork of SR-IOV type #6922

wenqiq · 2025-01-14T16:40:14Z

Add a script to create a Kubernetes testbed on AWS VM Nodes with SR-IOV interface configured.
Add e2e test codes for SR-IOV case to validate the traffic between two Pods with secondary network enabled.
Add a Jenkins job named cloud-antrea-test-aws-sriov-secondary-network in Jenkins service, and add github comment trigger to trigger the e2e tests for SecondaryNetwork with SR-IOV on demand.

The complete test process record:
https://jenkins.antrea.io/job/cloud-antrea-test-aws-sriov-secondary-network/16/

ci/aws/test-sr-iov-secondary-network-aws.sh

luolanzone · 2025-01-21T07:50:28Z

ci/aws/test-sr-iov-secondary-network-aws.sh

+        sudo systemctl restart containerd
+        sudo systemctl enable containerd
+
+        echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list


Can this v1.32 be a parameter of the script? then we can use the latest K8s if we need in the future through the Jenkins job.

luolanzone · 2025-01-21T07:54:26Z

ci/aws/test-sr-iov-secondary-network-aws.sh

+        sudo apt install -y docker.io
+        sudo docker --version
+
+        sudo apt-get update


I think this line can be removed.

It would fail during install the tools.

ci/aws/test-sr-iov-secondary-network-aws.sh

ci/jenkins/jobs/projects-cloud.yaml

luolanzone · 2025-01-21T08:23:43Z

test/e2e-secondary-network/infra/sriov-secondary-networks.yml

+  - cidr: "172.31.22.0/24"
+  subnetInfo:
+    gateway: "172.31.16.1"


Where are these IPs from? or it's totally a private CIDR which can be defined by us without any limitation?

test/e2e/framework.go

ci/aws/assign_ip_to_instance_networkinterface.sh

ci/aws/test-sriov-secondary-network-aws.sh

ci/aws/assign_ip_to_instance_networkinterface.sh

tnqn · 2025-01-22T14:02:47Z

test/e2e-secondary-network/secondary_network_test.go

+	resNum = sriovResNum
+	if pod.isSRIOV {
+		resNum = 1


I'm a bit confused, it seems resNum is set to the number specific to SRIOV in both cases.

In the testSecondaryNetwork test case it should set to 3 (sriovResNum = 3), and it is 1 in this test case:

apiVersion: v1 kind: Pod metadata: name: sample-pod-1 labels: app: sample-pod-1 annotations: k8s.v1.cni.cncf.io/networks: sriov-net1 spec: containers: - name: toolbox image: antrea/toolbox:latest resources: requests: intel.com/intel_sriov_netdevice: '1' limits: intel.com/intel_sriov_netdevice: '1'

It may make sense at this point to just remove the existing TestSriovNetwork (which was meant for baremetal) since we never run it, in order to avoid confusion. cc @jianjuns

I am fine to remove the baremetal test.

test/e2e-secondary-network/secondary_network_test.go

tnqn · 2025-01-22T14:18:57Z

test/e2e-secondary-network/secondary_network_test.go

+	t.Run("testCreateTestPodOnNode", func(t *testing.T) {
+		err := testData.createPods(t, e2eTestData.GetTestNamespace())
+		assert.NoError(t, err)
+	})
+	t.Run("testAssignPodIP", func(t *testing.T) {
+		err := testData.assignIP()
+		if err != nil {
+			t.Fatalf("Error when assign IP to ec2 instance: %v", err)
+		}
+	})
+	t.Run("testpingBetweenInterfaces", func(t *testing.T) {


These are 3 steps of a single test, shouldn't be 3 tests.

Updated. I saw the testSecondaryNetwork function use two tests and updated them.

ci/aws/test-sriov-secondary-network-aws.sh

antoninbas · 2025-01-30T23:50:52Z

ci/aws/test-sriov-secondary-network-aws.sh

+install_kubernetes() {
+    local node_ip=$1
+    echo "Installing Kubernetes on node $node_ip..."
+    ssh -o StrictHostKeyChecking=no -i "$AWS_EC2_SSH_KEY_NAME" ubuntu@$node_ip << EOF


would it be realistic to use the Ansible playbook we already have (over SSH) instead of having that long list of commands?

It seems that integrating Ansible involves significant changes and requires considerable effort. Given the scope of these modifications, it would be more efficient to implement them in a separate pull request (PR).

ci/aws/test-sriov-secondary-network-aws.sh

antoninbas · 2025-01-31T00:08:59Z

test/e2e-secondary-network/secondary_network_test.go

+	resNum = sriovResNum
+	if pod.isSRIOV {
+		resNum = 1


It may make sense at this point to just remove the existing TestSriovNetwork (which was meant for baremetal) since we never run it, in order to avoid confusion. cc @jianjuns

test/e2e-secondary-network/secondary_network_test.go

test/e2e/framework.go

antoninbas · 2025-01-31T00:25:37Z

test/e2e-secondary-network/secondary_network_test.go

+	return out.String(), nil
+}
+
+func (data *testData) assignIP() error {


it would be good if we could find a way to "isolate" the AWS-specific stuff in a single function and have the test script provide an "aws" flag when running the tests.

The function can iterate over the Pods, retrieve the Node, lookup the eni label and perform the IP assignment, all in one place. We don't need to build the mapping from Pod to eni ahead of time.

antoninbas · 2025-01-31T00:28:42Z

test/e2e-secondary-network/secondary_network_test.go

+}
+
+// createNetworkAttachmentDefinition creates a NetworkAttachmentDefinition in the specified namespace.
+func createNetworkAttachmentDefinition(client networkclient.Interface, namespace, name, config string) error {


could we let the script create the NetworkAttachmentDefinition with kubectl, like we do in ci/kind/test-secondary-network-kind.sh. This is also how we create the IPPool. I don't see anything "dynamic" here so I am not sure why we are doing it programmatically?

We should probably follow the existing model and create it in the default namespace?

Since the Namespace where the test Pod is located is randomly generated, the NetworkAttachmentDefinition needs to be in the same Namespace. Otherwise, it won`t work.

go.mod

Signed-off-by: Wenqi Qiu <[email protected]>

luolanzone added this to the Antrea v2.3 release milestone Jan 15, 2025

luolanzone added the area/secondary-network Issues or PRs related to support for secondary networks in Antrea label Jan 15, 2025

wenqiq force-pushed the add-sriov-e2e branch 4 times, most recently from 758c687 to 1f77538 Compare January 20, 2025 03:06

wenqiq changed the title ~~[WIP]Add e2e test case for SecondaryNetwork of SR-IOV type~~ Add e2e test case for SecondaryNetwork of SR-IOV type Jan 20, 2025

wenqiq mentioned this pull request Jan 20, 2025

Add e2e for SecondaryNetwork with SR-IOV #6915

Open

wenqiq marked this pull request as ready for review January 21, 2025 00:59

luolanzone requested review from antoninbas and tnqn January 21, 2025 07:33

luolanzone reviewed Jan 21, 2025

View reviewed changes

tnqn reviewed Jan 22, 2025

View reviewed changes

wenqiq force-pushed the add-sriov-e2e branch 2 times, most recently from aea2610 to 62000e7 Compare January 27, 2025 08:04

antoninbas reviewed Jan 31, 2025

View reviewed changes

luolanzone reviewed Feb 10, 2025

View reviewed changes

go.mod Outdated Show resolved Hide resolved

wenqiq force-pushed the add-sriov-e2e branch from e4f943c to 7162ed6 Compare February 10, 2025 06:49

Add e2e test case for SecondaryNetwork of SR-IOV type

b5e18b1

Signed-off-by: Wenqi Qiu <[email protected]>

wenqiq force-pushed the add-sriov-e2e branch from 277ec59 to b5e18b1 Compare February 11, 2025 05:37

wenqiq added 3 commits February 11, 2025 14:02

update

f207a49

Signed-off-by: Wenqi Qiu <[email protected]>

update

2cdffce

Signed-off-by: Wenqi Qiu <[email protected]>

update

5024367

Signed-off-by: Wenqi Qiu <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add e2e test case for SecondaryNetwork of SR-IOV type #6922

Add e2e test case for SecondaryNetwork of SR-IOV type #6922

wenqiq commented Jan 14, 2025 •

edited

Loading

luolanzone Jan 21, 2025

wenqiq Jan 23, 2025

luolanzone Jan 21, 2025

wenqiq Jan 23, 2025

luolanzone Jan 21, 2025

wenqiq Jan 23, 2025

tnqn Jan 22, 2025

wenqiq Jan 23, 2025

antoninbas Jan 31, 2025

jianjuns Feb 6, 2025

tnqn Jan 22, 2025

wenqiq Jan 23, 2025

antoninbas Jan 30, 2025

wenqiq Feb 11, 2025

antoninbas Jan 31, 2025

antoninbas Jan 31, 2025

wenqiq Feb 11, 2025

antoninbas Jan 31, 2025

wenqiq Feb 6, 2025

Add e2e test case for SecondaryNetwork of SR-IOV type #6922

Are you sure you want to change the base?

Add e2e test case for SecondaryNetwork of SR-IOV type #6922

Conversation

wenqiq commented Jan 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wenqiq commented Jan 14, 2025 •

edited

Loading