SR-IOV Network Operator 4.15.0-202410010035 | when setting linkType: IB the NIC get filtered out

[bbenshab]

when setting: linkType: IB on a SriovNetworkNodePolicy like in this example:

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
  name: mlnx-port-1
  namespace: openshift-sriov-network-operator
spec:
  deviceType: netdevice
  ibverbs: true
  isRdma: true
  linkType: IB
  nicSelector:
    pfNames:
    - ibs3f0
    vendor: 15b3
  nodeSelector:
    feature.node.kubernetes.io/network-sriov.capable: 'true'
  numVfs: 1
  priority: 99
  rdma: true
  resourceName: port1

the NIC gets filtered out as shown below openshift.io/port1= 0

oc get node intel-perf-27.perf.eng.bos2.dc.redhat.com -o json | jq .status.allocatable
{
  "cpu": "127500m",
  "ephemeral-storage": "213881594729",
  "hugepages-1Gi": "0",
  "hugepages-2Mi": "0",
  "memory": "526694060Ki",
  "nvidia.com/gpu": "2",
  "openshift.io/port1": "0",
  "pods": "250",
  "rdma/rdma_shared_device_a": "63"
}

the only workaround I found is to edit the config map:
oc edit configmap -n openshift-sriov-network-operator device-plugin-config

and then removing:
"linkTypes":["Infiniband"],

from:

apiVersion: v1
data:
  intel-perf-27.perf.eng.bos2.dc.redhat.com: '{"resourceList":[{"resourceName":"port1","selectors":{"vendors":["15b3"],"pfNames":["ibs3f0"],"linkTypes":["infiniband"],"IsRdma":true,"NeedVhostNet":false},"SelectorObj":null},{"resourceName":"port2","selectors":{"vendors":["15b3"],"pfNames":["ibs3f1"],"linkTypes":["infiniband"],"IsRdma":true,"NeedVhostNet":false},"SelectorObj":null}]}'
  perf-intel-6.perf.eng.bos2.dc.redhat.com: '{"resourceList":[{"resourceName":"port1","selectors":{"vendors":["15b3"],"pfNames":["ibs3f0"],"linkTypes":["infiniband"],"IsRdma":true,"NeedVhostNet":false},"SelectorObj":null},{"resourceName":"port2","selectors":{"vendors":["15b3"],"pfNames":["ibs3f1"],"linkTypes":["infiniband"],"IsRdma":true,"NeedVhostNet":false},"SelectorObj":null}]}'
kind: ConfigMap

however it get resets every 300 seconds.

for reference:
NetworkAttachmentDefinition:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
annotations:
k8s.v1.cni.cncf.io/resourceName: openshift.io/port1
name: network-port-1
namespace: default
spec:
config: "{\n "cniVersion": "0.3.1",\n "name": "network-port-1",\n
\ "type": "ib-sriov",\n "logLevel": "info",\n "ipam": {\n
\ "type": "whereabouts",\n "range": "192.168.1.2/24",\n
\ "exclude": [\n "192.168.1.1",\n "192.168.1.2"
,\n "192.168.1.254",\n "192.168.1.255"\n ],\n
\ "routes": [\n {\n "dst": "192.168.1.0/24"
\n }\n ]\n }\n}"

SriovIBNetwork:

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovIBNetwork
metadata:
name: sriov-ib-network-port-1
namespace: openshift-sriov-network-operator
spec:
pfNames:

• ibs3f0
  rdma: true
  resourceName: port1

[zeeke]

hi @bbenshab. can you please attach sriov logs and resources? Since it is an openshift cluster, you can get them with:

oc adm must-gather -- /usr/bin/gather_sriov

Also, you stated this happens on 4.15.0, any chance you can reproduce this issue with the latest sriov-network-operator version?

[adrianchiris]

lets try to reproduce with sriov-network-operator from this repo.

also it may be related to : #797

[SchSeba]

Hi @bbenshab,

can you please with latest sriov-operator that contains #797?

[SchSeba]

Hi @bbenshab any update?

[bbenshab]

hey @SchSeba I'm sorry but I currently don't have an IB setup to recreate this issue, however it's easily reproducible when following the steps I mentioned above.