[Scale 10k arp] Traffic drop while arp is being learned

[DavidZagury]

A PR recently enabled by default Receive Packet Steering (RPS) - #20211
It caused a degradation that can be seen between 202405 and 202411 branches.
When reverting this PR or disabling RPS by removing the linux configuration this PR set (no user friendly way to do so) - the test pass.

Steps to reproduce

configure l2 and l3 interfaces

config vlan add 363
config vlan member add -u 363 Ethernet176
config interface ip add Vlan363 101.1.0.1/24
config interface ip add Ethernet160 100.1.0.1/16
config save -y

modify copp policy to raise rate limits

sed -i '/"default": {/,/}/{s/"cir": "[0-9]"/"cir": "60000"/; s/"cbs": "[0-9]"/"cbs": "60000"/}' /usr/share/sonic/templates/copp_cfg.j2
sed -i '/"queue4_group2": {/,/}/{s/"cir": "[0-9]"/"cir": "60000"/; s/"cbs": "[0-9]"/"cbs": "60000"/}' /usr/share/sonic/templates/copp_cfg.j2
reboot

modify gc_thresholds to higher values

sysctl -w net.ipv4.neigh.default.gc_thresh1=65535
sysctl -w net.ipv4.neigh.default.gc_thresh2=65535
sysctl -w net.ipv4.neigh.default.gc_thresh3=65535

clear arp table. start traffic with 10000 pps and 10000 end hosts

sonic-clear arp
Send bidirectional l3 traffic (simulating a single host pinging 10k hosts on other side)

Observed behavior

A small percentage of traffic (<1%) is lost.

Expected behavior

There should not be any losses.