gNMIc is unable to scrape data on Port 9273

[arbabali999]

Hi,

I have successfully integrated Nokia and Juniper Routers with gnmic and successfully visualizing telemetry stats on Grafana. But with Huawei, i am unable to scrap data on port 9273 for Prometheus, hence, i am not able to visualize it on Grafana.

gnmic.yml:

10.1.1.1:
username: admin
password: admin
subscriptions:
- dplc_interface_stats

subscriptions:
dplc_interface_stats:
paths:
- "openconfig-interfaces:interfaces/interface[name=100GE1/0/2]/state/counters"
mode: stream
stream-mode: sample
sample-interval: 5s

Result:

sudo gnmic -a 10.1.1.1:57400 --insecure -u admin -p admin -d get --path openconfig-interfaces:interfaces/interface[name=100GE1/0/2]/state/counters

2024/10/10 11:11:49.187280 /home/runner/go/pkg/mod/google.golang.org/[email protected]/grpclog/logger.go:53: [gnmic] [core] [Channel #1] Channel Connectivity change to READY
[
{
"source": "10.1.1.1:57400",
"timestamp": 1728540709375863000,
"time": "2024-10-10T11:11:49.375863+05:00",
"updates": [
{
"Path": "openconfig-interfaces:interfaces/interface[name=100GE1/0/2]/state/counters",
"values": {
"openconfig-interfaces:interfaces/interface/state/counters": {
"in-broadcast-pkts": "3753",
"in-discards": "0",
"in-errors": "2252",
"in-fcs-errors": "1595",
"in-multicast-pkts": "12466908",
"in-octets": "8740684804016284",
"in-unicast-pkts": "17446988040687",
"in-unknown-protos": "0",
"last-clear": "1724223418253000000",
"out-broadcast-pkts": "3396",
"out-discards": "28727127",
"out-errors": "0",
"out-multicast-pkts": "15091273",
"out-octets": "23302406015041723",
"out-unicast-pkts": "20895639374896"
}
}
}
]
}
]

Here i can see the statistics but they are not scrapping on port 9273 i.e., gnmic.:9273/metrics where Nokia and Juniper stats are successfully exposing. Refer below snapshot.

Note: I have hide the device IPs that are scrapping data successfully for security reasons.

Can you please assist in resolving the issue.

[hellt]

your path in the gnmic config differs from the one you use in the cli example

Try removing the leading forward slash:

paths:
- "openconfig-interfaces:/interfaces/interface[name=100GE1/0/2]/state/counters"

[arbabali999]

That doesn't seem to be the issue, but I replicated the it regardless, and the problem still persists.

[pperesbr]

Hello everyone!!!

I'm also having problems getting the information collection to work correctly with Huawei equipment.

Based on the tests I've done, using the openconfig models doesn't work, so I downloaded the models that Huawei provides from the repository: https://github.com/Huawei/yang.

To do this collection, I managed to use their model, which is from the file: huawei-ifm.yang, as follows:

(.venv) g0004830@p-br-pr-cta-gt2-mk8s-01:~/huawei-yang/yang/network-router/8.22.0/ne40e-x8x16$ pyang -f tree huawei-ifm.yang
module: huawei-ifm
+--rw ifm
+--rw global
| +--rw statistic-interval? uint32
| +--rw ipv4-ignore-primary-sub? boolean
| +--rw sub-interface-link-trap-enable? boolean
| +--rw ipv4-conflict-enable!
| | +--rw preempt-enable? boolean
| +--rw ipv6-conflict-enable!
| +--rw preempt-enable? boolean
+--rw damp!

gnmic --timeout 60s -a 10.10.10.10:57400 -u admin -p admin --insecure get --path huawei-ifm:ifm/interfaces/interface[name=GigabitEthernet0/3/6]

However, it is not very complete, like the IP counters part is not included.

I don't really understand how YANG models work when they extend from another model using the augment option. There is another yang file in this same repository which is: huawei-ip-statistics.yang

Which has the following content:

module: huawei-ip-statistics

augment /ifm:ifm/ifm:interfaces/ifm:interface/ip:ipv4/ip:state:
+--ro statistics
+--ro in-receives? uint64
+--ro in-octets? uint64
+--ro in-rate-pkts? uint64
+--ro in-rate-bits? uint64
+--ro in-rate-byte-bandwidth-usage? string
+--ro in-ucast-pkts? uint64
+--ro in-ucast-octets? uint64
+--ro in-mcast-pkts? uint64 +--ro in-mcast-octets? uint64 +--ro in-bcast-pkts? uint64 +--ro in-bcast-octets? uint64 +--ro out-ucast-pkts? uint64 +--ro out-ucast-octets? uint64 +--ro out-mcast-pkts? uint64 +--ro out-mcast-octets? uint64 +--ro out-bcast-pkts? uint64 +--ro out-bcast-octets? uint64 +--ro out-rate-pkts? uint64 +--ro out-rate-bits? uint64 +--ro out-rate-byte-bandwidth-usage? string +--ro out-transmits? uint64 +--ro out-octets? uint64 +--ro out-frag-requires? uint64 +--ro out-frag-fails? uint32 +--ro out-frag-oks? uint64 +--ro out-frag-creates? uint64 augment /ifm:ifm/ifm:interfaces/ifm:interface/ip:ipv6/ip:state: +--ro statistics +--ro in-receives? uint64 +--ro in-octets? uint64 +--ro in-rate-pkts? uint64 +--ro in-rate-bits? uint64 +--ro in-rate-byte-bandwidth-usage? string +--ro in-ucast-pkts? uint64 +--ro in-ucast-octets? uint64 +--ro in-mcast-pkts? uint64 +--ro in-mcast-octets? uint64 +--ro out-ucast-pkts? uint64 +--ro out-ucast-octets? uint64 +--ro out-mcast-pkts? uint64 +--ro out-mcast-octets? uint64 +--ro out-rate-pkts? uint64 +--ro out-rate-bits? uint64 +--ro out-rate-byte-bandwidth-usage? string +--ro out-transmits? uint64 +--ro out-octets? uint64

I understand that it is extending what comes from huawei-ifm.yang, but what I couldn't do was define how the path is passed to query what is in the IP statistics, that is, how to use a model with augment.

I used gnmic generate but the path it provided me didn't work, example (path start with ifm example, but I need pass huawei-ifm before like yang file):

/ifm/interfaces/interface[name=]/ipv4/state/statistics/in-bcast-octets /ifm/interfaces/interface[name=]/ipv4/state/statistics/in-bcast-pkts /ifm/interfaces/interface[name=]/ipv4/state/statistics/in-mcast-octets /ifm/interfaces/interface[name=]/ipv 4/state/statistics/in-mcast-pkts /ifm/interfaces/interface[name=]/ipv4/state/statistics/in-octets / ifm/interfaces/interface[name=]/ipv4/state/statistics/in-rate-bits /ifm/interfaces/interface[name=]/ipv4/state/statistics/in-rate-byte-bandwidth-usage /ifm/interfaces/interface[name=]/ipv4/state/statistics/in-rate-pkts /ifm/interfaces/interface[name=]/ipv4 /state/statistics/in-receives /ifm/interfaces/interface[name=]/ipv4/state/statistics/in-ucast-octets

In summary I don't set path using a file with augment, the other files with no augment from Huawei working.

[pperesbr]

I got api from huawei... this example with rpc, how to convert to gnmic get path:

<?xml version="1.0" encoding="utf-8"?>
<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="">
  <get>
    <filter type="subtree">
      <ifm:ifm xmlns:ifm="urn:huawei:yang:huawei-ifm"
               xmlns:ip-stats="urn:huawei:yang:huawei-ip-statistics"
               xmlns:ip="urn:huawei:yang:huawei-ip">
        <ifm:interfaces>
          <ifm:interface>
            <ip:ipv4>
              <ip:state>
                <ip-stats:statistics/>
              </ip:state>
            </ip:ipv4>
          </ifm:interface>
        </ifm:interfaces>
      </ifm:ifm>
    </filter>
  </get>
</rpc>

Tks,

[pperesbr]

guys,

I managed to advance a little, I got to the huawei-ip module which is an augment of huawei-ifm, doing it like this:

gnmic --timeout 60s -a 10.10.10.10:57400 -u admin -p admin --insecure get --path huawei-ifm:ifm/interfaces/interface[name=GigabitEthernet0/3/6]/huawei-ip:ipv4/state/

But, this module huawei-ip-statistics not work, this status is:

gnmic --timeout 60s -a 10.10.10.10: 57400 -u admin -p admin --insecure get --path huawei-ifm:ifm/interfaces/interface[name=GigabitEthernet0/3/6]/huawei-ip:ipv4/state/huawei-ip-statistics:statistics/in-receives
target "200.204.204.13:57400" Get request failed: "200.204.204.13:57400" GetRequest failed: rpc error: code = NotFound desc = Not found.

[pperesbr]

Hello,

After several hours trying to understand and solve the problem, I would like to share here how I managed to do it. The path only works when the device's interface configuration has statistics enabled and in forward mode, which is not standard on Huawei, in fact, something that only Huawei does... lol.

Here is the interface configuration and the command to get the counters for both IPv4 and IPv6:

interface GigabitEthernet0/3/6
mtu 9100
undo shutdown
set flow-stat interval 10
control-flap
damp-interface enable
damp-interface level manual 20 2000 750 40
ipv6 enable
ip address x.x.x.x 255.255.255.254
ipv6 address x.x.x.x.x./127
ipv6 mtu 9100
trust upstream default
isis enable 1
isis ipv6 enable 1
isis circuit-type p2p
isis circuit-level level-2
isis authentication-mode keychain isis-pwd
isis ipv6 cost 600
isis cost 600
isis timer lsp-throttle 33
isis padding-hello
isis ldp-sync
isis timer ldp-sync hold-down 10
isis bfd enable
isis bfd min-tx-interval 100 min-rx-interval 100
isis suppress-reachability
isis ipv6 suppress-reachability
mpls
mpls mtu 9088
mpls ldp
mpls ldp timer hello-send 5
mpls ldp timer keepalive-hold 30
mpls ldp timer keepalive-send 10
undo dcn
port-queue-template QOS-TLF outbound
poison-reverse enable
statistic enable << default is not enabled
statistic mode forward << default is mac
mpls poison-reverse enable
clock synchronization enable
clock priority 69
clock bundle 1
clock ssm prc
ptp notslave disable
ptp enable

return

Huawei's doc.

The statistic mode forward command enables a device's all main physical interfaces and logical interfaces to collect traffic statistics based on IPv4 and IPv6 forwarding.

The undo statistic mode forward command restores the default traffic statistics collection mode.

By default, an interface collects traffic statistics based on the destination MAC address in packets.

gnmic --timeout 60s -a 10.10.10.10:57400 -u admin -p admin--insecure subscribe --path huawei-ifm:ifm/interfaces/interface[name=GigabitEthernet0/3/6]/huawei-ip:ipv6/state/huawei-ip-statistics:statistics --stream-mode sample
{
"source": "200.204.204.13:57400",
"subscription-name": "default-1730858018",
"timestamp": 1730858031666000000,
"time": "2024-11-06T01:53:51.666Z",
"updates": [
{
"Path": "huawei-ifm:ifm/interfaces/interface[name=GigabitEthernet0/3/6]/huawei-ip:ipv6/state/huawei-ip-statistics:statistics",
"values": {
"huawei-ifm:ifm/interfaces/interface/huawei-ip:ipv6/state/huawei-ip-statistics:statistics": {
"in-mcast-octets": 0,
"in-mcast-pkts": 0,
"in-octets": 5413024333,
"in-rate-bits": 56986872,
"in-rate-byte-bandwidth-usage": "0.56%",
"in-rate-pkts": 5646,
"in-receives": 4192815,
"in-ucast-octets": 5413024333,
"in-ucast-pkts": 4192815,
"out-mcast-octets": 0,
"out-mcast-pkts": 0,
"out-octets": 227212866,
"out-rate-bits": 12645488,
"out-rate-byte-bandwidth-usage": "0.12%",
"out-rate-pkts": 2537,
"out-transmits": 803007,
"out-ucast-octets": 227212866,
"out-ucast-pkts": 803007
}
}
}
],
"deletes": [
""
]
}
{
"sync-response": true
}

@arbabali999 I added you to my LinkedIn, if you want to exchange ideas about monitoring Huawei equipment, just call.

[arbabali999]

Thank you for your help. Unfortunately, my Huawei CE8850 device lacks a command specifically for "statistic mode forwarding." I did enable "statistic enable," and I can view the statistics using the get command. However, when I add the subscription in gnmic.yml, the statistics briefly appear on gnmic:9273/metrics. After refreshing the page, though, the data stops reflecting within a minute.

sudo gnmic -a 10.1.1.1:57400 --insecure -u admin -p admin -d get --path huawei-ifm:ifm/interfaces/interface[ifName=Eth-Trunk5]
2024/11/06 17:21:41.200149 /home/runner/go/pkg/mod/google.golang.org/[email protected]/grpclog/logger.go:53: [gnmic] [core] [Channel #1] Channel Connectivity change to READY
[
{
"source": "10.1.1.1:57400",
"timestamp": 1730895701591116000,
"time": "2024-11-06T17:21:41.591116+05:00",
"updates": [
{
"Path": "huawei-ifm:ifm/interfaces/interface[ifName=Eth-Trunk5]",
"values": {
"huawei-ifm:ifm/interfaces/interface": {
"ifAdminStatus": "up",
"ifCeIpStatiEnable": {
"ifIpStatiEnable": "Disable",
"ifIpStatiIpv4Enable": "Disable",
"ifIpStatiIpv6Enable": "Disable"
},
"ifClass": "mainInterface",
"ifClearedStat": {
"ifOperStatItvl": 300,
"inBitRate": "856118334",
"inByteRate": "107014791",
"inPacketRate": "220038",
"inUseRate": "0.85%",
"outBitRate": "3290595639",
"outByteRate": "411324454",
"outPacketRate": "382442",
"outUseRate": "3.29%",
"rcvBroadPacket": "898",
"rcvBroadPacketRate": "0",
"rcvDropPacket": "0",
"rcvErrorPacket": "1843",
"rcvMutiPacket": "29068506",
"rcvMutiPacketRate": "5",
"rcvUniPacket": "1446641410941",
"rcvUniPacketRate": "213124",
"receiveByte": "1013787103317884",
"receivePacket": "1538720216260",
"sendBroadPacket": "2441",
"sendBroadPacketRate": "0",
"sendByte": "3433702400167292",
"sendDropPacket": "0",
"sendErrorPacket": "0",
"sendMutiPacket": "23636849",
"sendMutiPacketRate": "7",
"sendPacket": "2796174853321",
"sendUniPacket": "2732612965330",
"sendUniPacketRate": "367698"
},
"ifControlFlap": {
"ifCtrlFlapEnbl": false
},
"ifDescr": "*** connected to testing link ***",
"ifDf": false,
"ifDynamicInfo": {
"ifCtrlFlapDamp": false,
"ifLinkStatus": "up",
"ifOperMac": "e00c-e5bf-29a1",
"ifOperSpeed": "100000000000",
"ifOperStatus": "up",
"ifOpertMTU": 1500,
"ifPhyStatus": "up",
"ifV4State": "up",
"ifV6State": "down",
"isOffline": false,
"lineProtocolUpTime": "2024-11-05T03:48:31Z"
},
"ifErrorDown": {
"errorDownCause": "invalid",
"recoveryTime": 0,
"remainderTime": 0
},
"ifIndex": 76,
"ifLinkProtocol": "ethernet",
"ifMtu": 1500,
"ifName": "Eth-Trunk5",
"ifNetworkLayerStatus": "ipv4andipv6up",
"ifNumber": "5",
"ifPhyType": "Eth-Trunk",
"ifRouterType": "Broadcast",
"ifServiceType": "None",
"ifStatistics": {
"in-realtime-bit-rate": "0",
"out-realtime-bit-rate": "0",
"rcvBroadPacket": "898",
"rcvDropPacket": "0",
"rcvErrorPacket": "1843",
"rcvMutiPacket": "29068493",
"rcvUniPacket": "1446640468161",
"receiveByte": "1013786975430492",
"receivePacket": "1538719970988",
"sendBroadPacket": "2441",
"sendByte": "3433701965810814",
"sendDropPacket": "0",
"sendErrorPacket": "0",
"sendMutiPacket": "23636839",
"sendPacket": "2796174440593",
"sendUniPacket": "2732611345111"
},
"ifTrapEnable": true,
"ipv4Config": {
"addrCfgType": "config"
},
"ipv6Config": {
"autoLinkLocal": false,
"enableFlag": false
},
"isL2SwitchPort": true,
"l2SubIfFlag": false,
"vrfName": "public",
"vsName": "Admin-VS"
}
}
}
]
}
]

[arbabali999]

PFB the problem i am having.

The stats briefly appear on the port that gNMIc exposes for Prometheus to retrieve data, but they stop displaying shortly afterward.

After a few moments, it stops displaying any data.