22.02.0

[cgrinds]

Releases

22.02.0 /

💥 Highlights of this major release include:

• Continued progress on the ONTAP REST config collector. Most of the template changes are in place and we're working on closing the gaps between ZAPI and REST. We've made lots of improvements to the REST collector and included 13 REST templates in this release. The REST collector should be considered early-access as we continue to improve it. If you try it out or have any feedback, let us know on Slack or GitHub. 📖 You can find more information about when you should switch from ZAPI to REST, what versions of ONTAP are supported by Harvest's REST collector, and how to fill ONTAP gaps between REST and ZAPI documented here

• Many of you asked for nightly builds. We have them. 🎊 We're also working on publishing to multiple Docker registries since you've told us you're running into rate-limiting problems with DockerHub. We'll announce here and Slack when we have a solution in place.

• Two new Data Protection dashboards

• bin/grafana cli should not overwrite dashboard changes, making it simpler to import/export dashboards, and enabling round-tripping dashboards (import, export, re-import)

• New include_contains plugin allows you to select a subset of objects. e.g. selecting only volumes with custom-defined ONTAP metadata

• We've included more out-of-the-box Prometheus alerts. Keep sharing your most useful alerts!

• 7mode workflows continue to be improved ❤️ Harvest now collects Qtree and Quotas counters from 7mode filers (these are already collected in cDOT)

• 28 bug fixes, 52 feature, and 11 documentation commits this release

IMPORTANT Admin node certificate file location changed. Certificate files have been consolidated into the cert directory. If you created self-signed admin certs, you need to move the admin-cert.pem and admin-key.pem files into the cert directory.

IMPORTANT In earlier versions of Harvest, the Qtree template exported the vserver metric. This counter was changed to svm to be consistent with other templates. If you are using the qtree vserver metric, you will need to update your queries to use svm instead.

IMPORTANT ‼️ After upgrade, don't forget to re-import your dashboards so you get all the new enhancements and fixes.
You can import via bin/harvest/grafana import cli or from the Grafana UI.

IMPORTANT The LabelAgent value_mapping plugin was deprecated in the 21.11 release and removed in 22.02.
Use LabelAgent value_to_num instead. See docs
for details.

Known Issues

The Unix collector is unable to monitor pollers running in containers. See #249 for details.

Enhancements

• Harvest should include a Data Protection dashboard that shows volumes protected by snapshots, which ones have exceeded their reserve copy, and which are unprotected feat: data protection dashboard - part1 - snapshots #664

• Harvest should include a Data Protection SnapMirror dashboard that shows which volumes are protected, how they're protected, their protection relationship, along with their health and lag durations.

• Harvest should provide nightly builds to GitHub and DockerHub feat: Nightly build publish #713

• Harvest bin/grafana cli should not overwrite dashboard changes, making it simpler to import/export dashboards, and enabling round-tripping dashboards (import, export, re-import) feat: revamp grafana import cli #831 Thanks to @luddite516 for reporting and @florianmulatz for iterating with us on a solution

• Harvest should provide a include_contains label agent plugin for filtering feat: include plugin label agent #735 Thanks to @chadpruden for reporting

• Improve Harvest's container compatibility with K8s via kompose. #655 See also and discussion

• The ZAPI cli tool should include counter types when querying ZAPIs feat: print counter type when querying zapi #663

• Harvest should include a richer set of Prometheus alerts Pre-defined rules for Prometheus alerting #254 Thanks @demalik for raising

• Template plugins should run in the order they are defined and compose better.
  The output of one plugin can be fed into the input of the next one. Predictable Harvest Plugin Ordering #736 Thanks to @chadpruden for raising

• Harvest should collect Antivirus counters when ONTAP offbox vscan is configured #346 Thanks to @burkl and @Falcon667 for reporting

• Document how to run Harvest with containerd and Rancher

• Qtree counters should be collected for 7-mode filers Qtree 7mode Template is not available in this repository #766 Thanks to @jmg011 for raising this issue and iterating with us on a solution

• Harvest admin node should work with pollers running in Docker compose #678

• Document how to run Harvest with Podman. Several RHEL customers asked about this since Podman ships as the default container runtime on RHEL8+.

• Harvest should include a Systemd service file for the HTTP service discovery admin node #656

• Document how ZAPI collectors, templates, and exporting work together. Thanks @jmg011 and others for asking for this

• Remove redundant dashboards (Network, Node, SVM, Volume) #703 Thanks to @mamoep for reporting this

• Harvest generate docker command should support customer-supplied Prometheus and Grafana ports. #584

• Harvest certificate authentication should work with self-signed subject alternative name (SAN) certificates. Improve documentation on how to use certificate authentication. Thanks to @edd1619 for raising this issue

• Harvest's Prometheus exporter should optionally sort labels. Without sorting, VictoriaMetrics marks metrics stale. Prometheus exporter - sort label order #756 Thanks to @mamoep for reporting and verifying

• Harvest should optionally log to a file when running in the foreground. Handy for instantiated instances running on OSes that have poor support for jounalctl feat: Harvest should support logging to file when running in foreground #813 and Support file logging for instantiated units #810 Thanks to @mamoep for reporting and verifying this works in a nightly build

• Harvest should collect workload concurrency #714

• Harvest certificate directory should be included in a container's volume mounts Cert directory needs to be included in container volume mounts #725

• MetroCluster dashboard should show path object metrics feat: add path object metrics into mcc dash #746

• Harvest should collect namespace resources from ONTAP feat: improve namespace monitoring #749

• Harvest should be more resilient to cluster connectivity issues PollData task Recovery handling is not correct #480

• Harvest Grafana dashboard version string should match the Harvest release Grafana dashboards should match release version #631

• REST collector improvements

  • Harvest REST collector should support ONTAP private cli endpoints Qtree 7mode Template is not available in this repository #766

  • REST collector should support ZAPI-like object prefixing feat: support zapi like prefixing object #786

  • REST collector should support computing new customer-defined metrics feat: rest gaps aggr, qtree, shelf #780

  • REST collector should collect aggregate, qtree and quota counters feat: rest gaps aggr, qtree, shelf #780

  • REST collector metrics should be reported in autosupport feat: asup rest #841

  • REST collector should collect sensor counters feat: rest sensor support for 9.10 and below #789

  • Collect network port interface information not available via ZAPI feature request: net-port-ifgrp-get-iter #691 Thanks to @pilot7777, @mamoep amd @wagneradrian92 for working on this with us

  • Publish REST collector document that highlights when you should switch from ZAPI to REST, what versions of ONTAP are supported by Harvest's REST collectors and how to fill ONTAP gaps between REST and ZAPI

  • REST collector should support Qutoa, Shelf, Snapmirror, and Volume plugins feat: quota,shelf rest plugin #799 and feat: snapmirror, volume plugin #811

• Improve troubleshooting and documentation on validating certificates from macOS #723

• Harvest should read its config information from the environment variable HARVEST_CONFIG when supplied. This env var has higher precedence than the --config command-line argument. feat: env var HARVEST_CONFIG has higher precedence than --config cm… #645

Fixes

• FlexGroup statistics should be aggregated across node and aggregates #706 Thanks to @wally007 for reporting

• Network Details dashboard should use correct units and support variable sorting #673 Thanks to @mamoep for reporting and reviewing the fix

• Harvest Systemd service should wait for network to start #707 Thanks to @mamoep for reporting and fixing

• MetroCluster dashboard should use correct units and support variable sorting #685 Thanks to @mamoep and @chris4789 for reporting this

• 7mode shelf plugin should handle cases where multiple channels have the same shelf id #692 Thanks to @pilot7777 for reporting this on Slack

• Improve template YAML parsing when indentation varies #704 Thanks to @mamoep for reporting this.

• Harvest should not include version information in its container name. #660. Thanks to @wally007 for raising this.

• Ignore missing Qtrees and improve uniqueness check on 7mode filers fix: check if qtree instance is found #782 and fix: handling vfiler for qtree 7mode #797. Thanks to @jmg011 for reporting

• Qtree instance key order should be unified between 7mode and cDOT fix: quota plugin qtreekey issue #807 Thanks to @jmg011 for reporting

• Workload detail volume collection should not try to create duplicate counters Error in workload_detail_volume #803 Thanks to @luddite516 for reporting

• Harvest HTTP service discovery node should not attempt to publish Prometheus metrics to InfluxDB #684

• Grafana import should save auth token to the config file referenced by HARVEST_CONFIG when that environnement variable exists #681

• bin/zapi should print output #715

• Snapmirror dashboard should show correct number of SVM-DR relationships, last transfer, and health status snapmirror dashboard bugs #728 Thanks to Gaël Cantarero on Slack for reporting

• Ensure that properties defined in object templates override their parent properties Fields(client_timeout, schedule, etc.) in child yaml are not taking precedence over default.yaml #765

• Increase time that metrics are retained in Prometheus exporter from 3 minutes to 5 minutes fix: remove custom schedule for snapshot and change default cachelimit #778

• Remove the misplaced SVM FCP Throughput panel from the iSCSI drilldown section of the SVM details dashboard SVM-Details dashboard -  #821 Thanks to @florianmulatz for reporting and fixing

• When importing Grafana dashboards, remove the existing id and uid so Grafana treats the import as a create instead of an overwrite fix: id issue in grafana import/export #825 Thanks to @luddite516 for reporting

• Relax the Grafana version check constraint so version 8.4.0-beta1 is considered >=7.1 Grafana latest (8.4.0-beta1) not recognized as >= 7.1 #828 Thanks to @ybizeul for reporting

• bin/harvest status should report running for pollers exporting to InfluxDB, instead of reporting that they are not running bin/harvest status incorrectly says poller is not running when using influx exporter #835

• Pin the Grafana and Prometheus versions in the Docker compose workflow instead of pulling latest fix: pin Grafana and Prometheus versions in compose workflow #822

---

This discussion was created from the release 22.02.0.

[cse-gh]

I'll start by admitting I don't understand containers very well, but following the upgrade instructions for the docker onecontainerperpoller installation, I've pulled the latest from docker hub, "docker pull rahulguptajss/harvest:22.02.0-4" and have also run "bin/harvest generate docker full --output harvest-compose.yml". When I start up the pollers again, they are still "v21.11.1" and the graphs in Grafana show as "21.11.1" also. I'm very likely doing something wrong here. Any help?

[cgrinds]

Hi @cse-gh

Maybe there's stale data. Can you double check the volumes section of your harvest-compose.yml? It should look something like this:

  u2:
    image: rahulguptajss/harvest:latest
    container_name: poller-u2
    restart: unless-stopped
    command: '--poller u2 --promPort 12990 --config /opt/harvest.yml'
    volumes:                          <====================================== interested in this section
      - /Users/cgrinds/opt/harvest/conf:/opt/harvest/conf
      - /Users/cgrinds/opt/harvest/cert:/opt/harvest/cert
      - /Users/cgrinds/opt/harvest/yml/harvest.openlab.yml:/opt/harvest.yml
    networks:
      - backend

Let's also double check the image - please paste the output of running

docker image ls -a
REPOSITORY                       TAG         IMAGE ID       CREATED         SIZE
rahulguptajss/harvest            22.02.0-4   80061bbe1c2c   5 hours ago     85.4MB
prom/prometheus                  v2.33.1     e528f02c45a6   13 days ago     204MB
grafana/grafana                  8.3.4       4a34578e4374   4 weeks ago     274MB
rancher/library-traefik          2.4.8       deaf4b1027ed   10 months ago   91.3MB
rancher/coredns-coredns          1.8.3       3885a5b7f138   11 months ago   43.5MB
rancher/local-path-provisioner   v0.0.19     148c19256271   14 months ago   42.4MB
rancher/klipper-lb               v0.2.0      465db341a9e5   15 months ago   6.1MB
rancher/metrics-server           v0.3.6      9dd718864ce6   2 years ago     39.9MB
rancher/pause                    3.1         da86e6ba6ca1   4 years ago     742kB

and the provisioned dashboards

cat grafana/dashboards/dashboard.yml
apiVersion: 1

providers:
  - name: 'Harvest cDOT'
    orgId: 1
    folder: 'Harvest main - cDOT'
    type: file
    disableDeletion: false
    editable: true
    allowUiUpdates: true
    options:
      path: /etc/grafana/provisioning/dashboards/cmode

[cse-gh]

Here we go...

cat grafana/dashboards/dashboard.yml
apiVersion: 1

providers:
  - name: 'Harvest cDOT'
    orgId: 1
    folder: 'Harvest 21.11 - cDOT'
    type: file
    disableDeletion: false
    editable: true
    allowUiUpdates: true
    options:
      path: /etc/grafana/provisioning/dashboards/cmode

docker image ls -a
REPOSITORY              TAG         IMAGE ID       CREATED        SIZE
rahulguptajss/harvest   22.02.0-4   80061bbe1c2c   5 hours ago    85.4MB
rahulguptajss/harvest   latest      80061bbe1c2c   5 hours ago    85.4MB
prom/prometheus         latest      a3d385fc29f9   8 weeks ago    201MB
grafana/grafana         latest      9b957e098315   2 months ago   275MB
rahulguptajss/harvest   <none>      a322d0d8a2d7   2 months ago   85MB

All pollers have this same config:

image: rahulguptajss/harvest:latest
    container_name: poller_u1sancl_v21.11.1
    restart: unless-stopped
    command: '--poller u1sancl --promPort 13000 --config /opt/harvest.yml'
    volumes:
      - type: bind
        source: /opt/netapp/harvest/conf
        target: /opt/harvest/conf
      - type: bind
        source: /opt/netapp/harvest/harvest.yml
        target: /opt/harvest.yml
    networks:
      - backend

[cgrinds]

thanks! I think I know what's going on. Is it possible you're running the docker compose commands shown above while the current working directory is version 21.11 of Harvest? I ask because the output of your grafana/dashboards/dashboard.yml seems to be coming from the 21.11 release directory instead of the 22.02 release directory.

Can you download 22.02 or cd into the directory that contains the 22.02 release and retry?

# stop containers
docker-compose -f prom-stack.yml -f harvest-compose.yml down

# restart once you've changed directory to 22.02 release
docker-compose -f prom-stack.yml -f harvest-compose.yml up -d --remove-orphans

[cse-gh]

Ok, thank you for this assistance! I see now why it didn't work, I didn't download the .tar.gz of the new version. I thought that was in the docker image. It all makes sense now. Up and running!