Multi-tenant support

[cgrinds]

See also

• Need documentation for --labels #1991
• Add multi-tenant SVM support #355

[slater0013]

Thanks for creating this !

#355 is more oriented on how to reduce harvested data to a specific tenant/SVM ! Which would solve the multi-tenant question but It also needs to multiply harvest agents and associated prometheus Databases.

In this configuration, many dashboards would be useless as they are related to cluster-wide metrics, out of the SVM-context.

In our context, client ABC can buy storage on our Netapp Clusters. (eg : SVM_ABC on Cluster1 for production storage, SVMs_ABC on Cluster 2 for backup and SVMm_ABC on Cluster 3 for metrocluster critical mirrored storage)

Our netapp context :

• 5 Netapp clusters
• 90+ SVMs

My idea, at first :

• 1 harvest instance
• 1 prometheus database
• 1 grafana server

Then, each tenant could have :

• his own Grafana Organization
• his own Grafana User Account (Viewer Role)
• access to pre-configured dashboards restricted to his own SVM(s)

Grafana import tool is almost complete, It would just need an extra parameter to give more control on dashboards variables, example :

Flags:
  --svm-variable-regex          SVM variable regex to filter SVM query results

This regex would be added on-the-fly to the JSON dashboards SVM variable, example with :

--svm-variable-regex "/^SVM[a-z]_ABC$/"

{
        "allValue": null,
        "current": {},
        "datasource": "${DS_PROMETHEUS}",
        "definition": "label_values(volume_labels{cluster=~\"$Cluster\"}, svm)",
        "description": null,
        "error": null,
        "hide": 0,
        "includeAll": true,
        "label": null,
        "multi": true,
        "name": "SVM",
        "options": [],
        "query": {
          "query": "label_values(volume_labels{cluster=~\"$Cluster\"}, svm)",
          "refId": "StandardVariableQuery"
        },
        "refresh": 1,
        "regex": "/^SVM[a-z]_ABC$/",
        "skipUrlSync": false,
        "sort": 1,
        "tagValuesQuery": "",
        "tagsQuery": "",
        "type": "query",
        "useTags": false
}

Could be replicated for DC and Cluster variables.

One "problem" with this solution : The filtering is "virtual" and grafana experienced user could easily handcraft API Calls to bypass It ! Not a big deal in our context, since the accessed metrics are not critical.

Hope It's clear 😃
GS.

[cgrinds]

Thanks @slater0013! Appreciate the details, we'll discuss it and get back to you

[rahulguptajss]

@slater0013 This is implemented via #2061. More details here