diff --git a/.gitignore b/.gitignore index eea678d..31533fb 100644 --- a/.gitignore +++ b/.gitignore @@ -31,3 +31,6 @@ temp-prometheus.yaml .susql-deploy-info.txt .susql-deploy-info-last.txt c3-susql-config.yaml +nohup.out +.env +node_modules diff --git a/samples/README.md b/samples/README.md index cfe62b7..d9151bb 100644 --- a/samples/README.md +++ b/samples/README.md @@ -20,6 +20,9 @@ - `training-job-2.yaml` - `rhosaij.yaml` +## directories +- `impact-framework` - sample files for using SusQL with the [Green Software Impact Framework](https://if.greensoftware.foundation). + ## License diff --git a/samples/impact-framework/README.md b/samples/impact-framework/README.md new file mode 100644 index 0000000..7edd94e --- /dev/null +++ b/samples/impact-framework/README.md @@ -0,0 +1,10 @@ +# Using SusQL with the Green Software Foundation's Impact Framework + +A large design point in using SusQL data with the Green Software Foundation's +Impact Framework is whether to run in a container ON the cluster with SusQL, or run +on a machine external to the cluster. + +This directory contains two directories with experiments into both approaches: + +- [onClusterApproach](onClusterApproach) +- [offClusterApproach](offClusterApproach) diff --git a/samples/impact-framework/offClusterApproach/.trent/README.md b/samples/impact-framework/offClusterApproach/.trent/README.md new file mode 100644 index 0000000..86478a9 --- /dev/null +++ b/samples/impact-framework/offClusterApproach/.trent/README.md @@ -0,0 +1 @@ +experimentation... diff --git a/samples/impact-framework/offClusterApproach/.trent/htest.yaml b/samples/impact-framework/offClusterApproach/.trent/htest.yaml new file mode 100644 index 0000000..d847beb --- /dev/null +++ b/samples/impact-framework/offClusterApproach/.trent/htest.yaml @@ -0,0 +1,24 @@ +name: hello plugin test +description: simple verification of plugin tech +tags: null +initialize: + plugins: + 'hello-plugin': + method: HelloPlugin + path: 'https://github.com/trent-s/hello-plugin' + config: + allow-padding: true + outputs: + - yaml +tree: + children: + child: + pipeline: + - hello-plugin + inputs: + - timestamp: 2023-08-06T00:00 + duration: 3600 + carbon: 30 + - timestamp: 2023-09-06T00:00 + duration: 3600 + carbon: 30 diff --git a/samples/impact-framework/offClusterApproach/.trent/pi1/pi1test.yaml b/samples/impact-framework/offClusterApproach/.trent/pi1/pi1test.yaml new file mode 100644 index 0000000..7b78b86 --- /dev/null +++ b/samples/impact-framework/offClusterApproach/.trent/pi1/pi1test.yaml @@ -0,0 +1,28 @@ +name: prometheus importer test +description: simple susql demo invoking prometheus-importer plugin +tags: null +initialize: + plugins: + prometheus-importer: + method: PrometheusImporter + path: 'https://github.com/Shivani-G/prometheus-importer' + global-config: + step: '1h' + start: '2024-09-16T23:50:30.000Z' + end: '2024-10-16T00:00:30.000Z' + query: 'rate(susql_container_joules_total)[1m])' + metricLabels: + - name + metricName: 'energy' + defaultLabels: + duration: 3600 + cloud/vendor: 'ibm' + cloud/instance-type: 'medium' + outputs: + - yaml +tree: + children: + child: + pipeline: + - prometheus-importer + inputs: diff --git a/samples/impact-framework/offClusterApproach/.trent/pi1/steps.txt b/samples/impact-framework/offClusterApproach/.trent/pi1/steps.txt new file mode 100644 index 0000000..28ed6cd --- /dev/null +++ b/samples/impact-framework/offClusterApproach/.trent/pi1/steps.txt @@ -0,0 +1,20 @@ +# in this directory +npm init -y +npm install -g husky +npm install -g @grnsft/if +# npm remove -g "https://github.com/trent-s/prometheus-importer" +npm remove -g prometheus-importer +npm remove prometheus-importer +npm install -g "https://github.com/Shivani-G/prometheus-importer" +npm update +echo "BEARER_TOKEN="$(oc whoami -t) >.env +echo HOST=https://$(oc get routes -n openshift-monitoring thanos-querier -o jsonpath='{.status.ingress[0].host}') >>.env +if-run --manifest pi1test.yaml --debug + + + + +## troubleshootign +# npm cache verify +# npm install npm@latest -g + diff --git a/samples/impact-framework/offClusterApproach/README.md b/samples/impact-framework/offClusterApproach/README.md new file mode 100644 index 0000000..2748b4d --- /dev/null +++ b/samples/impact-framework/offClusterApproach/README.md @@ -0,0 +1,36 @@ +# Using SusQL with the Green Software Foundation's Impact Framework + +This particular experiment was performed using a recent RHEL 9 x86 machine, +however, any OS that supports a sufficiently recent version of `nodejs` should +work in principle. + +The key to importing data from SusQL is a Prometheus Importer Plugin for the GSF Impact Framework. +Currently there are two such plugins described in the (IF Explorer)[https://explorer.if.greensoftware.foundation]: +- (`Prometheus Importer`)[https://github.com/andreic94/if-prometheus-importer/blob/main/README.md] by `andreic94`, et al. +- (`prometheus-importer`)[https://github.com/Shivani-G/prometheus-importer/blob/main/README.md] by `Shibani-G`. + +The following is an approach using the later plugin: + +### steps +- Ensure that you are logged into your cluster and can use the `oc` (or `kubectl` command). +- Make sure that a recent version of `node` is installed. (This test used v22.9.0) +- Install Impact Framework + - `npm install -g @grnsft/if` +- Install Prometheus Importer + - `npm install -g "https://github.com/Shivani-G/prometheus-importer"` +- Update, just to be sure: (Starting from the directory that contains this README...) + - `cd test; npm update; cd -` + +- Create required credential file, and attempt to use prometheus-importer: (Starting from the directory that contains this README...) +``` +cd test +echo "BEARER_TOKEN="$(oc whoami -t) >.env +echo HOST=https://$(oc get routes -n openshift-monitoring thanos-querier -o jsonpath='{.status.ingress[0].host}') >>.env + +if-run --manifest ../ptest.yaml --debug +cd - +``` + + +Note: +- `oc` and `kubectl` are (should be) interchangable in this context. diff --git a/samples/impact-framework/offClusterApproach/ptest.yaml b/samples/impact-framework/offClusterApproach/ptest.yaml new file mode 100644 index 0000000..ef96335 --- /dev/null +++ b/samples/impact-framework/offClusterApproach/ptest.yaml @@ -0,0 +1,27 @@ +name: prometheus importer test +description: simple susql demo invoking prometheus-importer plugin +tags: null +initialize: + plugins: + prometheus-importer: + method: PrometheusImporter + path: 'https://github.com/Shivani-G/prometheus-importer' + config: + step: '1h' + start: '2024-09-16T23:50:30.000Z' + end: '2024-10-16T00:00:30.000Z' + query: 'rate(susql_container_joules_total)[1m])' + metricLabels: + - name + metricName: 'energy' + defaultLabels: + duration: 3600 + cloud/vendor: 'ibm' + outputs: + - yaml +tree: + children: + child: + pipeline: + - prometheus-importer + inputs: diff --git a/samples/impact-framework/offClusterApproach/test/package-lock.json b/samples/impact-framework/offClusterApproach/test/package-lock.json new file mode 100644 index 0000000..1ff56e9 --- /dev/null +++ b/samples/impact-framework/offClusterApproach/test/package-lock.json @@ -0,0 +1,124 @@ +{ + "name": "test", + "version": "1.0.0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "test", + "version": "1.0.0", + "license": "ISC", + "dependencies": { + "prometheus-importer": "github:Shivani-G/prometheus-importer" + } + }, + "node_modules/@grnsft/if-core": { + "version": "0.0.7", + "resolved": "https://registry.npmjs.org/@grnsft/if-core/-/if-core-0.0.7.tgz", + "integrity": "sha512-+4w8Sq1KRPDb+Jy638wgrTXlXIZzewOBceT+rAy3Oaov1M/veY3gu3AV15SXcPHrsBoFmZ6QeI9g1rF3RKB0ww==", + "license": "MIT", + "dependencies": { + "typescript": "^5.1.6" + }, + "engines": { + "node": ">=18", + "npm": ">=8" + } + }, + "node_modules/dotenv": { + "version": "16.3.1", + "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.3.1.tgz", + "integrity": "sha512-IPzF4w4/Rd94bA9imS68tZBaYyBWSCE47V1RGuMrB94iyTOIEwRmVL2x/4An+6mETpLrKJ5hQkB8W4kFAadeIQ==", + "license": "BSD-2-Clause", + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/motdotla/dotenv?sponsor=1" + } + }, + "node_modules/node-domexception": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/node-domexception/-/node-domexception-1.0.0.tgz", + "integrity": "sha512-/jKZoMpw0F8GRwl4/eLROPA3cfcXtLApP0QzLmUT/HuPCZWyB7IY9ZrMeKw2O/nFIqPQB3PVM9aYm0F312AXDQ==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/jimmywarting" + }, + { + "type": "github", + "url": "https://paypal.me/jimmywarting" + } + ], + "license": "MIT", + "engines": { + "node": ">=10.5.0" + } + }, + "node_modules/node-fetch-commonjs": { + "version": "3.3.2", + "resolved": "https://registry.npmjs.org/node-fetch-commonjs/-/node-fetch-commonjs-3.3.2.tgz", + "integrity": "sha512-VBlAiynj3VMLrotgwOS3OyECFxas5y7ltLcK4t41lMUZeaK15Ym4QRkqN0EQKAFL42q9i21EPKjzLUPfltR72A==", + "license": "MIT", + "dependencies": { + "node-domexception": "^1.0.0", + "web-streams-polyfill": "^3.0.3" + }, + "engines": { + "node": "^12.20.0 || ^14.13.1 || >=16.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/node-fetch" + } + }, + "node_modules/prometheus-importer": { + "version": "v0.1.1", + "resolved": "git+ssh://git@github.com/Shivani-G/prometheus-importer.git#861a0a7ef3e5f81ec2904838f0b6905a94686a0d", + "license": "MIT", + "dependencies": { + "@grnsft/if-core": "^0.0.7", + "dotenv": "16.3.1", + "node-fetch-commonjs": "^3.3.2", + "typescript": "^5.5.2", + "zod": "^3.23.8" + }, + "engines": { + "node": ">=18", + "npm": ">=8" + } + }, + "node_modules/typescript": { + "version": "5.6.3", + "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.6.3.tgz", + "integrity": "sha512-hjcS1mhfuyi4WW8IWtjP7brDrG2cuDZukyrYrSauoXGNgx0S7zceP07adYkJycEr56BOUTNPzbInooiN3fn1qw==", + "license": "Apache-2.0", + "bin": { + "tsc": "bin/tsc", + "tsserver": "bin/tsserver" + }, + "engines": { + "node": ">=14.17" + } + }, + "node_modules/web-streams-polyfill": { + "version": "3.3.3", + "resolved": "https://registry.npmjs.org/web-streams-polyfill/-/web-streams-polyfill-3.3.3.tgz", + "integrity": "sha512-d2JWLCivmZYTSIoge9MsgFCZrt571BikcWGYkjC1khllbTeDlGqZ2D8vD8E/lJa8WGWbb7Plm8/XJYV7IJHZZw==", + "license": "MIT", + "engines": { + "node": ">= 8" + } + }, + "node_modules/zod": { + "version": "3.23.8", + "resolved": "https://registry.npmjs.org/zod/-/zod-3.23.8.tgz", + "integrity": "sha512-XBx9AXhXktjUqnepgTiE5flcKIYWi/rme0Eaj+5Y0lftuGBq+jyRu/md4WnuxqgP1ubdpNCsYEYPxrzVHD8d6g==", + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/colinhacks" + } + } + } +} diff --git a/samples/impact-framework/offClusterApproach/test/package.json b/samples/impact-framework/offClusterApproach/test/package.json new file mode 100644 index 0000000..78e39f8 --- /dev/null +++ b/samples/impact-framework/offClusterApproach/test/package.json @@ -0,0 +1,15 @@ +{ + "name": "test", + "version": "1.0.0", + "main": "index.js", + "scripts": { + "test": "echo \"Error: no test specified\" && exit 1" + }, + "keywords": [], + "author": "", + "license": "ISC", + "description": "", + "dependencies": { + "prometheus-importer": "github:Shivani-G/prometheus-importer" + } +} diff --git a/samples/impact-framework/onClusterApproach/.trent/htest.yaml b/samples/impact-framework/onClusterApproach/.trent/htest.yaml new file mode 100644 index 0000000..d847beb --- /dev/null +++ b/samples/impact-framework/onClusterApproach/.trent/htest.yaml @@ -0,0 +1,24 @@ +name: hello plugin test +description: simple verification of plugin tech +tags: null +initialize: + plugins: + 'hello-plugin': + method: HelloPlugin + path: 'https://github.com/trent-s/hello-plugin' + config: + allow-padding: true + outputs: + - yaml +tree: + children: + child: + pipeline: + - hello-plugin + inputs: + - timestamp: 2023-08-06T00:00 + duration: 3600 + carbon: 30 + - timestamp: 2023-09-06T00:00 + duration: 3600 + carbon: 30 diff --git a/samples/impact-framework/onClusterApproach/Dockerfile b/samples/impact-framework/onClusterApproach/Dockerfile new file mode 100644 index 0000000..aa18a52 --- /dev/null +++ b/samples/impact-framework/onClusterApproach/Dockerfile @@ -0,0 +1,24 @@ +FROM ubuntu:latest + +ENV NPM_CONFIG_PREFIX=/usr/local +ARG NPM_CONFIG_PREFIX=/usr/local + +WORKDIR /work +COPY ?test.yaml . + +RUN apt-get update -y && \ +apt-get upgrade -y && \ +apt-get install git curl vim lsb-release -y && \ +echo " --- Finished apt-get install" && \ +# npm install -g npm@latest && \ +# echo " --- Finished npm update" && \ +curl -fsSL https://deb.nodesource.com/setup_lts.x | bash && \ +apt-get upgrade -y && \ +apt-get install nodejs -y && \ +npm install -g npm@10.9.0 && \ +echo " --- Finished node update" && \ +npm install -g typescript rimraf husky @grnsft/if && \ +echo " --- Finished if install" && \ +npm install -g "https://github.com/Shivani-G/prometheus-importer" && \ +echo " --- Finished PI install" && \ +chown -R 1000:1000 /work /usr/local/ diff --git a/samples/impact-framework/onClusterApproach/README.md b/samples/impact-framework/onClusterApproach/README.md new file mode 100644 index 0000000..e85bc87 --- /dev/null +++ b/samples/impact-framework/onClusterApproach/README.md @@ -0,0 +1,46 @@ +# Using SusQL with the Green Software Foundation's Impact Framework + +The following README explains an approach to utilize SusQL data in the +Green Software Foundation's Impact Framework using a slightly modified version +of the PrometheusImporter plugin. The fundamental focus of this particular +approach is to run entirely on Cluster, and is based on OpenShift. + +The key to importing data from SusQL is a Prometheus Importer Plugin for the GSF Impact Framework. +Currently there are two such plugins described in the (IF Explorer)[https://explorer.if.greensoftware.foundation]: +- (`Prometheus Importer`)[https://github.com/andreic94/if-prometheus-importer/blob/main/README.md] by `andreic94`, et al. +- (`prometheus-importer`)[https://github.com/Shivani-G/prometheus-importer/blob/main/README.md] by `Shibani-G`. + +This approach uses the later plugin. + +However, this is a work in progress related to an apparent limitation +in the use of certain required global`nodejs` functionality within a container. + +### Using prebuilt container image: +- Ensure that you are logged into your cluster, then start the Impact Framework container: `oc apply -f impact-framework.yaml` +- Log into the newly created container: `oc rsh impact-framework bash` +- Try simple unit test: +``` +cd if +npm run if-run -- --manifest manifests/examples/builtins/sum/success.yml +``` +- Use SusQL data: (Edit HOST and BEARER_TOKEN as necessary.) +``` +cd if +echo "BEARER_TOKEN="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token) >>.env +echo "HOST=https://thanos-querier.openshift-monitoring.svc.cluster.local:9091" >>.env +npm run if-run -- --manifest ../ptest.yaml +``` + +### Building your own container image +For those who wish to build their own Impact Framework, log into both your +cluster and your container image repository, and run the following commands: +``` +export IMG=//