Carbon2 (#120)

* more documentation Signed-off-by: Scott Trent <[email protected]> * spelling Signed-off-by: Scott Trent <[email protected]> * fix grammar Signed-off-by: Scott Trent <[email protected]> * doc formatting Signed-off-by: Scott Trent <[email protected]> * fix quotation Signed-off-by: Scott Trent <[email protected]> * use consistent capitalization of LabelGroup Signed-off-by: Scott Trent <[email protected]> * tag command name Signed-off-by: Scott Trent <[email protected]> * include OpenShift AI example Signed-off-by: Scott Trent <[email protected]> * fix typography Signed-off-by: Scott Trent <[email protected]> * test branch image push Signed-off-by: MohammedAbdi <[email protected]> Signed-off-by: Scott Trent <[email protected]> * static carbon intensity starting point Signed-off-by: Scott Trent <[email protected]> * base for carbon phase 1 Signed-off-by: Scott Trent <[email protected]> * clean up. add carbon metric Signed-off-by: Scott Trent <[email protected]> * clean up phase 1 Signed-off-by: Scott Trent <[email protected]> * new keywords Signed-off-by: Scott Trent <[email protected]> * carbon phase 2 config input Signed-off-by: Scott Trent <[email protected]> * experiment with version Signed-off-by: Scott Trent <[email protected]> * migrate to operator-sdk 1.36.1 Signed-off-by: Scott Trent <[email protected]> * Prometheus/smon now to be configured by user. Signed-off-by: Scott Trent <[email protected]> * Revert "Prometheus/smon now to be configured by user." This reverts commit 2200ab1. Signed-off-by: Scott Trent <[email protected]> * use carbonintensity for all phases Signed-off-by: Scott Trent <[email protected]> * carbon prep Signed-off-by: Scott Trent <[email protected]> * dynamic carbon support Signed-off-by: Scott Trent <[email protected]> * static carbon intensity starting point Signed-off-by: Scott Trent <[email protected]> * base for carbon phase 1 Signed-off-by: Scott Trent <[email protected]> * clean up. add carbon metric Signed-off-by: Scott Trent <[email protected]> * clean up phase 1 Signed-off-by: Scott Trent <[email protected]> * Prometheus/smon now to be configured by user. Signed-off-by: Scott Trent <[email protected]> * Revert "Prometheus/smon now to be configured by user." This reverts commit 2200ab1. Signed-off-by: Scott Trent <[email protected]> * use carbonintensity for all phases Signed-off-by: Scott Trent <[email protected]> * dynamic carbon support Signed-off-by: Scott Trent <[email protected]> * consistent use of static Signed-off-by: Scott Trent <[email protected]> --------- Signed-off-by: Scott Trent <[email protected]> Signed-off-by: MohammedAbdi <[email protected]> Signed-off-by: Scott Trent <[email protected]> Co-authored-by: MohammedAbdi <[email protected]>
sustainable-computing-io · Aug 28, 2024 · 911e3a0 · 911e3a0
1 parent d87c60a
commit 911e3a0
Show file tree

Hide file tree

Showing 21 changed files with 353 additions and 50 deletions.
diff --git a/README.md b/README.md
@@ -10,11 +10,22 @@ SusQL is an operator that can be deployed in a Kubernetes/OpenShift cluster. You
 
 ## Carbon Dioxide Emission Calculation
 
-CO2 emission calculation is currently a work in progress. At the moment we use
-a static conversion factor based on
+There are two currently CO2 emission calculation methods available, and an additional method is under development.
+- The default CO2 emission calculation method is called "static", and simply uses a grams of CO2 per Joule 
+of electricity consumed coefficient to calculate grams of CO2 emitted. This value is user tunable by
+modifying the `CARBON-INTENSITY` configmap value.  The default value is based on
 [US EPA](https://www.epa.gov/energy/greenhouse-gases-equivalencies-calculator-calculations-and-references)
-data. Although users can configure this value to match their runtime environment, we are working on providing
-automatic up to date conversion functionality.
+data.
+- The "simpledynamic" method periodically queries the carbon intensity value for a user specified location to provide a more accurate estimation of CO2 emission.
+The configmap user configurable items are:
+  - `CARBON-METHOD` - The "simpledynamic" method is enabled when set to `simpledynamic`.
+  - `CARBON-INTENSITY` - This value is set automatically. User specified values will be overwritten.
+  - `CARBON-INTENSITY-URL` - Specifies a web API that returns carbon intensity. The default value works as of the date of this writing.
+  - `CARBON-LOCATION` - The location identifiers are defined by the API provider, but can be selected by he user.
+  - `CARBON-QUERY-RATE` - Interval in seconds at which the carbon intensity is queried. The data available from the source is updated less than hourly, so an interval of greater than 3600 seconds is recommended.
+  - `CARBON-QUERY-FILTER` - When the return value is embedded in a JSON object, this specification enables the extraction of the data. The default value matches the default provider.
+  - `CARBON-QUERY-CONV-2J` - If the carbon data provider does not provide data in the standard "grams of CO2 per Joule" then, this factor can be specified to normalize the units displayed. The default value ensures tha the default provider data is in the correct unit.
+- The third method "sdk" is still under development, and will offer an integration with the Green Software Foundation's [Carbon Aware SDK](https://github.com/Green-Software-Foundation/carbon-aware-sdk).
 
 ### Prerequisites
 
@@ -27,7 +38,7 @@ Follow these instructions to install the SusQL Operator with Helm.
 
 Follow these instructions to install the SusQL Operator from [OperatorHub](https://operatorhub.io).
 - [Installation with OperatorHub](doc/operatorhub-installation.md)
-
+ 
 
 ## Using SusQL
 
@@ -74,7 +85,7 @@ Energy of the group of pods is exposed in 2 ways:
 
 ## License
 
-Copyright 2024.
+Copyright 2023, 2024.
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.

diff --git a/VERSION b/VERSION
@@ -1 +1 @@
-0.0.24
+0.0.25
diff --git a/bundle/manifests/susql-operator.clusterserviceversion.yaml b/bundle/manifests/susql-operator.clusterserviceversion.yaml
@@ -22,15 +22,15 @@ metadata:
       ]
     capabilities: Basic Install
     categories: Monitoring
-    containerImage: quay.io/sustainable_computing_io/susql_operator:0.0.24
-    createdAt: "2024-08-26T05:08:49Z"
+    containerImage: quay.io/sustainable_computing_io/susql_operator:0.0.25
+    createdAt: "2024-08-28T09:06:10Z"
     description: 'Aggregates energy and CO2 emission data for pods tagged with SusQL
       labels '
     operators.operatorframework.io/builder: operator-sdk-v1.36.1
     operators.operatorframework.io/project_layout: go.kubebuilder.io/v4
     repository: https://github.com/sustainable-computing-io/susql-operator
     support: https://github.com/sustainable-computing-io/susql-operator/issues
-  name: susql-operator.v0.0.24
+  name: susql-operator.v0.0.25
   namespace: placeholder
 spec:
   apiservicedefinitions: {}
@@ -248,13 +248,79 @@ spec:
                       key: SUSQL-LOG-LEVEL
                       name: susql-config
                       optional: true
-                - name: STATIC-CARBON-INTENSITY
+                - name: CARBON-METHOD
                   valueFrom:
                     configMapKeyRef:
-                      key: STATIC-CARBON-INTENSITY
+                      key: CARBON-METHOD
                       name: susql-config
                       optional: true
-                image: quay.io/sustainable_computing_io/susql_operator:0.0.24
+                - name: CARBON-INTENSITY
+                  valueFrom:
+                    configMapKeyRef:
+                      key: CARBON-INTENSITY
+                      name: susql-config
+                      optional: true
+                - name: CARBON-INTENSITY-URL
+                  valueFrom:
+                    configMapKeyRef:
+                      key: CARBON-INTENSITY-URL
+                      name: susql-config
+                      optional: true
+                - name: CARBON-LOCATION
+                  valueFrom:
+                    configMapKeyRef:
+                      key: CARBON-LOCATION
+                      name: susql-config
+                      optional: true
+                - name: CARBON-QUERY-RATE
+                  valueFrom:
+                    configMapKeyRef:
+                      key: CARBON-QUERY-RATE
+                      name: susql-config
+                      optional: true
+                - name: CARBON-QUERY-FILTER
+                  valueFrom:
+                    configMapKeyRef:
+                      key: CARBON-QUERY-FILTER
+                      name: susql-config
+                      optional: true
+                - name: CARBON-INTENSITY
+                  valueFrom:
+                    configMapKeyRef:
+                      key: CARBON-INTENSITY
+                      name: susql-config
+                      optional: true
+                - name: CARBON-INTENSITY-URL
+                  valueFrom:
+                    configMapKeyRef:
+                      key: CARBON-INTENSITY-URL
+                      name: susql-config
+                      optional: true
+                - name: CARBON-LOCATION
+                  valueFrom:
+                    configMapKeyRef:
+                      key: CARBON-LOCATION
+                      name: susql-config
+                      optional: true
+                - name: CARBON-QUERY-RATE
+                  valueFrom:
+                    configMapKeyRef:
+                      key: CARBON-QUERY-RATE
+                      name: susql-config
+                      optional: true
+                - name: CARBON-QUERY-FILTER
+                  valueFrom:
+                    configMapKeyRef:
+                      key: CARBON-QUERY-FILTER
+                      name: susql-config
+                      optional: true
+                - name: CARBON-QUERY-CONV-2J
+                  valueFrom:
+                    configMapKeyRef:
+                      key: CARBON-QUERY-CONV-2J
+                      name: susql-config
+                      optional: true
+                image: quay.io/sustainable_computing_io/susql_operator:0.0.25
                 imagePullPolicy: IfNotPresent
                 livenessProbe:
                   httpGet:
@@ -340,6 +406,8 @@ spec:
   - energy
   - kepler
   - susql
+  - carbon dioxide (CO2) emissions
+  - green computing
   links:
   - name: SusQL Operator
     url: https://github.com/sustainable-computing-io/susql-operator
@@ -355,4 +423,4 @@ spec:
   provider:
     name: SusQL Operator Contributors
     url: https://github.com/sustainable-computing-io/susql-operator
-  version: 0.0.24
+  version: 0.0.25
diff --git a/cmd/main.go b/cmd/main.go
@@ -69,8 +69,14 @@ func main() {
 	var susqlPrometheusDatabaseUrl string = "https://thanos-querier.openshift-monitoring.svc.cluster.local:9091"
 	var samplingRate string = "2"
 	var susqlLogLevel string = "-5"
-	// Static Carbon Intensity Factor in grams CO2 / Joule
-	var staticCarbonIntensity string = "0.00000000011583333"
+	// Carbon Intensity Factor in grams CO2 / Joule
+	var carbonMethod string = "static" // options: static, simpledynamic, sdk
+	var carbonIntensity string = "0.0001158333333333"
+	var carbonIntensityUrl string = "https://api.electricitymap.org/v3/carbon-intensity/latest?zone=%s"
+	var carbonLocation string = "JP-TK"
+	var carbonQueryRate string = "7200"
+	var carbonQueryFilter string = "carbonIntensity"
+	var carbonQueryConv2J string = "0.0000002777777778"
 
 	// NOTE: these can be set as env or flag, flag takes precedence over env
 	keplerPrometheusUrlEnv := getEnv("KEPLER-PROMETHEUS-URL", keplerPrometheusUrl)
@@ -80,7 +86,13 @@ func main() {
 	samplingRateEnv := getEnv("SAMPLING-RATE", samplingRate)
 	probeAddrEnv := getEnv("HEALTH-PROBE-BIND-ADDRESS", probeAddr)
 	susqlLogLevelEnv := getEnv("SUSQL-LOG-LEVEL", susqlLogLevel)
-	staticCarbonIntensityEnv := getEnv("STATIC-CARBON-INTENSITY", staticCarbonIntensity)
+	carbonMethodEnv := getEnv("CARBON-METHOD", carbonMethod)
+	carbonIntensityEnv := getEnv("CARBON-INTENSITY", carbonIntensity)
+	carbonIntensityUrlEnv := getEnv("CARBON-INTENSITY-URL", carbonIntensityUrl)
+	carbonLocationEnv := getEnv("CARBON-LOCATION", carbonLocation)
+	carbonQueryRateEnv := getEnv("CARBON-QUERY-RATE", carbonQueryRate)
+	carbonQueryFilterEnv := getEnv("CARBON-QUERY-FILTER", carbonQueryFilter)
+	carbonQueryConv2JEnv := getEnv("CARBON-QUERY-CONV-2J", carbonQueryConv2J)
 	enableLeaderElectionEnv, err := strconv.ParseBool(getEnv("LEADER-ELECT", strconv.FormatBool(enableLeaderElection)))
 	if err != nil {
 		enableLeaderElectionEnv = false
@@ -93,7 +105,13 @@ func main() {
 	flag.StringVar(&samplingRate, "sampling-rate", samplingRateEnv, "Sampling rate in seconds")
 	flag.StringVar(&probeAddr, "health-probe-bind-address", probeAddrEnv, "The address the probe endpoint binds to.")
 	flag.StringVar(&susqlLogLevel, "susql-log-level", susqlLogLevelEnv, "SusQL log level")
-	flag.StringVar(&staticCarbonIntensity, "static-carbon-intensity", staticCarbonIntensityEnv, "Static Carbon Intensity Factor in grams CO2 / Joule")
+	flag.StringVar(&carbonMethod, "carbon-method", carbonMethodEnv, "Method used to calculate CO2 emissions")
+	flag.StringVar(&carbonIntensity, "carbon-intensity", carbonIntensityEnv, "Carbon Intensity Factor in grams CO2 / Joule")
+	flag.StringVar(&carbonIntensityUrl, "carbon-intensity-url", carbonIntensityUrlEnv, "URL used to query calculate carbon intensity")
+	flag.StringVar(&carbonLocation, "carbon-location", carbonLocationEnv, "Location identfier used in carbon intensity query")
+	flag.StringVar(&carbonQueryRate, "carbon-query-rate", carbonQueryRateEnv, "How often to query carbon intensity query (seconds)")
+	flag.StringVar(&carbonQueryFilter, "carbon-query-filter", carbonQueryFilterEnv, "Parameter to extract carbon intensity from JSON returned by query")
+	flag.StringVar(&carbonQueryConv2J, "carbon-query-conv-2j", carbonQueryConv2JEnv, "Factor to convert carbon intensity returned by query to grams CO2 / Joule")
 	flag.BoolVar(&enableLeaderElection, "leader-elect", enableLeaderElectionEnv,
 		"Enable leader election for controller manager. "+
 			"Enabling this will ensure there is only one active controller manager.")
@@ -121,7 +139,13 @@ func main() {
 	susqlLog.Info("susqlPrometheusDatabaseUrl=" + susqlPrometheusDatabaseUrl)
 	susqlLog.Info("samplingRate=" + samplingRate)
 	susqlLog.Info("susqlLogLevel=" + susqlLogLevel)
-	susqlLog.Info("staticCarbonIntensity=" + staticCarbonIntensity)
+	susqlLog.Info("carbonMethod=" + carbonMethod)
+	susqlLog.Info("carbonIntensity=" + carbonIntensity)
+	susqlLog.Info("carbonIntensityUrl=" + carbonIntensityUrl)
+	susqlLog.Info("carbonLocation=" + carbonLocation)
+	susqlLog.Info("carbonQueryRate=" + carbonQueryRate)
+	susqlLog.Info("carbonQueryFilter=" + carbonQueryFilter)
+	susqlLog.Info("carbonQueryConv2J=" + carbonQueryConv2J)
 
 	// if the enable-http2 flag is false (the default), http/2 should be disabled
 	// due to its vulnerabilities. More specifically, disabling http/2 will
@@ -164,15 +188,30 @@ func main() {
 		os.Exit(1)
 	}
 
+	// TODO: verify that carbonMethod is an expected value. If not log warning and set to default value.
+	// (static, simpledynamic, sdk)
+	// Note: assume that carbonIntensityUrl, carbonLocation, and carbonQueryFilter are OK. If not, we will log errors at runtime.
+
 	samplingRateInteger, err := strconv.Atoi(samplingRate)
 	if err != nil {
 		samplingRateInteger = 2
 	}
 
-	staticCarbonIntensityFloat, err := strconv.ParseFloat(staticCarbonIntensity, 64)
+	carbonQueryRateInteger, err := strconv.ParseInt(carbonQueryRate, 10, 64)
+	if err != nil {
+		carbonQueryRateInteger = 7200
+	}
+
+	carbonIntensityFloat, err := strconv.ParseFloat(carbonIntensity, 64)
+	if err != nil {
+		susqlLog.Error(err, "Unable to obtain initial carbon intensity value. Using 0.0.")
+		carbonIntensityFloat = 0.0
+	}
+
+	carbonQueryConv2JFloat, err := strconv.ParseFloat(carbonQueryConv2J, 64)
 	if err != nil {
-		susqlLog.Error(err, "Unable to obtain static carbon intensity value. Using 0.0.")
-		staticCarbonIntensityFloat = 0.0
+		susqlLog.Error(err, "Unable to obtain carbon query Conv 2J value. Using 0.0.")
+		carbonQueryConv2JFloat = 0.0
 	}
 
 	susqlLog.Info("Setting up labelGroupReconciler.")
@@ -185,7 +224,14 @@ func main() {
 		SusQLPrometheusDatabaseUrl: susqlPrometheusDatabaseUrl,
 		SusQLPrometheusMetricsUrl:  susqlPrometheusMetricsUrl,
 		SamplingRate:               time.Duration(samplingRateInteger) * time.Second,
-		StaticCarbonIntensity:      staticCarbonIntensityFloat,
+		CarbonMethod:               carbonMethod,
+		CarbonIntensity:            carbonIntensityFloat,
+		CarbonIntensityUrl:         carbonIntensityUrl,
+		CarbonIntensityTimeStamp:   0,
+		CarbonLocation:             carbonLocation,
+		CarbonQueryRate:            carbonQueryRateInteger,
+		CarbonQueryFilter:          carbonQueryFilter,
+		CarbonQueryConv2J:          carbonQueryConv2JFloat,
 		Logger:                     susqlLog,
 	}).SetupWithManager(mgr); err != nil {
 		susqlLog.Error(err, "unable to create controller", "controller", "LabelGroup")

diff --git a/config/manager/kustomization.yaml b/config/manager/kustomization.yaml
@@ -5,4 +5,4 @@ kind: Kustomization
 images:
 - name: controller
   newName: quay.io/sustainable_computing_io/susql_operator
-  newTag: 0.0.24
+  newTag: 0.0.25
diff --git a/config/manager/manager.yaml b/config/manager/manager.yaml
@@ -117,11 +117,47 @@ spec:
                 name: susql-config
                 key: SUSQL-LOG-LEVEL
                 optional: true
-          - name: STATIC-CARBON-INTENSITY
+          - name: CARBON-METHOD
             valueFrom:
               configMapKeyRef:
                 name: susql-config
-                key: STATIC-CARBON-INTENSITY
+                key: CARBON-METHOD
+                optional: true
+          - name: CARBON-INTENSITY
+            valueFrom:
+              configMapKeyRef:
+                name: susql-config
+                key: CARBON-INTENSITY
+                optional: true
+          - name: CARBON-INTENSITY-URL
+            valueFrom:
+              configMapKeyRef:
+                name: susql-config
+                key: CARBON-INTENSITY-URL
+                optional: true
+          - name: CARBON-LOCATION
+            valueFrom:
+              configMapKeyRef:
+                name: susql-config
+                key: CARBON-LOCATION
+                optional: true
+          - name: CARBON-QUERY-RATE
+            valueFrom:
+              configMapKeyRef:
+                name: susql-config
+                key: CARBON-QUERY-RATE
+                optional: true
+          - name: CARBON-QUERY-FILTER
+            valueFrom:
+              configMapKeyRef:
+                name: susql-config
+                key: CARBON-QUERY-FILTER
+                optional: true
+          - name: CARBON-QUERY-CONV-2J
+            valueFrom:
+              configMapKeyRef:
+                name: susql-config
+                key: CARBON-QUERY-CONV-2J
                 optional: true
         image: '<OPERATOR_IMG>'
         imagePullPolicy: IfNotPresent

diff --git a/config/manifests/bases/susql-operator.clusterserviceversion.yaml b/config/manifests/bases/susql-operator.clusterserviceversion.yaml
@@ -97,6 +97,8 @@ spec:
   - energy
   - kepler
   - susql
+  - carbon dioxide (CO2) emissions
+  - green computing
   links:
   - name: SusQL Operator
     url: https://github.com/sustainable-computing-io/susql-operator