Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add otel exporter option to override metric_reporting_period #50

Merged
merged 2 commits into from
Mar 10, 2025

Conversation

dhontecillas
Copy link
Contributor

@dhontecillas dhontecillas commented Feb 27, 2025

In case we have multiple OTEL metric exporters, we might want to tweak the reporting period for each one of them. (For example, if one of those is reporting to our on premises instance with a lower retention period, and another is sent to an external provider with longer retention period, and might be some rate limit).

With this PR, we add a custom_metric_reporting_period to the OTEL exporters config (docs about it here: https://www.krakend.io/docs/telemetry/opentelemetry/#exporters-otlp )

custom_metric_reporting_period : unsigned integer specifying the seconds to wait between sending report, 0 will not override the metric_reporting_period set for all metric exporters (https://www.krakend.io/docs/telemetry/opentelemetry/#metric_reporting_period)

In order to easily check the reporting periods, I've created two otel exporters with addresses pointing to non-existing services, so we can see an error in the log, each time it tries to report the metrics:

  • one at localhost:5005 that should be reporting each 5 seconds (global configuration)
  • one at localhost:5011 that should be reporting each 11 seconds (overriden configuration)
{
  "$schema": "https://www.krakend.io/schema/v3.json",
  "version": 3,
  "name": "My first API Gateway - KrakenD",
  "port": 8081,
  "host": ["http://my_service:8080"],
  "debug_endpoint": true,
  "extra_config": { 
    "telemetry/opentelemetry": {
      "service_name": "krakend_prometheus_service",
      "metric_reporting_period": 5,
      "trace_sample_rate": 1,
      "exporters": {
        "otlp": [
          {
            "disable_metrics": false,
            "disable_traces": true,
            "host": "localhost",
            "name": "five_seconds",
            "port": 5005,
            "use_http": true
          },
          {
            "disable_metrics": false,
            "disable_traces": true,
            "host": "localhost",
            "name": "eleven_seconds",
            "port": 5011,
            "use_http": true,
            "custom_metric_reporting_period": 11,
            "custom_reporting_period": 11
          }
        ]
      },
      "layers": {
        "global": {
          "disable_metrics": false,
          "disable_propagation": false,
          "disable_traces": false,
          "report_headers": true
        },
        "proxy": {
          "disable_metrics": false,
          "disable_traces": true,
          "report_headers": true
        },
        "backend": {
          "metrics": {
            "detailed_connection": true,
            "disable_stage": false,
            "read_payload": false,
            "round_trip": false
          },
          "traces": {
            "detailed_connection": false,
            "disable_stage": false,
            "read_payload": false,
            "report_headers": false,
            "round_trip": false
          }
        }
      }
    }
  },
  "endpoints": [
    {
      "endpoint": "/dashboard",
      "backend": [
        {
          "url_pattern": "/customer.json"
        },
        {
          "url_pattern": "/sales.json"
        }
      ]
    }
  ]
}
2025/03/03 09:58:56 KRAKEND ERROR: [SERVICE: OpenTelemetry] failed to upload metrics: Post "https://localhost:5005/v1/metrics": dial tcp 127.0.0.1:5005: connect: connection refused
2025/03/03 09:59:01 KRAKEND ERROR: [SERVICE: OpenTelemetry] failed to upload metrics: Post "https://localhost:5005/v1/metrics": dial tcp 127.0.0.1:5005: connect: connection refused
2025/03/03 09:59:02 KRAKEND ERROR: [SERVICE: OpenTelemetry] failed to upload metrics: Post "https://localhost:5011/v1/metrics": dial tcp 127.0.0.1:5011: connect: connection refused
2025/03/03 09:59:06 KRAKEND ERROR: [SERVICE: OpenTelemetry] failed to upload metrics: Post "https://localhost:5005/v1/metrics": dial tcp 127.0.0.1:5005: connect: connection refused
2025/03/03 09:59:11 KRAKEND ERROR: [SERVICE: OpenTelemetry] failed to upload metrics: Post "https://localhost:5005/v1/metrics": dial tcp 127.0.0.1:5005: connect: connection refused
2025/03/03 09:59:13 KRAKEND ERROR: [SERVICE: OpenTelemetry] failed to upload metrics: Post "https://localhost:5011/v1/metrics": dial tcp 127.0.0.1:5011: connect: connection refused
2025/03/03 09:59:16 KRAKEND ERROR: [SERVICE: OpenTelemetry] failed to upload metrics: Post "https://localhost:5005/v1/metrics": dial tcp 127.0.0.1:5005: connect: connection refused
2025/03/03 09:59:21 KRAKEND ERROR: [SERVICE: OpenTelemetry] failed to upload metrics: Post "https://localhost:5005/v1/metrics": dial tcp 127.0.0.1:5005: connect: connection refused
2025/03/03 09:59:24 KRAKEND ERROR: [SERVICE: OpenTelemetry] failed to upload metrics: Post "https://localhost:5011/v1/metrics": dial tcp 127.0.0.1:5011: connect: connection refused
2025/03/03 09:59:26 KRAKEND ERROR: [SERVICE: OpenTelemetry] failed to upload metrics: Post "https://localhost:5005/v1/metrics": dial tcp 127.0.0.1:5005: connect: connection refused
2025/03/03 09:59:31 KRAKEND ERROR: [SERVICE: OpenTelemetry] failed to upload metrics: Post "https://localhost:5005/v1/metrics": dial tcp 127.0.0.1:5005: connect: connection refused
2025/03/03 09:59:35 KRAKEND ERROR: [SERVICE: OpenTelemetry] failed to upload metrics: Post "https://localhost:5011/v1/metrics": dial tcp 127.0.0.1:5011: connect: connection refused
2025/03/03 09:59:36 KRAKEND ERROR: [SERVICE: OpenTelemetry] failed to upload metrics: Post "https://localhost:5005/v1/metrics": dial tcp 127.0.0.1:5005: connect: connection refused

@dhontecillas dhontecillas marked this pull request as draft February 27, 2025 16:10
@dhontecillas dhontecillas changed the title add otel exporter option to override the global setting for the metri… add otel exporter option to override metric_reporting_period Feb 27, 2025
@dhontecillas dhontecillas marked this pull request as ready for review March 3, 2025 13:27
@dhontecillas dhontecillas requested review from thedae, kpacha and taik0 March 3, 2025 13:28
@dhontecillas dhontecillas merged commit db42334 into main Mar 10, 2025
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants