Prometheus remote write proxy which marks timeseries with a Cortex tenant ID based on labels.
Cortex tenants (separate namespaces where metrics are stored to and queried from) are identified by a X-Scope-OrgID
HTTP header on both writes and queries.
This makes it impossible to use a single Prometheus (or an HA pair) to write to multiple tenants.
This proxy solves the problem using the following logic:
- Receive Prometheus remote write
- Search each timeseries for a specific label name and extract a tenant ID from its value. If the label wasn't not found then it can fall back to a configurable default ID. If none is configured then the write request will be rejected.
- Optionally removes this label the timeseries
- Groups timeseries by tenant
- Issues a number of parallel per-tenant HTTP requests to Cortex adding the tenant HTTP header (
X-Scope-OrgID
by default)
- Get
rpm
ordeb
for amd64 from the Releases page. For building see below.
- GET
/alive
returns 200 by default and 503 if the service is shutting down (iftimeout_shutdown
setting is > 0) - POST
/push
receives metrics from Prometheus - configure remote write to send here
Application expects the config file at /etc/cortex-tenant.yml
by default.
# Where to listen for incoming write requests from Prometheus
listen: 0.0.0.0:8080
# Profiling API, remove to disable
listen_pprof: 0.0.0.0:7008
# Where to send the modified requests (Cortex)
target: http://127.0.0.1:9091/receive
# Log level
log_level: warn
# HTTP request timeout
timeout: 10s
# Timeout to wait on shutdown to allow load balancers detect that we're going away.
# During this period after the shutdown command the /alive endpoint will reply with HTTP 503.
# Set to 0s to disable.
timeout_shutdown: 10s
tenant:
# Which label to look for the tenant information
label: tenant
# Whether to remove the tenant label from the request
label_remove: true
# To which header to add the tenant ID
header: X-Scope-OrgID
# Which tenant ID to use if the label is missing in any of the timeseries
# If this is not set or empty then the write request with missing tenant label
# will be rejected with HTTP code 400
default: foobar
remote_write:
- name: cortex_tenant
url: http://127.0.0.1:8080/push
scrape_configs:
- job_name: job1
scrape_interval: 60s
static_configs:
- targets:
- target1: 9090
labels:
tenant: foobar
- job_name: job2
scrape_interval: 60s
static_configs:
- targets:
- target2: 9090
labels:
tenant: deadbeef
This would result in job1
metrics ending up in the foobar
tenant in cortex and job2
in deadbeef
.
make build
should create you an amd64 binary.
If you want deb
or rpm
packages then install FPM and then run make rpm
or make deb
to create the packages.