Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Redis cluster endpoint with Mimir throws errors and unable to integrate with Mimir for Redis cluster cache under Kubernetes cluster environment #9301

Closed
rganeshpsgcs opened this issue Sep 14, 2024 · 3 comments
Closed

Redis cluster endpoint with Mimir throws errors and unable to integrate with Mimir for Redis cluster cache under Kubernetes cluster environment #9301

rganeshpsgcs opened this issue Sep 14, 2024 · 3 comments

Comments

@rganeshpsgcs
Copy link

Describe the bug

I have been using Mimir for my project and i have set up a Redis cluster using Bitnami Redis-cluster.

I configured the below end points in my values.yaml (added below as part of the Configuration options)
file for Redis Cluster (It is a cluster architecture and NOT Standalone or Sentinel)

i have deployed the Redis cluster in one of the Kubernetes cluster environment.

here is a pod details (Original IPs and namespace got redacted.)

kubectl get pods  -o wide -n myproject | grep cluster
test-redis-cluster-myproject-0                        1/1     Running   2 (6d6h ago)   6d6h   12.1.9.76     myproject-dev-worker-4   <none>           <none>
test-redis-cluster-myproject-1                        1/1     Running   2 (6d6h ago)   6d6h   12.1.10.54    myproject-dev-worker-2   <none>           <none>
test-redis-cluster-myproject-2                        1/1     Running   2 (6d6h ago)   6d6h   12.1.5.63     myproject-dev-worker-8   <none>           <none>
test-redis-cluster-myproject-3                        1/1     Running   1 (6d6h ago)   6d6h   12.1.3.76     myproject-dev-worker-1   <none>           <none>
test-redis-cluster-myproject-4                        1/1     Running   1 (6d6h ago)   6d6h   12.1.8.164    myproject-dev-worker-9   <none>           <none>
test-redis-cluster-myproject-5                        1/1     Running   2 (6d6h ago)   6d6h   12.1.4.173    myproject-dev-worker-5   <none>           <none>

using an internal pod, i can connect to the redis-cluster as below

kubectl run --namespace myproject redis-cluster-client --rm --tty -i --restart='Never' \
 --env REDIS_PASSWORD=$REDIS_PASSWORD \
--image docker.io/bitnami/redis-cluster:7.4.0-debian-12-r1 -- bash
$ redis-cli -c -h redis-cluster -a $REDIS_PASSWORD
redis-cluster:6379> SET NAME TCS
-> Redirected to slot [1450] located at 12.1.10.54:6379
OK
12.1.10.54:6379:6379> SET AGE 45
-> Redirected to slot [9895] located at 12.1.5.63:6379
OK

To Reproduce

Steps to reproduce the behavior:

However, when configured the below redis cluster service endpoints in mimir, i got below issues.

Configuration 1:

metadata_cache:
          backend: redis
          redis:
            username: ""
            password: "${METADATA_CACHE_PASSWORD}"
            endpoint: test-redis-cluster-myproject-0.test-redis-cluster-myproject-headless:6379, test-redis-cluster-myproject-1.test-redis-cluster-myproject-headless:6379, test-redis-cluster-myproject-2.test-redis-cluster-myproject-headless:6379, test-redis-cluster-myproject-3.test-redis-cluster-myproject-headless:6379, test-redis-cluster-myproject-4.test-redis-cluster-myproject-headless:6379, test-redis-cluster-myproject-5.test-redis-cluster-myproject-headless:6379

output

Perform **kubectl logs -f myproject-mimir-store-gateway-zone-a-0 -n mynamespace **


   ts=2024-09-11T11:28:29.223914275Z caller=mimir.go:882 level=info msg="Application started"
   ts=2024-09-11T11:28:47.172010345Z caller=gateway.go:322 level=info msg="synchronizing TSDB blocks for all users" reason=ring-change
   ts=2024-09-11T11:28:47.176867817Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup  test-redis-cluster-myproject-5.test-redis-cluster-myproject-headless: no such host" items=0
   ts=2024-09-11T11:28:47.191118469Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup  test-redis-cluster-myproject-2.test-redis-cluster-myproject-headless: no such host" items=0
   ts=2024-09-11T11:28:47.284307569Z caller=gateway.go:328 level=info msg="successfully synchronized TSDB blocks for all users" reason=ring-change
   ts=2024-09-11T11:28:59.135353685Z caller=gateway.go:322 level=info msg="synchronizing TSDB blocks for all users" reason=ring-change
   ts=2024-09-11T11:28:59.139842592Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup  test-redis-cluster-myproject-5.test-redis-cluster-myproject-headless: no such host" items=0
   ts=2024-09-11T11:28:59.151816083Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup  test-redis-cluster-myproject-3.test-redis-cluster-myproject-headless: no such host" items=0
   ts=2024-09-11T11:28:59.162099888Z caller=gateway.go:328 level=info msg="successfully synchronized TSDB blocks for all users" reason=ring-change
   ts=2024-09-11T11:29:58.957752913Z caller=gateway.go:322 level=info msg="synchronizing TSDB blocks for all users" reason=ring-change
   ts=2024-09-11T11:29:58.96193791Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup  test-redis-cluster-myproject-4.test-redis-cluster-myproject-headless: no such host" items=0
   ts=2024-09-11T11:29:59.041628636Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup  test-redis-cluster-myproject-1.test-redis-cluster-myproject-headless: no such host" items=0
   ts=2024-09-11T11:29:59.049628623Z caller=gateway.go:328 level=info msg="successfully synchronized TSDB blocks for all users" reason=ring-change
   ts=2024-09-11T11:30:10.922485224Z caller=gateway.go:322 level=info msg="synchronizing TSDB blocks for all users" reason=ring-change
   ts=2024-09-11T11:30:10.924259216Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="got 4 elements in cluster info address, expected 2 or 3" items=0
   ts=2024-09-11T11:30:10.945555701Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup  test-redis-cluster-myproject-3.test-redis-cluster-myproject-headless: no such host" items=0
   ts=2024-09-11T11:30:10.955025742Z caller=gateway.go:328 level=info msg="successfully synchronized TSDB blocks for all users" reason=ring-change
   ts=2024-09-11T11:30:22.886677908Z caller=gateway.go:322 level=info msg="synchronizing TSDB blocks for all users" reason=ring-change
   ts=2024-09-11T11:30:22.957847044Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup  test-redis-cluster-myproject-1.test-redis-cluster-myproject-headless: no such host" items=0
   ts=2024-09-11T11:30:22.976489253Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup  test-redis-cluster-myproject-3.test-redis-cluster-myproject-headless: no such host" items=0
   ts=2024-09-11T11:30:22.9929137Z caller=gateway.go:328 level=info msg="successfully synchronized TSDB blocks for all users" reason=ring-change
   ts=2024-09-11T11:31:22.707781151Z caller=gateway.go:322 level=info msg="synchronizing TSDB blocks for all users" reason=ring-change
   ts=2024-09-11T11:31:22.712186122Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup  test-redis-cluster-myproject-4.test-redis-cluster-myproject-headless: no such host" items=0
   ts=2024-09-11T11:31:22.724181797Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup  test-redis-cluster-myproject-4.test-redis-cluster-myproject-headless: no such host" items=0
   ts=2024-09-11T11:31:22.789579568Z caller=gateway.go:328 level=info msg="successfully synchronized TSDB blocks for all users" reason=ring-change

Below configured the POD IP address in Mimir's metadata_cache.

Configuration 2:

metadata_cache:
          backend: redis
          redis:
            username: ""
            password: "${METADATA_CACHE_PASSWORD}"
            endpoint: 12.1.9.76:6379, 12.1.10.54:6379, 12.1.5.63:6379, 12.1.3.76:6379, 12.1.8.164:6379, 12.1.4.173:6379

Perform **kubectl logs -f myproject-mimir-store-gateway-zone-a-0 -n mynamespace **

  ts=2024-09-11T12:31:21.675623797Z caller=main.go:226 level=info msg="Starting application" version="(version=2.12.0, branch=r288, revision=f31d942)"
  ts=2024-09-11T12:31:21.685815673Z caller=server.go:352 level=info msg="server listening on addresses" http=[::]:8080 grpc=[::]:9095
  ts=2024-09-11T12:31:21.714050668Z caller=remote.go:67 level=info msg="created remote index cache"
  ts=2024-09-11T12:31:21.716370571Z caller=module_service.go:82 level=info msg=starting module=activity-tracker
  ts=2024-09-11T12:31:21.71641562Z caller=module_service.go:82 level=info msg=starting module=usage-stats
  ts=2024-09-11T12:31:21.717760695Z caller=module_service.go:82 level=info msg=starting module=sanity-check
  ts=2024-09-11T12:31:21.717880943Z caller=sanity_check.go:32 level=info msg="Checking directories read/write access"
  ts=2024-09-11T12:31:21.737011817Z caller=sanity_check.go:37 level=info msg="Directories read/write access successfully checked"
  ts=2024-09-11T12:31:21.737050581Z caller=sanity_check.go:39 level=info msg="Checking object storage config"
  ts=2024-09-11T12:31:21.797706874Z caller=reporter.go:132 level=info msg="usage stats reporter initialized" cluster_id=08b46ce0-8dc4-4358-ade7-3c83e5824745
  ts=2024-09-11T12:31:21.875489972Z caller=sanity_check.go:44 level=info msg="Object storage config successfully checked"
  ts=2024-09-11T12:31:21.875535021Z caller=module_service.go:82 level=info msg=starting module=server
  ts=2024-09-11T12:31:21.87567789Z caller=module_service.go:82 level=info msg=starting module=memberlist-kv
  ts=2024-09-11T12:31:21.875700909Z caller=module_service.go:82 level=info msg=starting module=runtime-config
  ts=2024-09-11T12:31:21.875854728Z caller=module_service.go:82 level=info msg=starting module=store-gateway
  ts=2024-09-11T12:31:21.881738612Z caller=basic_lifecycler.go:295 level=info msg="instance found in the ring" instance=myproject-mimir-store-gateway-zone-a-0 ring=store-gateway state=LEAVING tokens=512 registered_at="2024-09-05 08:11:10 +0000 UTC"
  ts=2024-09-11T12:31:21.881804445Z caller=basic_lifecycler_delegates.go:69 level=info msg="not loading tokens from file, instance already in the ring"
  ts=2024-09-11T12:31:21.901253394Z caller=gateway.go:231 level=info msg="waiting until store-gateway is JOINING in the ring"
  ts=2024-09-11T12:31:21.901283427Z caller=gateway.go:235 level=info msg="store-gateway is JOINING in the ring"
  ts=2024-09-11T12:31:21.901292857Z caller=gateway.go:245 level=info msg="waiting until store-gateway ring topology is stable" min_waiting=1m0s max_waiting=5m0s
  ts=2024-09-11T12:32:21.901687827Z caller=gateway.go:249 level=info msg="store-gateway ring topology is stable"
  ts=2024-09-11T12:32:21.90173215Z caller=bucket_stores.go:179 level=info msg="synchronizing TSDB blocks for all users"
  ts=2024-09-11T12:32:21.909974786Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup  12.1.8.164: no such host" items=0
  ts=2024-09-11T12:32:21.978242307Z caller=bucket_stores.go:441 level=info user=myproject-dev13-ds msg="creating user bucket store"
  ts=2024-09-11T12:32:21.992374816Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup  12.1.8.164: no such host" items=0
  ts=2024-09-11T12:32:22.003851053Z caller=bucket.go:408 level=info user=myproject-dev13-ds msg="loaded new block" elapsed=41.714µs id=01J74E9PEA94C99ZH4XEEEMGXX
  ts=2024-09-11T12:32:22.004528201Z caller=bucket.go:408 level=info user=myproject-dev13-ds msg="loaded new block" elapsed=72.008µs id=01J7FT0WNSWHBDN25SZEP5JQVK
  ts=2024-09-11T12:32:22.020112448Z caller=spanlogger.go:109 user=myproject-dev13-ds method=indexheader.NewStreamBinaryReader level=info msg="loading sparse index-header from disk" id=01J7FK1HYW2ZYW7YSRH6NAV2QD path=/data/tsdb-sync/myproject-dev13-ds/01J7FK1HYW2ZYW7YSRH6NAV2QD/sparse-index-header
  ts=2024-09-11T12:32:22.02088313Z caller=spanlogger.go:109 user=myproject-dev13-ds method=indexheader.NewStreamBinaryReader level=info msg="loaded sparse index-header from disk" id=01J7FK1HYW2ZYW7YSRH6NAV2QD path=/data/tsdb-sync/myproject-dev13-ds/01J7FK1HYW2ZYW7YSRH6NAV2QD/sparse-index-header elapsed=780.722µs
  ts=2024-09-11T12:32:22.022417103Z caller=spanlogger.go:109 user=myproject-dev13-ds method=indexheader.NewStreamBinaryReader level=info msg="loading sparse index-header from disk" id=01J7FC2FR4W9V3XH835NNTZSG6 path=/data/tsdb-sync/myproject-dev13-ds/01J7FC2FR4W9V3XH835NNTZSG6/sparse-index-header
  ts=2024-09-11T12:32:22.022709352Z caller=bucket.go:408 level=info user=myproject-dev13-ds msg="loaded new block" elapsed=18.565397ms id=01J7FK1HYW2ZYW7YSRH6NAV2QD
  ts=2024-09-11T12:32:22.025583261Z caller=spanlogger.go:109 user=myproject-dev13-ds method=indexheader.NewStreamBinaryReader level=info msg="loaded sparse index-header from disk" id=01J7FC2FR4W9V3XH835NNTZSG6 path=/data/tsdb-sync/myproject-dev13-ds/01J7FC2FR4W9V3XH835NNTZSG6/sparse-index-header elapsed=3.334074ms
  ts=2024-09-11T12:32:22.02719689Z caller=bucket.go:408 level=info user=myproject-dev13-ds msg="loaded new block" elapsed=23.315907ms id=01J7FC2FR4W9V3XH835NNTZSG6
  ts=2024-09-11T12:32:22.035364604Z caller=bucket_stores.go:188 level=info msg="successfully synchronized TSDB blocks for all users"
  ts=2024-09-11T12:32:22.044266253Z caller=gateway.go:270 level=info msg="waiting until store-gateway is ACTIVE in the ring"
  ts=2024-09-11T12:32:22.044289963Z caller=gateway.go:274 level=info msg="store-gateway is ACTIVE in the ring"
  ts=2024-09-11T12:32:22.044329452Z caller=mimir.go:882 level=info msg="Application started"
  ts=2024-09-11T12:32:39.691942707Z caller=gateway.go:322 level=info msg="synchronizing TSDB blocks for all users" reason=ring-change
  ts=2024-09-11T12:32:39.698744127Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup  12.1.4.173: no such host" items=0
  ts=2024-09-11T12:32:39.745535387Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup  12.1.5.63: no such host" items=0
  ts=2024-09-11T12:32:39.754616372Z caller=gateway.go:328 level=info msg="successfully synchronized TSDB blocks for all users" reason=ring-change
  ts=2024-09-11T12:32:45.574307142Z caller=gateway.go:322 level=info msg="synchronizing TSDB blocks for all users" reason=ring-change
  ts=2024-09-11T12:32:45.581488796Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup  12.1.4.173: no such host" items=0
  ts=2024-09-11T12:32:45.603532627Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup  12.1.10.54: no such host" items=0
  ts=2024-09-11T12:32:45.630037299Z caller=gateway.go:328 level=info msg="successfully synchronized TSDB blocks for all users" reason=ring-change
  ts=2024-09-11T12:33:50.280264567Z caller=gateway.go:322 level=info msg="synchronizing TSDB blocks for all users" reason=ring-change
  ts=2024-09-11T12:33:50.287782186Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup  12.1.8.164: no such host" items=0
  ts=2024-09-11T12:33:50.395526515Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup  12.1.4.173: no such host" items=0
  ts=2024-09-11T12:33:50.403362765Z caller=gateway.go:328 level=info msg="successfully synchronized TSDB blocks for all users" reason=ring-change

Configuration 3:

By configuring only the FQDNs with kubernetes standard format, got the below error

i have configured this end point on mimir side. (Pls refer the end of this thread (bitnami/charts#29102 (comment))

endpoint: test-redis-cluster-myproject-0.test-redis-cluster-myproject-headless.myproject.svc.cluster.local:6379,
          test-redis-cluster-myproject-1.test-redis-cluster-myproject-headless.myproject.svc.cluster.local:6379,
          test-redis-cluster-myproject-2.test-redis-cluster-myproject-headless.myproject.svc.cluster.local:6379

here is the redis_client.go file

mimir/vendor/github.com/grafana/dskit/cache/redis_client.go

Line 250 in 5c60e89

level.Warn(c.logger).Log("msg", "failed to mget items from redis", "err", err, "items", len(resp))

Perform **kubectl logs -f myproject-mimir-store-gateway-zone-a-0 -n mynamespace **

ts=2024-09-12T16:53:44.817834546Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="got 4 elements in cluster info address, expected 2 or 3" items=0

Configuration 4:

By configuring only the host names as node names, got the below error.


        metadata_cache:
          backend: redis
          redis:
            username: ""
            password: "${METADATA_CACHE_PASSWORD}"
            endpoint: myproject-dev-worker-1:6379,myproject-dev-worker-2:6379,myproject-dev-worker-4:6379,myproject-dev-worker-5:6379,myproject-dev-worker-8:6379,myproject-dev-worker-9:6379

Perform **kubectl logs -f myproject-mimir-store-gateway-zone-a-0 -n mynamespace **

  ts=2024-09-12T03:43:43.40788838Z caller=main.go:226 level=info msg="Starting application" version="(version=2.12.0, branch=r288, revision=f31d942)"
  ts=2024-09-12T03:43:43.414584627Z caller=server.go:352 level=info msg="server listening on addresses" http=[::]:8080 grpc=[::]:9095
  ts=2024-09-12T03:43:43.449019055Z caller=remote.go:67 level=info msg="created remote index cache"
  ts=2024-09-12T03:43:43.450637087Z caller=module_service.go:82 level=info msg=starting module=usage-stats
  ts=2024-09-12T03:43:43.450795788Z caller=module_service.go:82 level=info msg=starting module=sanity-check
  ts=2024-09-12T03:43:43.450856637Z caller=module_service.go:82 level=info msg=starting module=activity-tracker
  ts=2024-09-12T03:43:43.450933473Z caller=sanity_check.go:32 level=info msg="Checking directories read/write access"
  ts=2024-09-12T03:43:43.47583219Z caller=sanity_check.go:37 level=info msg="Directories read/write access successfully checked"
  ts=2024-09-12T03:43:43.475864388Z caller=sanity_check.go:39 level=info msg="Checking object storage config"
  ts=2024-09-12T03:43:43.539730966Z caller=sanity_check.go:44 level=info msg="Object storage config successfully checked"
  ts=2024-09-12T03:43:43.539796158Z caller=module_service.go:82 level=info msg=starting module=server
  ts=2024-09-12T03:43:43.539813514Z caller=reporter.go:132 level=info msg="usage stats reporter initialized" cluster_id=08b46ce0-8dc4-4358-ade7-3c83e5824745
  ts=2024-09-12T03:43:43.53987765Z caller=module_service.go:82 level=info msg=starting module=memberlist-kv
  ts=2024-09-12T03:43:43.539902309Z caller=module_service.go:82 level=info msg=starting module=runtime-config
  ts=2024-09-12T03:43:43.540278817Z caller=module_service.go:82 level=info msg=starting module=store-gateway
  ts=2024-09-12T03:43:43.545928114Z caller=basic_lifecycler.go:295 level=info msg="instance found in the ring" instance=myproject-mimir-store-gateway-zone-a-0 ring=store-gateway state=LEAVING tokens=512 registered_at="2024-09-05 08:11:10 +0000 UTC"
  ts=2024-09-12T03:43:43.545966978Z caller=basic_lifecycler_delegates.go:69 level=info msg="not loading tokens from file, instance already in the ring"
  ts=2024-09-12T03:43:43.557532318Z caller=gateway.go:231 level=info msg="waiting until store-gateway is JOINING in the ring"
  ts=2024-09-12T03:43:43.557561858Z caller=gateway.go:235 level=info msg="store-gateway is JOINING in the ring"
  ts=2024-09-12T03:43:43.557569199Z caller=gateway.go:245 level=info msg="waiting until store-gateway ring topology is stable" min_waiting=1m0s max_waiting=5m0s
  ts=2024-09-12T03:44:43.558134462Z caller=gateway.go:249 level=info msg="store-gateway ring topology is stable"
  ts=2024-09-12T03:44:43.558168478Z caller=bucket_stores.go:179 level=info msg="synchronizing TSDB blocks for all users"
  ts=2024-09-12T03:44:43.660377359Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup myproject-dev-worker-8 on 18.18.0.11:53: no such host" items=0
  ts=2024-09-12T03:44:43.686367407Z caller=bucket_stores.go:441 level=info user=rias-dev13-ds msg="creating user bucket store"
  ts=2024-09-12T03:44:43.693950428Z caller=bucket_stores.go:441 level=info user=rias-dev13-sts msg="creating user bucket store"
  ts=2024-09-12T03:44:43.809658042Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup myproject-dev-worker-8 on 18.18.0.11:53: no such host" items=0
  ts=2024-09-12T03:44:43.829172676Z caller=bucket.go:408 level=info user=rias-dev13-ds msg="loaded new block" elapsed=70.598µs id=01J74E9PEA94C99ZH4XEEEMGXX
  ts=2024-09-12T03:44:43.829252664Z caller=bucket.go:408 level=info user=rias-dev13-ds msg="loaded new block" elapsed=23.381µs id=01J7FC2FR4W9V3XH835NNTZSG6
  ts=2024-09-12T03:44:43.829293994Z caller=bucket.go:408 level=info user=rias-dev13-ds msg="loaded new block" elapsed=28.529µs id=01J7HY72WH41TKJETZ36MZWES6
  ts=2024-09-12T03:44:43.960954241Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup myproject-dev-worker-9 on 18.18.0.11:53: no such host" items=0
  ts=2024-09-12T03:44:44.004860873Z caller=bucket.go:408 level=info user=rias-dev13-sts msg="loaded new block" elapsed=51.17µs id=01J7H6CK9244RYHVEZXP3EH3WJ
  ts=2024-09-12T03:44:44.004927391Z caller=bucket.go:408 level=info user=rias-dev13-sts msg="loaded new block" elapsed=16.245µs id=01J7H6CM6ZCR1K18BW25PC77EF
  ts=2024-09-12T03:44:44.007384648Z caller=bucket_stores.go:188 level=info msg="successfully synchronized TSDB blocks for all users"
  ts=2024-09-12T03:44:44.018901718Z caller=gateway.go:270 level=info msg="waiting until store-gateway is ACTIVE in the ring"
  ts=2024-09-12T03:44:44.124945957Z caller=gateway.go:274 level=info msg="store-gateway is ACTIVE in the ring"
  ts=2024-09-12T03:44:44.125008754Z caller=mimir.go:882 level=info msg="Application started"
  ts=2024-09-12T03:44:55.529909552Z caller=gateway.go:322 level=info msg="synchronizing TSDB blocks for all users" reason=ring-change
  ts=2024-09-12T03:44:55.617887719Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup myproject-dev-worker-8 on 18.18.0.11:53: no such host" items=0
  ts=2024-09-12T03:44:55.826589164Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup myproject-dev-worker-9 on 18.18.0.11:53: no such host" items=0
  ts=2024-09-12T03:44:56.06062763Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup myproject-dev-worker-2 on 18.18.0.11:53: no such host" items=0
  ts=2024-09-12T03:44:56.308508445Z caller=gateway.go:328 level=info msg="successfully synchronized TSDB blocks for all users" reason=ring-change
  ts=2024-09-12T03:45:06.933483785Z caller=gateway.go:322 level=info msg="synchronizing TSDB blocks for all users" reason=ring-change
  ts=2024-09-12T03:45:07.023252938Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup myproject-dev-worker-8 on 18.18.0.11:53: no such host" items=0
  ts=2024-09-12T03:45:07.147083244Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup myproject-dev-worker-2 on 18.18.0.11:53: no such host" items=0
  ts=2024-09-12T03:45:07.415238151Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup myproject-dev-worker-1 on 18.18.0.11:53: no such host" items=0
  ts=2024-09-12T03:45:07.422664735Z caller=gateway.go:328 level=info msg="successfully synchronized TSDB blocks for all users" reason=ring-change
  ts=2024-09-12T03:46:03.951881562Z caller=gateway.go:322 level=info msg="synchronizing TSDB blocks for all users" reason=ring-change
  ts=2024-09-12T03:46:04.035829908Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup myproject-dev-worker-9 on 18.18.0.11:53: no such host" items=0
  ts=2024-09-12T03:46:04.128405437Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup myproject-dev-worker-9 on 18.18.0.11:53: no such host" items=0
  ts=2024-09-12T03:46:04.295581305Z caller=redis_client.go:263 level=warn name=metadata-cache msg="failed to mget items from redis" err="dial tcp: lookup myproject-dev-worker-4 on 18.18.0.11:53: no such host" items=0
  ts=2024-09-12T03:46:04.317205636Z caller=gateway.go:328 level=info msg="successfully synchronized TSDB block

Expected behavior

Redis cluster should store the metrics as a key value pair (in our case meta data cache).

Environment

  • Infrastructure: Kubernetes cluster
  • Deployment tool: helm chart [bitnami/redis-cluster]

Additional Context

How to enable Cluster mode for Redis under Mimir?

Assuming it supports the Redis cluster, How to configure the Service end points for Redis cluster for Mimir?

Test case details (integrating Mimir with Redis cluster) or any documentation with examples of Configuring the
Redis cluster endpoints for Mimir.
@dimitarvdimitrov
Copy link
Contributor

redis cluster isn't supported at the moment. see #5896

i'll close this as a duplicate of #5896

@dimitarvdimitrov dimitarvdimitrov closed this as not planned Won't fix, can't repro, duplicate, stale Sep 18, 2024
@rganeshpsgcs
Copy link
Author

ok thanks for the updates @dimitarvdimitrov
Please notify once the support for Redis cluster is available from Mimir, can we expect this in the near future? or is it in the road map? for Mimir development.

@dimitarvdimitrov
Copy link
Contributor

not on the roadmap for now. you can subscribe to updates on the other issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dimitarvdimitrov @rganeshpsgcs