[Bug]: Redis Cluster mode is not support and other issues

[deepanshululla]

What happened?

A bug happened!

I was trying out redis and it seems there are big gaps in redis caching

1. it doesn't support redis username as an environment variable. It assumes admin user always. this can be bypassed by using redis url but they say they don't recommend using redis url for some reason.

2. It doesn't work in cluster mode which is not a production best practice
   so we would often see

redis.exceptions.ResponseError: CROSSSLOT Keys in request don't hash to the same slot

3.) We would often see errors like but this may be because of 2

redis.exceptions.DataError: Invalid input of type: 'NoneType'. Convert to a bytes, string, int or float first.

Relevant log output

we also see

redis.exceptions.ResponseError: Command # 1 (SET llmgateway.caching.caching:88dc28d0f030c55ed4ab77ed8faf098196cb1c05df778539800c9f1243fe6b4b::2025-01-31-22-18::request_count {"current_requests": 0, "current_tpm": 1444, "current_rpm": 41} EX 60) of pipeline caused error: ('MOVED 12346 dev-shared-llmgateway2-redis-rg-0001-001.dev-shared-llmgateway2-redis-rg.axogpv.usw2.cache.amazonaws.com:6379',)

our config

litellm_settings:
  json_logs: true # if true, logs will be in json format
  ssl_verify: false # 👈 KEY CHANGE
#  callbacks: [ "datadog"] # logs llm success + failure logs on datadog
#  service_callback: [ "datadog"] # logs redis, postgres failures on datadog
  success_callback: ["s3", "datadog_llm_observability"] # logs llm success logs on datadog
  failure_callback: ["s3", "datadog_llm_observability"] # logs llm success logs on datadog
  store_audit_logs: true
#  aws_role_name: arn:aws:iam::324247901622:role/rs-dev-shared-llmgateway-task-role
  s3_callback_params:
    s3_bucket_name: dev-shared-llmgateway   # AWS Bucket Name for S3
    s3_region_name: us-west-2              # AWS Region Name for S3
  cache: True          # set cache responses to True, litellm defaults to using a redis cache
  cache_params:
    mode: default_off
    ttl: 600
    type: redis
    namespace: "llmgateway.caching.caching"
#  callbacks: ["batch_redis_requests"] # 👈 KEY CHANGE!

model_list:
  - model_name: bedrock/*
    litellm_params:
      model: bedrock/*
#      aws_role_name: arn:aws:iam::324247901622:role/rs-dev-shared-llmgateway-task-role

general_settings:
  ssl_verify: false # 👈 KEY CHANGE
  allow_requests_on_db_unavailable: True
  proxy_batch_write_at: 60 # Batch write spend updates every 60s
  store_prompts_in_spend_logs: True
  database_connection_pool_limit: 10 # limit the number of database connections to = MAX Number of DB Connections/Number of instances of litellm proxy (Around 10-20 is good number)

Are you a ML Ops Team?

yes

What LiteLLM version are you on ?

main-v1.59.7

Twitter / LinkedIn details

No response