Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add operation on ShuffleSharding to filter READONLY ingesters #6517

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

danielblando
Copy link
Contributor

What this PR does:
We found an issue when a tenant has ingestion_tenant_shard_size lower than the number of ACTIVE ingesters or high number of READONLY ingesters when testing the new status of READONLY.

Eg:
Failed Push
Lets assume we have a ring with
10 ACTIVE ingesters
50 READONLY ingesters
tenantA ingestion_tenant_shard_size of 20

The current subRing of this tenant can be created with only READONLY ingesters. In this case, DoBatch will fail as there will be no health ingesters to send data.

Early throttle
Lets assume we have a ring with
80 ACTIVE ingesters
20 READONLY ingesters
tenantA ingestion_tenant_shard_size of 20

The current subRing can be created as a mix of ACTIVE and READONLY ingesters. This will cause a subRing of size 20 but only 15 ACTIVE ingesters supposedly. The localLimit for each ingesters will be calculated over 20 as the shard size but only 15 ingester are receiving all data. The new scenario will create a subRing over just the 80 ACTIVE ingesters.

This PR introduce a ShuffleShard ring filter by operation. We then create a new WriteShard operation to filter out READONLY ingesters of shuffleSharding subRings. The cache for sharding will be clean as usual when the ring changes or a READONLY status changes in the ring.

In the first case, we will create a subRing with only 10 ACTIVE ingesters avoiding causing 5xx on Push
The last scenario, we will create a subRing taking in consideration just the 80 ACTIVE ingesters avoiding READONLY ingester s to miss count the limit.

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@danielblando danielblando force-pushed the filter-ro-ingester-sharding branch from 4e75055 to d344999 Compare January 16, 2025 21:46
@danielblando danielblando marked this pull request as ready for review January 17, 2025 18:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant