feat(scheduler): add partial scheduling based on min replicas #6221
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
driev
approved these changes
Jan 24, 2025
driev
reviewed
Jan 27, 2025
| // also send an update if the model is not yet at desired replicas, if we have partial scheduling | ||
|
|
||
| if replicaStateUpdated || modelVersion.state.State == ScheduleFailed || model.IsDeleted() || modelVersion.state.State == ModelProgressing || | ||
| (modelVersion.state.State == ModelAvailable && len(modelVersion.GetAssignment()) < modelVersion.DesiredReplicas()) { |
There was a problem hiding this comment.
Oof.. this statement is becoming unwieldy (if it wasn't already). I wonder if it's worth having an explicit task to figure out how to refactor it.
There was a problem hiding this comment.
Yeah we should fix it, it keeps growing. I will add a todo in the code.
sakoush
force-pushed
the
INFRA-1188/partial_scheduling
branch
from
065a56e to
6170cdf
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
This PR introduces the ability to do partial scheduling for a given model if:
In this case we load the model on as many replicas as possible assuming it meets min replicas requirement; we mark the model as
Available/Ready(instead of theScheduleFailed) . We also make sure that when there is more capacity available we retry scheduling to make sure so that the model can potentially make use of this extra capacity.We also favour servers that have the models loaded to minimise ping-ping behaviour.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer: