Adding dynamic progress based starting of plots. #1091
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adding dynamic progress percent based starting. This waits until the average percent is greater than
(running + 1)/2 * 1/max_concurrent such as for a max concurrent of 8 it would use these values:
such as if there were 7 already running, the average would have to be 50% or greater such as,
12.5%, 25%, 37.5%, 50%, 62.5%, 75%, and 87.5 %
as they average out to 50%This should help in two cases:
First, when starting up the manager and plots are going faster than the average time fixed stagger minutes waste time.
Second, the user shouldn't have to try and guess/calculate the correct stagger minutes just their max parallel plots.
This change is completely passive, defaulting to false both in new and old configs without the option. All other config options such as max concurrent, max concurrent with early start, stagger minutes, and max for phase 1 still apply as well. I've documented the option in the config.yaml.default, however, I haven't added anything to the primary README.md yet as I wasn't sure if this would be a default option in the future.
I've been trying this out for about a day so far and plots seem to stagger well. I used a 10-minute minimum stagger so it doesn't delay extra and the average spacing is close to the 48-50 minutes I was using before once steady state is reached.
Info on not met percentages is also included in the INFO level logs similar to other config options and looks like this:
2021-06-02 21:39:23 [INFO]: Minimum percent not met, 51.97 vs 56.25