Rapid Failure Protection - How to implement

[mhsimkin]

Hi.
I have some windows services that I am updating to .NET 8 and adding support for Polly. Several of these services start long-running tasks.

There is currently custom logic in the services' ExecuteAsync method to

1. wait for all the long running tasks to complete.
2. restart a task if it failed for any reason besides the service being shutdown.

This restart logic is conceptually similar to the IIS App Pool Rapid Failure Protection.

If the long-running tasks fails, it should be restarted for x times within y minutes, with the delay between the restart increasing each time.

If its been greater than y minutes since the last restart, the retry counter needs to be reset to 0 and the cycle starts overs. However, if max retries is reached within y minutes then whole service needs to be aborted.

The Polly Retry Strategy will handle the restart and the track the failure counts, but it doesn't reset the retry counter until the task successfully completes. Which would never happen on a long-running tasks.

The other strategies don't seem to help with what I'm trying to achieve.

I am trying to figure out if this is something:

1. I can implement out of the box with Polly using existing strategies,
2. Will I need to build a custom strategy.
3. Polly is the wrong framework, and I should just continue to use the existing code.

For all I know, I could have the wrong approach or just be fixated on using the wrong tool.

Thank you for your advice.

-marc

[peter-csala]

The short answer is yes I think you can achieve something like what you have described.
In order to be able provide sample code for this I need to know which Polly API are you targeting V7 or V8?

[mhsimkin]

Hi @peter-csala, I am using the new V8 API.

Thanks -marc

[peter-csala]

I'll try to put together today or tomorrow a sample code.

[mhsimkin]

@peter-csala . Thank you.

[peter-csala]

Okay, lets build the sample code together. (If you are interested only about the end result then scroll to the end of this answer).

Start with some basic stuff

var retry = new ResiliencePipelineBuilder()
.AddRetry(new () {
    ShouldHandle = args => PredicateResult.True(),
    Delay = TimeSpan.FromSeconds(1),
    BackoffType = DelayBackoffType.Linear,
    MaxRetryAttempts = int.MaxValue,
    OnRetry = args =>
    {
        Console.WriteLine($"Attempt: {args.AttemptNumber} Delay: {args.RetryDelay} Duration: {args.Duration}");
        return default;
    }
})
.Build();

await retry.ExecuteAsync(ct => throw new MyException());

So, we have a retry strategy which retries unconditionally and indefinitely. The delay between retry attempts are linearly increasing.

The output looks something like this

Attempt: 0 Delay: 00:00:01 Duration: 00:00:00.0023585
Attempt: 1 Delay: 00:00:02 Duration: 00:00:00.0002090
Attempt: 2 Delay: 00:00:03 Duration: 00:00:00.0004294
Attempt: 3 Delay: 00:00:04 Duration: 00:00:00.0001497
...

So, here the Duration is representing the execution time of the retry attempt not the duration of the entire ExecuteAsync.

Use context for elapsed time management

ResiliencePropertyKey<DateTime> cycleStartKey = new(nameof(cycleStartKey)); // newly added
var context = ResilienceContextPool.Shared.Get(); // newly added

var retry = new ResiliencePipelineBuilder()
.AddRetry(new () {
    ShouldHandle = args => // rewritten
    {
        var cycleStart = args.Context.Properties.GetValue(cycleStartKey, DateTime.MinValue);
        var diff = DateTime.UtcNow - cycleStart;
        return diff <= TimeSpan.FromSeconds(10) ? PredicateResult.True() : PredicateResult.False();
    },
    Delay = TimeSpan.FromSeconds(1),
    BackoffType = DelayBackoffType.Linear,
    MaxRetryAttempts = int.MaxValue,
    OnRetry = args =>
    {
        Console.WriteLine($"Attempt: {args.AttemptNumber} Delay: {args.RetryDelay} Duration: {args.Duration}");
        return default;
    }
})
.Build();

context.Properties.Set(cycleStartKey, DateTime.UtcNow); // newly added
await retry.ExecuteAsync(ctx => throw new MyException(), context); // rewritten
ResilienceContextPool.Shared.Return(context); // newly added

So, we add UtcNow to the context before we kick off the initial attempt.
We retrieve the cycleStart (naming is hard...) inside the ShouldHandle and make a decision whether to perform a new retry.
If the whole duration of the ExecuteAsync exceeds the threshold (10 seconds in the example) then we stop retrying.

The output looks something like this

Attempt: 0 Delay: 00:00:01 Duration: 00:00:00.0022074
Attempt: 1 Delay: 00:00:02 Duration: 00:00:00.0004463
Attempt: 2 Delay: 00:00:03 Duration: 00:00:00.0003880
Attempt: 3 Delay: 00:00:04 Duration: 00:00:00.0001263
Unhandled exception. MyException: Exception of type 'MyException' was thrown.

So, before each retry attempt we assess whether we still have time to kick off a new attempt or not.

Retry the retries

ResiliencePropertyKey<DateTime> cycleStartKey = new(nameof(cycleStartKey));
var context = ResilienceContextPool.Shared.Get(); 

var retry = new ResiliencePipelineBuilder()
.AddRetry(new() {
    MaxRetryAttempts = int.MaxValue,
    OnRetry = args => // newly added
    {
        args.Context.Properties.Set(cycleStartKey, DateTime.UtcNow);
        Console.WriteLine($"Reset retry, new cycle {args.AttemptNumber + 1} will start");
        return default;
    }
})
.AddRetry(new () {
    ShouldHandle = args =>
    {
        var cycleStart = args.Context.Properties.GetValue(cycleStartKey, DateTime.MinValue);
        var diff = DateTime.UtcNow - cycleStart;
        return diff <= TimeSpan.FromSeconds(10) ? PredicateResult.True() : PredicateResult.False();
    },
    Delay = TimeSpan.FromSeconds(1),
    BackoffType = DelayBackoffType.Linear,
    MaxRetryAttempts = int.MaxValue,
    OnRetry = args =>
    {
        Console.WriteLine($"Attempt: {args.AttemptNumber} Delay: {args.RetryDelay} Duration: {args.Duration}");
        return default;
    }
})
.Build();

context.Properties.Set(cycleStartKey, DateTime.UtcNow);
await retry.ExecuteAsync(ctx => throw new MyException(), context);
ResilienceContextPool.Shared.Return(context);

As a final piece we added a new retry to the pipeline. If the inner retry exceeds the threshold then it lets the outer retry to handle the situation. The outer simply resets the cycleStart and that's it :D

The output looks something like this

Attempt: 0 Delay: 00:00:01 Duration: 00:00:00.0020737
Attempt: 1 Delay: 00:00:02 Duration: 00:00:00.0002517
Attempt: 2 Delay: 00:00:03 Duration: 00:00:00.0004992
Attempt: 3 Delay: 00:00:04 Duration: 00:00:00.0004996
Reset retry, new cycle 1 will start
Attempt: 0 Delay: 00:00:01 Duration: 00:00:00.0004460
Attempt: 1 Delay: 00:00:02 Duration: 00:00:00.0001938
Attempt: 2 Delay: 00:00:03 Duration: 00:00:00.0002350
Attempt: 3 Delay: 00:00:04 Duration: 00:00:00.0005258
Reset retry, new cycle 2 will start
Attempt: 0 Delay: 00:00:01 Duration: 00:00:00.0004981
Attempt: 1 Delay: 00:00:02 Duration: 00:00:00.0005560
...

---

Why don't we use Timeout strategy for time management?

That could be a really good question. This approach lets the attempt to finish (either succeed or fail). The timeout based approach would intrusively stop the execution of the given attempt.

Based on the original question I had the impression that you don't want to have a disruptive solution. Rather gently reset the retry counter after a threshold has been exceeded. If I misunderstood your question, please let me know.

[mhsimkin]

@peter-csala.

Thank you for the example.

Ideally, I don't want a disruptive solution. However, the current code base is disruptive and will stop the whole service if a long-running task fails too many time. That means there is most likely some issue where a resource (RabbitMQ, MongoDB, SQL database) is unavailable (network, load, etc.).

Do I need to maintain that approach? Not necessarily.

The MongoDB and SQL connections are already managed by each long-running task, with each Mongo call already wrapped by Polly.

The RabbitMQ Connection is managed by the class derived from the Microsoft.Extensions.Hosting.BackgroundService class. The connection is opened in the StartAsync method override and then a handler is invoked whenever a message is available to be consumed.

I will need to go and evaluate where is the best place to wrap the RabbitMQ calls with Polly, as they are currently not wrapped. In the service's StartAsync, or maybe inside the library that is used to manage RabbitMQ operations.

Anyhow, the code block in the "Retry the retries" section is how the final code should be structured.

To see if I understand.

There are two (2) retry blocks, where the first block will place the current time in the context block.

Then the second retry block evaluates whether too many retries have occurred during this cycle.

If we are within the execution window (counts and time) the Retry Strategies' OnRetry handler will be executed, and Polly will attempt to execute the provided callback again.

If we are outside the execution window, Polly will just clean up and let the exception bubble up to a try/catch block in my code.

Some questions:

1. Strategies are called in the reverse order they are added to the pipeline?
2. If I need to make this disruptive in the short term, I just need to add a ShouldHandle to the first Retry block and return false, if the execution window has elapsed?
3. Would using the circuit breaker strategy instead of a retry for the first block make sense? I would want to delay during a retry for awhile to see if the issue is resolved.

Finally, what are the teams thoughts on having nested Polly calls. As I mentioned the calls to MongoDB are already wrap in a Polly Retry Strategy, with a finite number of retries. If the operation is not completed successfully within that period, the error is logged and the exception is bubbled up.

Thank you for the code sample and all the help.

-marc

[peter-csala]

To see if I understand.

There are two (2) retry blocks, where the first block will place the current time in the context block.

Then the second retry block evaluates whether too many retries have occurred during this cycle.

Yes, correct.

If we are within the execution window (counts and time) the Retry Strategies' OnRetry handler will be executed, and Polly will attempt to execute the provided callback again.

Almost :D

If we are inside the execution window then only the second retry strategy's OnRetry delegate will be executed. After that the retry goes to sleep and then it issues a new attempt.

If we are outside the execution window, Polly will just clean up and let the exception bubble up to a try/catch block in my code.

No.

If we are outside the execution window then the second retry strategy will not trigger. Rather it will throw the MyException and let the first retry strategy to handle it. This mechanism is called escalation. The first retry strategy will be triggered because the default ShouldHandle predicate handles all exceptions except the OperationCanceledException.

[peter-csala]

Strategies are called in the reverse order they are added to the pipeline?

Yes. Here we have three sequence diagrams to depict how the registration order impacts the overall behavior of the resilience pipeline. I hope it clarifies how escalation works.

If I need to make this disruptive in the short term, I just need to add a ShouldHandle to the first Retry block and return false, if the execution window has elapsed?

The first retry strategy is now executed indefinitely (MaxRetryAttempts = int.MaxValue). If you want to disrupt this infinite loop then you can add a Timeout as a very first strategy. In the previous question I've mentioned sequence diagrams. The last one shows that we have a local and a global (overarching) timeout. The same thing can be used here:

.AddTimeout // Global timeout which is disruptive
.AddRetry // Indefinite retry which resets the cycle
.AddRetry // Time capped retry which executes new attempts until the cycle duration is not exceeded

Would using the circuit breaker strategy instead of a retry for the first block make sense? I would want to delay during a retry for awhile to see if the issue is resolved.

Circuit Breaker's main advantage is that once I've detected that downstream is misbehaving/unreachable then I don't let any outgoing requests. That means CB shares the wisdom that the downstream is unhealthy from the application perspective and prevents over-flooding that system with new requests until the downstream is self-healing.

So, it might make sense but I would first start with exponential back-off + jitter instead of introducing yet another strategy. I suggest to start with simpler pipelines and then add fine-tune them if needed.

[mhsimkin]

Understood. Thank you for all the help.

-marc

[mhsimkin]

Hi @peter-csala.

Referencing this feature request. Here is what I've run into.

I need to be able to do the following:

1. reset the attempt counter
2. capture a timestamp at the end of each execute of the callback.

Below is the code that I have come up with. I added a new class CycleStats that I used to capture the overall start time, and the start/stop of each attempt.

    public TimeSpan FailureInterval { get; }

    protected internal override async ValueTask<Outcome<T>> ExecuteCore<TState>(
        Func<ResilienceContext, TState, ValueTask<Outcome<T>>> callback, ResilienceContext context, TState state)
    {
        double retryState = 0;

        var cycleStats = new CycleStats();
        
        var attempt = 0;

        while (true)
        {
            var startTimestamp = _timeProvider.GetTimestamp();

           cycleStats.CycleStart();
            
            var outcome = await StrategyHelper.ExecuteCallbackSafeAsync(callback, context, state)
                .ConfigureAwait(context.ContinueOnCapturedContext);
            
            cycleStats.CycleStop();
            
            var shouldRetryArgs = new RetryPredicateArguments<T>(context, outcome, attempt);
            var handle = await ShouldHandle(shouldRetryArgs).ConfigureAwait(context.ContinueOnCapturedContext);
            var executionTime = _timeProvider.GetElapsedTime(startTimestamp);

            TelemetryUtil.ReportExecutionAttempt(_telemetry, context, outcome, attempt, executionTime, handle);

            if (context.CancellationToken.IsCancellationRequested) return outcome;

            // does the number of attempts need to be reset because the last execution period was greater than the failure interval.
            if (cycleStats.ElapsedCycleTime > FailureInterval)
            {
                attempt = 0;
                cycleStats.Reset();
            }
            
            if (IsLastAttempt(attempt, cycleStats, out var incrementAttempts) || !handle) return outcome;

            var delay = RetryHelper.GetRetryDelay(BackoffType, UseJitter, attempt, BaseDelay, MaxDelay, ref retryState,
                _randomizer);
            if (DelayGenerator is not null)
            {
                var delayArgs = new RetryDelayGeneratorArguments<T>(context, outcome, attempt);

                if (await DelayGenerator(delayArgs).ConfigureAwait(false) is TimeSpan newDelay &&
                    RetryHelper.IsValidDelay(newDelay)) delay = newDelay;
            }

            var onRetryArgs = new OnRetryArguments<T>(context, outcome, attempt, delay, executionTime);
            _telemetry.Report<OnRetryArguments<T>, T>(
                new ResilienceEvent(ResilienceEventSeverity.Warning, RetryConstants.OnRetryEvent), onRetryArgs);

            if (OnRetry is not null) await OnRetry(onRetryArgs).ConfigureAwait(context.ContinueOnCapturedContext);

            if (outcome.TryGetResult(out var resultValue))
                await DisposeHelper.TryDisposeSafeAsync(resultValue, context.IsSynchronous)
                    .ConfigureAwait(context.ContinueOnCapturedContext);

            // stryker disable once all : no means to test this
            if (delay > TimeSpan.Zero)
                try
                {
                    await _timeProvider.DelayAsync(delay, context).ConfigureAwait(context.ContinueOnCapturedContext);
                }
                catch (OperationCanceledException e)
                {
                    return Outcome.FromException<T>(e);
                }

            if (incrementAttempts) attempt++;
        }
    }

The IsLastAttempt method is modified to check to see if there have been too many attempts within the FailureInterval window.

    internal bool IsLastAttempt(int attempt, CycleStats currentStats, out bool incrementAttempts)
    {
        if (attempt == int.MaxValue)
        {
            incrementAttempts = false;
            return false;
        }

        if (currentStats.ElapsedOperationTime <= FailureInterval && attempt >= RetryCount)
        {
            incrementAttempts = false;
            return false;
        }
        
        incrementAttempts = true;
        return attempt >= RetryCount;
    }

The CycleStats class:

internal class CycleStats
{
    public DateTime CycleStarted { get; private set; }

    public DateTime CycleStopped { get; private set; }
    public TimeSpan ElapsedCycleTime => CycleStopped - CycleStarted;

    public TimeSpan ElapsedOperationTime => CycleStopped - OperationStarted;
    public DateTime OperationStarted { get; private set; } = DateTime.MinValue;

    public void CycleStart()
    {
        CycleStarted = DateTime.UtcNow;
        if (OperationStarted == DateTime.MinValue) OperationStarted = CycleStarted;
    }

    public void CycleStop()
    {
        CycleStopped = DateTime.UtcNow;
    }

    public void Reset()
    {
        OperationStarted = CycleStarted;
    }
}

After every execution of the callback, we need to check to see if the code ran longer than the FailureInterval. If so, the attempt counter, and the Operation start time need to be reset. If we haven't run longer than the FailureInterval, than we just carry on and let the logic in ShouldHandle and OnRetry determine how to proceed.

The idea behind the rapid failure is if the code fails x times in y minutes than we shouldn't restart or maybe add a longer delay. The current code base doesn't restart, due to an issue with the Rabbit MQ library being used.

I did look into using just a single retry strategy and two retry strategies. They work, however, I was managing multiple context properties, including my own failure counter.

I felt like I was overriding a lot of what retry strategy was doing, therefore it made more sense to implement my own strategy.

If I need to use the existing Retry Strategy, I would have code similar to this:

        ResiliencePropertyKey<DateTime> operationStartKey = new(nameof(operationStartKey));
        ResiliencePropertyKey<int> cycleRestartCountKey = new(nameof(cycleRestartCountKey));
        ResiliencePropertyKey<bool> resetCycleKey = new(nameof(resetCycleKey));

        var pipeline = new ResiliencePipelineBuilder()
            .AddRetry(new()
            {
                ShouldHandle = args =>
                {
                    var lastCycleStop = DateTime.UtcNow;
                    var operationStartTime = args.Context.Properties.GetValue(operationStartKey, DateTime.MinValue);

                    var diff = lastCycleStop - operationStartTime;

                    if (diff > rapidFailureProtection!.FailureInterval)
                    {
                        // we failed, but it has been longer than the specified interval between failures.
                        // therefore, we need to tell the OnRetry code to reset the cycle counts/timestamps
                        args.Context.Properties.Set(resetCycleKey, true);

                        return PredicateResult.True();
                    }

                    // need to make sure this is false, so the OnRetry code doesn't do reset when it is not needed.
                    args.Context.Properties.Set(resetCycleKey, false);

                    // we failed, we now need to determine how many times we failed within the window.
                    var restartCount = args.Context.Properties.GetValue(cycleRestartCountKey, 0);
                    
                    if (restartCount < rapidFailureProtection!.MaximumFailures) return PredicateResult.True();
                    
                    // we are at the allowed maximum number of failures for the window.
                    // log the error.
                    return PredicateResult.False();

                },
                Delay = TimeSpan.FromSeconds(30),
                BackoffType = DelayBackoffType.Exponential,
                MaxRetryAttempts = int.MaxValue,
                OnRetry = args =>
                {
                    var resetCycle = args.Context.Properties.GetValue(resetCycleKey, false);
                    var restartCount = resetCycle ? 0 : args.Context.Properties.GetValue(cycleRestartCountKey, 0);
                    
                    // maybe log some info.

                    if (resetCycle)
                        context.Properties.Set(operationStartKey, DateTime.UtcNow + args.RetryDelay);
                    
                    context.Properties.Set(cycleRestartCountKey, ++restartCount);
                    
                    return default;
                }
            })
            .Build();

        context.Properties.Set(operationStartKey, DateTime.UtcNow);
        await pipeline.ExecuteAsync(async ctx => await theEngine.Run().ConfigureAwait(false), context).ConfigureAwait(false);
        ResilienceContextPool.Shared.Return(context);

This is corporate development, I always assume the person that will be maintain my code is not me, therefore, it needs to follow the kiss principal. Having the rapid failure logic split over two different handlers is not optimal.

-marc

[peter-csala]

Hey Marc, I think you can expose this functionality via an extension method on the ResiliencePipelineBuilder like AddCyclicRetry or something like this. I try to post an example code here today or tomorrow.

[mhsimkin]

Hi Peter. I will give that a try later this morning. While that would make the code look cleaner, I would still be overriding what the Retry Strategy is doing by using context properties. I am also still making an estimate of the start timestamp for the next cycle.

[peter-csala]

Let me share with you my prototype:

1. Let's define a custom ResilienceStrategyOptions to capture the parameters of the cyclic retry

public class CyclicRetryStrategyOptions: ResilienceStrategyOptions
{
    public TimeSpan CycleLength {get; set;} = TimeSpan.FromSeconds(10);
    public TimeSpan BaseDelay {get; set;} = TimeSpan.FromSeconds(1);
    public ResiliencePropertyKey<DateTime> CycleStartKey {get;} = new(nameof(CycleStartKey));
}

2. Let's define a custom ResilienceStrategy to set the cycle start

public sealed class CycleStartStrategy: ResilienceStrategy
{
    private readonly CyclicRetryStrategyOptions options;

    public CycleStartStrategy(CyclicRetryStrategyOptions options)
    {
        this.options = options;   
    }

    protected override async ValueTask<Outcome<TResult>> ExecuteCore<TResult, TState>(
        Func<ResilienceContext, TState, ValueTask<Outcome<TResult>>> callback,
        ResilienceContext context,
        TState state)
        {
            context.Properties.Set(options.CycleStartKey, DateTime.UtcNow);
            Console.WriteLine($"Cycle start set to: {DateTime.UtcNow}");
            return await callback(context, state).ConfigureAwait(context.ContinueOnCapturedContext);
        }
}

3. Let's define an extension method AddCyclicRetry on the ResiliencePipelineBuilder to wire up all the strategies properly

public static class RetryExtensions
{ 
    public static ResiliencePipelineBuilder AddCyclicRetry(this ResiliencePipelineBuilder builder, CyclicRetryStrategyOptions options)
    {
        return builder
        .AddRetry(new() {
            ShouldHandle = new PredicateBuilder().Handle<MyException>(),
            MaxRetryAttempts = int.MaxValue,
            OnRetry = args =>
            {
                Console.WriteLine($"New cycle {args.AttemptNumber + 1} starts");
                return default;
            }
        })
        .AddStrategy(context => new CycleStartStrategy(options), options)
        .AddRetry(new() {
            ShouldHandle = args =>
            {
                var cycleStart = args.Context.Properties.GetValue(options.CycleStartKey, DateTime.MinValue);
                var diff = DateTime.UtcNow - cycleStart;
                return diff <= options.CycleLength ? PredicateResult.True() : PredicateResult.False();
            },
            Delay = options.BaseDelay,
            BackoffType = DelayBackoffType.Linear,
            MaxRetryAttempts = int.MaxValue,
            OnRetry = args =>
            {
                Console.WriteLine($"Attempt {args.AttemptNumber}th failed. Waiting {args.RetryDelay} seconds");
                return default;
            }
        });
    }
}

4. Let's use it

var retry = new ResiliencePipelineBuilder()
    .AddCyclicRetry(new CyclicRetryStrategyOptions())
    .Build();

await retry.ExecuteAsync(ctx => throw new MyException());

And that's it :D

Sample output

Cycle start set to: 4/4/2024 11:54:59 AM
Attempt 0th failed. Waiting 00:00:01 seconds
Attempt 1th failed. Waiting 00:00:02 seconds
Attempt 2th failed. Waiting 00:00:03 seconds
Attempt 3th failed. Waiting 00:00:04 seconds
New cycle 1 starts
Cycle start set to: 4/4/2024 11:55:12 AM
Attempt 0th failed. Waiting 00:00:01 seconds
Attempt 1th failed. Waiting 00:00:02 seconds
Attempt 2th failed. Waiting 00:00:03 seconds
Attempt 3th failed. Waiting 00:00:04 seconds
New cycle 2 starts
Cycle start set to: 4/4/2024 11:55:24 AM
Attempt 0th failed. Waiting 00:00:01 seconds
Attempt 1th failed. Waiting 00:00:02 seconds
Attempt 2th failed. Waiting 00:00:03 seconds
Attempt 3th failed. Waiting 00:00:04 seconds
New cycle 3 starts
Cycle start set to: 4/4/2024 11:55:36 AM
Attempt 0th failed. Waiting 00:00:01 seconds
...

[peter-csala]

@mhsimkin Here is a dotnet fiddle for this: https://dotnetfiddle.net/00vEpi

[mhsimkin]

Thank you. I will review and do some testing.

[mhsimkin]

@peter-csala , Thank you. This is closer to what I need. Still need to do a few tweaks.

[mhsimkin]

@peter-csala, my apologies for the delayed response. The code above does exactly what I need. Thank you again.