Polly's built-in OnHedging metric in resilience.polly.strategy.events doesn't match with custom metrics we record in onHedging action callback

[lvxiaoxin]

Hi Polly experts,

We recently adopt Polly resilience and chaos through the Microsoft.Extensions.Http.Resilience package (version 8.3), and applied the AddStandardHedgingHandler to our HttpClient. As a result, we automatically get the Polly built-in telemetry like resilience.polly.strategy.events. We start to do some chaos testing experiment, however, we noticed that the metrics count of event.name == "OnHedging" doesn't match metric we records in onHedging action while setting up the resilience handlers for HttpClient.

In our experiment, we used the _AddStandardHedgingHandler_ to our HttpClient and configured it with primary/secondary endpoint group. Then we introduced a chaos fault injection with failure ratio as 0.8. We could see the circuit breaker strategy kick in and take effects, then the hedging strategy works and switches the target endpoint from primary to secondary. Everything works well except for the resilience.polly.strategy.events metrics.

The chart is aggregated per 1 minute, and it shows in average there are only 6 OnHedging metrics, while actually there are about 2k+ requests per minute.

Then we did an experiment that we manually send a custom metric in the onHedging action callback while setting up the resilience policy, and its count match with the request traffic, but doesn't match above resilience.polly.strategy.events metrics.

So, I believe the hedging operation indeed happens and the functionality indeed works, but the only problem is unexpected resilience.polly.strategy.events that it's having too less metrics filtering by `event.name == "OnHedging".

Does my understanding of the built-in metrics wrong? Shouldn't it match with the onHedging action?

[martincostello]

@martintmk I feel you're well placed to answer this?

[martintmk]

If we look at this code:

Polly/src/Polly.Core/Hedging/Controller/TaskExecution.cs

Line 139 in 6f7edf8

private async Task HandleOnHedgingAsync(ResilienceContext primaryContext, int attemptNumber)

Notice, that path where OnHedging is called and where the telemetry is reported, is identical.

These metrics should be identical. Maybe you are collecting these differently?

[lvxiaoxin]

Thanks @martintmk and @martincostello. Yes you are right. We originally looked into the HandleOnHedgingAsync method and considered they should match from the code logic, but saw unexpected results from the final collected metrics.

It turns out that the difference comes from the way we collecting those metrics (one is sampling as count and another is not), leading to the mis-match between the built-in metric and my custom record metric. After I fixed that, the OnHedging metric of resilience.polly.strategy.events start to make senses and match our experiment traffic.

Thanks a lot for the help here!