Bug - Recoverability HeaderCustomization : Unable to access custom message headers

[amarn2]

Describe the bug

Description

We were upgrading our project from NServiceBus 7.3.2 to 8.2.3 and found possibly one breaking change.

In our project, we had setup a few incoming and outgoing custom headers which were used to run some logic when doing the HeaderCustomization but we are unable to fetch those headers after upgrading to 8.2.3. We suspect it could have been broken long time back probably since the build - 7044043

Expected behavior

Should be able to access all the headers on the message so that we are able to modify them before saving the message to error queue.

Actual behavior

It shows some fixed 19 headers and none of the custom headers are included in that list.

Versions

8.2.3
Please list the version of the relevant packages or applications in which the bug exists.
NServiceBus

Steps to reproduce

1. Set customization for header when message is being sent to error queue-
   recoverability.Failed(
   failed =>
   {
   failed.HeaderCustomization(headers =>
   {
   if (headers.ContainsKey("CustomHeader"))
   {
   headers["CustomHeader"] = "Value updated at HeaderCustomization";
   }
   });
   });

2. Run a case with scenario - MoveToError queue.

Relevant log output

Additional Information

Workarounds

Possible solutions

Additional information

[andreasohlund]

Thanks for the detailed bug report @amarn2 , this definitely looks like a bug on our end. To help reproduce this can you share the code where you are setting those custom headers?

[amarn2]

@andreasohlund We are setting the custom headers using IMutateOutgoingMessages and also tried IMutateOutgoingTransportMessages and using SendOptions as well but all the headers added there are not available in HeaderCustomization.
While those custom headers are available on the Recoverability > OnMessageSentToErrorQueue event with the failedMessage.Headers field.

[andreasohlund]

I was able to reproduce the issue here #7273

We are looking into the issue, stay tuned

[andreasohlund]

@amarn2 what is the urgency on your end?

Can you explain a bit what those headers are used for and why you need to change/look at them in the HeaderCustomizations ?

[amarn2]

There is no urgency for now as we are going to apply some work arounds, but it would be good to get it fixed considering it might have been broken for long time and some other users like me might be facing the similar issue.

We are adding some custom header say "xyz" if the outgoing message belongs to a particular type say "mySpecificMessageType". Since our service is dealing with different kind of messages, the custom header is checked to identify "mySpecificMessageType" messages and act accordingly on the HeaderCustomization because these type of messages need separate treatment than others when they are being sent to error queue.

Also at HeaderCustomization, in the end we are removing this custom header "xyz" so that the message is saved without this header in the error queue.

[andreasohlund]

@amarn2 as a workaround for now (and likely a better solution in the long run) can you try to move your logic to a recoverability pipeline behaviour (there you have full access to the incoming message and can control the emitted headers via the Metadata property(.

Eg if you want to remove a header:

            class RecoverabilityPipelineBehavior : Behavior<IRecoverabilityContext>
            {
                public override Task Invoke(IRecoverabilityContext context, Func<Task> next)
                {
                    context.Metadata.Remove("NServiceBus.ExceptionInfo.ExceptionType");
                    return next();
                }
            }

with this approach, you should be able to skip the mutator and HeaderCustomization and put all your logic in a single place.

Let me know if that works