Fix delayed message null activity

[jpalac]

Addresses #7209

[jpalac]

There is a sample provided to test this scenario: https://github.com/jonesr-out/nservicebus-otel-bug

It seems that when the StartActivity runs with default values, there is no active listener so a null activity is returned from this call.

This PR makes a change to pass in the traceFlags of the spanContext into the new root activity context. It seems to solve the issue reported in the sample.

I have created a test to try and reproduce the issue, however I was unable to get the StartActivity code to return null the way it does in the demo.
I believe that may be due to our testing setup of the activity listener.
I'm not sure if there's another way that this could be setup to be tested.

@lailabougria @SzymonPobiega can you see anything wrong with the proposed solution here? Any possible side effects?

[lailabougria]

The traceflags represent whether the original span was recorded or not (eg whether it was sampled in or out). When the user instructs to start a new trace, I don't think we should respect the previous decision, as this is a completely new trace, and therefore, a separate decision should be made. By passing the trace flags from the original send span, we're now accepting those previous decisions when I believe we shouldn't.

However, what I find odd is that with the previous code, you ended up with a null activity. Did that code contain any type of sampling settings? Did it have open telemetry enabled?

If a call is more helpful happy to jump on one.

[jpalac]

@lailabougria - further to our discussion the other day, I can get the activity to not be null by either:

• changing the CreateNewRootActivityContext method to static ActivityContext CreateNewRootActivityContext() => new(default, default, default, default); - so not creating the traceid but using default instead.

OR

• setting the Activity.Current to null before calling activity = ActivitySources.Main.CreateActivity(name: ActivityNames.IncomingMessageActivityName, ActivityKind.Consumer, remoteParentActivityContext); - so in this instance it's like the example we were looking at, and using default for parentContext instead of trying to create it as above

The second method is similar to what has been suggested in that original post and verified as "safe to use" - this is the code they were referring to: https://github.com/mu88/Repro_OpenTelemetry_Baggage/blob/main/src/Web/ActivityExtensions.cs#L13-L29

I've updated the PR to use the second method - what do you think? Any undesired side effects you can think of?

[lailabougria]

@jpalac If we can validate that:

• The sampling decision is made again and is independent of the parent context
  • This could be double-checked by splitting up the sample into two endpoints, where the receiving endpoint sets the alwaysonsampler and the sending endpoint sets it to exclude it from the sample
• The new activity is part of a new trace (different trace id than the one passed in the incoming message)
• And the new activity links back to the original trace
  Then I believe we're good!