Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SpanListener do not get invoked when native OTel tracer is used via Inject or unwrap #9079

Open
vasanth-bhat opened this issue Aug 1, 2024 · 8 comments · May be fixed by #9610
Open

SpanListener do not get invoked when native OTel tracer is used via Inject or unwrap #9079

vasanth-bhat opened this issue Aug 1, 2024 · 8 comments · May be fixed by #9610
Assignees
Labels
4.x Version 4.x enhancement New feature or request must-have P2 tracing
Milestone

Comments

@vasanth-bhat
Copy link

vasanth-bhat commented Aug 1, 2024

Environment Details

  • Helidon Version: 4.x
  • Helidon SE or Helidon MP. : SE & MP
  • JDK version: 21
  • OS: Linux x64

Problem Description

Helidon 4.x supports the direct use of direct native OpenTelemetry Tracer ( io.opentelemetry.api.trace.Tracer) by services code in addition to use of Helidon's wrapper API's ( io.helidon.tracing.Tracer ) .

There are multiple ways the native Otel tracer can be used. For. such use cases since he wrapper is not involved the SpanListener do not get invoked. This will break the solutions ( RIC telemetry-jar) that are implemented based on Span Listener life cycle events.

  1. By injecting native OTel tracer and use that create Spans.

private io.opentelemetry.api.trace.Tracer tracer;

@Inject
 CustomResource(Tracer tracer) {        
    this.tracer = tracer; 
     Span span = tracer.spanBuilder("custom").startSpan();
}. 
  1. By unwrapping the Helidon Tracer

    var tracer = io.helidon.tracing.Tracer.global();
    io.opentelemetry.api.trace.Tracer unwrapped = tracer.unwrap(io.opentelemetry.api.trace.Tracer.class);
    Span span = unwrapped.spanBuilder("custom").startSpan();

The ask here is to explore the options, on SpanListeners invoked.

For example , For. usecase #1 above, where native OTel tracer is injected.
would it make sense to have a separate wrapped Tracer ( that actually implements io.opentelemetry.api.trace.Tracer and adds the SpanListener feature ) injected instead of the injecting the instance of “io.opentelemetry.sdk.trace.SdkTracer”

For #2, Similarly, Can the unwrap() of. Helidon Tracer unwrap to new wrapped Tracer that implements io.opentelemetry.api.trace.Tracer instead of “io.opentelemetry.sdk.trace.SdkTracer” ?

@m0mus m0mus added enhancement New feature or request tracing P3 4.x Version 4.x labels Aug 5, 2024
@m0mus m0mus added this to Backlog Aug 12, 2024
@m0mus m0mus moved this to Normal priority in Backlog Aug 12, 2024
@vasanth-bhat
Copy link
Author

Hi @m0mus
Any update on this JIRA. When is this going to be considered for implementation?

@m0mus
Copy link
Contributor

m0mus commented Nov 14, 2024

The task has been categorized as a normal priority item. We do not currently have an estimated completion date, but considering the number of high priority tasks in the queue, it is unlikely to be completed in the immediate future. Is there a specific urgency associated with this task that we should be aware of?

@vasanth-bhat
Copy link
Author

it's required for spectra module

@m0mus m0mus moved this from Normal priority to Triage in Backlog Nov 19, 2024
@m0mus m0mus added P2 and removed P3 labels Nov 21, 2024
@m0mus m0mus moved this from Triage to Sprint Scope in Backlog Nov 21, 2024
@tjquinno
Copy link
Member

You are referring to two different technology implementations: Helidon tracing (which supports, among others, OpenTelemetry tracing) and Helidon's implementation of the MicroProfile Telemetry 1.1 spec which is layered on OpenTelemetry tracing.

First, a clarification. I do not see how the current Helidon code can be "breaking" any solutions in an app or library. I understand that you are asking Helidon to provide some additional functionality, but "breaking" refers to changes which cause working code to stop working. The code you are referring to could not ever have worked the way you want because, as you point out, the injected OTel tracers or spans have never participated in the separate Helidon tracing span listener feature.

Now, as to your specific suggestions and requests...

  1. This does not seem as if it would work as described.

    Helidon cannot do exactly what you suggest in the way you suggest in a backward-compatible way.

    For the injected Tracer or Span objects to call back to span listeners, Helidon would have to inject its own implementations of OTel's Tracer and Span interfaces. Existing app or library code that injects these types might already use code such as injectedTracer instanceof ExtendedTracer. It's not feasible for Helidon to have its own implementations which extend the OTel ones and, in fact, ExtendedTracer for example is final so Helidon could not have its own subclass.

    Realistically, there is probably very little if any user code that relies on specific OTel types that way, but such code might exist.

    There might be other techniques we could use instead while still injecting the native OTel types. I am in the process of looking into some already.

    A different approach

    Another quite different possibility could be for Helidon to support injecting the Helidon tracing API types (in addition to injecting the OTel types as it does today). That way the injected objects would automatically call back to any span listeners. This would have the added advantage of working for any of the tracing providers, not just OTel.

    Would this approach work for your library and apps?

  2. The Helidon Tracer.unwrap method takes a parameter, the type you want to unwrap as. So myTracer.unwrap(io.opentelemetry.api.trace.Tracer.class) should work if the tracer is actually an OTel SdkTracer because SdkTracer implements OTel's Tracer interface.

    Do you have a reproducer where that does not work?

@vasanth-bhat
Copy link
Author

Thanks for reviewing the requirements .

This requirement is for a library that relies on SpanListener for emitting Spans as events in JFR. This helps us to co-related JVM events such as GC pauses, socket reads, even method samples etc with associated Span and do deeper analysis. This functionality is very critical for root cause analysis of slow requests both for production as well during performance tests.

. Helidon to support injecting the Helidon tracing API types (in addition to injecting the OTel types as it does today). That way the injected objects would automatically call back to any span listeners.

I thought this already works. The MixedTelemetryGreetResource example uses such an approach.

However the services directly use MP Telemetry.,and native OTel tracers. For this usage where services directly use native OTel API , as one would expect the SpanListener is not invoked and those spans will not make it to JFR as events. So the solution we have breaks for such usages. This is what I meant when I said breaks the solution.

myTracer.unwrap(io.opentelemetry.api.trace.Tracer.class) should work
Yes, this works fine. My point here was when one does an unwrap and uses the native OTel tracer this way, the SpanListener callbacks would get skipped. This is just another usage that can cause SpanListener to be skipped.

So net we would need a solution, where we need the Helidon SpanListener methods to be invoked , even when services are directly using the OTel API’s, either by injecting OTel tracer or when OTel tracer is obtained by unwrapping a Helidon Tracer.

@tjquinno
Copy link
Member

Would the following work for you...

Suppose Helidon provided a way--perhaps through configuration--for you to select that the injected MP Telemetry types:

  • would implement the OpenTelemetry interfaces (as required by the spec),
  • would notify span listeners (as you are requesting), but
  • would NOT match the OTel implementation types (as they do today).

This means that uses of instanceof or SomeOtelType.cast(injectedObject) that might work today would not work if you configured this behavior.

@vasanth-bhat
Copy link
Author

vasanth-bhat commented Dec 2, 2024

Sure, we can try it out if a prototype implementation is provided.

This means that uses of instanceof or SomeOtelType.cast(injectedObject) that might work today would not work

If the wrappers implement standard OpenTelemetry api interfaces , instanceof or cast to to standard Otel interfaces ( io.opentelemetry.api.trace.Tracer, io.opentelemetry.api.trace.Span ). should still work, right?

I guess the. instanceof or cast to implementation types ( io.opentelemetry.sdk.trace.sdkTracer, io.opentelemetry.sdk.trace.SdkSpan etc.) will break.

The Helidon provided injected MP Telemetry wrappers, can maintain a reference to original implementation(SdkSpan, SdkTracer) as a delegate, can provide unwrap() method to get the native OTel types.

This approach is used in the implementation of unwrap() method ( below for ref) io.helidon.tracing.providers.opentelemetry.OpenTelemetrySpan

  @Override
    public <T> T unwrap(Class<T> spanClass) {
        if (spanClass.isInstance(delegate)) {
            return spanClass.cast(delegate);
        }
        if (spanClass.isInstance(this)) {
            return spanClass.cast(this);
        }
        throw new IllegalArgumentException("Cannot provide an instance of " + spanClass.getName()
                                                   + ", telemetry span is: " + delegate.getClass().getName());
    }

Can a similar approach be done here as well?

@tjquinno
Copy link
Member

tjquinno commented Dec 2, 2024

  1. Yes, instanceof referring to the OTel Tracer or Span interfaces would of course work. I was simply highlighting that existing code that refers to one of the OTel implementation classes would no longer work if the proposed behavior is configured
  2. Any change along these lines would indeed probably require changes to unwrap.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment