Creating a Log Appender for Rainbow Gum and standard way to get OpenTelemetry instance for libraries.

[agentgt]

So I'm the author of a modern Java logging library called Rainbow Gum: https://github.com/jstachio/rainbowgum

I'm creating a direct OpenTelemetry log appender that I will maintain such that one does not to have to use post processing of json or whatnot.

How to implement one seems pretty straight forward and I plan on augmenting Rainbow Gum core based on what is missing that Open Telemetry likes. It looks like the log4j2 appender is a good starting point for me

Now the question is the best way to get OpenTelemetry instance. I see most examples do this programmatically by creating it.

Luckily Rainbow Gum loves programmatic configuration and is designed for it. So the folks that go that route I can easily support.

For the Spring Boot route that gets a little tricky but my logging framework has support so I'm fairly sure I will figure I can lookup and wire OpenTelemetry bean from the context: https://jstach.io/doc/rainbowgum/current/apidocs/#spring_boot (this is not ideal though as the Spring Boot LoggingSystem only gives Environment but there are some hacks I can do)

Now comes the folks that are using other apps. I suppose I will make some ServiceLocator for folks to provide OpenTelemetry to me. Probably with Rainbow Gum's ServiceRegistry.

Anyway I was surprised that OpenTelemetry does not have its own ServiceLocator provider or some standard way of discovery of a pseudo global OpenTelemetry. If an application author uses the instrumentation is there a way to find the the OpenTelemetry instance?

Finally I noticed that log4j2 and logback have byte code instrumentation. I assume that instrumentation is to get metrics on logging itself and is independent of log appending correct?

---

EDIT I think instead of the bridge api I can go direct: https://opentelemetry.io/docs/languages/java/instrumentation/#logs-sdk

and it appears that doc also shows the discovery way of finding an OpenTelemetrySdk instance.

So now I guess should use the Logs SDK directly or use the bridge API?

[jkwatson]

This repo can't really provide any kind of service locator, since that will be totally dependent on the application framework that is being used. We have our GlobalOpenTelemetry, but really, if you're not in the javaagent context, you shouldn't use it, as it doesn't have an easy-to-deal-with lifecycle, or any way to defer initialization.

The OTel javaagent does provide an instance pre-configured in GlobalOpenTelemetry, and the spring-boot-starter over in the instrumentation repository provides a Spring bean for it. Anything else would need to be application framework specific, and hence doesn't really belong in this particular repository, which is only providing the low-level building blocks (and the auto-configuration extension based on SPIs).

[agentgt]

By ServiceLocator I actually meant Java.util.ServiceLoader which is a JDK standard and not application or framework specific.

However I can see why that would still not be provided by module.

I get the hint that I may have posted in the wrong repository but
as a point of critique there are so many open telemetry repos I’m not sure where is the best place to ask implementer questions.

[jkwatson]

A good place to go is the CNCF slack, the #otel-java channel.

[jkwatson]

This question seems to have a lot of questions in it, and I'm not sure any more which you need help with. Can you break your issues down a little bit?

[agentgt]

This question seems to have a lot of questions in it, and I'm not sure any more which you need help with. Can you break your issues down a little bit?

Yes apologies. I don't think I need help with anything and it was more confirmation. I should have broken this up but I wasn't sure because it is all kind of connected and there are multiple repositories like instrumentation which includes the log4j2/logback appenders which I would not count as instrumentation. Should I ask log appender question over there? (this rhetorical as I now know to use the discord).

First before I ask the main question understand that logging is similar to metrics (and instrumentation) in that it has a chicken/egg configuration initialization problem. It needs to be available before anything else. This is why logging is very difficult to have Spring initialize and say wire in the components because Spring itself uses logging. I need to have OTEL ready to go very early and the canonical approach is to use some SPI via the ServiceLoader.

You said there was no Service Locator but this is most likely what I want (edit but not really as it creates every time):

OpenTelemetrySdk sdk = AutoConfiguredOpenTelemetrySdk.initialize()
    .getOpenTelemetrySdk();

The above would happen only if they did not wire in OpenTelemetry to the logging framework prior to something like Logger logger = LoggerFactory.getLogger(...).

Otherwise I have to do what Log4j2 and Logback do where this is some queue that will get replayed and you use a global static method to set the OTEL instance. I think that is nasty solution but if it is the canonical approach I suppose I can implement it:

https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/2d5775ae8a4d5c2d630292889e3f1d02ac1a2ab9/instrumentation/log4j/log4j-appender-2.17/library/src/main/java/io/opentelemetry/instrumentation/log4j/appender/v2_17/OpenTelemetryAppender.java#L74

So I guess I don't have an easy question and its more of a dialog of what is the best practice for a log appender fetching the OTEL instance. Anyway thanks for being patient and I'm sorry for the confusion. I will join the discord later but I think I have enough based on existing code to get something working even if its possibly not the right choice.

EDIT I see there really is no choice. I have to do similar to what is happening in logback and log4j2 especially because of Spring Boot support:

https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/spring/spring-boot-autoconfigure/src/main/java/io/opentelemetry/instrumentation/spring/autoconfigure/internal/instrumentation/logging/OpenTelemetryAppenderAutoConfiguration.java#L33

If nothing calls the static methods then you are just queueing messages forever. I suppose I can set a limit on the queue.

You see because I thought OTEL was low enough and needed so early in initialization similar to logging that there would be a global singleton (and indeed there sort of is but I should not use it). The reason I thought this is I always saw it being configured with env variables. So it appears every framework has to figure out a way to resolve it. I suppose it is less work for you guys and more flexible.

Let me put it this way. Imagine if you had to inject logging everywhere. I think what you have chosen is probably correct but I will have to replicate all the code that you do for Spring Boot for logback and log4j and whatever other frameworks.... which is not a problem but there might be weird conflicts.

[jkwatson]

I think this is actually a question for the instrumentation repo, because that's where we have chosen to house the appender implementations. And, yes, initializing appenders is fraught, because none of them are designed for late-binding, nor do they really have hooks built into them for instrumentation. I definitely recommend starting a discussion on the CNCF slack, and/or in the instrumentation repository, where the appender implementations are located.