Personas and getting started

[owais]

OpenTelemetry and observability in general has two distinct groups of users. Developers or service owners and SREs. This discussion will try to document the current and desired workflows to get started with OpenTelemetry Python for each persona.

Goals

• Identify user personas and use-cases.
• Help guide the discussion about distros and configuration.

Non-goals

• Prescibe responsibility should be divided between packages.
• Prescribe what a distro is or should do.

Developers/Service owners

Some teams prefer developers to integrate and own observability. This means observability is integrated into services during development and often with code.

Current

Today developers have to configure the OpenTelemetry Python as follows:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
    BatchSpanProcessor,
    ConsoleSpanExporter,
)
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.redis import RedisInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor


provider = TracerProvider()
processor = BatchSpanProcessor(ConsoleSpanExporter())
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)


FastAPIInstrumentor.instrument(app)
RedisInstrumentor().instrument()
RequestsInstrumentor().instrument()
# other instrumentations

While this is not overly complex, it is still quite a bit of code that can be completely avoided. It also means developers have to import and setup all instrumentations they want to use.

Desired

It'd be great if we could provide a convenience function that allows to configure the SDK and instrumentations both

from opentelemetry.sdk.setup import setup_tracing

setup_tracing()

The function can accept arguments to customize behavior such as use a different
provider, processor, exporter and allow to enable/disable all or some instrumentations. For example,

# use a different exporter
setup_tracing(exporter=MyCustomExporter)

# disable all instrumentations
setup_tracing(instrumentations=False)

# enable only a few
setup_tracing(instrumentations=["falcon", "requests"])

This above snippet would essentially do the same thing as the previous code sample. It'd setup a tracing pipeline with the default provider, processor and exporter, and also load+activate any installed instrumentation packages (using the entry points mechanism).

The example uses opentelemetry.sdk. import path but it can really live anywhere and doesn't have to be tied to the SDK.

SREs

Some teams prefer for SREs to integrate observability. This means developers hand over the services to the SRE team who are then responsible for instrumenting the service. This usually happens as part of some CI process and no code changes to the core service are highly desired.

Current

Today we support this use-case with the help of the opentelemetry-instrument command. Any service can be instrumented by running it with the opentelemetry-instrument command. For example, the following will load all installed instrumentation packages and activate them.

pip install opentelemetry-sdk opentelemetry-instrumentation opentelemetry-distro

Install instrumentation packages

opentelemetry-bootstrap -a install

opentelemetry-instrument python main.py

However, it'll not configure tracing pipeline (SDK) and as a result won't be able to export to tracing data anywhere.

Today this command provides some hooks (distro or pre/post) that allow some other packages (distro) to kick in and configure tracing WHEN installed. This means there is an optional runtime dependency on the distro package and the command does not work intuitively as long as that dependency is satisfied i.e, some opentelemetry distro is installed.

Desired

I think it'd be great if we could provide a single package (single pip install) to get everyone started with opentelemetry. Ideally something like the following should work:

pip install opentelemetry

Install instrumentation packages

opentelemetry-bootstrap -a install

opentelemetry-instrument python main.py

The above should be all that is required by an SRE team to setup tracing pipelines and enable relevant instrumentations. The proposed opentelemetry package would be specific to the official OpenTelemetry SDK and instrumentations. Anyone wanting to get started with official OpenTelemetry Python should only have to install this single package. It'd include/ship the instrument and boostrap commands and depend on the API, SDK, instrumentation and any other official package necessary to provide the above mentioned UX.

[NathanielRN]

Sorry what is an SRE?

Google says it's a "Site-reliability Engineer" 🙂

setup_tracing(exporter=MyCustomExporter)

This a cool idea, but I feel like it will get long quickly. As in we will need setup_trace(exporter=MyCustomExporter, insecure=True, exporter_endpoint=localhost:4317) and more for TracerProvider and instrumentations, and etc.

Not the end of the world because if you want more customization you rightfully need more parameters.

It would be cool if you could configure it from a .yml file like the collector?

Like

client: # (or sdk: ?)
  exporter:
    otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
  instrumentations:
    - flask
    - requests
    - falcon
  resource-detectors:
    - ec2

and then

setup_tracing(from_config='./sdk-config.yml')

With the ProxyTracer solution we have we could feasible configure at any time in runtime before the app starts?

opentelemetry-bootstrap -a install

I wonder if we should provide a opentelemetry-bootstrap -a uninstall...

[owais]

I don't think we should discuss what the API should look like here but just agree on if we want to ship this user experience. If we do, it might influence our decisions on what exactly goes into distro and what goes into SDK/instrumentaiton. That was the goal. Once we agree on the goal, we can then discuss the exact APIs we'd want to provide.

---

But to answer your question,

This a cool idea, but I feel like it will get long quickly. As in we will need setup_trace(exporter=MyCustomExporter, insecure=True, exporter_endpoint=localhost:4317) and more for TracerProvider and instrumentations, and etc.

This should be easy to avoid and actually must be avoided if we want to support 3rd party components being passes to this function. Generally, I like to accept "factories" or callables that create the actual instances. For example, it could look like

ExporterFactory = Callable[[], SpanExporter]
ProcessorFactory = Callable[[SpanExporter], SpanProcessor]
ProviderFactory = Callable([], ProviderFactory)


def setup_tracing(
     provider_factory: ProviderFactory=default_provider_factory,
     exporter_factory: ExporterFactory=default_exporter_factory,
     processor_factory: SpanProcessorFactory=default_processor_factory
):
       provider = provider_factory()
       provider.add_span_processor(processor_factory(exporter_factory()))

[lzchen]

Java and dot net both have simple tracing pipeline setup APIs because they use the builder. If users want to configure anything specific, do it like so.

So I do think it will be great to have an easier way for users to setup a tracing pipeline that is easily configurable as well. However, I would like to keep our API surface as small as possible. Might be a dumb question but is builder pattern not common in Python?

[ocelotl]

I feel like this discussion intends to happen in a high level of abstraction, I'll try to keep my replies in such level as well. Please let me know if I need to be more specific about any point I mention here.

Identify user personas and use-cases.

This feels like a natural way of finding the overall direction of this project, first find what kind of users we expect to have, then create features that will make these kind of users happy. I am not against this approach, nevertheless, it is new to me. This is what I have had in mind so far:

OpenTelemetry provides several options to configure itself in order to satisfy the application requirements, these options can be seen as a spectrum which goes from one end (let's call it left end) providing more control and requiring more code from the end user, to another end (let's call it right end) providing less control and requiring less code from the user. For example, on the left end of the spectrum we provide functionality that allows the user to create their own spans directly, this offers great control on where the spans start and end and such but requires more code from the user. On the right end of the spectrum we provide functionality that allows the user to just run a command and have that command automatically instrument their application with no control on the spans themselves but without requiring any code from the user. In between both ends of the spectrum there are other options like using XInstrumentor().instrument(app).

I feel like the intention of defining user personas is similar to understanding this part of the project as a spectrum, the developers are closer to the left end, SREs are closer to the right end.

I just wanted to introduce this other approach because I feel like it is possible that we end up finding other personas that don't align exactly with personas we had defined previously but overlap with some others, so maybe instead of trying with a discrete approach we can try with a continuous one which is more "flexible".

Help guide the discussion about distros and configuration.

I am having some trouble understanding this goal. Is it different from the discussion we have in #2005?

[phillipuniverse]

My 2c here - I think Datadog's ddtrace strikes a nice balance with this as they expose a patch_all operation. They have 2 supported ways of running:

1. Do everything automatically with ddtrace-run (synonymous with opentelemetry-instrument)
2. Manual invocation of patch_all() (which is what ddtrace-run uses under the covers)

I use the manual patch mode and also do my own patching of FastAPI, so I use it like this:

from ddtrace import patch_all

# Because we have custom middleware, we are going to do our own patching of fastapi to ensure that the tracing
# happens as the very first middleware
patch_all(fastapi=False, starlette=False)

This is slightly more complicated in the OpenTelemetry world though as there is a lot more to configure than just libraries, which is what Datadog's patch_all is designed to do. What if the concerns were split in OpenTelemetry and there were separate functions to control each type of config? Meaning, 1 function operated at the opentelemetry-contrib level to instrument all libraries and another at the opentelemetry-python level that intitialized the SpanProcessor/Exporter/etc configuration? Something like:

from opentelemetry.api import configure as api_config

api_config(span_processors=[...], exporter=[...]) 

from opentelemetry.contrib import patch_libs

patch_libs(fastapi=False)