Need advice to understand what's wrong with getting metrics

[maryanchuk04]

Hello everyone! I need to add telemetry to my service and understand why it doesn't work, I have created a file where I have telemetry settings, I would like to know what is wrong.

I want to get information about HTTP requests.

import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { PrometheusExporter } from '@opentelemetry/exporter-prometheus';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { HttpInstrumentation } from '@opentelemetry/instrumentation-http';
import { RedisInstrumentation } from '@opentelemetry/instrumentation-redis';
import { Resource } from '@opentelemetry/resources';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { NodeSDK, NodeSDKConfiguration } from '@opentelemetry/sdk-node';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';

const serviceName = 'MyService';

const prometheusExporter = new PrometheusExporter({ port: 9090, endpoint: '/metrics', appendTimestamp: true });

const redisInstrumentation = new RedisInstrumentation({ enabled: true });
const httpInstrumentation = new HttpInstrumentation({ enabled: true });

/**
 * @summary start open telemetry for service
 */
export const startOpenTelemetry = () => {
  const sdk = new NodeSDK({
    traceExporter: new OTLPTraceExporter({ url: process.env.TRACING_DESTINATION_URL }),
    resource: new Resource({ [SemanticResourceAttributes.SERVICE_NAME]: serviceName }),
    metricReader: prometheusExporter,
    instrumentations: [httpInstrumentation, redisInstrumentation, getNodeAutoInstrumentations()],
  });

  sdk.start();
};

• This only affects the JavaScript OpenTelemetry library
• This may affect other libraries, but I would like to get opinions here first

[maryanchuk04]

@jagregory @smithclay @jsuereth @dmathieu Can someone please help me?

[project0]

Do you start the sdk after initializing other imports? Afaik there is a problem with registering the metrics instrumentation before the sdk is started.
I think it could be related with open-telemetry/opentelemetry-js#4065

[maryanchuk04]

@project0 I'm actually new to this library and I just need to add metrics for http requests and I'd like it to be automatic.
I tried to use getNodeAutoInstrumentations(), but it doesn't help

[maryanchuk04]

It would be greatly appreciated if someone could show a piece of code how to do this

[project0]

you are lucky, i just spent time today into the same topic as well 😅

This seems to work for me, important is the auto register happens on your first imports before you initialize any classes so they get patched before.
Metrics are not yet auto configured as well, so built something simple at my own.

I am using it in conjunction with nestJS and config module and it seems to work fine.

instrumentation file:

import { AwsInstrumentation } from '@opentelemetry/instrumentation-aws-sdk';
import { FastifyInstrumentation } from '@opentelemetry/instrumentation-fastify';
import { GenericPoolInstrumentation } from '@opentelemetry/instrumentation-generic-pool';
import { HttpInstrumentation } from '@opentelemetry/instrumentation-http';
import { LruMemoizerInstrumentation } from '@opentelemetry/instrumentation-lru-memoizer';
import { NestInstrumentation } from '@opentelemetry/instrumentation-nestjs-core';
import { NetInstrumentation } from '@opentelemetry/instrumentation-net';
import { PinoInstrumentation } from '@opentelemetry/instrumentation-pino';

import * as metricsExporterOtlpGrpc from '@opentelemetry/exporter-metrics-otlp-grpc';
import * as metricsExporterOtlpHhttp from '@opentelemetry/exporter-metrics-otlp-http';
import * as metricsExporterOtlpProto from '@opentelemetry/exporter-metrics-otlp-proto';
import {
  PeriodicExportingMetricReader,
  ConsoleMetricExporter,
} from '@opentelemetry/sdk-metrics';
import { PrometheusExporter } from '@opentelemetry/exporter-prometheus';

import * as opentelemetry from '@opentelemetry/sdk-node';

export const instrumentations = [
  new HttpInstrumentation({
    ignoreIncomingRequestHook: (req) => req.url!.includes('/health'),
  }),
  new AwsInstrumentation({
    enabled: true,
    suppressInternalInstrumentation: true,
  }),
  new FastifyInstrumentation(),
  new GenericPoolInstrumentation(),
  new LruMemoizerInstrumentation(),
  new NestInstrumentation(),
  new NetInstrumentation(),
  new PinoInstrumentation(),
];

// exxtract headers from env var: key=value,key2=value2
const otelHeaders = (val: undefined | string): Record<string, any> => {
  const headers = {};
  (val || '').split(',').forEach((curr) => {
    const [key, value] = curr.split('=', 1);
    headers[key] = value;
  });
  return headers;
};

// follow OTEL defaults
// https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/exporter.md
const getMetricsExporter = (): opentelemetry.metrics.MetricReader => {
  if (process.env.OTEL_METRICS_EXPORTER === 'none') return undefined;
  if (process.env.OTEL_METRICS_EXPORTER === 'prometheus')
    return new PrometheusExporter({ port: 9600 });

  const exporter = () => {
    switch (process.env.OTEL_METRICS_EXPORTER) {
      case 'console': {
        return new ConsoleMetricExporter();
      }

      // OTEL
      default: {
        const conf = {
          headers: otelHeaders(
            process.env.OTEL_EXPORTER_OTLP_METRICS_HEADERS ||
              process.env.OTEL_EXPORTER_OTLP_HEADERS,
          ),
        };
        switch (
          process.env.OTEL_EXPORTER_OTLP_METRICS_PROTOCOL ||
          process.env.OTEL_EXPORTER_OTLP_PROTOCOL
        ) {
          case 'grpc':
            return new metricsExporterOtlpGrpc.OTLPMetricExporter(conf);
          case 'http/json':
            return new metricsExporterOtlpHhttp.OTLPMetricExporter(conf);
          // http/protobuf
          default:
            return new metricsExporterOtlpProto.OTLPMetricExporter(conf);
        }
      }
    }
  };

  // https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#metrics-sdk-configuration
  return new PeriodicExportingMetricReader({
    exporter: exporter(),
    exportTimeoutMillis: parseInt(
      process.env.OTEL_METRIC_EXPORT_TIMEOUT || '30000',
      10,
    ),
    exportIntervalMillis: parseInt(
      process.env.OTEL_METRIC_EXPORT_INTERVAL || '60000',
      10,
    ),
  });
};

const createNewSDK = (): opentelemetry.NodeSDK => {
  // most stuff is automatically configured by the SDK and with env vars
  return new opentelemetry.NodeSDK({
    // tracer is automatically wired
    // https://github.com/open-telemetry/opentelemetry-js/blob/main/experimental/packages/opentelemetry-sdk-node/src/TracerProviderWithEnvExporter.ts
    traceExporter: undefined,
    // not yet implemented, so we do it manually
    metricReader: getMetricsExporter(),
    autoDetectResources: true,
    instrumentations: instrumentations,
  });
};

export default createNewSDK;

your app/main file:

import { instrumentations } from './instrumentation';
import { registerInstrumentations } from '@opentelemetry/instrumentation';

// needs to be loaded early to ensure instrumentation is recognized
registerInstrumentations({
  instrumentations: instrumentations,
});

// app bootstrap
bootstrap()

[maryanchuk04]

@project0 Yep, thanks. I do something like that after your advice.

But still, nothing changed, once a little earlier, a few days ago, it worked, and this is the result

Code

import { PrometheusExporter } from '@opentelemetry/exporter-prometheus';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { HttpInstrumentation } from '@opentelemetry/instrumentation-http';
import { Resource } from '@opentelemetry/resources';
import { NodeSDK } from '@opentelemetry/sdk-node';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';

const serviceName = 'MyService';

const prometheusExporter = new PrometheusExporter({ port: 9090, endpoint: '/metrics', appendTimestamp: true });

export const instrumentations = [new HttpInstrumentation({ enabled: true })];

/**
 * @summary start open telemetry for service
 */
export const getOpenTelemetryNodeSDK = (): NodeSDK => {
  return new NodeSDK({
    traceExporter: new OTLPTraceExporter({ url: process.env.TRACING_DESTINATION_URL }),
    resource: new Resource({ [SemanticResourceAttributes.SERVICE_NAME]: serviceName }),
    metricReader: prometheusExporter,
    autoDetectResources: true,
    instrumentations: [instrumentations],
  });
};

---

and main.ts

import { registerInstrumentations } from '@opentelemetry/instrumentation';
import { instrumentations, getOpenTelemetryNodeSDK } from './settings/open-telemetry-settings.js';
registerInstrumentations({
  instrumentations: instrumentations,
});

const sdk = getOpenTelemetryNodeSDK();
sdk.start();

NOW

3 days before

[maryanchuk04]

@project0 It is also possible that this does not work because I am using the HTTP module and not Express?
What do you think?

[project0]

It shouldnt matter, the express instrumentation only adds traces.
However, i cannot really tell you whats wrong in your case, but your "diContainer.load" in the screenshot rings a bell for me. you should really load instrumentation before anything else as it hooks into the classes by modifying/patching the modules/classes.

[maryanchuk04]

@project0 Thank you! I'm very grateful for your help, there was a problem with the order of loading modules!