Synchronous Memory Metrics

[rupinder10]

I have borrowed code from this:

https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/runtime-metrics/library/src/main/java/io/opentelemetry/instrumentation/runtimemetrics/MemoryPools.java

to export metrics to Prometheus. But in my case I dont use MemoryMxBean as I have my own data structure that has all the data. So I dont want to use the Asynchronous meters LongUpDownSumObserver. I replaced those with the Synchronous equivalent LongUpDownCounter. But the view show the values constantly increasing and never dropping for heap values that actually go up and down.

What is metric to replace with in this case ? Or is there a reset somewhere that I would have to add explicitly ?

[jkwatson]

Can you link to the code that you've written? It will be helpful to see precisely what you're doing.

[rupinder10]

I have created a very simple simulated scenario to depict this. The memory is sawtooth like i.e goes from 100 too 900 and then back again. But when I see this in prometheus it just keep rising. Using a LongUpDownSumObserver shows the correct pattern.

import java.util.Collections;

import io.opentelemetry.api.metrics.GlobalMetricsProvider;
import io.opentelemetry.api.metrics.LongUpDownCounter;
import io.opentelemetry.api.metrics.Meter;
import io.opentelemetry.api.metrics.MeterProvider;
import io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricExporter;
import io.opentelemetry.sdk.metrics.SdkMeterProvider;
import io.opentelemetry.sdk.metrics.export.IntervalMetricReader;

public final class JvmMetrics {

	private static LongUpDownCounter memoryAreaMetrics;
	private static long memoryUsed = 0;

	public static MeterProvider initOpenTelemetryMetrics() {
		OtlpGrpcMetricExporter metricExporter = OtlpGrpcMetricExporter.builder().build();
		SdkMeterProvider meterProvider = SdkMeterProvider.builder().buildAndRegisterGlobal();

		IntervalMetricReader intervalMetricReader = IntervalMetricReader.builder().setMetricExporter(metricExporter)
				.setMetricProducers(Collections.singleton(meterProvider)).setExportIntervalMillis(1000).build();

		Runtime.getRuntime().addShutdownHook(new Thread(intervalMetricReader::shutdown));

		Meter meter = GlobalMetricsProvider.getMeter(JvmMetrics.class.getName());
		memoryAreaMetrics = meter.longUpDownCounterBuilder("runtime.jvm.memory.area")
				.setDescription("Bytes of a given JVM memory area.").setUnit("By").build();

		return meterProvider;
	}

	public static void addEvent(long memory) {
		memoryAreaMetrics.add(memory);
	}

	public static void main(String[] args) throws InterruptedException {
		JvmMetrics.initOpenTelemetryMetrics();
		for (int i = 0; i < 100; i++) {
			addEvent(getMemory());
			Thread.sleep(10000); // Wait 10 seconds
		}
	}

	private static long getMemory() {
		long mem = (memoryUsed += 100) % 1000;
		System.out.println(mem);
		return mem;
	}

	private JvmMetrics() {
	}
}

[jkwatson]

I'll take a look

[jkwatson]

so... I'm not sure what you expected this to do. you're calling add() on a counter, and always giving it a positive value. That will be always monotonically increasing and hence never decrease.

[rupinder10]

That was exactly my understanding. I want to set the current value. add is the only method that LongUpDownCounter has. All I did was replace the LongUpDownSumObserver with LongUpDownCounter and replace the observe method with add.

I would hate to have to record the last value and then do addition that way as that would require a lot of housekeeping. That is why my original question was about the synchronous equivalent of LongDownSumObserver.

[jkwatson]

I think the closest equivalent is the LongValueRecorder. It'll aggregate as a summary (at the moment), though, which may or may not be what you want. There is nothing in the current API that acts like a synchronous gauge, though.

[jkwatson]

One thing you could do is to register a view for your LongValueRecorder that aggregates as a "last value" aggregation:

    InstrumentSelector selector = InstrumentSelector.builder()
        .setInstrumentType(InstrumentType.VALUE_RECORDER)
        .setInstrumentNameRegex("runtime\\.jvm\\.memory\\.area")
        .build();
    meterProvider.registerView(selector, AggregatorFactory.lastValue());

This then acts like a gauge and will only report the latest recorded value when metrics are exported. Would that work?

[rupinder10]

I think that will work although I am worried about building something that will not work as we move forward. I am going to see if there is a way for me to keep a record of the last value of my object and then have the observer pattern read the value from there. If it doesn't work or surprises me somehow, then I will fall back to this.

[rupinder10]

Since I am taking an approach to keep track of my events and extract the information from those when the observer is ready to observe, I would like to specify the original time of capture of the information and not when it was observed by the observer. What is the best way to do that ? Something like what we do with setStartTimstamp() in the span for traces. Does metrics API also have that capability ?

[jkwatson]

Just FYI, if you're worried about things not working moving forward, you probably should avoid using the metrics API right now. It's still alpha, and the OpenTelemetry Metrics SIG is doing a lot of re-thinking on everything right now. I can almost certainly guarantee that things will change in the future.

We will almost certainly have views in the future, so it should be relatively safe to use that approach, if it works.

As to keeping track of timestamps and passing them into the API somehow. No, there's nothing in the API to support this use-case right now. If you're interested in influencing the API design, however, please feel free to join the aforementioned metrics effort. :)

[rupinder10]

Let me know how to join and I would certainly be interested.

…

On Thu, Mar 18, 2021 at 5:47 PM John Watson ***@***.***> wrote: Just FYI, if you're worried about things not working moving forward, you probably should avoid using the metrics API right now. It's still alpha, and the OpenTelemetry Metrics SIG is doing a lot of re-thinking on everything right now. I can almost certainly guarantee that things will change in the future. We will almost certainly have views in the future, so it should be relatively safe to use that approach, if it works. As to keeping track of timestamps and passing them into the API somehow. No, there's nothing in the API to support this use-case right now. If you're interested in influencing the API design, however, please feel free to join the aforementioned metrics effort. :) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3040 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB56UZ76KSAB46P5FLWIZJLTEJYGFANCNFSM4ZMXT7CQ> .

[jkwatson]

If you head over here: https://github.com/open-telemetry/community#special-interest-groups , you should be able to see the Metrics Specification SIG. All the meetings are open to the public, and on the public calendar on that same page. Let me know if there's anything else I can do!