Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Perf event writes metric #4334

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

geyslan
Copy link
Member

@geyslan geyslan commented Sep 27, 2024

1. Explain what the PR does

c033971 chore(performance): update tracee dashboard
5c35758 chore: add perfbuf metric per event (METRICS=1)

5c35758 chore: add perfbuf metric per event (METRICS=1)

Enabled only when built with METRICS=1.

BPFPerfEventSubmitAttemptsCount and BPFPerfEventSubmitFailuresCount
count the number of events processed by the eBPF programs and written to
or attempted to be written to the perf buffer.

It is incremented right after the attempt of writing the event to the
perf buffer, making it possible to measure if the that event was
successfully written to the perf buffer or not.

This metric can be used to monitor the performance of individual eBPF
events and to detect potential bottlenecks.

image

2. Explain how to test it

METRICS=1 make tracee

sudo ./dist/tracee -e socket_dup,execve --metrics --pyroscope --pprof

make -f builder/Makefile.performance dashboard-start

3. Other comments

go.mod Outdated Show resolved Hide resolved
api/v1beta1/diagnostic.proto Outdated Show resolved Hide resolved
Makefile Outdated Show resolved Hide resolved
@geyslan

This comment was marked as outdated.

@geyslan geyslan force-pushed the perf-event-writes-metric branch 2 times, most recently from b3a3181 to fadf5a8 Compare October 1, 2024 20:39
@geyslan geyslan marked this pull request as draft October 7, 2024 10:49
@geyslan
Copy link
Member Author

geyslan commented Oct 7, 2024

I'm working on a stress tool on top of this branch. Gonna let you know soon.

@NDStrahilevitz
Copy link
Collaborator

After offline discussion, registering another way of approaching this: by enabling the metric with a flag instead of requiring a rebuild.

It's feasible considering an increment of program sizes, an additional branch check (metric enabled) and a dangling map.

I actually prefer having this in a debug build precisely so that we don't include these overheads in regular user runtimes.

@geyslan geyslan force-pushed the perf-event-writes-metric branch 2 times, most recently from fac164c to 7633d97 Compare October 15, 2024 14:35
@geyslan geyslan marked this pull request as ready for review October 15, 2024 14:36
@geyslan geyslan force-pushed the perf-event-writes-metric branch 2 times, most recently from a7412c1 to d58139d Compare October 15, 2024 14:39
pkg/metrics/collector.go Outdated Show resolved Hide resolved
@@ -11,7 +9,7 @@ services:
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /:/rootfs:ro
- /:/rootfs:ro,rslave
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opportunistic change to accommodate new version:

Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "rprivate"

Enabled only when built with METRICS=1.

BPFPerfEventSubmitAttemptsCount and BPFPerfEventSubmitFailuresCount
count the number of events processed by the eBPF programs and written to
or attempted to be written to the perf buffer.

It is incremented right after the attempt of writing the event to the
perf buffer, making it possible to measure if the that event was
successfully written to the perf buffer or not.

This metric can be used to monitor the performance of individual eBPF
events and to detect potential bottlenecks.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants