Add missed probes metrics #1941

olsajiri · 2024-01-08T09:31:33Z

netlify · 2024-03-28T13:22:36Z

✅ Deploy Preview for tetragon ready!

Name	Link
🔨 Latest commit	`02900fe`
🔍 Latest deploy log	https://app.netlify.com/sites/tetragon/deploys/66b0d848057b810008ebb48b
😎 Deploy Preview	https://deploy-preview-1941--tetragon.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

pkg/metrics/probemetrics/probemetrics.go

lambdanis · 2024-03-29T17:56:05Z

pkg/metrics/probemetrics/probemetrics.go

+	MissedProbes = metrics.NewBPFCounter(prometheus.NewDesc(
+		prometheus.BuildFQName(consts.MetricsNamespace, "", "missed_probes_total"),
+		"The total number of Tetragon probe missed per policy,probe.",
+		[]string{"policy", "attach"}, nil,


What is the attach label exactly?

I'm asking because tetragon_policy_events_total metric has a hook label with values like kprobe:__sys_setfsgid. Is attach essentially same or only similar? I'm thinking of standardizing some labels.

so attach is kernel function name for single kprobe and kprobe_multi (3 functions) for kprobe multi
similar for uprobes.. I'm not sure it's good idea to standardize this, we might want to change in future ;-)

pkg/metrics/probemetrics/probemetrics.go

lambdanis · 2024-03-29T18:36:47Z

pkg/metrics/probemetrics/probemetrics.go

+}
+
+var (
+	MissedProbes = metrics.NewBPFCounter(prometheus.NewDesc(


To have this metric documented in the metrics reference, we need to create a separate dummy collector that reports this metrics with example labels. See 861489a for an example, and consts.go for example labels.

This is not ideal tbh. I'm working on a better metrics library for Tetragon use cases, so hopefully soon it will be more straightforward.

sure, will add that

lambdanis

Looks good :)

kkourt

Thanks, LGTM.

One question what exactly is this missed metric? :)
Is it when a link exists without a program? Or is it something else?

olsajiri · 2024-07-31T12:06:30Z

Thanks, LGTM.

One question what exactly is this missed metric? :) Is it when a link exists without a program? Or is it something else?

so it's incremented any time we don't execute the bpf program due to recursion in kernel,
like when you have 2 kprobe programs trying to run on top of each other on the same cpu

there was a case +- year ago where we were loosing exit events for this reason due to wrong
hook we used.. now if that happens again we will see that in the metrics

olsajiri · 2024-07-31T18:00:56Z

@lambdanis could you please recheck, I changed few things on metrics side because the tetragon base changed

lambdanis

A few comments:

Ideally we should use MustNewCustomCounter and NewCustomCollector from pkg/metrics here. See how other collectors were defined in a05fbc7 or 7309c9b. If it's a trouble for some reason, I'm ok with merging as-is, but then we should have a follow-up item to refactor it.
The new metrics are not documented. Using helpers from 1. should provide an easy way to include them in the generated reference.
I'm not sure kprobe_multi (3 functions) is a useful label value. Seeing such metric, how do I find out which functions were missed?

olsajiri · 2024-08-02T08:24:19Z

A few comments:

1. Ideally we should use `MustNewCustomCounter` and `NewCustomCollector` from `pkg/metrics` here. See how other collectors were defined in [a05fbc7](https://github.com/cilium/tetragon/commit/a05fbc71ba49225c045a6354737724810409e4c7) or [7309c9b](https://github.com/cilium/tetragon/commit/7309c9ba22a2b07bae182e5ea8b10aa214be13fd). If it's a trouble for some reason, I'm ok with merging as-is, but then we should have a follow-up item to refactor it.

ok, used that in new version

2. The new metrics are not documented. Using helpers from 1. should provide an easy way to include them in the generated reference.

ok

3. I'm not sure `kprobe_multi (3 functions)` is a useful label value. Seeing such metric, how do I find out which functions were missed?

we don't have any more insight than that.. we get the counter per link/program that is attached to multiple functions,
and the missed counter spans over all attached functions.. we discussed some time ago to add a feature that would break down the counter to specific functions, but there was no need so far

lambdanis · 2024-08-02T15:18:39Z

ok, used that in new version

I see you use MustNewCustomCounter but the collector is defined as a separate type, not using NewCustomCollector. It would be nice to have the new metrics documented (in docs generated by make metrics-docs), and using NewCustomCollector should make it easy (you need to pass a collectForDocs function reporting fake metrics).

At the moment we keep unloaded programs in AllPrograms. Making AllPrograms local and locked by mutex. Adding new helpers progsAdd and progsCleanup to add and cleanup programs (on sensor unload) respectively. Signed-off-by: Jiri Olsa <[email protected]>

This just matters for log output, changing from: time="2024-03-28T13:08:34Z" level=info msg="Loading registered BPF probe" Attach="3 functions" Program=... into: time="2024-03-28T13:08:34Z" level=info msg="Loading registered BPF probe" Attach="kprobe_multi (3 functions)" Program=... It has more info and will help in following changes where the Attach string will serve as the metric label. Signed-off-by: Jiri Olsa <[email protected]>

Storing link in program so we can access link's stats and store them as metrics. Signed-off-by: Jiri Olsa <[email protected]>

Storing ebpf.Program instance in program.Program to allow ebpf related stats retrieval in following changes. Signed-off-by: Jiri Olsa <[email protected]>

Adding support to set policy name for program so we can use it as a metric label in following changes. We are going to support missed stats retrieval for kprobes in following changes so at ATM we set policy for kprobe programs which are in base sensor and generic kprobe sensor. Signed-off-by: Jiri Olsa <[email protected]>

Adding HasMissedStatsKprobe/Multi functions to detect support of kprobe missed stats. Checking this by searching BTF for needed structs, because AFAICS there's no 'functional' test possible. Signed-off-by: Jiri Olsa <[email protected]>

Adding metrics for missed runs on program and link level to kprobemetrics package and logic to store and collect missed stats. The missed stats are supported for all programs and kprobe/kprobe-multi links. They are stored per 'attach name' and 'policy name'. For programs (not just kprobes): tetragon_missed_prog_probes_total{attach="__x64_sys_linkat",policy="sys-linkat-passwd"} 68 tetragon_missed_prog_probes_total{attach="acct_process",policy="__base__"} 60 tetragon_missed_prog_probes_total{attach="sched/sched_process_exec",policy="__base__"} 64 tetragon_missed_prog_probes_total{attach="security_bprm_committing_creds",policy="__base__"} 66 tetragon_missed_prog_probes_total{attach="wake_up_new_task",policy="__base__"} 62 For kprobe and kprobe-multi links: tetragon_missed_link_probes_total{attach="__x64_sys_linkat",policy="sys-linkat-passwd"} 45 tetragon_missed_link_probes_total{attach="acct_process",policy="__base__"} 39 tetragon_missed_link_probes_total{attach="security_bprm_committing_creds",policy="__base__"} 43 tetragon_missed_link_probes_total{attach="wake_up_new_task",policy="__base__"} 41 tetragon_missed_prog_probes_total{attach="acct_process",policy="__base__"} 40 tetragon_missed_prog_probes_total{attach="kprobe_multi (1 functions)",policy="sys-linkat-passwd"} 48 tetragon_missed_prog_probes_total{attach="sched/sched_process_exec",policy="__base__"} 44 tetragon_missed_prog_probes_total{attach="security_bprm_committing_creds",policy="__base__"} 46 tetragon_missed_prog_probes_total{attach="wake_up_new_task",policy="__base__"} 42 Note changing the healthMetrics group to be created as not constrained, so it can carry new metrics. It will be addressed in future by adding debug metrics group. Signed-off-by: Jiri Olsa <[email protected]>

Adding test for tetragon_missed_prog_probes_total metric. At the moment I'm able to trigged missed event only for kprobe_multi and for prog recursion miss. We will need to do special setup to hit other missed counts, adding that on my todo list. Signed-off-by: Jiri Olsa <[email protected]>

lambdanis

looks good :)

olsajiri added the release-note/minor This PR introduces a minor user-visible change label Jan 8, 2024

olsajiri force-pushed the pr/olsajiri/missed branch from de8076f to d53cdd7 Compare January 22, 2024 13:59

olsajiri force-pushed the pr/olsajiri/missed branch from d53cdd7 to 6e23002 Compare February 2, 2024 11:23

olsajiri mentioned this pull request Feb 2, 2024

Add support to retrieve missed counts from probes cilium/ebpf#1295

Merged

olsajiri force-pushed the pr/olsajiri/missed branch 2 times, most recently from f3305dc to 97ca33b Compare March 28, 2024 13:21

olsajiri force-pushed the pr/olsajiri/missed branch 2 times, most recently from 1c504b7 to d270fd3 Compare March 28, 2024 14:47

lambdanis added the area/metrics Related to prometheus metrics label Mar 28, 2024

lambdanis reviewed Mar 29, 2024

View reviewed changes

olsajiri force-pushed the pr/olsajiri/missed branch from d270fd3 to 9209527 Compare March 30, 2024 22:03

olsajiri force-pushed the pr/olsajiri/missed branch 4 times, most recently from 97b720d to 62c22cf Compare April 9, 2024 13:23

olsajiri force-pushed the pr/olsajiri/missed branch from 62c22cf to 17d8993 Compare April 28, 2024 19:35

olsajiri force-pushed the pr/olsajiri/missed branch 3 times, most recently from 62db9fa to cc05052 Compare May 11, 2024 19:04

olsajiri force-pushed the pr/olsajiri/missed branch 2 times, most recently from b24768d to a87a9e2 Compare May 28, 2024 11:04

olsajiri force-pushed the pr/olsajiri/missed branch 5 times, most recently from 59b3588 to 6966716 Compare June 19, 2024 10:17

lambdanis approved these changes Jun 21, 2024

View reviewed changes

olsajiri force-pushed the pr/olsajiri/missed branch 2 times, most recently from 744570d to 22437f4 Compare July 11, 2024 19:24

olsajiri force-pushed the pr/olsajiri/missed branch 2 times, most recently from b161f68 to 9ef973e Compare July 31, 2024 09:39

olsajiri marked this pull request as ready for review July 31, 2024 10:15

olsajiri requested a review from a team as a code owner July 31, 2024 10:15

olsajiri requested a review from kkourt July 31, 2024 10:15

kkourt approved these changes Jul 31, 2024

View reviewed changes

lambdanis reviewed Aug 1, 2024

View reviewed changes

olsajiri force-pushed the pr/olsajiri/missed branch 3 times, most recently from 80c5f4d to 42d1674 Compare August 2, 2024 08:21

olsajiri force-pushed the pr/olsajiri/missed branch from 42d1674 to 457c7e2 Compare August 2, 2024 12:32

olsajiri force-pushed the pr/olsajiri/missed branch from 457c7e2 to f712d75 Compare August 5, 2024 11:57

olsajiri added 6 commits August 5, 2024 13:33

tetragon: Store link in program.Program

191c724

Storing link in program so we can access link's stats and store them as metrics. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Store ebpf.Program instance in program.Program

dfef686

Storing ebpf.Program instance in program.Program to allow ebpf related stats retrieval in following changes. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Add HasMissedStatsKprobe/Multi functions

26b1be1

Adding HasMissedStatsKprobe/Multi functions to detect support of kprobe missed stats. Checking this by searching BTF for needed structs, because AFAICS there's no 'functional' test possible. Signed-off-by: Jiri Olsa <[email protected]>

olsajiri force-pushed the pr/olsajiri/missed branch from f712d75 to 997de7c Compare August 5, 2024 13:35

olsajiri added 2 commits August 5, 2024 13:48

olsajiri force-pushed the pr/olsajiri/missed branch from 997de7c to 02900fe Compare August 5, 2024 13:48

olsajiri requested a review from mtardy as a code owner August 5, 2024 13:48

lambdanis approved these changes Aug 6, 2024

View reviewed changes

olsajiri merged commit 5ed9f84 into main Aug 6, 2024
47 checks passed

olsajiri deleted the pr/olsajiri/missed branch August 6, 2024 08:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add missed probes metrics #1941

Add missed probes metrics #1941

olsajiri commented Jan 8, 2024 •

edited

Loading

netlify bot commented Mar 28, 2024 •

edited

Loading

lambdanis Mar 29, 2024

olsajiri Mar 30, 2024

lambdanis Mar 29, 2024

olsajiri Mar 30, 2024

lambdanis left a comment

kkourt left a comment

olsajiri commented Jul 31, 2024

olsajiri commented Jul 31, 2024

lambdanis left a comment

olsajiri commented Aug 2, 2024

lambdanis commented Aug 2, 2024

lambdanis left a comment

Add missed probes metrics #1941

Add missed probes metrics #1941

Conversation

olsajiri commented Jan 8, 2024 • edited Loading

netlify bot commented Mar 28, 2024 • edited Loading

✅ Deploy Preview for tetragon ready!

lambdanis Mar 29, 2024

Choose a reason for hiding this comment

olsajiri Mar 30, 2024

Choose a reason for hiding this comment

lambdanis Mar 29, 2024

Choose a reason for hiding this comment

olsajiri Mar 30, 2024

Choose a reason for hiding this comment

lambdanis left a comment

Choose a reason for hiding this comment

kkourt left a comment

Choose a reason for hiding this comment

olsajiri commented Jul 31, 2024

olsajiri commented Jul 31, 2024

lambdanis left a comment

Choose a reason for hiding this comment

olsajiri commented Aug 2, 2024

lambdanis commented Aug 2, 2024

lambdanis left a comment

Choose a reason for hiding this comment

olsajiri commented Jan 8, 2024 •

edited

Loading

netlify bot commented Mar 28, 2024 •

edited

Loading