Initial support for BEAM (Erlang/Elixir)

[GregMefford]

Note

The first commit in the PR is so that this branch stacks on top of #288 while I work on this one.

I have begun to work on support for BEAM languages like Erlang and Elixir, and wanted to open this PR early as a draft, so that I can get any feedback you may have to help the process go more smoothly. I don't have much experience with Go or eBPF, so any feedback you have is very welcome. What I have so far is mostly based on digging through the existing support for other languages as well as the BEAM / OTP source code, and trying to understand how all the parts fit together.

I have also been digging through the BEAM / OTP source code, and also the gdb scripts that it includes for working directly with the memory image of a running system or core dump.

So far, I am able to see the logs from my Go code coming through, and confirming that it's working correctly as far as loading and attaching the interpreter support, like this:

$ sudo ./ebpf-profiler -collection-agent=127.0.0.1:11000 -disable-tls
INFO[0000] Starting OTEL profiling agent v0.0.0 (revision main-67f28f2b, build timestamp 1735697412)
INFO[0000] Interpreter tracers: perl,php,python,hotspot,ruby,beam
INFO[0000] Determined PAC mask to be 0x007F000000000000
INFO[0000] Found offsets: task stack 0x38, pt_regs 0x3eb0, tpbase 0x1c30
INFO[0000] Supports generic eBPF map batch operations
INFO[0000] Supports LPM trie eBPF map batch operations
INFO[0000] eBPF tracer loaded
INFO[0000] Attached tracer program
INFO[0000] Attached sched monitor
INFO[0026] BEAM interpreter found: [beam.smp]
INFO[0026] read symbol value etp_otp_release: 27
INFO[0026] read symbol value etp_erts_version: 15.1.3
INFO[0026] BEAM loaded, otp_version: 27, interpRanges: [{718368 718496}]
INFO[0026] BEAM interpreter attaching

However, I can't seem to get any tracing logs out of the eBPF program, so I suspect that it's never being run. If I modify the native eBPF script to write the same kind of log there, I can confirm that I'm seeing it in /sys/kernel/tracing/trace_pipe after doing the following to narrow down the logs I want to see, but the same doesn't work for my beam program:

# echo 1 > /sys/kernel/tracing/tracing_on
# echo 0 > /sys/kernel/tracing/events/enable
# echo 1 > /sys/kernel/tracing/events/bpf_trace/bpf_trace_printk/enable

I was thinking that this was because OTP 27 includes a JIT, so the interpreter might never be used, but I am also not seeing any frames for the native JIT code executing, so I'd love any advice you may have there in terms of how I might go about troubleshooting that. Maybe the native unwinder is just missing some heuristic that's needed for the way the ASMJIT / BEAMJIT works? I'm not clear on how the profiler resolves symbols for JIT code or how those should show up in devfiler, so maybe it is working and I just don't know how to use to the tool... 😅 But from what I can tell, I don't think the frames are showing up there for anything but the C code for Erlang itself (and built-in C functions). I was expecting to be able to see which Erlang code was running, for example.

I also tried building OTP 27 with the JIT disabled to confirm my theory that it just wasn't working, but it behaved the same (though with a different memory address showing for the interpRanges, which confirms that it really did build a different set of code).

[fabled]

I was thinking that this was because OTP 27 includes a JIT, so the interpreter might never be used, but I am also not seeing any frames for the native JIT code executing, so I'd love any advice you may have there in terms of how I might go about troubleshooting that.

Typically JIT is on mmaped anonymous memory. You will need to add hooks to call your unwinder for this memory mappings. For a generic catch it all example, see the v8 unwinder's code at https://github.com/open-telemetry/opentelemetry-ebpf-profiler/blob/main/interpreter/nodev8/v8.go#L544

If you can extract the exact memory area where JIT code exists directly from the VM, you can refer to hotspot unwinder code at https://github.com/open-telemetry/opentelemetry-ebpf-profiler/blob/main/interpreter/hotspot/instance.go#L784

Maybe the native unwinder is just missing some heuristic that's needed for the way the ASMJIT / BEAMJIT works?

The native winder will not have heuristic for it. You need to implement the code to hook your unwinder for the memory areas where JIT code is at (see above).

After that you'll need to have eBPF code that actually unwinds the JIT code. It might be simple if the JIT frame layout is frame pointer based, see e.g. v8 unwinder https://github.com/open-telemetry/opentelemetry-ebpf-profiler/blob/main/support/ebpf/v8_tracer.ebpf.c, or highly complicated if there is a custom frame layout, see e.g. hotspot unwinder https://github.com/open-telemetry/opentelemetry-ebpf-profiler/blob/main/support/ebpf/hotspot_tracer.ebpf.c. The unwinder will need to collect the extra needed by the symolization in the next step.

I'm not clear on how the profiler resolves symbols for JIT code or how those should show up in devfiler, so maybe it is working and I just don't know how to use to the tool... 😅

Once the unwinding is done, the core will code interpreter plugins symolization code which will need to extract the symbol data from the target process. Again, see some examples how its done for the hotspot https://github.com/open-telemetry/opentelemetry-ebpf-profiler/blob/main/interpreter/hotspot/instance.go#L860 or v8 https://github.com/open-telemetry/opentelemetry-ebpf-profiler/blob/main/interpreter/nodev8/v8.go#L1689.

But from what I can tell, I don't think the frames are showing up there for anything but the C code for Erlang itself (and built-in C functions). I was expecting to be able to see which Erlang code was running, for example.

Correct, you will need to implement both the unwinding and symbolization yourself. Depending on the VM internals, this can be highly complicated and extensive work that is needed to cover all the corner cases within the ebpf constraints.

[GregMefford]

Typically JIT is on mmaped anonymous memory. You will need to add hooks to call your unwinder for this memory mappings. For a generic catch it all example, see the v8 unwinder's code at https://github.com/open-telemetry/opentelemetry-ebpf-profiler/blob/main/interpreter/nodev8/v8.go#L544

If you can extract the exact memory area where JIT code exists directly from the VM, you can refer to hotspot unwinder code at https://github.com/open-telemetry/opentelemetry-ebpf-profiler/blob/main/interpreter/hotspot/instance.go#L784

Aha! Thanks, this was the connection I was missing. I was thinking that since I can't statically know about all the JIT code that might be generated in the future, I can't possibly add it all to the maps, but I believe the BEAM does have ways to pretty easily locate the memory of all the JITted code, so I'll dig into that and the v8 example.

The BEAM does use frame pointers, so I believe it should be relatively straightforward to figure out.

Thanks for the tips! ❤️ 🚀

[fabled]

INFO[0026] BEAM loaded, otp_version: 27, interpRanges: [{718368 718496}]

The range for the interpreter looks suspiciously small - only 128 bytes. This can be valid, if its just a small stub but guaranteed to be on stack for interpreter frames.

Alternatively, this could be a function doing something else that is not necessarily on stack when executing interpreted code.

You might want to double check which functions are on stack when executing interpreted code. If it can be a set of multiple functions (e.g. several functions with same signature tailcalling each other -- compiler can convert call to jump), you need to extract the range that covers all of these. It would become a problem if these functions are not contiguously in the executable area.

Aha! Thanks, this was the connection I was missing. I was thinking that since I can't statically know about all the JIT code that might be generated in the future, I can't possibly add it all to the maps, but I believe the BEAM does have ways to pretty easily locate the memory of all the JITted code, so I'll dig into that and the v8 example.

No problem. But in short, you'll need to manually extract those areas and then call UpdatePidInterpreterMapping. Internally, the ebpf "misuses" the network prefix lookup by looking up memory address from a prefix lookup map. So these calls should properly preprocess memory ranges to prefix lists as shown by the examples.

You can also provide little bit of context data for each memory area. This could be useful if there's some auxiliary data connected to each memory area the unwinder needs.

The BEAM does use frame pointers, so I believe it should be relatively straightforward to figure out.

Nice! Then v8 or dotnet unwinders are closest equivalents. The hotspot unwinder implements a custom frame layout. v8 is simplest because all the extra data is accessible directly from stack data. The dotnet is a step more complicated as mapping from PC to auxiliary data is non-trivial step.

Thanks for the tips! ❤️ 🚀

You're welcome. Looking forward to the BEAM support! Thank you for working on this!

[florianl]

FYI: I have opened a open-telemetry/semantic-conventions#1735 with OTel semconv to add a type for beam.

[GregMefford]

I had some more time today to make progress, by copying the v8 implementation of SynchronizeMappings, and also by using a JITDUMP file that the BEAM outputs when it's running with the frame pointer support that we'll want anyway for unwinding (It's the +JPperf true Erlang runtime flag). Either way, it looks like I'm seeing these new frames showing up in DevFiler, so I think that's forward progress, but I don't see any evidence that my EBPF code is getting called still.

With the "catch-all" mappings, I get logs like this out of the agent, so I assume that means it's working and there's just one large region of memory it's treating as BEAM JIT code:

INFO[0000] Enabling BEAM for 0xec8d54800000/0x4000000

With the fancier JITDump code, I get logs like this, so I am pretty sure it's parsing the file correctly. Those function names look sensible to me and they're function names I recognize as things I did include in my test app I'm running.

INFO[1164] JITDump Code Load 'Elixir.Mint.HTTP1.Parse':token_list_sep_downcase/2 @ 0xec8d55485834 (364 bytes)
INFO[1164] JITDump Code Load 'Elixir.Mint.HTTP1.Parse':transfer_encoding_header/1-CodeInfoPrologue @ 0xec8d554859a0 (52 bytes)

I was hoping to see logs from the eBPF code using the following (as root), but nothing is showing up:

echo 0 > /sys/kernel/tracing/events/enable
echo 1 > /sys/kernel/tracing/events/bpf_trace/bpf_trace_printk/enable
echo 1 > /sys/kernel/tracing/tracing_on
cat /sys/kernel/tracing/trace_pipe

This is the kind of thing I'm seeing in DevFiler, so it's looking promising that it's doing something, but I'm not clear on where to look next for signs of life / why the eBFP unwinder does not seem to be getting called like I was expecting.

[florianl]

I was hoping to see logs from the eBPF code using the following (as root), but nothing is showing up:

To get log lines using the DEBUG_PRINT macro, like here, you need to compile the eBPF code with the target debug-amd64 in support/ebpf before compiling the Go part of the agent code. Once done, you should be able to see output via bpftool prog tracelog.
Just using make debug-agent from the main Makefile will not call the debug-amd64 target in support/ebpf iirc.

[GregMefford]

To get log lines using the DEBUG_PRINT macro [...]

Ah, thanks! I wasn't sure how to make that work, which is why I did it this way instead, which did work when I put it in one of the other eBPF programs, but I don't see anything coming from my program still.

[florianl]

With #145 things changed a bit and I missed that part. Sorry that I have missed this one in the first place.

Here are steps I used to generate output to bpftool prog tracelog:

$ make debug-amd64 -C support/ebpf
$ make debug-agent
$ sudo ./ebpf-profiler -collection-agent=127.0.0.1:11000 -disable-tls -v

I missed, that -v is now coupled with the debug eBPF blobs that produce the output via DEBUG_PRINT. Hope this helps.

[GregMefford]

I spent some more time today to make progress on this, and was able to get past where I was stuck before, and now I am able to see that my eBPF unwinder is running (not sure yet if it's doing the correct thing, but it's doing something) and my Symbolize function is getting called on the Golang side. I think I can see what some of the next steps are in terms of making it work correctly, but I think I might be stuck on something that I'm still unsure how to solve.

In the output form the ebpf-profiler agent, I get occasional errors like the following, and I believe it's preventing from the trace showing up at all in DevFiler (I tried downloading the latest 0.11.0 as well, so that's the version I'm using).

ERRO[0037] Request failed: rpc error: code = InvalidArgument desc = unsupported frame kind: beam

Is this because of the PR you mentioned opening to add that as a supported type in the OTEL spec, and it hasn't yet been pulled into DevFiler? Or is there just somewhere in the ebpf-profiler code that it doing some validation that I need to update in my PR to allow that as a valid frame kind? I wasn't able to find that error string in the code, so I'm guessing it's coming from somewhere else, and given that it's an rpc error, I'm guessing it's DevFiler.

Thanks again for all your help!

[florianl]

Is this because of the PR you mentioned opening to add that as a supported type in the OTEL spec, and it hasn't yet been pulled into DevFiler?

Yes - devfiler uses a filter on frame types. This might change in the future. For the mean time and to enable you to continue your work, I have created a version of devfiler that handles beam frame types:

curl -L -H 'Authorization: 6ac0e3549c910483' -o 'devfiler-v0.11.0-beam.tar.gz' https://upload.elastic.co/d/7a3f64a4545109bdcc99a85ec80038f55f8b4522aa5d3978e2d56575a0246b79

Working on live processes can be tricky and reproducing edge cases can be hard. For that reason, I recommend looking into coredump. With the tool coredump one can import a core dump of a process and run all the Go and eBPF code in user space just like a regular Go test. Please feel free to ping me, if you need help.

[GregMefford]

Sorry it took so long to make some time to actually give it a try, but when I tried running the custom devfiler build on my M1 Mac using the apple-silicon version, it crashes with the following error. I'm not familiar with this kind of error, but it looks like it's trying to dynamically load a library that I don't have installed - should that have been statically compiled into the app instead?

-------------------------------------
Translated Report (Full Report Below)
-------------------------------------

Process:               devfiler [75900]
Path:                  /Applications/devfiler.app/Contents/MacOS/devfiler
Identifier:            org.nixos.devfiler
Version:               ???
Code Type:             ARM-64 (Native)
Parent Process:        launchd [1]
User ID:               501

Date/Time:             2025-02-09 14:05:58.3904 -0500
OS Version:            macOS 15.3 (24D60)
Report Version:        12
Anonymous UUID:        3B6A222A-F595-8AE9-0C62-BC866F9E1776

Sleep/Wake UUID:       CB6571CC-1695-408A-BF33-2158F4604E6C

Time Awake Since Boot: 190000 seconds
Time Since Wake:       2159 seconds

System Integrity Protection: enabled

Crashed Thread:        0

Exception Type:        EXC_CRASH (SIGABRT)
Exception Codes:       0x0000000000000000, 0x0000000000000000

Termination Reason:    Namespace DYLD, Code 1 Library missing
Library not loaded: /nix/*/libc++abi.1.0.dylib
Referenced from: <FBBC286B-508A-36A4-9257-CC5C94E09504> /Applications/devfiler.app/Contents/MacOS/devfiler
Reason: tried: '/nix/*/libc++abi.1.0.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/nix/*/libc++abi.1.0.dylib' (no such file), '/nix/*/libc++abi.1.0.dylib' (no such file), '/usr/local/lib/libc++abi.1.0.dylib' (no such file), '/usr/lib/libc++abi.1.0.dylib' (no such file, not in dyld cache)
(terminated at launch; ignore backtrace)

I'm planning to try to figure out how to get the core dump testing thing working, though, so not blocking at the moment. I just wanted to see whether it got me further along with the path I was on before, having a version of devfiler with support for the beam type.

[florianl]

sorry - this was not supposed to happen 🙏

[GregMefford]

With the updated version of devfiler that supports the Beam type, I think it's actually pretty close to working! I just need to sort out the actual symbolization, because for now I'm just sending a static "Some Bogus Name" for the file to see that it's working at all.