Convert BCC style source to libbpf style source and enable AOT/CO-RE for BCC framework

[yunwei37]

Hi! I have some working PoC and ideas, and I would like to get some comments or feedbacks before I going on this.

motivation

• Alrough the use of bcc for new BPF applications is strongly discouraged, there still exists a lot of tools base on bcc jit compile framework in the community. A number of bcc tools havn't been converted to libbpf in this repo yet. see [DRAFT Proposal] bcc / tools repo evolution #3976.
• A source to source converter can speed up python->libbpf tool conversions, which are encouraged [DRAFT Proposal] bcc / tools repo evolution #3976. Compared to generated libbpf style ELF file, some smaller code issues can be found and fix mannually in the converted libbpf source.

A Prove of Concept converter for convert BCC style kernel source to libbpf style kernel source

we have created a source to source converter base on bcc frontend: see https://github.com/eunomia-bpf/bcc/tree/master/src/cc/converter

This work will introduce:

• A new clang frontend action: https://github.com/eunomia-bpf/bcc/blob/master/src/cc/frontends/clang/libbpf_frontend_action.cc, modified on the exising b_frontend_action.
• Some new include headers: https://github.com/eunomia-bpf/bcc/tree/master/src/cc/converter/bcc, modified on existing bcc internal headers.

A new flag aot_mode and an example converter may also works:

• (BPF.h#L116)[https://github.com/eunomia-bpf/bcc/blob/a4e1c71cedbab57f75b0a617a40323c831be73f8/src/cc/api/BPF.h#L116]
• (ConvertBCCtoLibbbpf)[https://github.com/eunomia-bpf/bcc/blob/master/examples/cpp/ConvertBCCtoLibbbpf.cc]

The BPF(bool aot_mode = true) may be used to enable the converter and aot build.

The converter may include two passes to generate a libbpf source from bcc source:

1. libbpf_frontend_action:

   • change the map access to libbpf style map access, for example, bpf_map_update_elem
   • change the bpf_probe_read* to bpf_core_read*
   • change the a->b->c access to BPF_CORE_READ(a, b, c)
   • change some bcc internal helpers to libbpf helpers, for example, bpf_map_lookup_or_try_init
   • etc...

2. preprocesseo pass: work like clang -E -P -C -nostdinc source_rewrite.bcc.c 1> pre_output.bpf.c

   • change the bcc style maps to libbpf style maps
   • process and cleanup the bcc specified macros
   • note: for this poc, the current preprocessor pass is done through makefile and clang cmd, which should be embed in bcc as the internal bcc includes in export.
   • After the preprocessor pass, some libbpf specified includes will be added into the generated source file.

example

A bcc source:

#include <linux/ptrace.h>

struct query_probe_t {
  uint64_t ts;
  pid_t pid;
  char query[100];
};

BPF_HASH(queries, struct query_probe_t, int);

int probe_mysql_query(struct pt_regs *ctx, void* thd, char* query, size_t len) {
  if (query) {
    struct query_probe_t key = {};

    key.ts = bpf_ktime_get_ns();
    key.pid = bpf_get_current_pid_tgid();

    bpf_probe_read_user_str(&key.query, sizeof(key.query), query);
    bpf_trace_printk("Hello, World! Here I did a sys_clone call! %s\n", key.query);

    int one = 1;
    queries.update(&key, &one);
  }
  return 0;
}

will results in:

#include <bpf/bpf_core_read.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <vmlinux.h>

#include "bits.bpf.h"
#include "maps.bpf.h"

struct query_probe_t {
  uint64_t ts;
  pid_t pid;
  char query[100];
};
struct {
  __uint(type, BPF_MAP_TYPE_HASH);
  __uint(max_entries, 10240);
  __type(key, struct query_probe_t);
  __type(value, int);
} queries SEC(".maps");
SEC("kprobe")
int probe_mysql_query(struct pt_regs *ctx) {
  void *thd = (void *)ctx->di;
  char *query = (char *)ctx->si;
  size_t len = (size_t)ctx->dx;
  if (query) {
    struct query_probe_t key = {};
    key.ts = bpf_ktime_get_ns();
    key.pid = bpf_get_current_pid_tgid();
    bpf_core_read_user_str(&key.query, sizeof(key.query), query);
    bpf_printk("Hello, World! Here I did a sys_clone call! %s\n", key.query);
    int one = 1;
    bpf_map_update_elem(&queries, &key, &one, BPF_ANY);
  }
  return 0;
}

/* No license defined, using GPL
 * You can define your own BPF_LICENSE in your C code */
char LICENSE[] SEC("license") = "GPL";

which can be compile with clang and CO-RE enabled, and load with libbpf loader. The compiled BPF code can pass the verifier.

Unsolved problems

1. TracepointFrontendAction support for libbpf source
2. fix more bcc maps and helpers support
3. tests on more existing bcc style code

AOT support

I have tried to AOT load the compiled libbpf code through the bpf-loader in eunomia-bpf: https://github.com/eunomia-bpf/eunomia-bpf, maybe I can add a similar libbpf CO-RE ELF loader in bcc? It should not be difficult to add one with the high level libbpf API, compare to the bcc frontend implement.

(TODO: add more detail design and api examples for aot build)

reference

• compiling BCC stype C code into ELF format #3090
• Ahead-of-time compilation #1887
• support load objects #2564
• https://dxuuu.xyz/aot-bpftrace.html

[anakryiko]

int probe_mysql_query(struct pt_regs *ctx) {
  void *thd = (void *)ctx->di;
  char *query = (char *)ctx->si;
  size_t len = (size_t)ctx->dx;

this part will be x86-specific for not good reason. Consider utilizing libbpf's BPF_KPROBE/BPF_UPROBE macros, which in this case would be:

int BPF_UPROBE(probe_mysql_query, void *thd, char* query, size_t len) {
}

Note that BPF_UPROBE is just an alias to BPF_KPROBE, added very recently, so depending on how recent libbpf you have, you might need to stick to BPF_KPROBE instead.

[yunwei37]

Thanks! I will continue to improve this.