Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
For now, there isn't a way to set and get per-function metadata with a low overhead, which is not convenient for some situations. Take BPF trampoline for example, we need to create a trampoline for each kernel function, as we have to store some information of the function to the trampoline, such as BPF progs, function arg count, etc. The performance overhead and memory consumption can be higher to create these trampolines. With the supporting of per-function metadata storage, we can store these information to the metadata, and create a global BPF trampoline for all the kernel functions. In the global trampoline, we get the information that we need from the function metadata through the ip (function address) with almost no overhead. Another beneficiary can be ftrace. For now, all the kernel functions that are enabled by dynamic ftrace will be added to a filter hash if there are more than one callbacks. And hash lookup will happen when the traced functions are called, which has an impact on the performance, see __ftrace_ops_list_func() -> ftrace_ops_test(). With the per-function metadata supporting, we can store the information that if the callback is enabled on the kernel function to the metadata. Support per-function metadata storage in the function padding, and previous discussion can be found in [1]. Generally speaking, we have two way to implement this feature: 1. Create a function metadata array, and prepend a insn which can hold the index of the function metadata in the array. And store the insn to the function padding. 2. Allocate the function metadata with kmalloc(), and prepend a insn which hold the pointer of the metadata. And store the insn to the function padding. Compared with way 2, way 1 consume less space, but we need to do more work on the global function metadata array. And we implement this function in the way 1. We implement this function in the following way for x86: With CONFIG_CALL_PADDING enabled, there will be 16-bytes(or more) padding space before all the kernel functions. And some kernel features can use it, such as MITIGATION_CALL_DEPTH_TRACKING, CFI_CLANG, FINEIBT, etc. In my research, MITIGATION_CALL_DEPTH_TRACKING will consume the tail 9-bytes in the function padding, and FINEIBT + CFI_CLANG will consume the head 7-bytes. So there will be no space for us if MITIGATION_CALL_DEPTH_TRACKING and CFI_CLANG are both enabled. In x86, we need 5-bytes to prepend a "mov %eax xxx" insn, which can hold a 4-bytes index. So we have following logic: 1. use the head 5-bytes if CFI_CLANG is not enabled 2. use the tail 5-bytes if MITIGATION_CALL_DEPTH_TRACKING is not enabled 3. compile the kernel with extra 5-bytes padding if MITIGATION_CALL_DEPTH_TRACKING and CFI_CLANG are both enabled. In the third case, we compile the kernel with a function padding of 21-bytes, which means the real function is not 16-bytes aligned any more. And in [2], I tested the performance of the kernel with different padding, and it seems that extra 5-bytes don't have impact on the performance. However, it's a huge change to make the kernel function unaligned in 16-bytes, and I'm not sure if there are any other influence. So another choice is to compile the kernel with 32-bytes aligned if there is no space available for us in the function padding. But this will increase the text size ~5%. (And I'm not sure with method to use.) We implement this function in the following way for arm64: The per-function metadata storage is already used by ftrace if CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS is enabled, and it store the pointer of the callback directly to the function padding, which consume 8-bytes, in the commit baaf553 ("arm64: Implement HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS") [3]. So we can directly store the index to the function padding too, without a prepending. With CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS enabled, the function is 8-bytes aligned, and we will compile the kernel with extra 8-bytes (2 NOPS) padding space. Otherwise, the function is 4-bytes aligned, and only extra 4-bytes (1 NOPS) is needed. However, we have the same problem with Mark in [3]: we can't use the function padding together with CFI_CLANG, which can make the clang compiles a wrong offset to the pre-function type hash. He said that he was working with others on this problem 2 years ago. Hi Mark, is there any progress on this problem? I tested this function by setting metadata for all the kernel function, and it consumes 0.7s for 70k+ functions, not bad :/ Maybe we should split this patch into 3 patches :/ Link: https://lore.kernel.org/bpf/CADxym3anLzM6cAkn_z71GDd_VeKiqqk1ts=xuiP7pr4PO6USPA@mail.gmail.com/ [1] Link: https://lore.kernel.org/bpf/CADxym3af+CU5Mx8myB8UowdXSc3wJOqWyH4oyq+eXKahXBTXyg@mail.gmail.com/ [2] Signed-off-by: Menglong Dong <[email protected]>
- Loading branch information
1 parent
16566af
commit dbc51df