The main BTFhub repository serves as a comprehensive resource, housing documentation, tools, and examples to guide users on how to leverage the BTF files effectively. However, the actual BTF files are stored separately in the BTFhub-Archive repository. This separation ensures a clean and organized structure, with each repository focusing on its designated role.
The Extended Berkeley Packet Filter (eBPF) is esteemed for its portability, a primary attribute of which is due to the BPF Type Format (BTF). More details about BTF can be discovered in this comprehensive guide.
Before the advent of Compile Once-Run Everywhere (CO-RE), developers working with eBPF had to compile an individual eBPF object for each kernel version they intended to support. This stipulation led toolkits, such as iovisor/bcc, to depend on runtime compilations to handle different kernel versions.
However, the introduction of CO-RE facilitated a significant shift in eBPF portability, allowing a single eBPF object to be loaded into multiple differing kernels. This is achieved by the libbpf loader, a component within the eBPF's loader and verification architecture. The libbpf loader arranges the necessary infrastructure for an eBPF object, including eBPF map creation, code relocation, setting up eBPF probes, managing links, handling their attachments, among others.
Here's the technical insight: both the eBPF object and the target kernel contain BTF information, generally embedded within their respective ELF (Executable and Linkable Format) files. The libbpf loader leverages this embedded BTF information to calculate the requisite changes such as relocations, map creations, probe attachments, and more for an eBPF object. As a result, this eBPF object can be loaded and have its programs executed across any kernel without the need for object modification, thus enhancing portability.
Regrettably, the BPF Type Format (BTF) wasn't always readily available. This can be attributed to either a lack of kernel support or the absence of userland tools capable of interpreting the BTF format. As a result, certain Linux distributions ended up releasing kernels without embedded BTF information.
This is the precise reason behind the existence of BTFhub. BTFhub's primary function is to supply BTF information for those Linux kernels that were released by distributions without this information embedded. Instead of requiring you to recompile your eBPF code for each existing Linux kernel that lacks BTF support, your code will be relocated by libbpf based on the available BTF information fetched from BTFhub's files.
The libbpf's support for external (raw) BTF files, which started with this commit, enables us to supply libbpf with an external BTF file corresponding to the kernel you intend your eBPF code to run on. It's important to note that each kernel requires its unique BTF file.
Please note that BTFhub's use is not universally necessary. If your intent is to support your eBPF CO-RE application solely on the most recent kernels, you will not need BTFhub. However, if you aim to support all released kernels, which include versions from some Long Term Support Linux distributions, then BTFhub may prove to be an indispensable resource.
This is a list of existing distributions and their current status on eBPF and BTF support.
Tracee, a runtime security and tracing tool for Linux, serves as a leading example of effective utilization of BTFhub. Tracee incorporates a script that downloads the contents of both the BTFhub and BTFhub Archive repositories.
This script then uses the BTFhub scripts to generate tailored BTF files that are exceptionally small in size. The result is a streamlined integration process and an efficient method for handling BTF files, demonstrating the power of BTFhub and its scripts when used effectively.