\documentclass[../thesis.tex]{subfiles}
\begin{document}

\chapter{eBPF} \label{ebpf}

\epigraph{Super powers have finally come to Linux.}{Brendann Gregg}

\emph{eBPF} is an instrumentation framework allowing to run sandboxed programs in a privileged context like in the Linux kernel~\cite{WhatEBPFIntroduction}. It allows to extend the capabilities of the kernel without changing the kernel source code or loading custom kernel modules. Efficiency and security of the executed programs are ensured thanks to JIT\footnote{Just-In-Time} compilation and a verification engine. We will use the technology in Chapter \ref{sysrapl} for the implementation of profiling capabilities of the energy profiler and thus we first briefly introduce its capabilities and features.

\section{Overview} \label{ebpf-overview}

At the core of \emph{eBPF} stands a virtual machine running in a kernel of an operating system. This virtual machine has its own instruction set and clearly defined rules for programs running in it. Rules are set because \begin{enumerate*}[\emph{\alph*)}] \item all \emph{eBPF} programs have to be guaranteed to finish, \item memory accesses are typed and bounded, and \item programs can have at maximum \verb|BPF_MAXINSNS| instructions (4096 by default)\end{enumerate*}. These rules are necessary for the verifier to be able to reliably and quickly check all loaded programs. A program can be loaded using the \verb|bpf()| system call which accepts \emph{eBPF} \emph{bytecode}. Such bytecode can either be hand-crafted or generated. The standard compiler for generating the bytecode is \emph{Clang} compiler containing a \emph{eBPF} back-end.

\emph{eBPF} programs run in kernel-space and very often they need to be configured before running or share data at runtime with user-space. For such purposes one can use so called \emph{maps}. They are generic storage types in kernel-space available to \emph{eBPF} programs and user-space. They can be used as hash tables, arrays or stacks. We can also use them for storing per-CPU data. A special map is ring buffer (\verb|BPF ringbuf|). It is ideal for continuous data delivery from kernel-space to user-space (e.g., for logging). It offers great performance and ensures data order. 

We mentioned that an \emph{eBPF} program bytecode needs to be loaded and often initialized before running. Instead of accomplishing that manually using the \verb|bpf()| system call this can also be accomplished using a \emph{loader}. A loader takes care of loading an \emph{eBPF} program, its initialization and how it is run.

After a \emph{eBPF} program is loaded and verified, it is then further submitted to JIT\footnote{Just In Time} compilation that optimizes the performance of the program. Overall, \emph{eBPF} is an instrumentation framework with excellent performance, high reliability and security, clear boundaries for use, and straightforward ways to use. 

\begin{figure}[h]
    \centering
    \includegraphics[width=\textwidth]{ebpf-architecture}
    \caption{A diagram of \emph{eBPF} architecture and how technologies utilizing it are structured.~\cite{WhatEBPFIntroduction}}
\end{figure}

%\section{History of eBPF}

%\emph{eBPF} historically standed for \emph{extended BPF} but that has not been the case for a while now. While originating in the \emph{Berkley Packet Filters} (\emph{BPF}) it has evolved into a technology of its own.

%Originally, \emph{BPF} was designed as a very simple virtual machine intended for filtering packets in kernel-space. The provided packet filters were compiled into programs which were executed in a register-based virtual machine running inside of a kernel. This has allowed to transition from the until then common approach of implementing packet filters in user-space which required copying of data across the kernel/user-space protection boundary. The created technology yielded up to 20 times faster filter evaluation and up to 100 times better performance than Sun’s \emph{NIT}\footnote{Network Interface Tap}.~\cite{mccannelBSDPacketFilter1993}

%A virtual machine with registers and close integration with the kernel provided a good foundation for further extensions of the module in the Linux kernel. Specifically, the virtual machine architecture was updated to match more closely modern architectures, the number of registers increased and the number of kernel subsystems interfacing with the virtual machine increased as well. In Linux 3.18, the \verb|bpf()| syscall was introduced allowing for loading and running eBPF programs in the form of a bytecode during system runtime. In Linux 4.4 the \verb|bpf()| syscall was made available to non-root users as well by extending and fortifying the eBPF verifier that checks all loaded eBPF programs~\cite{ThoroughIntroductionEBPF,BPFSystemCall,UnprivilegedBpfLWN}.

%\subsection{Adoption of eBPF} \label{ebpf-adoption}

%\emph{BPF} was originally~\cite{mccanneBSDPacketFilter1993} designed for UNIX systems and \emph{eBPF} was a result of work done on the Linux kernel. By design, it is also meant to live as close as possible to the kernel but the popularity and practicality of the technology has motivated the creation of projects like \emph{ubpf}~\cite{UBPF2023} which is an implementation of a \emph{eBPF} virtual machine in userspace licensed under the Apache license. Microsoft also introduced an implementation of \emph{eBPF} for Microsoft Windows called \emph{eBPF for Windows} which utilizes \emph{ubpf}~\cite{EBPFWindows2023,gaddehosurMakingEBPFWork2021, ImplementingEBPFWindows}.

\section{Developer toolchains}

Writing \emph{eBPF} bytecode by hand is not a recommended venture since the kernel ecosystem keeps evolving at an incredible pace. A possible solution is to leverage abstractions built on top of \emph{eBPF} like \emph{bcc} (\ref{ebpf-bcc}), \emph{bpftrace} (\ref{ebpf-bpftrace}), or \emph{libbpf} (\ref{ebpf-libbpf}), providing a more expressive environment for writing \emph{eBPF} programs. The abstractions universally depend on the Clang/LLVM toolchain implementing an \emph{eBPF} bytecode backend for compilation of \emph{eBPF} programs.

\subsection{BCC} \label{ebpf-bcc}

\begin{wrapfigure}{r}{0.20\textwidth}
    \includegraphics[width=0.20\textwidth]{ebpf-bcc-logo}
\end{wrapfigure}

\emph{BPF Compiler Collection} (\emph{BCC}) is a toolchain providing C, Python and Lua interface for writing \emph{eBPF} programs. The \emph{BCC} toolchain is based on the LLVM toolchain. It provides a BPF-specific front-end for C, making it easier to write valid \emph{eBPF} programs. Along the C language front-end, additional front-ends for Python and Lua are provided~\cite{BPFCompilerCollection2022}.

Since 2020, the Python interface of \emph{BCC} is considered deprecated for writing new performance tools~\cite{brendanBPFBinariesBTF2020}. It is recommended to use the C interface and \emph{bpftrace} for quick \emph{eBPF} scripting.

\subsection{bpftrace} \label{ebpf-bpftrace}

\emph{bpftrace} is a high-level frontend for \emph{eBPF} utilizing \emph{BCC} for interfacing with the \emph{eBPF} Linux system. It is ideal for writing \emph{one-liners} and short scripts. The \emph{bpftrace} language is inspired by \emph{awk}, \emph{C} language and tracers like \emph{DTrace} and \emph{SystemTap}.~\cite{Bpftrace}

An example of a \emph{bpftrace} one-liner counting the number of page faults by process is:

\begin{lstlisting}[language=sh]
$ bpftrace -e 'software:faults:1 { @[comm] = count(); }'
\end{lstlisting}

\subsection{libbpf} \label{ebpf-libbpf}

\emph{libbpf} is a user-space \emph{C}/\emph{C++} \emph{eBPF} loader library providing APIs for interacting with the \emph{eBPF} Linux system. \emph{libbpf} bootstraps the loaded \emph{eBPF} object file, allowing for hooking up to different phases of \emph{eBPF} program lifetime, and providing useful API similarly to \emph{BCC} (albeit not as high-level). The development of \emph{libbpf} over the years lead to closing the gap in features between \emph{BCC}, fixing issues in the API or addressing scenarios of incompatibilities between different versions of the kernel. This work culminated in the release of version 1.0 in August 2022.~\cite{nakryikoJourneyLibbpf2022}
    
The lifetime of \emph{eBPF} programs has four phases for which \emph{libbpf} provides a way to define them and execute them. These phases are~\cite{nakryikoBCCLibbpfConversion2020}:

\begin{enumerate}[\arabic*)]
    \item \textbf{Open phase} loads up the \emph{eBPF} object file into the memory, upon all present \emph{eBPF} programs, maps and global variables are made available. It allows one to make adjustments to the structures because the program has not been loaded into the kernel, yet.

    \item \textbf{Load phase.} creates \emph{eBPF} maps, verifies the programs and loads them into the kernel, but the programs are not executed, yet. At this phase it is still possible to adjust the state of \emph{eBPF} maps.

    \item \textbf{Attachment phase.} starts the \emph{eBPF} program by attaching it to its defined hook points.

    \item \textbf{Teardown phase.} detaches and unloads \emph{eBPF} programs from the kernel. All \emph{eBPF} resources (e.g., maps) are freed.
\end{enumerate}

\subsubsection{bpftool} \label{ebpf-bpftool}

To make it convenient for programmers to work with \emph{eBPF} programs exists the \emph{bpftool} tool. It serves for inspecting and manipulating \emph{eBPF} programs and maps. One can also use it to generate a \emph{skeleton header file} from an object file of an \emph{eBPF} program to ease the development of \Gls{userspace} programs. The generated header file contains the \emph{eBPF} program bytecode, provides functions for loading it and provides typed access to all maps, programs, global variables and such.

\section{BPF CO-RE}

\emph{BPF Compile Once \textendash Run Everywhere} (\emph{BPF CO-RE})~\cite{nakryikoBPFCORECompile2020} is an approach to creating \emph{eBPF} programs that solves many issues with portability of \emph{eBPF} programs. The technology was introduced at the LSF conference in 2019~\cite{LinuxKernelDevelopers}. Many \emph{eBPF} tools originally written using BCC or other toolchains have already been converted to \emph{BPF CO-RE}.~\cite{nakryikoBPFCORECompile2020}

Portability of \emph{eBPF} programs is troublesome. The user-written \emph{eBPF} programs need to work within the set kernel environment over which they have little to no control. Any program requiring access to raw internal kernel data can not rely on any notion of stability from the kernel. \emph{BCC} tackles this problem by embedding an \emph{eBPF} program in a user-space binary and compiling it on-the-fly when requested. The compilation uses \emph{Clang/LLVM}, embedded in \emph{BCC}, and kernel headers. Programs created by \emph{BCC} then require it to be installed on the system to compile the programs on-the-fly and the programs are inherently unstable due to possible \Gls{abi}/\Gls{api} change in the Linux kernel (e.g., they can contain unhandled cases of renamed structures across kernel versions). Note that, kernel headers are not installed on systems by default which are required for \emph{Clang/LLVM} to be able to compile an \emph{eBPF} program.~\cite{nakryikoBPFCORECompile2020}

\emph{BPF CO-RE} overcomes the portability problem of \emph{eBPF} programs by leveraging several functionalities in the used components: kernel info format (\emph{BTF}), compiler (\emph{Clang}), and user-space \emph{eBPF} loader library (\emph{libbpf}).

\subsection{BTF}

\emph{BTF} stands for \emph{BPF Type Format}. It is a debugging data format which is an alternative to the \emph{DWARF} format. It achieves up to 100x size reduction over \emph{DWARF} while containing all necessary type information of C programs. Having such a space-efficient data format makes it feasible for including it in Linux operating systems for the Linux kernel by default. The information is available at path \verb|/sys/kernel/btf/vmlinux| and can be used to generate a C header file using the tool \verb|bpftool|.~\cite{nakryikoBPFCORECompile2020}

\subsection{Clang}

The \emph{Clang} compiler was extended with a certain feature. It makes it possible to track field existence, removals, offset relocations or size changes. These bits of information are then emitted for the \emph{eBPF} program loader to pick up and use it for making the necessary adjustments.~\cite{nakryikoBPFCORECompile2020,nakryikoBCCLibbpfConversion2020}

\subsection{libbpf}

Before loading an \emph{eBPF} program and then submitting it to the verification engine, \emph{libbpf} inspects the \emph{eBPF} object file and, if necessary, makes adjustments to it to match the \emph{BTF} information of the running kernel. These adjustments are done completely transparently and automatically.~\cite{nakryikoBPFCORECompile2020}

\section{eBPF in practice}

Since its creation \emph{eBPF} has been constantly growing in popularity and the number of use cases for it only keep growing. Nowadays it is possible to not just monitor and profile but also change the behaviour of the kernel at runtime~\cite{HIDBPFLinuxKernel}. Some noteworthy \emph{eBPF} projects are:

\begin{itemize}
    \item \emph{libbpf-tools} and \emph{bcc-tools} are collections of small but very useful \emph{eBPF} programs contributed by the members of the community made available for all users to use for profiling.~\cite{BPFCompilerCollection2022}
    \item \emph{ebpf\_exporter} is a metrics collector/exporter for Prometheus~\cite{prometheusPrometheusMonitoringSystem} created by CloudFlare.~\cite{EbpfExporter2023}
    \item \emph{bpfilter} is a direct competitor of \emph{iptables} and \emph{nftables}. It offers a compatibility layer translating iptables rules into \emph{eBPF} programs while also offering the possiblity of writing firewall rules in C.~\cite{corbetBPFComesFirewalls2018}
    \item \emph{wachy} is a dynamic tracing profiler providing a TUI\footnote{Text-based User Interface}.~\cite{Wachy2022}
\end{itemize}

\section{Alternative technologies}

In this chapter we've introduced the \emph{eBPF} instrumentation framework, but there exist alternative instrumentation frameworks. In this section we'd like to introduce some of the alternatives we've considered when selecting the appropriate technology for the implementation of tracing capabilities of the created energy profiler.

% table for comparison

\begin{table}[h]
\centering
\begin{tabular}{|c|c|c|c|} 
    \hline
    & eBPF & SystemTap & Pin \\ 
    \hline
    Kernel-space instrumentation & \checkmark & \checkmark & \\
    \hline
    User-space instrumentation & \checkmark & \checkmark & \checkmark \\
    \hline
    Instrumentation stability & \checkmark & & \checkmark \\
    \hline
    Instruction set specific support & & & \checkmark \\
    \hline
    Requires kernel debuginfo & & \checkmark & \\
    \hline
    Relies on recent kernel version & \checkmark & \checkmark & \\
    \hline
\end{tabular}
\caption{A comparison of instrumentation frameworks \emph{eBPF}, \emph{SystemTap} and \emph{Pin}}
\label{table:ebpf-alternatives}
\end{table}

\paragraph{SystemTap} is a general-purporse tracing and profiling framework for the Linux kernel. It supports a large number of mechanisms for gathering data about both the kernel and processes running on the system. In general it is regarded as one of the most powerful profiling frameworks. \emph{SystemTap} instruments code by loading a a custom kernel module into the Linux kernel. These modules are created automatically by \emph{SystemTap} from custom scripts which are then first transpiled into the C language. For most of its functionality, \emph{SystemTap} requires for the Linux kernel to be present together with the \emph{SystemTap} program. And because \emph{SystemTap} is implemented out-of-tree of the Linux kernel and is not designed with safety/resiliency first, it has a history of causing system lock ups, freezes or even kernel panics.~\cite{FITBT23847}

\paragraph{Intel Pin} is a dynamic binary instrumentation framework developed by Intel enabling the creation of dynamic program analysis tools. \emph{Pin} is mainly used in tracing of \Gls{userspace} applications without the need to recompile the traced binaries. It also provides cross-platform support and in general offers stellar performance. \emph{Pin} instruments code on the instruction level by using \Gls{jit} compiler. \emph{Pin} ensures that the original behaviour of the analysed software is kept even during profiling. \emph{Pin} operates on a level above the operating system and thus can only capture user-space code. It also supports only a limited number of instruction sets (IA-32, x86-64 and MIC).~\cite{FITBT23847,PinDynamicBinary}

\end{document}