\documentclass[../thesis.tex]{subfiles}
\begin{document}

\chapter{Existing Methods and Technology} \label{existing-methods-technology}

In Chapter \ref{background-related-work} we introduced a number of methods for energy profiling. In this Chapter we will further describe them in the context of this work. We will emphasize on their strengths and limitations and discuss their potential usage for the created energy profiler.

\section{RAPL} \label{existing-methods-technology-rapl}

\emph{Running Average Power Limit} (\emph{RAPL}) is an interface introduced by Intel in the Sandy Bridge architecture of Core processors. It allows controlling the power consumption limits. This ability to control also comes with the ability to monitor. Originally this feature was backed by purely software models of consumption. With the Haswell architecture the interface became backed by hardware power meters embedded in the chip which significantly raised the accuracy of the power readings. The interface is made of non-architectural \Glspl{msr}.~\cite{khanRAPLActionExperiences2018}

This thesis is set in the context of the Linux kernel, so the following paragraphs will use its terminology for the \emph{RAPL} interface (mainly domains).

The \emph{RAPL} interface exposes several domains (visualized in Figure \ref{fig:rapl-domains}):

\begin{enumerate}[\emph{\alph*)}]
    \item \textbf{cores} shows the power consumption of all cores in a package,
    \item \textbf{pkg} shows the power consumption of all cores in a package together with \Gls{llc-cache},
    \item \textbf{gpu} shows the power consumption of so called \emph{uncore} device (on desktop this typically corresponds to integrated GPU),
    \item \textbf{dram} shows the power consumption of DRAM,
    \item \textbf{psys} shows the power consumption of the whole platform or System-on-Chip (SoC).
\end{enumerate}

\begin{figure}[h]
    \centering
    \includegraphics[width=0.4\textwidth]{rapl-domains}
    \caption{Power domains supported by \emph{RAPL} (visualization~based~on~\cite{khanRAPLActionExperiences2018})}
    \label{fig:rapl-domains}
\end{figure}

The different domains provide a reasonable level of insight into a systems energy consumption. But it does not cover the whole system. The interface is closely tied to the CPU and the components closely tied to it. It does not provide insight into other areas of the system including peripherals, Wi-Fi modules, Bluetooth modules and more. Relying purely on the \emph{RAPL} interface as the only source of energy consumption will cause the resulting profile not to have all energy consumption information of the system. Despite the fact, prior art~\cite{10.5555/2735522.2735546,aggarwalGreenAdvisorToolAnalyzing2015,pathakWhereEnergySpent2012} shows that the power readings correlate well with the whole system's power consumption and thus the data incompleteness is not fatal.

% at first only for their devices then made into more generic interface in Linux
The interface was first made available in the Linux kernel in version 3.14~\cite{weaverLinuxSupportPower} and it was an interface implemented only for Intel CPUs. In a later version of the Linux kernel the interface has been generalized so that other hardware drivers can make use of it. This made it possible for AMD to add support for their AMD Ryzen processors (though in a limited form). The AMD implementation originally exposed only the \emph{package} domain and recently exposed also the \emph{core} domain. If the other \Glspl{msr} for other domains are present, they are deemed unreliable enough to not be made available.

% precision? (mention papers); trustworthy? (mention papers)
The precision of the \emph{RAPL} interface is important for assuring high quality profiling data. It does differ across vendors and architectures. The interface exposes for all domains three values: \begin{enumerate*}[\emph{\alph*)}]\item an actual \emph{value} \item \emph{unit} usually as a string with the name of the unit \item \emph{scale} used for scaling the \emph{value} to fit the \emph{unit}\end{enumerate*}. E.g., on Intel Skylake processor architecture the \emph{unit} is \verb|Joules| and its \emph{scale} is \verb|2.3283064365386962890625e-10|, making a single increment of \emph{value} an increment of \verb|0.23 nJ| (nano-joule).

High precision of the \emph{RAPL} interface is not a guarantee of high accuracy. A number of researches~\cite{hackenbergEnergyEfficiencyFeature2015,khanRAPLActionExperiences2018,garciaExplorationEnergyConsumption2019} on the accuracy of the \emph{RAPL} has already been conducted. The results generally show that the interface provides data with high level of correlation with data measured by AC power meters~\cite{khanRAPLActionExperiences2018,garciaExplorationEnergyConsumption2019}. The quality of data provided by the interface has even been increased, since its first introduction in the Sandy Bridge architecture, thanks to the transition from pure software models to \emph{fully integrated voltage regulators} (\emph{FIVR})~\cite{hackenbergEnergyEfficiencyFeature2015}.

\subsection{Reading the values}

In the Linux kernel there exist several ways of reading the hardware information. In the following, we will list these ways together with their basic principles and caveats.

For the purposes of this thesis a single optimal approach needed to be chosen.

\subsubsection{sysfs}

The first approach uses of \emph{sysfs}, a pseudo file system in the Linux Kernel available at the \verb|/sys| path in. In particular, the \emph{RAPL} interface is available under \texttt{/sys/class/powercap/intel-rapl}. This approach is the one with the lowest entry barrier as it only involves reading the content of files and thus requires no special knowledge. But for the purposes of this work this approach is not optimal due to its inefficiency for high frequency measurements. The inefficiency is caused by the file \Gls{api} not being designed for high frequency re-reads of the same file.

\subsubsection{perf\_events} \label{existing-methods-technology-rapl-perf-events}

% what is perf, what it is used for; why not use it directly?

\emph{Perf} is a tracing and profiling tool developed in parallel with the Linux kernel allowing instrumentation of CPU performance counters, tracepoints, kprobes\footnote{kernel probes}, uprobes\footnote{user probes} and a lot more. Alternatively it can be referred to as \verb|perf_events|. One can make use of \emph{perf} directly using the \emph{perf} CLI application which has a broad offering of features or using the \verb|linux/perf_event.h| C header file to create individual applications (for more advanced usage). \verb|perf_events| are quite efficient at being used for high frequency tracing.

A minimal example of using \verb|perf_events| from C code is:

\begin{lstlisting}[language=C]
#include <linux/perf_event.h>
#include <sys/syscall.h>
#include <unistd.h>

struct perf_event_attr attr;
long long clock;
int fd;

attr.type = PERF_TYPE_SOFTWARE;
attr.config = PERF_COUNT_SW_CPU_CLOCK;
attr.freq = 1;
attr.sample_period = frequency;

fd = syscall(SYS_perf_event_open, &attr, 0, 0, -1, 0);
read(fd, &clock, sizeof(clock));
close(fd);
\end{lstlisting}

% how to know what is supported
\emph{RAPL} perf events belong to a certain category of events: Kernel PMU\footnote{Performance Monitoring Unit} events. Since this category of events depends on the hardware architecture, their availability is not assured. One can check the availability of different perf events using the \emph{perf} CLI application by using the \verb|perf list| command or one can search the content of \emph{sysfs} (\verb|/sys|) for the different events.

% dedicated hardware counters -> no need for multiplexing

The issue of watching multiple PMU perf events is that it cause so called \emph{time multiplexing}. During the multiplexing the kernel switches which event uses the capabilities of the hardware. This phenomenon skews the monitored data which need to be scaled to at least approximate of the true data. Thankfully, this phenomenon does not happen with \emph{RAPL} because at least on Intel architectures the interface has its own dedicated counters.

\subsubsection{Comparing the approaches}

To demonstrate the two approaches to reading energy consumption via the \emph{RAPL} interface, we will conduct a series of benchmarks using different sample programs utilizing the different approaches. In total we will conduct three benchmarks:

\begin{enumerate}[\textbf{\alph*)}]
    \item \textbf{sysfs}, where a C~program reads at a set interval (using \texttt{<sys/timerfd.h>}) of 9999Hz the content of \emph{RAPL} files in \texttt{sysfs},
    \item \textbf{perf-userspace}, where a C~program reads at a set interval (using \texttt{<sys/timerfd.h>}) of 9999Hz perf events of all \emph{RAPL} domains,
    \item \textbf{perf-ebpf}, where a C~program reads at a set interval of 9999Hz perf events of all \emph{RAPL} domains using an \emph{eBPF} program.
\end{enumerate}

For this benchmark we use the same testing environment as in Chapter \ref{experiments}. In the benchmarks the provider of the results are the \texttt{perf} and \texttt{bpftool} utilities. The \texttt{perf} utility is run as \texttt{perf stat -e cycles -e instructions -e L1-dcache-loads -e LLC-load-misses -p <benchmark-pid> --timeout 5000} where \texttt{cycles}, \texttt{instructions}, \texttt{L1-dcache-loads}, and \texttt{LLC-load-misses} are the benchmarked statistics, \texttt{<benchmark-pid>} is the \Gls{pid} of the benchmarked program which was started prior to the start of the \texttt{perf} utility, and \texttt{5000} is the duration in milliseconds for which the \texttt{perf} utility is run. The \texttt{bpftool} utility is run as \texttt{bpftool prog profile name collect\_rapl\_info duration 5 cycles instructions l1d\_loads llc\_misses} where \texttt{collect\_rapl\_info} is the name of the profiled \emph{eBPF} program, \texttt{5} is the duration in seconds for which the \texttt{bpftool} utility is run, and \texttt{cycles}, \texttt{instructions}, \texttt{l1d\_loads}, and \texttt{llc\_misses} are the benchmarked statistics. The used statistics are based on the \texttt{bpftool} utility which only supports a limited number of statistics and can only measure four at a single time. In benchmarks \emph{sysfs} and \emph{perf-userspace} we only use the \texttt{perf} utility as the source of the results but in benchmark \emph{perf-ebpf} we use both the \texttt{perf} and \texttt{bpftool} utilities because effectively there are two programs running during the benchmark. One in \Gls{userspace} and the other in \Gls{kernelspace}.

Table \ref{table:existing-methods-technology-performance} shows the results of the benchmarks. The \emph{perf-ebpf} test case contains two rows of values where the top row contains the statistics for both the \Gls{userspace} and the \Gls{kernelspace} (\emph{eBPF}) programs and the bottom row contains the statistics for only the \emph{eBPF} program. The results show that the most inefficient approach to high-frequency reading energy consumption via the \emph{RAPL} interface is the \texttt{sysfs} approach. Of the two programs utilizing perf events the more efficient is the one using \emph{eBPF} reading the perf event counters in \Gls{kernelspace}.

Based on the results of the benchmarks we will use for the implementation of the energy consumption reading capabilities in Chapter \ref{sysrapl} the \emph{perf\_event} approach implemented using the \emph{eBPF} technology.

\begin{table}[h]
\centering
\begin{tabular}{|l|r r r r|} 
    \hline
    Benchmark & cycles & instructions & L1-dcache-loads & LLC-load-misses \\
    \hline
    \textbf{sysfs} & $3,795,412,368$ & $3,069,819,731$ & $777,323,949$ & $56,504$ \\
    \hline
    \textbf{perf-userspace} & $1,174,388,152$ & $769,703,008$ & $191,072,023$ & $50,263$ \\
    \hline
    \multirow{2}{6em}{\textbf{perf-ebpf}} & $760,739,586$ & $656,435,608$ & $158,702,720$ & $32,285$ \\
    & $49,995$ & $370,994,417$ & $52,538,065$ & $8,491$ \\
    \hline
\end{tabular}
\caption{Results of the benchmarks the efficiency of the different approaches to reading energy consumption values via the \emph{RAPL} interface.}
\label{table:existing-methods-technology-performance}
\end{table}

\section{System calls} \label{existing-methods-technology-syscalls}

System calls are an interface for services of the operating system made available to the user. The purpose of system calls varies greatly but they can be roughly classified into six major groups: \begin{enumerate*}[label=\emph{\alph*)}]\item process control, \item file manipulation, \item device manipulation, \item information maintenance, \item communications, and \item protection\end{enumerate*}~\cite{silberschatzOperatingSystemConcepts2013}. All programs, even if not directly, make use of system calls. This makes them a good candidate for giving energy usage the context of why is energy being used at a certain point of time.

% tracking of system calls; what does it provide and what can it be used for
% Why not strace?

System calls can be traced in multiple ways. The commonly known approach uses the \verb|ptrace()| system call. This approach is for example used in \emph{strace}, a tool dedicated to tracing system calls. \emph{strace} is a great tool but does not serve our purposes as we are also tracing other domains than system calls. Another approach can make use of \emph{eBPF} by attaching an \emph{eBPF} program to the \verb|tracepoint/raw_syscalls/sys_{enter,exit}| tracepoints. The details of using \emph{eBPF} for tracing system calls is in Chapter \ref{sysrapl}.

% what info to read

While tracing, we're interested in the system call ID, its entry and exit times, the cpu they're running on and their associated \Gls{pid} and \Gls{tgid}. We can use the entry and exit times for rating the single system calls using energy values (\emph{RAPL}). Using the CPU number, \Gls{pid} and \Gls{tgid} we can further classify this information.

In this thesis we will only use the entry and exit time of system calls to calculate their energy score. But more detailed analysis of system calls in the context of energy consumption can lead to much more detailed and interesting results. We will describe on a high level some of these analysis methods.

\subsection{I/O operation bundles}

% Difference between reading everything at once and reading everything over a prolonged period of time
% Many reads but for a single purpose

All I/O operations in an operating systems are carried out using system calls. System calls can be used to track single I/O operations but those operations often do not happen isolated. Instead, often I/O operations happen in bulk. Reading a big file, reading multiple files for the same purpose, writing to files in set intervals,\dots. The purpose of I/O operations differs in the number of involved files/streams, how often they happen, in what intervals and what is the size of read/written data. Detecting such \emph{bundles} could help understand the energy behaviour of the profiled software.~\cite{pathakWhereEnergySpent2012}

\subsection{Asynchronous Power Behaviour}

Apart from the use of system calls for I/O or process management, they are used for general interaction with all hardware components. And all hardware components have their own energy behaviour. And only a selected few of these components make their energy behaviour known to the kernel. And often they might not know the information themselves as they might not have a dedicated power measurement module. Among such components can be Wi-Fi modules, Bluetooth modules, GPS modules or fingerprint readers.

Pathak et al.~\cite{pathakWhereEnergySpent2012} discuss how all the different hardware components often operate in different \emph{modes}. Each of these \emph{modes} causes the component to enter a different \emph{power state} changing its energy consumption. For example, in a GPS module some of the different \emph{modes} could be: \begin{enumerate*}[label=\emph{\alph*)}]\item idle \item locating satellites \item location found (low accuracy) \item location found (high accuracy)\end{enumerate*}.

% tail state, detecting using syscalls

Some modes of operation can be directly requested by the user (e.g., switching a Wi-Fi module on/off) but some modes are assumed without being directly requested (e.g., a spinning drive spinning after already reading a file). Pathak et al.~\cite{pathakWhereEnergySpent2012} write about such modes as \emph{tail state}. Such states can contribute in a significant way to the total energy consumption. System calls could be used to detect changes in modes of operation of these hardware components, track their current state in finite state machines and use this information to improve the accuracy of energy profiling.  

\end{document}