\documentclass[../thesis.tex]{subfiles}
\begin{document}

\chapter{Experiments} \label{experiments}

% What's the chapter about, what are testing and why?
% The experiments can have many goals but we are not trying to show the maximum possible accuracy

This chapter will demonstrate the use of \verb|sysrapl| and \verb|sysrapl-report| (introduced in Chapter \ref{sysrapl}) in a series of experiments. The goal of these experiments is not only to show the capabilities of the created tooling but also to show its behaviour under different conditions.

\section{Methodology} \label{experiments-methodology}

% Structure of experiments. What is given and what changes across the experiments. How is it presented in the paper.

Instead of profiling a large number of projects across several experiments, we chose to profile a single project\,---\, GNOME Shell~\cite{GNOMEShell2023}\,---\, under different runtime conditions. Because \verb|sysrapl| is an energy profiler, the runtime conditions can affect the results and we will hopefully see a high value in showing those effects.

GNOME Shell is a \Gls{graphical shell} using Mutter~\cite{Mutter2023} as a \Gls{display server}. The fact that it is a non-trivial project with complex behaviour and that it makes even use of all basic hardware components (CPU, RAM, GPU) makes it a good choice for our experiments.

For the runtime of the experiments we define three variables which we use to create a matrix of test cases. The variables and their possible values are:

\begin{itemize}
    \item \textbf{System installation.}
        The state of an operating system differs from user to user. Most users don't have complete control over their system's configuration. Moreover, this can be a potential source of energy consumption anomalies in the form of, e.g., software running continuously in background or software assuming a different mode of operation. 
    
        \begin{itemize}
            \item \textbf{\emph{existing}} represents an existing Fedora Workstation 38 system with several years worth of upgrades, personalized selection of installed software and possible fine-tunings of the behaviour of the system. This environment is far from the ideal environment for continuous testing.
            \item \textbf{\emph{live}} represents a system as booted from a Live ISO on a USB flash drive. The booted system is Fedora Workstation 38. This environment is close to the ideal environment for continuous testing.
        \end{itemize}

    \item \textbf{Workload.}
        The most visible cause of energy consumption is the software users run on their system. Peripheral devices like Bluetooth modules or Wi-Fi modules can also contribute significantly to the system's energy consumption. In the experiments we want to see how different workloads contribute to the system's consumption.
        
        In both workloads a single terminal (GNOME Terminal) window is open and it is used to start the test session.

        \begin{itemize}
            \item \textbf{\emph{clean}} does not run any graphical applications with only the default set of background services running. The laptop is in \emph{air-plane} mode (Wi-Fi and Bluetooth modules are disabled). To ensure the cleanliness of the environment the profiling was performed moments after the system's boot and login into the graphical session.
            \item \textbf{\emph{busy}} has a series of applications running including: \begin{enumerate*}[(\arabic*)]\item calendar app (GNOME Calendar), \item maps application (GNOME Maps), and, \item Firefox with 5+ tabs open and loaded, where one plays a video on YouTube in 720p quality \end{enumerate*}. Both Wi-Fi and Bluetooth modules are enabled and the laptop is connected to a 5GHz access point.
        \end{itemize}

    \item \textbf{Power mode.}
        % Explain power modes, the controlling project and what does it actually tweak.
        Modern hardware devices (i.e., CPU or GPU) do not have a single mode of operation. The drivers of these devices often take into consideration external factors like temperature and energy consumption (e.g., using interfaces like \emph{RAPL}), or configuration preference. The configuration preference is often presented to users in the form of \emph{power modes}.
        
        Fedora Workstation uses the \emph{power-profiles-daemon}~\cite{noceraPowerprofilesdaemon2023} project since version 35~\cite{noceraChangesPowerProfiles2021} to provide a way to set the system's mode of operation. The project (based on the used performance scaling driver) sets used scaling governors or energy biases. The project understands three power modes: \begin{enumerate*}[\alph*)]\item \textbf{\emph{balanced}}, \item \textbf{\emph{powersave}}, and, \item \textbf{\emph{performance}} (needs to be supported by the scaling driver)\end{enumerate*}.
\end{itemize}

% Detailed list of test steps. Why the same steps? ydotool

All experiments share a single testing scenario. It involves basic interaction with the graphical shell simulating use by a user. To ensure consistency between the different test cases, we automated the testing scenario by implementing a test script. The script is written in the Bash scripting language. At the beginning it starts the \verb|sysrapl| profiler and then start executing the test scenario where the steps are:

\begin{enumerate}
    \item Open overview (press \verb|Start|).
    \item Open application menu (press \verb|Start+A|).
    \item Close overview (press \verb|Start|).
    \item Switch to workspace on the right (press \verb|Ctrl+Alt+Right|).
    \item Switch to workspace on the left (press \verb|Ctrl+Alt+Left|).
\end{enumerate}

Between every step we delay for 5 seconds. The keystrokes are send to the shell using the \verb|ydotool|~\cite{notmoeYdotool2023} tool. The tool requires a \Gls{daemon} to be running\footnote{Which the test script ensures it does}. The testing script needs to be run with root privileges as both \verb|sysrapl| and the \verb|ydotool| daemon require them.

During the testing we used an external power meter (Elektrobock EMF-1) for correlating the measured values with the values measured by \verb|sysrapl| using the \emph{RAPL} interface. The power meter has a display updating the power readings every second with values with one decimal place. The power meter was configured to display current power consumption in Watts. Because the power meter does not offer a way to count the total energy consumed, we captured every experiment on a video and retrospectively summed the measured values into Joules.

\section{Testing environment}

All experiments were carried out on a single reference laptop:

\begin{center}
\begin{tabular}{l l}
    Model & \verb|Lenovo ThinkPad T460| \\
    Arch & \verb|x86_64| \\
    CPU & \verb|Intel Core i5-6300U CPU @ 2.40GHz| \\
    GPU & \verb|Intel HD Graphics 520| \\
    RAM & \verb|16GB DDR3 @ 1600MHz| \\
    \hline
    OS & \verb|Fedora Workstation 38| \\
    Kernel & \verb|6.2.12| \\
    \hline
    GNOME Shell & \verb|44.0| \\
    Mutter & \verb|44.0| \\
\end{tabular}
\end{center}

The laptop contains two battery packs: \begin{enumerate*}[\alph*)]\item internal, and, \item external\end{enumerate*}. During the experiments the external battery pack was removed and the laptop was plugged into the power meter which was plugged into a wall socket.

\section{Experiments}

Table \ref{table:experiments-testcases} shows all test cases that were conducted as part of our experiment.

\begin{table}[h]
\centering
\begin{tabular}{|c|l|l|l|} 
    \hline
    \textbf{Test case} & \textbf{System installation} & \textbf{Workload} & \textbf{Power mode} \\ 
    \hline
    ECB & existing & clean & balanced \\
    \hline
    ECS & existing & clean & powersave \\
    \hline
    ECP & existing & clean & performance \\
    \hline
    EBB & existing & busy & balanced \\
    \hline
    EBS & existing & busy & powersave \\
    \hline
    EBP & existing & busy & performance \\
    \hline
    LCB & live & clean & balanced \\
    \hline
    LCS & live & clean & powersave \\
    \hline
    LCP & live & clean & performance \\
    \hline
    LBB & live & busy & balanced \\
    \hline
    LBS & live & busy & powersave \\
    \hline
    LBP & live & busy & performance \\
    \hline
\end{tabular}
\caption{Test cases with specified runtime variables defined in Section \ref{experiments-methodology}}
\label{table:experiments-testcases}
\end{table}

% Summary of the experiments

% Based on the consumption ranking make a remark on how the state of the installation can affect energy consumption

Table \ref{table:experiments-energy} shows all test cases with the total energy consumed sorted in descending order based on the total consumed energy. The order of the test cases shows that, as expected, the highest ranking test cases are in general the ones with \emph{busy} workloads. These two groups are then ordered generally by the power mode with \emph{performance} at the top, followed by \emph{balanced} and \emph{powersave}. The system installation has the lowest impact on the overall rankings but the results show significant difference in energy consumption in test cases with different \emph{system installation} variable but with all other variables equal (e.g, test cases EBP and LBP, or test cases EBB and LBB). Test cases EBS and LBS are an exception to this phenomenon which makes us believe that multiple runs of the same experiments are required to fully exaplain this phenomenon.

The energy consumption of the whole laptop measured by the hardware monitor (see Section \ref{experiments-methodology}) is uniformly higher than the values measured in the \emph{psys} energy domain, which reflects the energy consumption of the whole SoC, by about 100 Joules. This suggests that while the SoC contributes the most to the reference system's energy consumption, there are other components that also contribute significantly. Such components possibly are the display, speakers, WiFi module, Bluetooth module and more. If we were to order the test cases in Table \ref{table:experiments-energy} in descending order based on the \emph{hardware monitor} values, we would get almost the same order with the exception of the last four test cases. This suggest that the majority of variance in energy consumption across the different test cases comes from the SoC rather than other hardware components in the reference system.

\begin{table}[h]
\centering
\begin{tabular}{|c|r r r r r|r|} 
    \hline
    Test case & \textbf{psys [J]} & pkg [J] & cores [J] & ram [J] & gpu [J] & \textbf{hardware monitor [J]} \\
    \hline % existing
    EBP & \textbf{496.551} & 229.442 & 162.999 & 71.836 & 27.852 & \textbf{600.0} \\
    \hline % live
    LBO & \textbf{439.894} & 184.968 & 134.792 & 57.929 & 14.889 & \textbf{542.4} \\
    \hline
    EBB & \textbf{353.226} & 120.749 & 62.753 & 72.870 & 19.238 & \textbf{443.6} \\
    \hline
    LBB & \textbf{335.378} & 101.770 & 52.966 & 59.050 & 12.379 & \textbf{415.8} \\
    \hline
    LBS & \textbf{292.539} & 72.099 & 26.069 & 55.302 & 9.734 & \textbf{379.1} \\
    \hline
    EBS & \textbf{283.364} & 70.898 & 22.761 & 63.006 & 10.913 & \textbf{357.3} \\
    \hline
    ECP & \textbf{201.514} & 60.171 & 35.206 & 27.641 & 0.855 & \textbf{304.9} \\
    \hline
    ECB & \textbf{185.740} & 43.225 & 18,342 & 26.384 & 0.684 & \textbf{285.5} \\
    \hline
    ECS & \textbf{167.512} & 30.951 & 6.989 & 24.457 & 0.439 & \textbf{266.7} \\
    \hline
    LCP & \textbf{166.271} & 35.657 & 12.077 & 24.610 & 0.895 & \textbf{284.5} \\
    \hline
    LCB & \textbf{155.632} & 28.195 & 4.929 & 23.971 & 0.839 & \textbf{272.0} \\
    \hline
    LCS & \textbf{150.826} & 25.794 & 3.050 & 22.983 & 0.623 & \textbf{267.6} \\
    \hline
\end{tabular}
\caption{Total energy consumed during test cases. The test cases are sorted in descending order based on the \emph{psys} energy domain}
\label{table:experiments-energy}
\end{table}

% Show some graphs from selected tests output and explain what can be seen there.
% Add numbered marks to the images and explain them in text. (What step happened, why is there a gap?)

Figure \ref{fig:experiments-heatmaps} shows two heatmaps from two experiments showing clear difference in the energy profile of the test cases.

\begin{figure}[ht]
    \centering
    \begin{subfigure}{0.48\textwidth}
        \includegraphics[width=\linewidth]{experiments-heatmap-clean} 
        \caption{Test case ECB}
        \label{fig:experiments-heatmaps-clean}
    \end{subfigure}
    \begin{subfigure}{0.48\textwidth}
        \includegraphics[width=\linewidth]{experiments-heatmap-busy}
        \caption{Test case EBB}
        \label{fig:experiments-heatmaps-busy}
    \end{subfigure}

    \caption{Heatmaps from test cases \textbf{ECB} and \textbf{EBB} showing energy consumption of system calls across energy domains in time with highlighted steps of the test scenario and highlighted noteworthy areas in the heatmaps.}
    \label{fig:experiments-heatmaps}
\end{figure}

In Figure \ref{fig:experiments-heatmaps-clean} most of the "hot spots" in energy consumption across all system calls are at the times when the different steps (marked by arrows) in the test scenario are carried out. Areas marked by number 1 in Figure \ref{fig:experiments-heatmaps-clean} highlight how the steps working with the "Overview" and "Application Menu" show "hot spots" mainly for the \verb|ioctl| system call (i.e., input/output control; for configuring (mainly) hardware devices) while the steps switching workspaces show "hot spots" mainly for the \verb|poll| system call (i.e., for waiting for file descriptor to perform I/O operations). This suggests a difference in workloads and the capability of \verb|sysrapl| to detect these differences.

But in Figure \ref{fig:experiments-heatmaps-busy} the test scenario steps are not as clear. That suggest that \Gls{compositing} windows is a significantly more expensive activity than the short-lived actions of interacting with the shell. In the figure there are two notable areas. The first one is marked by number 2 where the energy consumption of the \verb|ioctl| system call has increased. In this case the Shell renders additional graphical elements while compositing scaled windows on the screen leading to probable higher energy consumption. In the are marked by number 3 the energy consumption of the \verb|ioctl| system call was lowered significantly. We believe this is due to an optimization in Shell where an invisible window (Firefox playing a YouTube video), hidden after switching a workspace, is no longer composited as it is no longer visible on the screen.

\begin{center}
\begin{table}[h]
\centering
\begin{tabular}{|c|l l l l l l|} 
    \hline
    Test cases & ECB & ECS & ECP & EBB & EBS & EBP \\ 
    \hline
    \multirow{5}{8em}{Top 5 consuming system calls}
    & ioctl & poll & poll & ioctl & ioctl & ioctl \\
    & poll & ioctl & ioctl & write & write & poll \\
    & read & read & read & read & read & write \\
    & mprotect & write & write & poll & poll & read \\
    & write & getpid & getpid & epoll\_wait & epoll\_wait & recvmsg \\
    \hline
    \multirow{5}{8em}{Top 5 consuming system calls values (\emph{psys}) [mJ]}
    & \mc1r{$100.641$} & \mc1r{$91.593$} & \mc1r{$73.750$} & \mc1r{$831.667$} & \mc1r{$1,137.011$} & \mc1{r|}{$687.751$} \\
    & \mc1r{$70.060$} & \mc1r{$72.700$} & \mc1r{$28.805$} & \mc1r{$694.437$} & \mc1r{$869.479$} & \mc1{r|}{$541.777$} \\
    & \mc1r{$19.717$} & \mc1r{$28.861$} & \mc1r{$12.380$} & \mc1r{$607.970$} & \mc1r{$828.454$} & \mc1{r|}{$495.398$} \\
    & \mc1r{$14.982$} & \mc1r{$28.364$} & \mc1r{$10.272$} & \mc1r{$482.460$} & \mc1r{$463.981$} & \mc1{r|}{$426.071$} \\
    & \mc1r{$14.554$} & \mc1r{$11.331$} & \mc1r{$5.480$} & \mc1r{$152.292$} & \mc1r{$197.497$} & \mc1{r|}{$124.903$} \\
    \hline
\end{tabular}

\bigskip

\begin{tabular}{|c|l l l l l l|} 
    \hline
    Test cases & LCB & LCS & LCP & LBB & LBS & LBP \\ 
    \hline
    \multirow{5}{8em}{Top 5 consuming system calls}
    & poll & poll & poll & write & write & poll \\ 
    & ioctl & ioctl & ioctl & read & read & write \\
    & read & read & read & poll & ioctl & ioctl \\
    & write & write & write & ioctl & poll & read \\
    & mprotect & getpid & getpid & epoll\_wait & epoll\_wait & recvmsg \\
    \hline
    \multirow{5}{8em}{Top 5 consuming system calls values [mJ]}
    & \mc1r{$58.396$} & \mc1r{$88.072$} & \mc1r{$83.324$} & \mc1r{$700.995$} & \mc1r{$902.970$} & \mc1{r|}{$463.572$} \\
    & \mc1r{$57.762$} & \mc1r{$76.289$} & \mc1r{$24.350$} & \mc1r{$508.048$} & \mc1r{$644.264$} & \mc1{r|}{$362.542$} \\
    & \mc1r{$24.863$} & \mc1r{$34.246$} & \mc1r{$11.899$} & \mc1r{$398.288$} & \mc1r{$516.286$} & \mc1{r|}{$265.104$} \\
    & \mc1r{$21.064$} & \mc1r{$30.144$} & \mc1r{$10.709$} & \mc1r{$373.766$} & \mc1r{$240.162$} & \mc1{r|}{$253.515$} \\
    & \mc1r{$20.820$} & \mc1r{$13.512$} & \mc1r{$5.459$} & \mc1r{$174.855$} & \mc1r{$229.607$} & \mc1{r|}{$142.546$} \\
    \hline
\end{tabular}
\caption{Top consuming system calls and the values consumed (\emph{psys}) in all test cases.}
\label{table:experiments-syscalls}
\end{table}
\end{center}

% Are there differences in the highest consumers across test cases? Why?
% Are the scores accurate?

Table \ref{table:experiments-syscalls} shows the top consuming system calls across all test cases. The results show that while there is some small variation in the ranking of the system calls, the lists always contain the \emph{ioctl}, \emph{poll}, \emph{read} and \emph{write} system calls. Their presence is expected as they are all used for I/O operations which Shell does use a lot in order to draw content on the display.

The results also show clear difference in energy consumption between test cases with the \emph{clean} and \emph{busy} workloads. This shows the capability of the profiler to recognize differences in workloads.

% Weird consumption between power modes

From the results it is apparent that the scoring algorithm described in Section \ref{sysrapl-postprocessor} is indeed quite trivial and large number of system calls and the length of their runtime affect the energy score significantly. This is very noticeable in test cases EBB, EBS, and EBP where the total consumption of the different system calls is the highest for the \emph{powersave} performance mode.

% Jump in consumption between clean and busy environments

In test cases with \emph{busy} workload it is unclear what is the extent of the effect of the noise (i.e., Firefox window playing a YouTube video) on the resulting energy scores of system calls. A mechanism for filtering out noise could increase the quality of the results. But for the purposes of this thesis the results are adequate because the scores managed to properly locate spots in the profiles with higher/lower energy consumption.
\end{document}