\chapter{Emulator design and implementation}
\label{chap:wenux}
WEnux stands for WAN Emulator for Linux. It has been developed as an open software by Jozef Urbanovský, under the GNU General Public License v3.0. 

\begin{figure}[H]
	\centering
	\includegraphics[scale=0.35]{../images/wenux.png}
	\caption{Logo of WEnux}
	\label{wenux}
\end{figure}

\section{Research of existing technologies}
As an inspiration for WEnux, was software emulator called \textit{WANem} \cite{wanem}, which is frankly over engineering its use case and makes itself seem way more complicated than it should be. WANem is no longer developed or maintained, but it does present a good concept how network emulation can be performed on Linux machine. 

WANem was more or less researched from the user perspective, how does it present itself and what are its capabilities. However upon further inspection, it is mostly written in a Bash, with usage of userspace calls to control selected kernel modules.

WEnux was implemented before the code inspection of WANem happend and there was a revelation that WANem does emulation on almost equal terms as WEnux. Both are using \textit{iptables}, \textit{iproute} and ultimately \textit{tc} utilities from userspace, to manipulate kernel parameters in \textit{netfilter}, \textit{conntrack}, \textit{tbf} and \textit{netem} modules. WEnux in comparison to WANem is written in much more sensible way, where it uses object oriented approach in Python3 and code is easily manageable and extensible at any time. On the other hand WANem internals are written mostly in Bash with PHP frontend.


\section{Experimenting and prototyping}
Due to native Linux environment with full access to the system, there is no need to reinvent the wheel, by writing everything from a scratch. It makes a lot of sense to use already predefined kernel modules dedicated to this purpose, such as netem and its userspace configuration utility tc. 

First experiments were conducted with basic configuration of \textit{tc-netem} to achieve delay with egress \textit{qdisc}. This was done manually to ensure, that tc is a viable way to proceed. After successful tests with regards to delay, jitter and loss, decision regarding programming language had to be made. There were no constraints from instructions and speed of emulation would fall on kernel itself, making the implementation independent from the programming language.

Python as a candidate sounded suitable for work of this kind as it was more or less supposed to provide a wrapper around userspace utility and possible extension to frontend interface later on. As an extremely versatile object oriented language, this pick turned out right.

At the beginning of the WEnux, the code itself was written in a procedural structural way, without employing any of Python's object oriented strengths. As a proof of concept it was sufficient and tested with regards to \textit{iperf3} generator, \textit{ping} and http speedtest. Egress qdisc was first to be supported with binding to a specified NIC, as well as, ability to include or exclude network and transport layer identifiers. Egress qdisc was extended with netem to manipulate outgoing packet attributes, such as loss, duplication, jitter and delay. Soon after support for tbf was added with ability to limit the rate, size of token bucket itself and latency time, for which packets can stay queue before being dropped. 

For simplification of ingress traffic IFB has been created to handle incoming traffic instead of a filter that would serve as a policer. This allows ingress traffic to be redirected to IFB, and be treated as if it was egress traffic. This allows more code reuse in a form of shared configuration of the qdisc hierarchy. 

First released implementation of WEnux was using command line interface for control, with extensive argument parser to handle all the possible configuration options. This result was submitted as a semestral thesis. It was able to attach itself on an interface and emulate WAN-like conditions for passing traffic. It was able to manipulate almost every parameter, that made sense to support, in netem and tbf submodules in tc.

Implementation has been separated to \texttt{Tc} module, which contains code related wrapper of tc, netem and tbf. There was also a \texttt{Utils} module that handles processes unrelated to networking. Entry point to WEnux was through \texttt{wenux.py} script that contained an argument parser for all implemented parameters. 
It also handled signals to allow for easier control of application.

\newpage
\section{Final application}
WEnux was designed from the start, to be extended of graphical user interface, to allow for easy configuration and interaction with user.
This section discussed the final form of application, that is being submitted and decisions that lead to it. Code itself is documented with \emph{docstring} style comments, with only some parts highlighted here.

\subsection{Graphical user interface}
As WEnux is meant to run on Linux system, it was not required to think of portable solution that could run on a server or on any other system. As previously mentioned WANem had front-end implemented as a web interface, with its own server, running remotely or locally. As WEnux is not trying to be as robust to run remotely, it did not make sense to pick web interface along with web server, which could possibly take even more resources from emulation itself. To also allow for easy maintenance and minimal knowledge of different programming languages Python GUI framework made the most sense.

Main contenders for GUI framework were \emph{PyQt5}, \emph{PyGUI}, \emph{Gooey}, \emph{PySimpleGui} and \emph{Tkinter}.
In research phase PyGUI and PySimpleGui were favorites, due to being lightweight and easy to work with. PySimpleGui is actually a wrapper for Tkinter and Qt and allows user to pick between these to frameworks in a minimalist way. Tkinter was considered due to being lightweight object-oriented layer on top of Tk. It has the advantage of being included with the Python standard library, making it the most convenient and compatible toolkit to program with. PyQt5 is one of the most widely used framework for GUI development, developed under GPL license, offering robust tools, that can not only serve for creating user interface. PyQt5 also offers Qt Designer, which is extremely convenient builder of graphical user interface that uses WYSIWYG editor. 
On the other hand Gooey is not as well known framework, but it builds upon programs that utilize argument parsers.


At the beginning there were experiments mostly with Gooey, as WEnux already had quite robust argument parser, which Gooey was trying to create a GUI out of, but it wasn't configurable enough in some cases. 
PyQt5 with its Qt Designer quickly replaced Gooey as preferred framework for development of GUI.

\begin{figure}[H]
	\centering
	\includegraphics[scale=0.45]{../images/wenux_empty.png}
	\caption{Fresh start of WEnux application}
	\label{wenux_empty}
\end{figure}

\begin{figure}[h]
	\centering
	\includegraphics[scale=0.45]{../images/wenux_full.png}
	\caption{Sample configuration and execution of WEnux application}
	\label{wenux_full}
\end{figure}

\subsubsection{PyQt5 and Qt Designer}
PyQt5 is a library, that lets user utilize \emph{Qt GUI framework} from Python. Qt itself is written in C++ and is wide spread cross-platform framework for development. 
Such an approach allows for easy and fast development under Python, while using all of the advantages of C++, especially its speed, being compiled language. 
PyQt5 offers not only graphical user interface API classes, but also XML handling, network communication, regular expressions, threads, SQL database, multimedia and other technologies available in Qt. 
All of those classes are contained withing a top-level Python package called PyQt5, which is required to run WEnux, as well.
     
Main components used from PyQt in WEnux are:
\begin{itemize}
	\item \emph{QtCore} module contains the core non-GUI classes, mainly used as fundamental enablers. Qt uses aforementioned enables to give use higher-level UI and application development components. It includes the core event loop, signals and slot mechanisms. As a cross-platform framework it provides abstraction for Unicode, threads, mapped files, shared memory, regular expressions and many more. 
	\item \emph{QtWidgets} is more of a traditional module with UI elements to create classic desktop-style user interface. It allows user to render to the screen, handle user input events. All UI elements provided by Qt are either subclasses of QtWidgets or use some form of connection with a QWidgets class \cite{pyqt5, qtdoc}.
\end{itemize}


Core usage and rendering example can be also seen in the \texttt{main} part of WEnux application, showcased in the listing \ref{pymain}. 

\begin{lstlisting}[language=Python, caption={Top-level script of WEnux with PyQt5 usage}, basicstyle=\linespread{1}\footnotesize, label={pymain}]
from Gui.Gui import WenuxGUI
from PyQt5 import QtWidgets

	if __name__ == "__main__":
	app = QtWidgets.QApplication([])
	...
	application = WenuxGUI()
	application.show()
	sys.exit(app.exec())
\end{lstlisting}

\emph{Qt Designer} is a tool for designing and building Qt graphical user interfaces with QtWidgets. Once can create and heavily customize windows, dialogs or buttons in a WYSIWYG editor, that serves also as a testing tool. GUI created by Qt Designer is easy to integrate with Python code, using Qt's signals and slot mechanisms.

Result of UI, designed in Qt Designer is an XML \texttt{.ui} file format without any additional code. Therefore, Qt includes \texttt{uic} utility to generate a C++ code that actually creates the GUI. \texttt{QUiLoader} class is the one, that allows an application to load a \texttt{.ui} file and to create the corresponding UI dynamically.

However, PyQt5 does not provide a wrapper for \texttt{QUiLoader}, but instead incorporates \texttt{uic} Python module, which can load \texttt{.ui} files as well to dynamically generate user interface. 

WANem was mostly designed and tested in Qt Designer and with UI itself was generated by following utility - \texttt{pyuic5 qt5.ui -o Qt5.py}.
After this step, it was possible to utilize all UI related objects in Python code and connect previously written WANem implementation.

\subsubsection{Bridge}
In order to simplify emulation to one point of configuration, dedicated interfaces for emulation are bridged together. This process creates a shared interface and requires \texttt{bridge-utils} package, containing userspace utility \texttt{brctl}.

For this purpose a dedicated shell script has been written, that takes three arguments containing names of mentioned interfaces and intent of the operation, that can be either setup or teardown.
Script \texttt{bridge.sh} cleans up previous configuration, sets up a new bridge with two given interfaces and assigns it an IP address. After the WEnux finishes, the script is called again for a teardown on created bridge.

\subsubsection{Core logic}
Depending on the direction of impairment, type of device is selected - either IFB in ingress path, replacing default qdisc, or real device with egress path.
To allow higher granularity of configuration prio qdisc is created in order to allow some traffic to bypass impairments. Next in order are the traffic control filters for inclusion, exclusions, different IP versions and ports. 
These filters are referring to their parent root qdisc, where \texttt{flowid 1:3} is used for inclusion, while \texttt{flowid 1:2} serves exclusion of specified filter rules.
Impairments themselves follow in the next step, binding new qdisc to their parent root qdisc with all user defined arguments. Available impairments to be emulated are - packet loss, duplication, delay, jitter, reordering, link rate and lastly packet overhead. 
All of aforementioned impairments can be selectively applied on included IP addresses and ports in both directions of traffic flow.

\begin{lstlisting}[language=Python, caption={Method netem in the TcWrapper class}, basicstyle=\linespread{1}\footnotesize, label={tcwrapper}]
def netem(self, operation):
	...
	loss = Loss(self.loss_ratio,
		    self.loss_corr)
	
	dupl = Duplicate(self.dupl_ratio,
		    	 self.dupl_corr)
	
	delay = Delay(self.delay,
		      self.jitter,
		      self.delay_jitter_corr)
	
	reorder = Reorder(self.reorder_ratio,
			  self.reorder_corr)
	
	corrupt = Corrupt(self.corrupt_ratio)
	
	rate = Rate(self.limit,
		    self.overhead)
	
	netem_collection = NetemCollection(self.nic)
	netem_collection.extend([loss, dupl, delay, 
				reorder, corrupt, rate])
	netem_collection.apply_cmd(operation=operation)
\end{lstlisting}
                                              
Core logic is contained mostly in the \texttt{TcWrapper} module, that does the initialization of prio qdisc and additional ingress related operations if needed. It defines \texttt{tbf} and \texttt{netem} methods dedicated for the direct control of their tc userspace utility and fabrication of correct command for the given impairment. This class creates an instance for each of the configurable netem parameter as a child dataclass to the \texttt{NetemCmd} and hands them over to the \texttt{NetemCollection} class, that aggregates all netem parameters and their generated command to master one, which is executed through method \texttt{apply\_cmd}.
As an example in the figure \ref{delayclass}, \texttt{Delay} dataclass is shown with basic functionality to compose the partial command. 

\newpage
\begin{lstlisting}[language=Python, caption={Delay dataclass that inherits from the NetemCmd}, basicstyle=\linespread{1}\footnotesize, label={delayclass}]
@dataclass(eq=True, frozen=True)
class Delay(NetemCmd):
	...
	_time: int = 0
	_jitter: int = 0
	_correlation: float = 0.0
	
	def make_cmd(self):
		_time = self.format(self._time, "ms")
		_jitter = self.format(self._jitter, "ms")
		_correlation = self.format(self._correlation, "%")
	
	return f"delay {_time} {_jitter} {_correlation}"

\end{lstlisting}

\texttt{NetemCollection} is a child class to \texttt{TcCollection} and its main use is to allow for application of aggregated command. This is done through keeping the reference to all sub-commands classes created in the \texttt{TcWrapper}'s method \texttt{netem}. To aggregate all the created commands, the \texttt{apply\_cmd} method looks for all of its own child dataclasses of \texttt{NetemCmd}. If dataclass is not empty, its result, consisting of partial netem command, is taken and appended to the final command to be run. The inner workings can be seen in the listing \ref{collection}.

\begin{lstlisting}[language=Python, caption={NetemCollection class used for aggregation of NetemCmd classes and execution of netem command}, basicstyle=\linespread{1}\footnotesize, label={collection}]
class NetemCollection(TcCollection):
	...	
	def apply_cmd(self, operation):
	...
	_used_dtcls = set()
	
	self._operation_check(operation)
	
	for dtcls in self:
		for param_val in dtcls.__dict__.values():
			if param_val:
				_used_dtcls.add(dtcls)
	
	...
	
	check_call(f"tc qdisc {operation} dev {self._nic} 
	parent 1:3 handle 30: netem " + 
	" ".join([x.make_cmd() for x in _used_dtcls]))

\end{lstlisting}

	\begin{figure}[h]
		\centering
		\includegraphics[scale=0.85]{../images/design.pdf}
		\caption{Final specimen design of WEnux}
		\label{design}
	\end{figure}

\subsection{Installation}
WEnux requires few extra modules to function properly, such as its ingress datapath impairment. As application was developed under Fedora 30 and 31, with some testing done on RHEL8.1 and RHEL8.2, it makes the most sense to explain the installation process with regards to RPM-based Linux distributions. 

Due to the WEnux code using \emph{fstrings} and \emph{dataclasses} it is required to run either \texttt{Python3.6} with dataclasses installed additionally through \emph{pip} or natively \texttt{Python3.7} that has dataclasses included in the built-in packages.

Core package that does the emulation in WEnux is contained in \texttt{iproute-tc}, including tc module with netem, tbf, filter and many others. For ingress impairments \texttt{ifb} module is required to be loaded into the kernel. However there is a Fedora specific package that user needs to get before being able to successfully manipulate ingress traffic. 
Depending on the type of installation of RPM-based distribution packages that need to be present are \texttt{kernel-modules-extra} and \texttt{kernel-debug-modules-extra}.
For the bridging capability user needs to install \texttt{bridge-utils} package, containing userspace utility \texttt{brctl}, utilized in the \texttt{bridge.sh} script used in the WEnux initialization. 

For the graphical interface of WEnux it is required to have \texttt{PyQt5} package, that can be installed either with \emph{pip} under the name of \texttt{pyqt5} or directly through system repository through \emph{dnf} utility under the name of \texttt{python3-qt5}.

WEnux as a whole can be installed either through \emph{pip}, located in Python package index - \emph{PyPI} repository available online. Or it can cloned from \emph{git} repository. Last option is available to get WEnux is as an appendix in the thesis itself, navigating into \texttt{python} folder. WEnux does not have any dependency on other Python libraries and it only uses built-in modules that already come with standard installation of Python. 

Lastly WEnux requires root access due to manipulation with kernel.

Installation and running process (some steps might be redundant based on distribution and presence of WEnux package):
\begin{enumerate}
	\item \texttt{dnf install kernel-modules-extra kernel-debug-modules-extra -y}
	\item \texttt{dnf install python3.7 -y}
	\item \texttt{dnf install iproute-tc -y}
	\item \texttt{dnf install bridge-utils -y}
	\item \texttt{dnf install python3-qt5 -y}
	\item \texttt{mkdir wenux; cd wenux/; pip3 install -t .; chmod u+x wenux.py} 
	\item \texttt{./wenux.py \{interface1\} \{interface2\}}
\end{enumerate}


\chapter{Testing}
This chapter concludes a small form of validation phase that has been done to the WEnux. Different parts of emulation are briefly tested with sample CPU and memory usage.

First off, there is a need to explain expected outcome of the network emulator. 
Selection of the language does not play any role here with current implementation, because it does not control the packet flow or quality of service parameters. Traffic control is done directly through kernel with heavily optimized software implementation, which speed would not be surpassed without using some form of hardware acceleration or for example defining XDP. Also, software emulation is extremely costly procedure when it comes to high speed networking and one should not expect that it will replace a hardware emulator, with dedicated and optimized hardware setup with acceleration of different parts of emulation. 

As WEnux is directly using kernel modules to control ingress and egress traffic, the implementation and emulation itself is extremely accurate up to a point, where kernel has enough resources to manipulate the traffic, such as storing it to cause delay, jitter, reorder and other types of impairments. 

Due to the use of IFB in the ingress flow scenario, both flows are similar in terms of resource usage. In this case, redirecting traffic from an interface's ingress to the IFB's egress is done directly by stealing the packet and placing it straight to the egress of the IFB device. Moreover, this operation is not that costly in term of resources.

\begin{figure}[h]
	\centering
	\includegraphics[scale=1]{../images/ifb_path.pdf}
	\caption{Packet's traversal through kernel with the use of IFB}
	\label{ifb_path}
\end{figure}

Even though WEnux is directly plugging into kernel, emulation itself introduces some deviation, caused by processing and execution of additional code introduced by either \texttt{netem} or other extensions of \texttt{tc}.
One thing to note is that some parameters, for example rate or limit throttling, delay and others are affected by several factors, such as kernel clock granularity, which causes imperfections in shaping in some scenarios. This can be seen in an bursts of packets, known as artificial packet compression. Another limitation, which introduces artificial delay, is in the network adapter buffers.

\section{Testing methodology}
\label{methodology}
\emph{WEnux} test were conducted on a \emph{ThunderX2 CN9980 - Cavium} \cite{thunder} ARM server, using microprocessor based on Vulcan microarchitecture. This is well known high-performance ARM server, manufactured by \emph{TSMC}, using standard \texttt{64 bit} word size, with 32 cores and 128 threads per socket. This server was chosen due to being available at the time for testing, and to also prove, that WEnux is going to function under \emph{kernel-alt}. Used system was RHEL8 General Availability release, with default kernel options and parameters. 

However WEnux was developed under the \emph{kernel-x86-64} and locally tested on Fedora 30 and 31 distributions. 

Implementation of emulator was tested with artificially generated UDP\footnote{UDP stream was used due to TCP having congestion control algorithms, which would interfere with the results} traffic with packet size of 1000 bytes at the set packet rate of 300 packets per second, with in-between packet delay of 3.33ms, producing a total traffic flow of 300 kilobytes per second. This stream was generated with iperf3\footnote{\url{https://iperf.fr/}} framework. 

Based on recommendations from the RFC 2679 \cite{metrics} one-way delay was defined as $$Delay=T_{r}-T_{s}$$ where $T_{s}$ represents the time, when source sends the first bit of the packet, and $T_{r}$ is the time, when last bit of the packet is received by the destination.

Definition of one-way delay jitter is derived from the delay definition itself, measured as the absolute value of the difference between the delay of two consecutive packets. $$Jitter=\vert D_{n}-D_{n-1}\vert$$ formula is straight-forward, where $D_{n}$ describes the one-way delay of the \emph{n}-th packet and $D_{n-1}$ is the exact same delay of the \emph{(n-1)}-th packet respectively.

Packet loss is calculated as a portion of lost packets for a sliding window of one second in length. 

Standard deviation ($\sigma$) of delay, implemented as \emph{correlation} parameter in \emph{WEnux}, is calculated based on the formula of the one-way delay as $$\sigma=\sqrt{{1\over N-1}\sum\limits_{i=1}^{N}(D_{i}-\bar{D})^{2}}$$ where $N$ is the number of delay samples used in $\sigma$ calculation, $D_{i}$ represents delay of the \emph{i}-th sample of delay, and lastly the $\bar{D}$ is the mean value of the one-way delay samples.

WEnux was tested for delay generation accuracy by setting a reference value in emulator and later the value of one-way delay was measured for comparison. Tests were run for 30 seconds, sample rate was set to $1/10$ of a second for total of 300 samples. It might seem that sample rate, as well as, test duration are insufficient to evaluate the emulator, but for the sake of this work it should be enough to estimate the behaviour of the implementation. To stress the buffers and kernel itself, test was run with reconfiguration of WEnux after 5 seconds, to double the parameter value of previous iteration, with seed value set to 1 ms. 

\section{Evaluation}
Recorded and reference values for each of measured parameters are shown in figures \ref{delay_graph}, \ref{loss_graph} and \ref{jitter_graph}.
All of these graphs are showing mostly extreme values, where kernel has issues handling almost real-time emulation with minimal impairments, with high level of error. As parameters rose, accuracy of emulation was getting higher as well, with error rate dropping below 1\% at around 50 ms of delay. 

This study presents an extreme cases in terms of low parameter values, which most probably wont be used, as they are not real scenarios in WAN. However, going above the 50 ms is presenting such stable results, that it does not make sense to evaluate the accuracy of emulation. 

On the other end of spectrum, using values above 1000 ms, can stress resources available, but considering the server used for test, the change was marginal, or even negligible. 

\begin{figure}[h]
	\centering
	\includegraphics[scale=0.7]{../images/delay_graph.pdf}
	\caption{Measured and configured delay}
	\label{delay_graph}
\end{figure}


\begin{figure}[h]
	\centering
	\includegraphics[scale=0.7]{../images/loss_graph.pdf}
	\caption{Measured and configured packet loss}
	\label{loss_graph}
\end{figure}

\begin{figure}[h]
	\centering	
	\includegraphics[scale=0.7]{../images/jitter_graph.pdf}
	\caption{Measured and configured delay with 10\% jitter}
	\label{jitter_graph}
\end{figure}

Moreover, to evaluate the CPU and RAM consumed by the WEnux, there is need to change the whole methodology, to push the maximum throughput available through the NIC and gradually reduce the rate to find the threshold of rate that the server is able to handle with WEnux. Anyhow, rate is only one parameter, creating one combination of a large scale, that WEnux currently offers. In depth evaluation of resource usage is beyond the scope of this work. 

However, some combinations were tested and their respective values were recorded in a table \ref{resources}. Chosen methodology was very simple, consisting of recording the baseline CPU and RAM usage before using the WEnux and running iperf3 UDP stream limited to 100Mbps. Then, the same test was run with WEnux enabled with different combination of parameters and unlimited UDP stream.

Difference between baseline value and WEnux test value is recorded in the table for respective parameters. 


\begin{table}[H]
	\centering
	\caption{CPU and RAM usage of WEnux emulations}
	\label{resources}
	\renewcommand{\arraystretch}{1.3}
	\begin{tabular}{l l l}
		\hline \hline
		
		\textbf{Combination of WEnux parameters} & 
		\textbf{CPU [\%]} & 
		\textbf{RAM [GB]} \\ \hline 
		
		\emph{\texttt{limit} 100000, \texttt{delay} 200, \texttt{jitter} 20} & 
		1.3\% & 
		0.02 \\ \hline 
		
		\emph{\texttt{limit} 100000, \texttt{delay} 500, \texttt{jitter} 20} & 
		1.5\% & 
		0.06 \\ \hline 
		
		\emph{\texttt{limit} 100000, \texttt{delay} 1000, \texttt{jitter} 20} & 
		1.7\% & 
		0.11 \\ \hline 
		
		\emph{\texttt{limit} 100000, \texttt{loss\_ratio} 1} & 
		0.3\% & 
		0.00 \\ \hline 
		
		\emph{\texttt{limit} 100000, \texttt{reorder\_ratio} 1} & 
		2.0\% & 
		0.02 \\ \hline 
	\end{tabular}
\end{table}

\


