\chapter{Linux network stack}
\label{chap:network}
First chapter provides comprehensive knowledge of Linux networking for understanding the main concepts of the thesis. It mainly describes \emph{netfilter} with almost all of its components and packet traversal through the network stack, explaining step by step actions the packet takes. There is also emphasis on a routing subsystem, as well as, traffic control in both general and Linux kernel sense, due to it being a core of this thesis. One must know network stack, to understand how network can be emulated and also to comprehend the implementation of the WEnux emulator, described in chapter \ref{chap:wenux}.

\section{Network model}
\label{net_stack}
International Organization for Standardization (ISO) created an Open Systems Interconnection (OSI) reference model \ref{iso_osi_model}, which defines a network as 7 separate layers. Core concept of this model is based on abstraction and standardization of different responsibilities of each layer. 
This thesis mostly focuses on the Ethernet based implementation, along with low-level kernel processes and userspace\footnote{Segment of system's virtual memory dedicated to running user applications} applications \cite{kernel_net}.

\begin{figure}[H]
	\centering
	\label{iso_osi_model}
	\includegraphics[scale=0.975]{../images/model.pdf}
	\caption{Model of individual ISO OSI layers with their functionality, protocol examples and data units on each layer}
\end{figure}

Linux kernel stack is handling incoming frames from the second layer upwards to the fourth layer, where it passes the control over segment to the userspace.
Kernel handles many network operations on these layers, such as:
\begin{itemize}
	\item Changes to packet due protocol rules (IP security (IPsec) or Network Address Translation (NAT) rules)
	\item Discarding the packet (routing not having the entry for given endpoint)
	\item Handling the network errors and sending error messages (Internet Control Message Protocol (ICMP) sending the control messages)
	\item Fragmentation of packet (if the packet extends the Maximum transmission unit (MTU))
	\item Checksum calculation (redundancy check for data integrity)
\end{itemize}

\section{Network device}
The link layer of Linux kernel is resided by the network device drivers, which are handling the incoming or outgoing traffic to the physical network interface card (NIC). Network device in kernel is represented by important attributes described in the \texttt{net\_device} structure. Understanding the network device is crucial knowledge to comprehend the network stack\cite{kernel_net}. 

Device parameters can be following:
\begin{itemize}
	\item \textbf{IRQ\footnote{Interrupt Request} number} - describes the device priority of interrupt request on the CPU
	\item \textbf{MTU} - determines the size of maximum transmission unit coming in or out of the device
	\item \textbf{MAC\footnote{Media Access Control} address} - logical hardware identification address
	\item \textbf{Name} - symbolic name containing used to reference the device
	\item \textbf{Flags} - characterizes the state of the device 
	\item \textbf{Features} - list of supported features by the device
	\item \textbf{Queues} - numbers of Tx and Rx queues of the device
\end{itemize}

\subsection{New API}
With the influx of high speed networking the previous implementation of network drivers, based on interrupts for each packet received, was not efficient enough. In order to combat this problem, the new software technique was developed, call the New API (NAPI), based on polling\cite{kernel_net}.

\textbf{Interrupt requests} - straightforward method of handling each and every packet is to issue and interrupt directly to kernel. However, servicing IRQs is a costly procedure in terms of the CPU resources and time. 

\begin{figure}[H]
	\centering
	\includegraphics[scale=1]{../images/interrupt.pdf}
	\caption{Timeline of interrupt or exception handling \cite{exception}}
	\label{interrupt}
\end{figure}

\textbf{Polling} - developed as an alternative to the interrupt-based processing. Kernel periodically checks whether there are incoming packets on the NIC, thus eliminating the need of processing the interrupts. Efficiency of polling is ultimately based on its frequency. Sub-optimal period can either lead to wasted CPU cycles, or on the other side of spectrum to packet loss, due to incoming buffer being full.

As a compromise, the Linux kernel uses the interrupt-driven mode by default and only chooses to switch to the polling mode, when the packet flow exceeds a certain threshold, known as \emph{weight} of the interface.

Result of these two methods is explained in figure \ref{modern}.
Whenever a packet arrives, it is added to the network device's receive ring-buffer, and an interrupt is raised on the CPU associated with given memory segment. However, typical network device does have multiple ring-buffers and their respective interrupt numbers are spread across various CPUs. Nowadays, NICs use advanced feature called an adaptive interrupt coalescing (AIC) to reduce the number of hardware interrupts raised during heavy load. This allows the CPU to run uninterrupted for longer time, thus processing more information.

\texttt{softirq daemon\footnote{Computer program that runs as a background process}} is crucial part of the lower parts of the network stack, as it handles the mechanism that copies the network data from the device's memory into a kernel socket buffer. In case of constant interrupts to softirq process from the device, the kernel would fail to move the packets to the internal socket buffer memory, therefore increasing the possibility of packet loss. Softirq process is using NAPI polling, whenever it is trying to dequeue a batch of packets from the NIC, by momentarily masking and disabling interrupts, that might occur \cite{kernel_net, net_driver}.


\begin{figure}[h]
	\centering
	\includegraphics[scale=1]{../images/modern.pdf}
	\caption{Processing of incoming packets in modern Linux kernel using both polling and interrupts \cite{modern_image}}
	\label{modern}
\end{figure}

\subsection{Role of network driver in kernel}
To understand the network emulation, one must know the important role, that network driver plays in it.

Each time kernel needs to send a packet to an interface, first it is enqueued into the queuing discipline (qdisc), configured for that specific interface.
Afterwards, the kernel tried is handling the transport of packets between the qdisc and the network driver \cite{tc-man}. 

Thus main responsibility of the network driver is to:
\begin{itemize}
	\item Receive incoming packets destined to the host, while passing them upwards to the network and transport layer
	\item Transit outgoing packets generated on the host and sent to an external network, or to forward traffic received by the host
\end{itemize}



\section{Netfilter}
In high-level overview, a netfilter subsystem is a framework to register callbacks in different positions of packets in the network stack, also called netfilter hooks, explained in section \ref{hooking}, and perform diverse operations on packets, such as changes to addresses or ports, dropping them completely, logging and many other modifications. So called netfilter hooks, allow various netfilter kernel modules to register these callbacks to perform numerous tasks on the netfilter subsystem.

\begin{figure}[h]
	\centering
	\includegraphics[scale=1]{../images/flow.pdf}
	\caption{Packet flow in the network layer through the netfilter \cite{tc_image}}
	\label{flow}
\end{figure}


As shown in figure \ref{flow}, the netfilter subsystem provides the following functionalities: Packet selection, filtering and mangling, which is mostly executed by the \emph{iptables}, \emph{nftbales} and \emph{bpfilter} modules. Network address translation is also present in netfilter, which is done through NAT table and can be considered as a packet mangling in a way, where it only manipulates the addresses. 

Netfilter also tracks connections with the \emph{conntrack}, which keeps a state table to determine and identify connections that belong together through the tracking of IP addresses, ports, etc. NAT facilities also make use of connection tracking, to translate all packets from a flow the exact same way. 

Gathering of network statistic is also present as a netfilter module, that is able to log all actions of netfilter frameworks \cite{kernel_net}. 



\subsection{Netfilter hooks}
\label{hooking}
Hooking is extremely potent tool for monitoring software behaviour or extending functionality without altering the original code. The core concept is to intercept specific system events or calls and use them to initiate custom code.

Hooks can be found in five points of the network stack, shared between both IPv4 and IPv6. These points in packet's traversal through given protocol stack are very well-defined, and netfilter registers these callbacks in order to parse, change and use these packets \cite{hooks}.

\begin{table}[H]
	\centering
	\caption{Netfilter hooks}
	\renewcommand{\arraystretch}{1.3}
	\begin{tabular}{l p{9,5cm}}
		\hline \hline
		
		\textbf{Name of hook} & \textbf{When is hook called} \\ \hline 
		
		1. \texttt{NF\_INET\_PRE\_ROUTING} & Packet arrives to the netfilter\\ 
		
		2. \texttt{NF\_INET\_LOCAL\_IN} & Destination of packet is local host\\
		
		3. \texttt{NF\_INET\_FORWARD} & Destination of packet is other interface\\
		
		4. \texttt{NF\_INET\_POST\_ROUTING} & Packet leaves the netfilter and sent to network\\
		
		5. \texttt{NF\_INET\_LOCAL\_OUT} & Packet is generated locally and destined out\\
	\end{tabular}
	\label{hooks}
\end{table}

First hook can be found in the \texttt{ip\_rcv()} function in IPv4, and respectively in the \texttt{ipv6\_rcv()} function in IPv6. These functions are protocol handlers and also point that all incoming packets reach, before they enter routing subsystem for the lookup.

Second hook located in the \texttt{ip\_local\_deliver()} function in IPv4, and in the \linebreak\texttt{ip6\_input()} function for IPv6. All incoming packets, where the lookup in routing subsystem decided, that they are destined to local host, reach this hook point after first passing through the \texttt{NF\_INET\_PRE\_ROUTING} hook point.

Third hook is extremely similar to second hook, which is based in the \linebreak \texttt{ip\_forward()} and the \texttt{ip6\_forward()} functions for their respective protocols. All forwarded packets, meaning that lookup in routing subsystem was a forward decision, reach this hook point after getting here via the \texttt{NF\_INET\_PRE\_ROUTING} hook.

Fourth hook is the last netfilter hook, that packet reaches, when it is leaving this subsystem, in his respective functions \texttt{ip\_out()} and \texttt{ip6\_finish\_output2()}. Forwarded packets, that already left the routing subsystem reach this final point. The \texttt{NF\_INET\_POST\_ROUTING} hook additionally serves fifth hook as well, to help packets exit the netfilter framework.

Fifth and the last defined hook is implemented as a stack unique function \linebreak \texttt{\_\_ip\_local\_out()} in IPv4 and \texttt{\_\_ip6\_local\_out()} in IPv6. Indicated hook is reached when packet is generated locally on the host and later gets to fourth post routing hook to leave the netfilter\cite{kernel_net}.


\subsection{nftables, iptables and bpfilter}
Another crucial part of netfilter is a kernel module called iptables and its successor nftables. Both of these netfilter extensions have their userspace utilities to configure their kernel counterpart, however, this section only talks about the kernel portion. There is also a third contender for packet filtering, known under the name of bpfilter, which is supposed to soon replace both nftables and iptables\cite{kernel_net}.

\subsubsection{Internals of packet filtering}
From higher perspective, iptables\cite{iptables} and nftables modules allow modification of the kernel module and its parameters through the userspace utility, and for definition of tables containing chains of rules for how each packet should be treated. Each table is associated with different kind of packet processing based on the netfilter hook points. Packets are typically processed by sequentially matching with rules in the chain. Nonetheless a chain can contain a goto or jump rule, which allows for table nesting. However, the main difference between the goto and chain, is that jump rule remembers the point of call, therefore it is bound to return after the nesting rule is explored. As a result of this system, every packet arriving or leaving the Linux host passes through at least one chain.

Figure \ref{flow} explains the existence of multiple tables, where chains are referenced. 
Chains consist of rules, where they specify which packets are supposed to get matched. Essential part of iptables rules are so called targets and verdicts, which can either extend the chain which packet is going to traverse and respectively explicitly make a direct decision, whether the packet should be dropped or allowed through. Packets are matched based on the ISO/OSI model layers, ranging from the data link layer up to the transport layer. 

A packet is traversing the chain of rules until either of these events occur:
\begin{enumerate}
	\item Rule matches the packet and gives the verdict whether to take call \texttt{ACCEPT} or \texttt{DROP}, or a module that will carry this verdict
	\item Rule calls the \texttt{RETURN} verdict, therefore packet has to go back to the processing chain
	\item Rule does not match the packet and end of the chain is reached, thus traversal continues in the parent chain or in the base chain policy, where the verdict is given
\end{enumerate}


Not only verdict specifies the fate of packet, but targets can also provide verdict, whether to \texttt{ACCEPT} the packet, consequently using the \texttt{NAT} module, or to \texttt{DROP} the packet by calling the \texttt{REJECT} module. Main difference between the target and verdict is that it can also rule \texttt{CONTINUE}, for example to the \texttt{LOG} module, to continue with the next rule as if no target nor verdict was given.

\subsubsection{Implementation specifics}
These examples are concerning IPv4 network namespace object, known as the \linebreak \texttt{netns\_ipv4}. First of all, every table has to be registered with the \linebreak \texttt{ipt\_register\_table()} or respectively unregistered with the \linebreak \texttt{ipt\_unregister\_table()} function. After registration namespace object knows a reference to the \texttt{xt\_table} object, thus the \texttt{netns\_ipv4} structure also has pointers to the tables, such as the \texttt{iptable\_filter}, \texttt{iptable\_mangle}, \texttt{nat\_table}, etc \cite{kernel_net}. 

To simplify the explanation of iptables and its internal working, example in form of packet flow chart \ref{iptables} is presented. 

Packet arrives to network layer and is immediately received by the \texttt{ip\_rcv()} function, which also contains the first hook, packet is going to encounter, in form of \linebreak the \texttt{NF\_INET\_PRE\_ROUTING} hook. However, filter table's callback does not registered this hook, as it is not present in previously mentioned table. Following the chart, the packet leaves the \texttt{ip\_rcv\_finish()} function and goes to routing subsystem \ref{routing} and performs lookup. In this specific scenario packet is either intended to be delivered locally or forwarded. Routing subsystem, based on its decision, initiates:

\begin{itemize}
	\item {\textbf{Local delivery} by calling the \texttt{ip\_local\_deliver()} function, which returns \linebreak the \texttt{NF\_INET\_LOCAL\_IN} hook. Filter table contains this hook, so the \texttt{NF\_HOOK()} (callback) macro will invoke the \texttt{iptable\_filter\_hook()} function. This filter hook is only an auxiliary function, to call the \texttt{ipt\_do\_table()} function, which invokes the LOG target callback, known as the \texttt{ipt\_log\_packet()} function. Resulting in the packet header being written to the syslog, as there are no more rules to be called. Thus, the \texttt{ip\_local\_deliver\_finish()} function is invoked, and packet's traversal advances to the the transport layer, where it will get handled by its respective socket.
	}
\begin{lstlisting}[language=c, caption={net/ipv4/ip\_input.c \cite{linux}}, basicstyle=\linespread{1}\footnotesize]
int ip_local_deliver(struct sk_buff *skb)
{
  ...
  return NF_HOOK(NFPROTO_IPV4, NF_INET_LOCAL_IN,
  		net, NULL, skb, skb->dev, NULL,
  		ip_local_deliver_finish);
};
\end{lstlisting}
	
	\item {\textbf{Forwarding}: by calling the \texttt{ip\_forward()} function, that returns \linebreak \texttt{NF\_INET\_FORWARD} hook, similar to the local delivery. This call pattern is intended to be the same, as for the local delivery by continuing with another auxiliary hook call in form of the \texttt{iptable\_filter\_hook()} function, which calls the \texttt{ipt\_do\_table()} function.
	The packet is once again logged to the syslog, by the \texttt{ipt\_log\_packet()} function. However, at this point the forward table no longer shares its path with the filter table, by having a the \texttt{ip\_forward\_finish()} function in the \texttt{NF\_HOOK()} argument, which is known as the continuation function. As there is no \texttt{NF\_INET\_POST\_ROUTING} hook present, the \texttt{ip\_output()} function is invoked, which continues to the \texttt{ip\_finish\_output()} function, that hands over the packet to the network driver.

	}

\begin{lstlisting}[language=c, caption={net/ipv4/ip\_forward.c \cite{linux}}, basicstyle=\linespread{1}\footnotesize]
int ip_forward(struct sk_buff *skb)
{
  ...
  return NF_HOOK(NFPROTO_IPV4, NF_INET_FORWARD,
  		net, NULL, skb, skb->dev, rt->dst.dev,
  		ip_forward_finish);
  ...
};
\end{lstlisting}
\end{itemize}

\begin{figure}[h]
	\centering
	\label{iptables}
	\includegraphics[scale=1]{../images/iptables.pdf}
	\caption{Netfilter's iptables example traversal with forwarded and local destined packets \cite{kernel_net}}
\end{figure}


\subsubsection{nftables}
So why was nftables created, if it basically does the same thing as iptables? First of all it simplifies Linux kernel application binary interface and reduces the code duplication, highly present in iptables. As mentioned above, packets are usually inspected sequentially, which is not always true. nftables provides a new implantation over prehistoric iptables, which supports generic set infrastructure, that allows for maps constructions, as well as, concatenations. This fact alone grants nftables advantage of faster inspection, due to the possibility to create multidimensional tree of ruleset, thus reducing the rules packet has to explore. iptables are also scattered based on protocol used, such as iptables for the IPv4, ip6tables for the IPv6, arptables for the Adress Resolution Protocol (ARP) and ebtables for the Ethernet.
Crucial difference is that nftables implementation added a simple virtual machine into kernel that executes the bytecode for the packet filtering. The virtual machines does operations so simple, it is all the way down on the arithmetic, bitwise and comparison logical operations level, with basic packet parameters and its metadata. This virtualization allows for manipulations with multiple sets of data, replacing full set lookup\cite{nftables}.


\subsubsection{bpfilter}
With the rise of high speed networking the performance of both iptables and nftables is insufficient in case of network speeds higher than 10 Gbits per second. Its extended functionality also covers high-performance load-balancing, Denial of Service (DoS) mitigation, firewalling, safe instrumentation of kernel and userspace code, etc.

From higher perspective, one notable design difference is that the BPF, does not allow user to tap into raw network, but instead it creates a pseudo-device, acting like a staged controlled area.
This pseudo-device is later bound to a network interface and it acts as a network device in the kernel sense, thus reading the buffers directly from the NIC, as well as, writing or injecting new packets to it.

As a result of this implementation is safe and flexible programming environment in various different contexts, such as, networking datapaths, kernel probes, perf events any many more.
Previous realization of packet filtering with iptables and nftables added a kernel module, which introduced significant risk, in case something would go bad, which is not the case for BPF programs, as they are verified at load time to ensure no out-of-bounds accesses can occur. 
BPF is much faster than older iptables and nftables implementation due to just-in-time compilation of its bytecode to the native instruction set of the CPU.

XDP is closely tied to the BPF, as it modifies the packet flow through the kernel network stack and in some cases, it allows the packet to explicitly bypass whole network stack and be directly transmitted to the outbound NIC. Previously, the network datapath was set chain of actions, which is no longer the case with XDP eBPF\footnote{Extended BPF, implementation of BPF in Linux kernel} combination. 

To elaborate how XDP eBPF changes the packet flow, we can use \ref{flow} as a reference. The packet arrives to the NIC and is directly sent to the XDP eBPF, where it decides whether to \texttt{XDP\_ACCEPT} packet and call the \texttt{alloc\_skb}, to \texttt{XDP\_REDIRECT} the packet, thus sending it to the userspace and creating a XDP specific packet with the \texttt{sll\_family} parameter set to \texttt{AF\_XDP} or lastly, to bypass the kernel network stack altogether using the operation \texttt{XDP\_RX} \cite{bpf}.


\section{Routing subsystem}
\label{routing}
This section is explaining few advanced ideas of routing, as well as, how routing works in the Linux network stack.

Linux routing subsystem is not only used in the Linux distributions designed to servers or clients, but also systems developed for routers, ranging from SOHO routers, all the way up to the core layer, used in the Internet backbone. This section mainly talks about routing tables, FIB TRIE and much more. TRIE is derived from word \emph{retrieval} and describes a data structure, defined as a special tree that replaced the FIB hash table. Crucial part of routing subsystem is route lookup, how and when ICMP Redirect messages are generated and lastly about the removal of the routing cache code.

\subsection{FIB}
Primary responsibility of a routing subsystem is maintenance of forwarding database and forwarding of traversing packets. SOHO networks might have their FIB managed by an administrator, because of the topology being static. However, as we move up the ladder to bigger networks, topology becomes dynamic and maintaining of FIB by hand is rather impossible. In such cases, the FIB is managed by the userspace routing daemons, which alter kernel routing tables, sometimes in conjunction with special hardware enhancements. 

\subsubsection{FIB table}
The core routing subsystem data structure is known as the \texttt{fib\_table}, which is basically a routing table. Another important part of a Linux routing model is the \texttt{fib\_info} structure, that contains similar characteristics shared between multiple routes. All of those structures are stored in a hash table called \texttt{fib\_info\_hash}, that is dynamically resized based on the amount of \texttt{fib\_info} structures currently employed in the routing subsystem. 

Maximum amount of routing tables is $2^8$, where default tables are known under the \texttt{tb\_id} 255, for \texttt{RT\_TABLE\_LOCAL} and 254 for \texttt{RT\_TABLE\_MAIN}. The rest of tables are not initialized and can only be utilized with the Policy Based Routing (PBR). Routing table also contains info about the number of default routes in the table under the \texttt{tb\_num\_default} value, as well as, placeholder for a routing entry in form of complete \texttt{trie} structure, that points through the \texttt{fib\_alias} to theW \texttt{fib\_info}. Between all of those structures are multiple \texttt{key\_vector} structures, which are either internal nodes or a leaf for trie \cite{kernel_net, trie}.


\begin{lstlisting}[language=c, caption={include/net/ip\_fib.h \cite{linux}}, basicstyle=\linespread{1}\footnotesize]
struct fib_table {
  struct hlist_node	tb_hlist;
  u32			tb_id;
  int			tb_num_default;
  struct rcu_head	rcu;
  unsigned long 	*tb_data;
  unsigned long		__data[0];
};
\end{lstlisting}

\subsubsection{FIB info}
This structure is meant to represent a routing entry, but in fact, it often aggregates multiple entries into one. 

\begin{lstlisting}[language=c, caption={include/net/ip\_fib.h \cite{linux}}, basicstyle=\linespread{1}\footnotesize]
struct fib_info {
  struct hlist_node	fib_hash;
  struct hlist_node	fib_lhash;
  struct net		*fib_net;
  int			fib_treeref;
  ...
  unsigned char		fib_protocol;
  unsigned char		fib_scope;
  ...
  u32			fib_priority;
  ...
  unsigned int		fib_offload_cnt;
  struct rcu_head	rcu;
  struct fib_nh		fib_nh[0];
  #define fib_dev	fib_nh[0].nh_dev
 };
\end{lstlisting}


It contains important routing entry parameters, such as:
\begin{itemize}
	\item \texttt{fib\_dev} - outgoing network interface
	\item \texttt{fib\_protocol} - routing protocol identifier of this route, defines how this routing entry was created
	\item \texttt{fib\_scope} - identification of destination distance, describes whether the destination is directly attached, locally available or global
	\item \texttt{fib\_nh[0]} - specification of nexthop member, defined as array due to the Multipath Routing
\end{itemize}

\subsubsection{Caching}
Route lookup can be performed numerous amount of times per second, therefore it is in the best interest to optimize its performance in the routing subsystem. Results of route lookup are typically cached in a nexthop structure, known as the \texttt{fib\_nh}. However, there are many exception, due to different types of routes, which are beyond the scope of this work.

\subsubsection{FIB nexthop}
The \texttt{fib\_nh} structure basically represents the decision of the routing subsystem with all required information to perform packet hand off to the lower layer. 

\newpage
\begin{lstlisting}[language=c, caption={include/net/ip\_fib.h \cite{linux}}, basicstyle=\linespread{1}\footnotesize]
struct fib_nh {
  struct net_device	*nh_dev;
  struct hlist_node	nh_hash;
  struct fib_info	*nh_parent;
  unsigned int		nh_flags;
  unsigned char		nh_scope;
  ...
  int			nh_oif;
  __be32		nh_gw;
  __be32		nh_saddr;
  ...
};
\end{lstlisting}

One of the most important parameters it contains, is a direct reference to the \texttt{net\_device} in form of the \texttt{nh\_dev} parameter, specifying the outgoing nexthop interface.

Extremely important part of the FIB nexthop are also its exception, which are meant to handle cases, where routing entry has to be changed by the state of network invoked by the ICMP redirect message or 
Path MTU discovery. 

\subsubsection{FIB alias}
Due to the nature of LPC trie implementation, described in section \ref{trie}, it allows for the node reuse and compression by multiple prefixes with similar traits. If route entries differ only in the Type of Service (ToS), FIB generates an alias. 

\begin{lstlisting}[language=c, caption={net/ipv4/fib\_lookup.h \cite{linux}}, basicstyle=\linespread{1}\footnotesize]
struct fib_alias {
	struct hlist_node	fa_list;
	struct fib_info		*fa_info;
	u8			fa_tos;
	u8			fa_type;
	u8			fa_state;
	u8			fa_slen;
	u32			tb_id;
	s16			fa_default;
	struct rcu_head		rcu;
};
\end{lstlisting}

This means, that there is no need to create the \texttt{fib\_info} structure for each routing entry, but rather the \texttt{fib\_alias} which links them together and speeds us the route lookup, as its footprint is smaller than info's.
As per of code listing above from the kernel, it aggregates multiple \texttt{fib\_info} with same parameters seen in the structure.


\subsubsection{FIB relations and memory representation of LPC trie}
\begin{figure}[H]
	\centering
	\includegraphics[scale=1]{../images/fib_all.pdf}
	\caption{Graphical representation of memory using the LPC trie \cite{fib}}
	\label{fib_all}
\end{figure}

\subsection{Route lookup}
Crucial part of being able to communicate over the IP is the route selection to the destination. The selection of a route path is traditionally made on a hop-by-hop basis based exclusively on the destination of the packet.
The Linux routing subsystem can be easily compared to a conventional routing device, such as router, multi layer switch, firewall, etc. However, it can be configured to be much more flexible, for example to prioritize routes based on specific packet characteristics.

\subsubsection{Trie}
\label{trie}
Currently the routing subsystem is using the \emph{Level and Path compressed trie}, known as the LPC trie for route lookup. This type of trie detects and compresses densely populated parts of the trie and replaces them with a single node and an associate vector of $2^k$ children. Such node can handle $k$ input bits instead of a single one. 

The LPC trie is a structure that allows \emph{longest prefix match} technique to be utilized much faster than the hash based solution. This technique ensures that the most specific route will be selected. Use of this method allows routes for large network to be overridden by more specific host or network routes. Conversely, the longest prefix match also allows specific networks or hosts to be aggregated into the larger network address, due to the Classless Inter-Domain Routing (CIDR) \cite{trie}. 

There is also an exception, where destination of the packet does not decide the route, but rather predefined policy, known as the policy-based routing.

\subsubsection{Policy-based routing}
With the raising complexity of network scenarios, a destination address-based routing is no longer sufficient in those cases, so the PBR takes over. Its native Linux support through the use of multiple routing tables and the routing policy database allows for granual control over the routing path of any packet.

The PBR is an extremely potent tool for the routing, due to its ability of selectors to use any attribute of a packet passing through the routing subsystem. Its attributes can range from the source address, ToS or Differentiated Services (DS) flags, \emph{fwmark} and interface name, where the packet was received, can be used as selectors. Mentioned attributes determine which routing table is going to be used, allowing for higher control over the routing.

When the routing subsystem is trying to determine the outgoing route for a packet, the kernel always used to consult routing cache first. Nowadays, there is no routing cache due to many problems it had, most notable being the DoS. 

The routing chain begins to search the Routing Policy Database (RPDB) by the priority. For each matching entry in the RPDB, the specified routing table is searched for the longest prefix match based on the destination address. If the kernel is successful it will select the route and forward the packet. Otherwise, search through the RPDB continues until match is found or all of the sub-routing tables are exhausted \cite{ip_guide}.

\subsubsection{Implementation specifics}
To begin this subsection, one must know, that each packet triggers lookup in routing subsystem, in either Rx or Tx path.

Route lookup for the destination address begins with the \texttt{fib\_lookup()} function. When it finds a proper entry in the routing subsystem, it builds the \texttt{fib\_result} object, consisting of different routing parameters. The lookup function call contains important \texttt{flowi4} or \texttt{flowi6} object respectively, which stores packet specific attributes, such as the destination address, source address, ToS, etc. Based on this object the key for lookup is defined and it should be initialized prior to the function call. The \texttt{fib\_lookup()} starts its search in the local FIB table. If the lookup succeeds, the \texttt{dst} object is built, containing the \texttt{dst\_entry} structure and the destination cache. This object is embedded into a \texttt{rtable} structure, which is known as the routing entry with reference to the SKB. The \texttt{dst\_entry} is utterly important in routing process as it contains two callbacks named the \texttt{input} and \texttt{output}, where only these two can manipulate with the SKB.

To understand how route is propagated through the kernel, there is need to delve deeper into the attributes of the \texttt{rtable}. First off, the \texttt{rt\_flags} parameters contains flags, describing the type of destination address, whether it is a broadcast, multicast, local or redirect address. The \texttt{rt\_uses\_gateway} parameter defines if nexthop is gateway or a direct route. Last important parameter is the \texttt{rt\_iif} specifying the \texttt{ifindex} identifier of the incoming interface.

Next step in the routing subsystem is to determine where the input callback is going to be set. 
\begin{itemize}
	\item incoming unicast packets with local host destination - \texttt{ip\_local\_deliver()}
	\item incoming unicast packets with remote destination - \texttt{ip\_forward()}
	\item locally generated packets with remote destination - \texttt{ip\_output()}
	\item multicast packets - \texttt{ip\_mr\_input()}
	\item filtered packets by the kernel or user - \texttt{ip\_error()}
\end{itemize}

Last step to decide the result of routing is known as the structure \texttt{fib\_result} with following important parameters specifying the outgoing route. It contains a pointer to the \texttt{fib\_info} object, that holds a reference to the \texttt{fib\_nh}. Another crucial pointer is to the FIB table, as well as, reference to the \texttt{fib\_alias} list, containing all objects associated with given route \cite{kernel_net}. Relations between these structures can be seen in figure \ref{fib_all}.

\subsubsection{Summary}
This section described the routing subsystem in detail, with Linux implementation specifics, how are the incoming and outgoing packets handled and organization of memory regarding the FIB routing structures.
However, various advanced routing topics, such as the multicast, multipath and PBR, in detail, are beyond the scope of this work.

\section{Traffic control}
This section will not only talk about the specifics in the Linux network stack, but also explore this topic in general sense, as it is the core concept of the thesis.

Quality of service, or so called traffic control (TC) is the name given to the sets of queuing systems and mechanism by which packets are received and transmitted on a device.
TC is one of the last segment of the Linux network stack before packets are sent to the network driver or one of the first segments when incoming packets are handed over to the network stack from the network driver. It includes a decision regarding the incoming and outgoing packets, whether to accept them at what rate, order, etc. 
For the purpose of this work, policies are created from given set of traffic control systems, which are meant to cripple or degrade the network in specified way, to emulate the WAN.

Queue organization is essential part of packet-switched network due to their nature of being stateless. Statelessness is one of the fundamental strengths of IP networks, where different types of flows are not differentiated. This alone, creates a requirements, where serialized outbound data have to be managed by a queue.

One of the main advantages of traffic control is more predictable network usage and less volatile contention for network resources. The network is set to meet the goals of the traffic control configuration, that has been communicated with users. This policy is known to users and they know what to expect from the network. 

On the other side of the spectrum traffic control is extremely complex discipline. Not only in terms of sheer knowledge required to properly configure the policies, but also the computing resources, which can be quickly exhausted or over utilized, as the system has to manage all the structures required for handling the traffic control\cite{tc-tldp, demyst}.

When a packet is sent in the egress path, the TC is going to be applied in three phases in the following predefined order, if they contain user-defined policies: 
\textbf{Classification}, \textbf{Scheduling} and \textbf{Shaping}.
A packet coming in the ingress path will only pass through \textbf{Policer}. There is also a possibility for the ingress path to apply other types of traffic control, explained in section \ref{sec:ifb}.

\begin{figure}[h]
	\centering
	\includegraphics[scale=1]{../images/tc_path.pdf}
	\caption{Sequence of different algorithms applied during the packet's traversal \cite{demyst}}
	\label{tc_path}
\end{figure}

Not necessarily every ingress or egress path contains marking or dropping actions, but they are certainly important parts of traffic control. 
\textbf{Dropping} discards either an entire packet, flow or even classification. 
\textbf{Marking} is a mechanism by which a packet is altered in a known predefined way.

\subsection{Flows}
A flow is defined as a sequence of packets from a distinct source to a distinct unicast, anycast or multicast destination, that the source desires to label as a flow. 
It can contain all packets in one direction in a specific transport connection or a media stream, but it does not necessarily map a transport connection in a 1:1 ratio \cite{flow_rfc}.
A flow is describes as a 5-tuple consisting of packets with the same: 
\begin{itemize}
	\item source address
	\item destination address
	\item source port
	\item destination port
	\item transport protocol
\end{itemize}

One of traffic control mechanism is ability to separate the traffic into classes of flows, known as the classifier. Flows can be aggregated later in the TC pipeline and transmitted together. 
Another approach is to divide the bandwidth equally based on the individual flows, but this might be extremely challenging task, as each flow competes against one another and some applications might build a large number of flows on purpose.

\subsection{Tokens and buckets}
\label{tokenbucket}
Another crucial part of quality of service, based on tokens and buckets, is known as the shaper.

Dequeuing can be done multiple ways. Simplest way is to create a counter of packets being dequeued as each item is taken from the queue, which results into complex usage of timers and measurements to limit this rate accurately. Smarter and indeed used way in traffic control, is to generate a tokens at a desired fixed rate, and only dequeue packets if a token is available. Generated tokens are stored in a bucket with fixed size, where they can pile up, if they are not needed, up to a bucket size, and then be used in a burst. This is core concept of rate-limiting or shaping, to support such bursty traffic as HTTP.

To bridge this with the Linux kernel, Token Bucket Filter (TBF) qdisc is an example of a shaper. 
It works on the exact same way as previously described token and bucket combination. The TBF generates \texttt{rate} tokens per second and \texttt{burst} defines the size of bucket. Regular traffic, that is highly predictable can be easily handled by smaller bucket size. However, burstier traffic needs larger buckets to be satisfied, unless the burstiness reduction of the flow is required.

Combination of tokens and buckets is used in many qdisc, such as specified classless TBF and classful Hierarchy Token Bucket (HTB).

\begin{figure}[h]
	\centering
	\includegraphics[scale=1]{../images/tbf.pdf}
	\caption{UML diagram of relations between qdiscs and classes \cite{demyst}}	
	\label{tbf}
\end{figure}

Figure \ref{tbf} describes the activity diagram of a token bucket, where multiple values in diagram are abbervations standing for:

\begin{itemize}
	\item \texttt{limit-at} - guaranteed upload/download data rate to a target
	\item \texttt{max-limit} - maximal upload/download data rate that is allowed for a target
	\item \texttt{burst-limit} - maximal upload/download data rate that is allowed for a target while the burst is active
\end{itemize}

Maximal token rate is defined at any point in time as the highest of mentioned values, with an exception to burst-limit, that is only active in the burst state. In case where limit-at is the highest value and extra tokens need to be issued to compensate for all missing tokens that were not borrowed from it's parent queue \cite{mikrotik}.


\subsection{Classification}
Classifier is meant to sort or separate traffic into queues, by identifying individual flows and determining which policies to use.

\emph{Classifying} is the mechanism to separate packets to be segregated and treated differently, possibly putting them to different egress queues. 
All of this is based on network flows, which are distinguished based on the metadata and packet headers, allowing relevant policy to be dispatched.

Such identification can be done through the traffic control by itself, conntrack or newly by eBPF. TC classifier's implementation is the simplest of the bunch, but also offers the least options. It can classify packets mostly based on metadata of the individual packet and there is also a possibility to extract certain information from the packet header. 
The conntrack is much more complex system, which can be used to divide group of packets into flows, track flow's state and more. This classification can simplify application of specified policy, which can be applied on an aggregated flow, instead of individual packets.

Classification can also include packet marking, which happens on a boundary of a single traffic control domain or it can be done on each hop separately.

\subsection{Scheduling}
Scheduler typically organizes and sometimes reorganizes packet order meant for the output. As it turns out, the main part of this section is a queue management, specifically how are packets sent and how they are treated when the queues are full. 

\emph{Scheduling} is the mechanism to arrange or rearrange packets between the input and output of a specific queue. Most common scheduler by far is also one of the simplest, known as the First in First Out (\textbf{FIFO}) scheduler, which does not compensate for various network conditions and does not prefer any flow in a default state. When granular traffic control is required, other generic scheduling algorithms are mostly used, for example:
\begin{itemize}
	\item \textbf{SFQ} - Stochastic Fairness Queueing is a fair queuing algorithms that attempts to prevent any single client or flow from network usage domination.
	\item \textbf{WRR} - Weighted Round Robin is a round-robin algorithm which gives each flow or client a change to dequeue packets.
	\item \textbf{RED} - Random Early Detection is congestion avoidance algorithm mostly utilized in backbone.
\end{itemize}

\subsection{Shaping}
\label{shap}
Shapers are meant to delay packets and release them at a disciplined pace to enforce policies such as rate limits.

\emph{Shaping} is the mechanism of packet delay in an output queue before a transmission to meet a desired output rate. 
Most common use case is to limit the bandwidth and smooth out spikes in the network. Shaping mechanism can be viewed as a work-consuming operation, where it requires resources in order to delay packets. Thus, queuing mechanism performing shaping function is a work-consuming operation. A work-conserving queuing mechanism such as \emph{PRIO} is not capable of delaying a packet. 

One of the main advantages of bandwidth shaping is the ability to control latency of packets. Shaping algorithms are typically a variation of previously mentioned token bucket algorithm \ref{tokenbucket}, allowing it to control the rate. 

\subsection{Policing}
Policers measure and enforce predefined bandwidth and burst limit in a particular queue. They can be found in an ingress path of packet and perform very similar function as shapers \ref{shap}, however the key difference is not only in the packet's flow direction, but in their respective actions. Policers can only take on a single corrective operation, known as dropping.

\emph{Policing} is the mechanism by which traffic can be limited, in the sense of traffic control. Its most frequent usage is to keep the peer in check regarding its bandwidth usage, on the network border. A policer is going to accept traffic until a specified rate, and then drop all the traffic exceeding this limit. Nonetheless the traffic does not necessarily have to be dropped, but it can also get reclassified. It is an extremely simple mechanism to control an ingress queue state. If the packet has arrived to NIC and has been scheduled to enter the queue, its rate is compared with predefined limit and depending on the result, it is either allowed to enqueue or not. Although the mechanism seem simple from the black box perspective, internally it uses token bucket implementation, but does not posses capability to delay a packet as previously mentioned.

\subsection{Components of Linux traffic control}
Described general elements of traffic control can be directly interconnected with the Linux implementation.

\begin{table}[H]
	\centering
	\caption{Correlation between traffic control in general and the Linux sense}
	\label{correlation}
	\renewcommand{\arraystretch}{1.3}
	\begin{tabular}{l p{9,5cm}}
		\hline \hline
		
		\textbf{Traditional element} & \textbf{Linux component} \\ \hline 
		
		\textbf{Classifier} & \texttt{filter} identifies as a part of \texttt{classifier} object\\ 
		
		\textbf{Scheduler} & \texttt{qdisc}, can also contain classes and other qdisc\\
		
		\textbf{Shaper}  & \texttt{class}\\
		
		\textbf{Policer}  & Exists as a component of a \texttt{filter}\\
	\end{tabular}
\end{table}

\subsubsection{filter}
\label{filter}
\texttt{Filter} can be described as one of the most complex component in the Linux traffic control. Its implementation provides a mechanism, which aggregates multiple key elements of traffic control. As per table \ref{correlation}, most obvious role is to classify packets into an output queues with use of a single or multiple filters defined by the user.
Filters cannot exist on their own and have to be attached either to classes or classful qdiscs. However, enqueued packets will always enter the root qdisc and after its traversal, the packet may be directed to any subclasses, where further classification can be performed.

Policer can be considered as a part of filter, where it calls one action above the threshold and other action below the specified threshold. Result of this implementation can lead to a simulation of three-color meter.

Arriving packets are classified as either green, if they fully comply to the requested guaranties, yellow if they are between the minimum guaranties and peak threshold, or red if their values are not acceptable. Such implementation in the network congestion allows red packets to be dropped before yellow ones and so on.

Most notable fact about policing, is that it is not physically capable of delaying packets, as it does not have an access to any queue, thus packets cannot be dropped in a sensible way. This leads to random packet drops, whenever policy is violated, later resulting in a bad behaviour in protocols such as TCP, where retransmission timeouts are going to get triggered, thus introducing latency issue when multiple retransmissions occur.

\begin{figure}[h]
	\centering
	\includegraphics[scale=1]{../images/uml-tbf.pdf}
	\caption{UML class diagram of relations between qdiscs and classes \cite{demyst}}
	\label{uml-tbf}
\end{figure}

\subsubsection{qdisc}
\label{qdisc}
The Linux scheduler is know an \texttt{qdisc} in the kernel. Whether user wants it or not, every interface does have a scheduler of some kind and Linux uses a FIFO by default. This scheduler is the simplest implementation possible, which does not interfere with traffic order nor contain any rules. To be exact the default qdisc is \texttt{pfifo\_fast}. 
It is based on a conventional FIFO qdisc, with added prioritization. It provides three different bands, known as the FIFO queues for separating the traffic. 
The band 0 is filled with highest priority traffic and always serviced first. By analogy the band 1 is emptied of pending packets before the band 2 can be dequeued \cite{linux, tc-tldp}.

A queuing discipline can be divided based on its relation to classes.
\begin{itemize}
	\item \textbf{Classful qdiscs} can contain classes and provide a handle for filters to attach to. They can be also used without child classes at the expense of resources, without added functionality.
	\item \textbf{Classless qdiscs} cannot contain classes and and do not provide handle, which makes it impossible for filters to attach. They are used to accept data and only reschedule, delay or drop them, without any subdivisions.
\end{itemize}

Each interface defines a location onto which a traffic control structures can be attached for either an egress or ingress path. The most used is the egress qdisc, known as root dqisc, which is able to contain any other qdisc and class structures. It allows for an extended configuration and creation of the child classes. Therefore, outgoing traffic is always passed through the egress or root qdisc.

On the other side of spectrum, incoming traffic is handled by the ingress qdisc. Compared to the root qdisc, it comes with limited capabilities, where no child classes are allowed and it only exists as an object, that provides a handle for the filter. The ingress qdisc is mostly used as a convenient object for policer to limit the incoming bandwidth.

In conclusion, the egress qdisc is much more useful in overall traffic control scheme, as it contains a real qdisc with possibility to classify, schedule and shape the traffic, as opposed to the ingress qdisc that can only apply specified policies.

\subsubsection{class}
\label{class}
Classful qdisc, such as the HTB, is a predisposition for \texttt{class} existence. Classes are known in the Linux as a form of shaper and offer great flexibility, as they can contain either single child qdisc or multiple children classes. To increase the complexity of traffic control scenarios, one can also define a classful qdisc itself, inside a class.

The amount of filters attached to a class is not limited, which allows for the selection of a child class or for the use of a filter to reclassify or drop traffic entering an individual class. Inner class or root class is defined as a class that contains a child class. This hierarchy is terminated in a leaf class, which ends the chain by containing a qdisc but never any child class \cite{tc-tldp}.

\subsubsection{IFB}
\label{sec:ifb}
Intermediate Functional Block is a virtual network device with inverted egress and ingress paths allowing to shape incoming traffic by attaching arbitrary qdisc to the IFB. Device creation is tied to an ifb kernel module, which has to be loaded. After the IFB has been created, an ingress traffic can be redirected from the physical NIC directly to the newly created device. The conntrack can also play a role here, by calling the \texttt{CONNMARK} target to restore the connection mark which has been modified by the flow redirection \cite{ifb}. 

\begin{figure}[hb]
	\centering
	\includegraphics[scale=1]{../images/ifb.pdf}
	\caption{Ingress data path with IFB network device \cite{demyst}}
	\label{ifb}
\end{figure}

\chapter{Virtualization of network}
\label{virt}
This chapter elucidates the virtualization of network and closely explains concepts of simulations and emulations. Since the main focus of the thesis is an emulation of transmission network, the section elaborating on the emulators is much more extensive. However, there is a need to explain the correlation between the model and simulation as well, to better understand the virtualization as a whole.

\section{Models}
\label{models}
A model can be described, as an object that aims to re-create something from the real world. Specifically computer model can be represented as a set of algorithms and equations used to capture the behaviour of the system being modelled. 
There are many attributes on which one can categorize the computer models, most notable being: time, entropy and dynamicity \cite{models}.

\begin{itemize}
	\item \textbf{Continuous model} represents continuous data, where they have a potentially infinite number, divisibility and attributes.
	Their best description can be given through differential equation. 
	 
	\item \textbf{Discrete model} is analogue to continuous model. State variables can only take on a countable set of values, such as integers. Discrete models define the smallest unit of time supported, and events may only occur at discrete timeline.
	
	\item \textbf{Stochastic model} allows for random variants in variables over time, which serves as a primary tool for estimation of probability distribution of potential outcomes of the system.
	
	\item \textbf{Deterministic model} is a model, in which the values for the dependant variables of the system are entirely determined by the parameters of model. In contrast, stochastic, or probabilistic models introduce randomness, best represented as probability distributions.
	
	\item \textbf{Dynamic model} is dependant on time, allowing for interactions between variables over time, as well as, examination of relationships, that could not be sorted out otherwise. A valid example can be a computer network running over several hours, where the state of each network component varies based on traffic generated.
	
	\item \textbf{Static model} is described as, invariant to the course of time, can be considered to only work with snapshots\footnote{State of system at a given point in time} of the system. Model represents a system at a given point in time or compares the system at different points in time.
\end{itemize}

\section{Simulators}
Due to the thesis being focused on emulation, this section will only elaborate on network simulators, to understand the difference. 

Simulations are based on different types of models described in section \ref{models}. Every model leverages some strengths and weaknesses when it comes to the network simulation.

First off, network simulator, compared to a real network, is implemented only as a software program modelling the behaviour of a network by calculating the interaction between different routing entities. This fact, forces any network simulator to implement all parts of network, ranging from electrical signals, protocol stack to network appliances. Depending on the network under simulation, various parts of network have to be simulated, while others can be omitted. 
 
Typical network simulator is a dynamic DES, where events are scheduled dynamically with passing time. On the other side of spectrum are static Monte Carlo based simulators, where repeated sampling is used to compute the result \cite{simulator}. 

Network simulators are commonly used in research, engineering and development of the various parts of network. \emph{Cisco} is well known for its \emph{Packet Tracer} simulator, mostly used for education, as its support of protocols is limited and proprietary restricted. For professional use in engineering and development, common simulators are \emph{GNS3}, \emph{EVE-NG}, \emph{NS3} and others.

\section{Emulators}
An emulator is a piece of hardware or software, which is aiming to duplicate (or emulate) the functionality of a specific system, it models, resulting in a replacement of the given system. 

Emulator's main goal is to reproduce the external behaviour of the original system, which can be viewed as a black box\footnote{A device, system or object which can be viewed in terms of its inputs and outputs, without the knowledge of internal workings}. 
Therefore, it has to accept the same type of input parameters and produce the same output as the real-world counterpart. For the external observer it has to look like as if the emulator is performing the same function as the original system. 
However, the implementation of the given function does not have to reflect the real-world system, it just has to behave live the original function\cite{emulation}. 

One can describe emulator in a way, where one computer called \emph{the host} is enabled to behave as another computer described as \emph{the guest}. In this sense, it allows the host system to run software or use peripheral devices designed for the guest system. Good example can be obsolescence, where new versions of Windows, operating system, drop support for specific functions from older releases and functionality of applications using these obsolete functions are left to be emulated, instead of rewriting the code to use a newer functions or any other type of maintenance. As another valid example, can be a fact, that many computer printers are designed to emulate Hewlett-Packard LaserJet printers, due to the sheer amount of the software written for HP printers, thus making it easier to develop the printer itself, without much need to care about the accompanying software\cite{emulator}. 

Well-known computability theory or so called Church-Turing thesis\cite{church-turing}, implies that any operating environment can be emulated within any other environment, assuming that enough memory is available. In reality, it can be extremely difficult to replicate the exact behaviour without proper documentation, hence reverse engineering has to take place. However, that it not the main concern with this theorem in practice, where the timing constraints exist. If the emulator speed does not match the original system, either by running on worse hardware or slower implementation, it might run to timer interrupts, possibly altering behaviour.

To get to the core of this thesis, we need to explore the network emulators. 
Due to the native Linux environment, traffic control along with its enhancement Netem are great examples of network emulators. Netem\cite{netem} allows for modifications of delay, packet loss, duplication and many other network traffic properties, to emulate characteristics of a wide area network. Netem, in black box sense, is acting as an actual network to external systems.
