\chapter{Implementation}
\label{ch:implementation}

The proposed system implementation has its origin in the Audio Spectrogram Transformer architecture described in Section~\ref{sec:vision-transformer-and-audio-processing}. This chapter is dedicated to the implementation details of the entire system, which includes speech recording preprocessing and augmentation, model implementation, model training techniques, and evaluation process. Python was selected as the main programming language due to its strong connections with deep learning. The author of the thesis chose the PyTorch library\footnote{\url{https://pytorch.org/}} and its ecosystem for input processing, training, and testing the model because of its stability, reliability, regular maintenance and development, a large ecosystem with a lot of implemented features and methods, and last but not least, its increasing popularity among academic works in recent years.

The implementation description is divided into sections that follow the flow of the resulting system. The description begins with the preprocessing of speech input data, which produces a dataset ready for use in model training. The next subsection is devoted to the model implementation details with a consecutive description of its training loop preparation. The last sections comprise the implementation of evaluation metrics

% speech data and augmentation 
\section{Speech recording preprocessing}

The preprocessing of input data is divided into two parts: the transformation of the speech recording signal into the mel spectrogram, and data augmentation, which serves to add some random noise or obfuscation to the data samples that changes them in a way that preserves their core properties while enlarging the training dataset. The result is the implementation of a particular class that serves as the data source for the model during training.

The pipeline for mel spectrogram extraction consists of a few steps. The first step is to load the audio file with the appropriate sampling rate (expressed in Hertz [Hz]), which determines the number of samples (data points) of amplitude that represent one second of recording. The selected value for this purpose is 16000 Hz, as it strikes a balance between quality and the recording size (the higher the sample rate, the higher the quality but the higher the memory requirements).
The second stage of the mel spectrogram construction pipeline is framing and windowing. Inspired by other works that used these methods, the implementation in this work partitions the loaded recording into frames of length 25 milliseconds with a 10 millisecond hop (shift) size, or frames of size 400 samples with 160 sample hop length when the sampling rate is 16000 Hz. The frame size value is chosen intentionally because research has shown that speech is invariant for twenty to thirty milliseconds~\cite{Akcay-SER-review}. The hop size value is set to 10 milliseconds to create a big enough overlap between neighbouring frames to prevent information loss during the windowing when the frame ends are smoothed by a windowing function to eliminate discontinuities that could influence the results of further processing (detailed explanation of framing and windowing is in Subsection~\ref{subsec:speech-preprocessing}). The implementation in this thesis utilizes the Hanning windowing function, one of the most widely used functions in other works during the speech preprocessing stage~\cite{Akcay-SER-review}.

The third pipeline step consists of Short-time Fourier transformation (STFT). In this step, each frame is transformed from the time to frequency domain. The transformed frames are then filtered with a mel filter bank involving \(n\) mel band filters. The value \(n\) is a hyperparameter that can be set according to demand. After the mel filter bank is applied to the frames, the construction of the mel spectrogram is done. In this phase, it has a shape of \(n\times m'\).

In the last stage, the spectrogram is adjusted to the selected number of frames (mel spectrogram columns), denoted as \(m\). This modification is required to unify the length of each mel spectrogram to construct mini-batches with the same shape (more about training in Section~\ref{sec:training-implementation}). If \(m < m'\), then the \(m' - m\) number of columns is removed from the right side of the spectrogram. In the other case, when  \(m > m'\), new columns filled with neutral values are added to the right side of the spectrogram. Again, \(m\) is a hyperparameter that can be selected according to preferences. All in all, the final mel spectrogram has a shape of \(n \times m\), where \(n\) represents the number of rows (mel bands), and \(m\) implies the number of columns (time frames).


\begin{figure}
    \centering
    \includegraphics[width=0.97\textwidth]{obrazky-figures/implementation/mel_spectrogram_construction.pdf}
    \captionsetup{justification=centering}
    \caption[Mel spectrogram construction.]{Flow of mel spectrogram cosntruction.\\Step 1. represents loading recording with given sampling frequency.\\ Step 2. denotes signal augmentation of original recording. \\ Step 3. shows transformation of augmented signal to mel spectrogram.\\Step 4. is mel spectrogram augmentation.}
    \label{fig:spectrogram-construction}
\end{figure}


To augment the dataset, there is implemented five augmentation types that work based on Bernoulli random variable \(X\), where the probability for \(X=1\), denoted as \(p\), is an arbitrary value that needs to satisfy the following criterion: \(0 \leq p \leq 1\) . This implementation considers \(p = 0.25\). The value is chosen empirically and it is one of the topics for further discussion.

There are implemented five different augmentation strategies. Three are focused on raw speech recording and the remainder two modify the mel spectrogram. The recording augmentation includes adding random white (gaussian) noise, which is represented as a vector with the same shape as speech recording. Values in this vector are random numbers generated from the Gaussian distribution with zero mean and the variance argument is equal to the recording's variance. Before the noise is added to the original recording, it is multiplied by \textit{noise factor} (small decimal number) serving as regularization of noise impact. The thesis implementation considers three noise factors (0.1, 0.15, 0.2). When augmentation is applied, one of the noise factors is randomly chosen as a regularization value.

The second augmentation technique is speed change. The main idea is to either stretch or shrink the original recording by chosen factor. The implementation considers four different factors (0.85, 0.9, 1.1, 1.15) and when the augmentation is applied, one of them is randomly chosen.

The next method for artificially increasing the number of recordings by editing their signals is \textit{room impulse response} which adds an echo effect and creates the feeling of conference room sound. It is implemented according to the tutorial from the official PyTorch documentation page\footnote{\url{https://pytorch.org/audio/main/tutorials/audio\_data\_augmentation\_tutorial.html}}.

The fourth and fifth implemented augmentation methods are focused on adjusting the mel spectrogram. They are called time and frequency masking. Their purpose is to change a certain portion of the time respectively frequency domain to the value symbolizing that either in a given time is an amplitude of the original signal equal to zero or for given a frequency band that it is not present in the original signal. Both augmentations strategies receive \textit{mask\_param} argument determining the maximum value they can mask in a mel spectrogram. The actual value is computed in runtime and it is sampled from interval \(<0, mask\_param)\), which has uniform probabilistic distribution. The implementation of time and frequency masking is inspired by the PyTorch documentation\footnote{\url{https://pytorch.org/audio/master/generated/torchaudio.transforms.FrequencyMasking.html}}.

% model description
\section{Model implementation}
\label{sec:model-implemenation}

Based on the description in subsection~\ref{subsec:deep-learning-model}, the implementation utilizes the Audio Spectrogram Transformer (AST) by Gond et al.~\cite{audio-spectrogram-transformer}, which is freely available in GitHub\footnote{\url{https://github.com/YuanGongND/ast}} under the 3-Clause BSD license\footnote{\url{https://opensource.org/license/bsd-3-clause/}}, which allows the author to use this code for purpose of this thesis. The thesis implementation contains minor changes without any fundamental impacts compared with the original AST implementation and therefore this section is focused more on the selection of one of offered pretrained ViT models.

According to the original paper that proposed AST, its implementation utilizes the standard ViT architecture proposed in~\cite{vision-transformer}, which was properly described in Section~\ref{sec:vision-transformer}. In fact, this implementation uses the models from the library PyTorch Image Models (Timm)\footnote{\url{https://github.com/huggingface/pytorch-image-models/tree/main/timm}}~. The provided code offers four types of pretrained ViT, which differ in their number of trainable parameters that directly implies their size and memory requirements. The list of offered models contains \textbf{tiny224} (\(\approx 5.8\) million parameters), \textbf{small224} (\(\approx 22.2\) million parameters), \textbf{base224} (\(\approx 86.8\) million parameters) and \textbf{base384} (same number of parameters as base224). Obviously, the model name represents its relative size and the number denotes the image size that was used during ImageNet pretraining phase, e.g. type tiny224 was pretrained on the images of size \(224\times 224\). Since the utilization of SER models is focused on small or even embedded devices, the thesis author decided to choose the model of tiny224. Another reason to use type tiny224 is the size of available datasets that do not demand a bigger model type to obtain better performance. The further implication of this decision is the potentially less time-consuming training of the model, which is highly influenced by the number of trainable parameters. 

The selected classificator for the pretrained model is a multi-layer perceptron that consists of one normalization layer and an output layer of shape equal to the number of considered emotions for a given dataset.

% model training
\section{Training implementation}
\label{sec:training-implementation}
% hyper parameters -> select appropriate learning rate for problem according to PL ligthining manual https://pytorch-lightning.readthedocs.io/en/1.9.2/advanced/training_tricks.html#learning-rate-finder
This implementation follows the backpropagation algorithm that is currently state-of-the-art in the training of neural networks. The training maintains this flow:
\begin{enumerate}
    \item forward propagation of inputs,
    \item compute loss (error) as the difference between predictions and expected values (ground truths),
    \item backpropagate error throw the network in order to adjust parameters.
\end{enumerate}

\subsubsection*{Loss function and optimizer definition}
First, there is a need to define the function responsible for computing loss (error) as the difference between the outputs of the neural network and expected values. Because dealing with classification task, it is selected \textbf{Cross Entropy} function designed for this purpose. The algorithm \textbf{Adam} is selected as a learning rate optimizer that takes care of adjusting the learning rate value during the training process.

\subsubsection*{Dataset partitioning}
Before the beginning of training, the dataset is divided into 10 equal parts called folds according to the method \textbf{stratified k-fold cross-validation}. Each fold contains two subsets of data samples split in a ratio of 9:1. The first subset contains data samples for training and the second for testing. The logic behind such partitioning is to have a subset of data samples that serve exclusively for training and the other one for testing purposes and final evaluation. In the code, there is used class \texttt{StratifiedKFold}\footnote{\url{https://scikit-learn.org/stable/modules/generated/sklearn.model\_selection.StratifiedKFold.html}} to achieve described partitioning.

The implication of this approach is the need to create 10 different models and run 10 training sessions independent of each other. The results of this way should secure enough evaluation data to decide, if the model performance is dependent on dataset partitioning or not.

Each subset is then further divided into mini-batches of size \(n\), which is an arbitrary number. The purpose of the mini-batch is to tackle the problem of model parameter update frequency. The lower the \(n\) is, the higher frequency of parameter updates, but it can lead to some local optimum of loss followed by the suboptimal model results. On the other hand, the low frequency of updates can lead to slowing the training process and the high memory requirements. To handle the trade-off between these risks, the \(n\) is set to 32.

\subsubsection*{Training loop}
The implementation of the training loop follows the steps mentioned at the beginning of this section. It includes an inner mini-batch loop, which first computes neural network inference for input data from the mini-batch, computes loss value and backpropagates the error via the network. Then the loss value together with reached classification accuracy for this mini-batch is stored to epoch evidence. Both values are preserved in the mentioned evidence and at the end of the epoch is logged their mean as model performance representation of the current epoch. 

During the training loop is also tracked the carbon footprint since the most computationally-extensive operation are performed in it. For the collection of carbon footprint data is utilized the library Codecarbon\footnote{\url{https://codecarbon.io/}}. The tracking frequency is set per epoch and values are stored to global evidence for evaluation.

The tracker class provided by this library automatically detects the type of computational resources and measures their consumption of power in kilowatt-hours (kWh). The only watched computational resoruce is grahpical processing unit. The energy consumption of other computational units is not tracked because the training is run on computer with shared resources except GPU that is assigned exclusively to one user. 

The measured consumed energy is multiplied by the coefficient, which is interpreted as the amount of emitted carbon dioxide mass necessary to produce this energy. The coefficient is determined automatically based on the computational resource location. For Czechia, the coefficient has the value of 386 grams of carbon dioxide per kilowatt-hour of consumed energy (gCO\(_2\)/kWh)\footnote{\url{https://github.com/mlco2/codecarbon/blob/master/codecarbon/data/private_infra/eu-carbon-intensity-electricity.csv}}. 

Another implemented feature is model checkpointing, which monitors the
value of loss function on training data after each epoch. At the end of the training, it returns the model with a setting that achieved the lowest loss value. This model is considered the best from this training and is stored for evaluation. Since there is performed 10 training runs for one dataset, the ten best models (one from each run) are saved for evaluation. 

The implementation of training loop and model checkpointing is realized via the library PyTorch Lightning\footnote{\url{https://lightning.ai/docs/pytorch/stable/}} that offers convenient interface and implementation guidelines. This library also takes care of the proper utilization of available computational resources (e.g. GPUs) with the possibility of configuration. To track, log and store the training data is selected library wandb, which provides an online tool for visualization of measured data with the possibility of its real-time monitoring\footnote{\url{https://wandb.ai/}}.

% real use of the model

\section{Evaluation implementaion}
\label{sec:evaliation-impelementation}

After the training, each model makes predictions on its test part of the dataset, which results are stored and used as input for functions that compute metrics. The list of considered metrics includes: \textbf{accuracy} (weighted and unweighted), \textbf{precision}, \textbf{recall}, \textbf{f1 score} and \textbf{specificity}. Except for accuracy, the rest of the metrics are computed in three ways: per class, unweighted and weighted per dataset. For accuracy, there is provided only weighted and unweighted per dataset values.


Before metrics computation, the first step is to build the \textbf{confusion matrix}, which example is depicted in Figure~\ref{fig:confusion-matrix-example}. This matrix is starting point for further calculations. The formulas used to gain metrics values are shown in equations ~\ref{eq:accuracy-formula}--~\ref{eq:specificity-formula}. In each of these formulas \(TP\) (\textit{true positive}) means the number of correctly classified samples into the \textit{positive} category, \(TN\) (\textit{true negative}) represents the number of correctly classified samples into the \textit{negative} category, \(FP\) (\textit{false positive}) means the number of incorrectly classified samples into the \textit{positive} category and \(FN\) (\textit{false negative}) denotes the number of incorrectly classified samples into the \textit{negative} category.

\begin{figure}[h!]
    \centering
    \includegraphics[width=0.6\textwidth]{obrazky-figures/implementation/sample_confusion_matrix.pdf}
    \captionsetup{justification=centering}
    \caption[Confusion matrix example.]{Example of the confusion matrix. The horizontal axis represents the predicted classes and the vertical target classes (ground truths). The values in diagonal cells represent the number of correctly predicted data samples for a given class.}
    \label{fig:confusion-matrix-example}
\end{figure}

To enumerate given formulas, there is need to substitute \(TP\), \(TN\), \(FP\) and \(FN\) with specific values. They are computed with an approach called \textit{One-vs-Rest} (OvR) that selects one class as \textit{positive} (referential) and others are considered as \textit{negative}. Then, \(TP\) is equal to the value of correctly classified samples for the referential class, \(TN\) is equal to the number of samples that are marked as negative and their predicted class is negative too. \(FP\) is the sum of samples that are classified as positive, but their true label is negative. \(FN\) is the sum of cells that represents negative predictions for samples with a positive true label. This process is repeated for each class. The per-dataset values for precision, recall, F1 score and specificity are obtained as weighted and unweighted averages of per-class metrics. The implementation utilizes the Scikit-learn and Torchmetrics\footnote{\url{https://torchmetrics.readthedocs.io/en/stable/}} libraries for metrics calculation.

\begin{equation}
    Accuracy = \frac{TP + TN}{TP + FP + TN + FN}
    \label{eq:accuracy-formula}
\end{equation}

\begin{equation}
    Precision = \frac{TP}{TP + FP}
    \label{eq:precision-formula}
\end{equation}

\begin{equation}
    Recall = \frac{TP}{TP + FN}
    \label{eq:recall-formula}
\end{equation}

\begin{equation}
    F1 \, score = \frac{2 \times Precision \times Recall}{Precision + Recall}
    \label{eq:recall-formula}
\end{equation}

\begin{equation}
    Specificity = \frac{TN}{TN + FP}
    \label{eq:specificity-formula}
\end{equation}


\bigskip

This chapter contained implementation details of the program that performs speech recording preprocessing, model initialization and training. At the beginning of the chapter was described the dataset class that provides data samples for the AST model in demanded form. It includes the entire process from reading the speech recording data from the file to editing them using augmentation techniques. The following section designates the model implementation details. At first, it was described the partitioning of mel spectrograms into the sequence of patches enriched by values determining their position in the original input. Then the inference process summary was provided, which outputs an vector with probabilities for each of the considered emotional classes. The last two sections contained a description of the training methodology and the evaluation of training outputs.
