% \chapter{Vision Transformer and speech processing}
\chapter{Proposed methodology}
\label{ch:solution-proposal}

% This chapter analyzes a solution concept that is a blueprint for the implementation. The baseline of this concept is \textbf{Vision Transformer (ViT)} architecture examined in the following subsections. For a complete understanding of the ViT, it is necessary to understand the \textbf{Transformer} architecture from which is derived.

% After examination of the Transformer, further work is focused on the introduction of ViT and what are the possibilities to use it in the context of audio and speech processing. The final section of this chapter is dedicated to the suggested plan for implementation part.

This chapter analyzes a solution concept that is a blueprint for the implementation. The baseline of this concept is \textbf{Vision Transformer (ViT)} architecture, derivation of Transformer architecture. Therefore, for a complete understanding of the whole solution, it is necessary to study ViT as well as Transformer. 

After the examination of mentioned deep learning concepts, further work is focused on the possibilities to use ViT in terms of audio and speech processing. The final section of this chapter devotes to the suggested plan for implementation part.

\section{Transformer neural network}
\label{sec:transformer-neural-network}

The idea of a Transformer was first proposed by the Vaswani et al.~\cite{transformers-NIPS2017_3f5ee243}. Its principal purpose is to process sequential inputs. The original model was used for the machine translation task (part of natural language processing), where it performed at the state-of-the-art level according to the results in~\cite{transformers-NIPS2017_3f5ee243}. 

The Transformer architecture is depicted in Figure~\ref{fig:transformer-neural-network}. It is \textit{encoder-decoder} deep neural network architecture, which does not contain convolutional or recurrent units. According to the mentioned figure, each block occurs \(N\) times in the architecture. Both blocks have common parts, particularly \textbf{positional encoding} and \textbf{multi-head attention mechanism}. This section is dedicated to the explanation of these two components, whereas they form the skeleton of each of these blocks. It is important to mention that each of the blocks can create a standalone neural network. The models that contain only encoders are primarily used for encoding and are used for another task (e.g. classification). On the other hand, models that consist only of decoders are used for the prediction of the next value in a sequence based on the previous input and thus are considered autoregressive.
  

\begin{figure}[h!]
    \centering
    \includegraphics[width=0.5\textwidth]{obrazky-figures/solution_proposal/transformer_neural_network.png}
    \caption[Transformer neural network architecture.]{Transformer neural network architecture~\cite{transformers-NIPS2017_3f5ee243}.}
    \label{fig:transformer-neural-network}
\end{figure}


The first step of the input processing is its encoding in a way that assigns each input part, called a token, at index \(i\) in a sequence the length of \(N\), a value that determines its position. This step is called \textit{positional encoding (PE)}, and its output is called \textit{embedding}. The transformation process of the input values to the embeddings is expressed by equations~\ref{eq:pe-even-indices} and \ref{eq:pe-odd-indices}. Each equation takes as the input the token denoted as \(pos\) with index \(i\), which needs to be adjusted to create position awareness that determines its order in a sequence for further network processing. The parameter \(d_{model}\) is the size of the embedding, which was for the original Transformer model set to value 512. These embeddings are further connected to input matrix \(I\in\mathbb{R}^{N\times d_{model}}\).

\begin{equation}
      PE_{(pos, 2i)} = sin(pos/10000^{2i/d_{model}})
      \label{eq:pe-even-indices}
  \end{equation}  

\begin{equation}
      PE_{(pos, 2i+1)} = cos(pos/10000^{2i+1/d_{model}})
      \label{eq:pe-odd-indices}
\end{equation}


After the input is encoded to the suitable form, it is further processed by the encoder block, which uses \textbf{a multi-head self-attention layer} with a feedforward layer. The examination of self-attention is related to the prior work in Section~\ref{subsec:deep-learning-approach}, where the author briefly described the attention mechanism. The self-attention mechanism can be described as the computation process, which 
computes a score among the segments of input to express their relationships. In Figure~\ref{subfig:self-attention}, there is a particular type of the self-attention mechanism, which is called \textbf{scale dot-product attention} and is used in the original model. Its inputs are the matrices \(Q\in\mathbb{R}^{N\times d_{model}}\), \(K\in\mathbb{R}^{N\times d_{model}}\) and \(V\in\mathbb{R}^{N\times d_{model}}\), which are results of linear projections of input matrix \(I\) with projection matrices 
\(W^{Q}\in\mathbb{R}^{d_{model}\times d_{model}}\), \(W^{K}\in\mathbb{R}^{d_{model}\times d_{model}}\) and \(W^{V}\in\mathbb{R}^{d_{model}\times d_{model}}\). The computation of these matrices is depicted by the equation~\ref{eq:self-attention-inputs-computation}.

\begin{subequations}
    \begin{equation}
        Q = I \times W^{Q}
    \end{equation}
    \begin{equation}
        K = I \times W^{K} 
    \end{equation}
    \begin{equation}
        V = I \times W^{V}
    \end{equation}
      \label{eq:self-attention-inputs-computation}
\end{subequations} 

The final matrix \(A\in\mathbb{R}^{N\times d_{model}}\) of the self-attention is expressed in the equation~\ref{eq:self-attention-output-computation}. The matrices \(W^{Q}\), \(W^{K}\) and \(W^{V}\) are learnable parameters that are adjusted during the training of the neural network.

\begin{equation}
      A=softmax(\frac{QK^T}{\sqrt{d_{model}}})V
      \label{eq:self-attention-output-computation}
\end{equation}

\begin{figure}[H]
    \centering
    \begin{subfigure}[b]{0.45\textwidth}
        \includegraphics[width=\textwidth]{obrazky-figures/solution_proposal/scaled_dot_product.png}
        \captionsetup{justification=centering}
        \caption{\textbf{Scale Dot-Product Attention} - version of the self-attention mechanism used in original Transformer architecture.} 
        \label{subfig:self-attention}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.45\textwidth}
        \includegraphics[width=\textwidth]{obrazky-figures/solution_proposal/multihead_attention.png}
        \caption{Multi-head attention mechanism.}
        \label{subfig:multi-head-attention}
    \end{subfigure}
    \captionsetup{justification=centering}
    \caption[The Attention mechanism components.]{The Attention mechanism components~\cite{transformers-NIPS2017_3f5ee243}.}
    \label{fig:attention-mechanism}
\end{figure}

The concept of the multi-head attention mechanism, depicted in Figure~\ref{subfig:multi-head-attention}, can be treated as the extension of the self-attention mechanism. The term \textit{multi-head} refers to the fact that input is processed by multiple self-attention blocks enabling the model to focus on different parts of the input at the same time (in parallel). As shown in Figure~\ref{subfig:multi-head-attention}, there are \(h\) blocks of the self-attention called \textit{heads}, and each of them processes the input independently of one another. In particular, there are \(h\) heads and each process the input of size that is equal to a proportion of the embedding size and the number of heads (\(d_{model}/h\)). It implies that the size of the projection matrices of each head is reduced by the factor of \(h\).   

After each head produces its results, they are concatenated together to the embedding of the original size \(d_{model}\). The final output of the multi-head attention is produced by the fully-connected layer that has the purpose to link information among concatenated embeddings. In the original paper the authors used~8 heads implying that each one processed embedding of size 64 (\(512 / 8 = 64\)).



\section{Vision Transformer}
\label{sec:vision-transformer}

Dosovitskiy et al.~\cite{vision-transformer} presented a neural network architecture based on the Transformer architecture designed to handle computer vision tasks. The inspiration for ViT design came from the success of the Transformers in NLP tasks, especially a neural network called BERT~\cite{devlin2019bert}. Figure~\ref{fig:vision-transformer-neural-network} depicts the original architecture of ViT, which consists only of the encoder part of the Transformer. 

This architecture provides the linear transformation layer that projects the input image to fixed-size patches. They are converted to the \(1D\) tokens in the next step. Another property that ViT brings is the extra learnable token that is denoted in Figure~\ref{fig:vision-transformer-neural-network} as \texttt{[class]}. It serves classification purposes and is inspired by the BERT model. It is connected to the beginning of the token sequence and represents a summary of the whole image. At the start of the training, \texttt{class} is initialized with random values that are adjusted during the training. Its output plays a key role as it is connected to the classification head, which returns the final network result.

The first stage in the model inference is a partitioning of the input image with shape \({H\times W \times C} \) to patches \(\textbf{x}_{p}\) with shape \({P^2\cdot C}\), where \((H, W, C)\) represents height, width and number of colour channels of the image, \(P\) is the size of a patch, \(N = (H \cdot W) / P^2\). After this step, each patch \(\textbf{x}_{p}\) is fed to the linear projection layer, where the shape of each patch is adjusted to the value \(D\) that is equal to the token size. This linear layer is composed of learnable parameters tuned during the fitting process. After projection, \texttt{[class]} token is inserted at the start of the sequence. Then to each token is added a positional encoding value, which results in the embeddings sequence ready to enter the encoder block. Its only used output is \texttt{[class]} token output that is further fed to the MLP head, which produces the final classification result.

\begin{figure}[h!]
    \centering
    \includegraphics[width=\textwidth]{obrazky-figures/solution_proposal/vision_transformer.png}
    \caption[Vision Transformer neural network architecture.]{Vision Transformer neural network architecture~\cite{vision-transformer}.}
    \label{fig:vision-transformer-neural-network}
\end{figure}

The authors of~\cite{vision-transformer} implemented and evaluated three types of Vision Transformer, which differ in the size of an image patch, the number of heads of multi-head attention, the number of encoders and the size of the final classification module. The evaluation was done on the image classification benchmarks such as ImageNet, CIFAR-100\footnote{\url{cs.toronto.edu/~kriz/cifar.html}}, Oxford-IIIT Pets\footnote{\url{https://www.robots.ox.ac.uk/~vgg/data/pets/}} etc. The observed metric was accuracy. The obtained results were compared to the models \textbf{ResNet152x4} and \textbf{EfficientNet-L2}. Each of the ViT models reached better results than the other compared models. The authors also observed the training time measured in TPUv3-core-days, and each ViT model needed less training time than their opponents. In two cases was a difference in the order of magnitude.


\section{Audio Spectrogram Transformer}
\label{sec:vision-transformer-and-audio-processing}

Based on the ViT architecture, Gong et al.~\cite{audio-spectrogram-transformer} created \textbf{Audio Spectrogram Model (AST)} utilized for the classification of audio recordings based on their Mel spectrograms. Figure~\ref{fig:ast} depicts its overall structure and outlines its working principle, which mostly matches the ViT. The original idea was to create a model based on transformer architecture that is able to process the audio and speech data without convolutions. Its additional interesting property is the transfer learning approach because the AST uses in its core the pretrained ViT on the ImageNet dataset.

\begin{figure}[h!]
    \centering
    \includegraphics[width=0.7\textwidth]{obrazky-figures/solution_proposal/ast.png}
    \caption[Audio Spectrogram Transformer architecture.]{Audio Spectrogram Transformer architecture~\cite{audio-spectrogram-transformer}.}
    \label{fig:ast}
\end{figure}

The ViT accepts the RGB images that are adjusted to fixed shape (e.g. \(224\times224\) or \(384\times384\)) which is not typical for the mel spectrogram, which tends to vary in both dimensions. Therefore, the first step is the reduction of the input channel number from three to one. This is done by averaging the weights of the linear projection layer.

Another issue with the shape is connected to positional encoding. The ViT deals only with the sequences of fixed sizes even though it is a Transformer model. This is caused by the utilization of learnable positional encoding that is pretrained. As mentioned earlier, the mel spectrograms tend to vary and there is a need to choose the size of the mel filter bank and the number of frames that should spectrogram contain in order to adjust positional encoding to demanded shape. To account for this fact, the authors of AST proposed a technique, which modifies positional encoding with respect to the selected values, particularly, each dimension is reduced or lengthened with \textit{bi-linear interpolation} method. Let's suppose for demonstration purposes ViT model with patch shape \(16\times 16\) that was trained with images of shape \(224\times 224\). Then, each image is partitioned to \(14\times 14\) and projected to a token of size \(768\). This implies that positional encoding generates value for \(192\) tokens to create position-aware embeddings. If the same model should be used as AST with inputs of size \(1024\times128\) (\(64\times8\) patches), the positional encoding needs to be adjusted to be able to process sequence of size \(512\), where horizontal (\(x\)) part of positional encoding is reduced to the value \(8\) and vertical (\(y\)) part is interpolated to \(64\).

As mentioned in the section beginning, the Gong et al.~\cite{audio-spectrogram-transformer} proposed model was pretrained on the ImageNet dataset and fine-tuned on the following datasets AudioSet\footnote{\url{https://ieeexplore.ieee.org/abstract/document/7952261}}, ESC-50\footnote{\url{https://dl.acm.org/doi/10.1145/2733373.2806390}} and Speech Commands V2\footnote{\url{https://arxiv.org/abs/1804.03209}}. While the results of the AudioSet did not reach state-of-the-art, the model performed better than state-of-the-art models on the rest two datasets. Besides these results, the model shows that it is possible to utilize the pure transformer model pretrained on the image dataset, fine-tune it on an audio dataset and obtain intriguing performance.

\section{Architecture of the proposed SER system}
\label{sec:proposed-architecture}

This section comprehends the description of the proposed SER system together with the evaluation methodology. The system's explanation includes the emotional model, the database(s), preprocessing and feature extraction methods and the deep learning model.

The primary goal of the designed system is to tackle the following challenges mentioned in~\ref{subsec:challanges-in-the-SER-research}: 
\begin{itemize}
    \item \textbf{pure Transformer model} - use model that is based only on the Transformer architecture without any convolutions and recurrent cells as in other recognition fields that deep learning emerged in,
    \item \textbf{transfer learning} - use pretrained ViT model, 
    \item \textbf{environmental impact} - track the greenhouse effect produced during the fitting process.
\end{itemize}

\subsection{Emotional model and databases}
\label{subsec:emotional-model-and-databases}
The thesis author decided to choose a discrete (categorical) emotional model because of the properties such as good interpretability and comprehensibility. Another reason, for such a decision, is the wide offer of the datasets labelled discretely.

Regarding datasets, the author wanted to do a comprehensive evaluation that includes five different emotional corpora, particularly Emo-DB, EMOVO~\cite{emovo} and RAVDESS. This set contains three languages (English, German and Italian). The usage of more emotional corpora results in a more robust estimation of SER system capabilities.


\subsection{Preprocessing and feature extraction methods}
\label{subsec:preprocessing-and-feature-extraction-methods}

Due to using the skeleton of the deep learning model from~\cite{audio-spectrogram-transformer}, the feature extraction stage has to correspond with the process described in this paper. Its results are fixed-size mel spectrograms. In Figure~\ref{fig:the-architecture-of-the-solution} is this part depicted by the \textbf{Mel spectrogram extractor} part. 

Each recording is divided into frames of length 25 milliseconds with 10 milliseconds hop size. On each frame is applied the windowing function with Hanning window. The Short-time Fourier transform (STFT) is used for the transformation of each frame from time to frequency domain and its results are filtered by a mel filter bank with 32 filters, which values are representing the mel spectrogram.

\subsection{Deep learning model}
\label{subsec:deep-learning-model}
The proposed deep learning model consists of two logical parts, particularly the pretrained AST model and the custom classification part. In Figure~\ref{fig:the-architecture-of-the-solution} are these parts depicted as \textbf{Audio Spectrogram Transformer} and \textbf{Classification head}.

\begin{figure}[h!]
    \centering
    \includegraphics[width=\textwidth]{obrazky-figures/solution_proposal/model_picture.pdf}
    \captionsetup{justification=centering}
    \caption[The proposed SER system.]{The proposed SER system as a pipeline of modules. Its result is only illustrative.}
    \label{fig:the-architecture-of-the-solution}
\end{figure}

The AST part is taken from the~\cite{audio-spectrogram-transformer} as a pretrained model and it serves as a backbone of the model. The classification head is represented by a fully connected layer series with the output layer that returns the probabilities of emotion labels. The parameters of the classification head are fine-tuned during the fitting process on the mentioned datasets.  


\subsection{Evaluation methodology}
\label{subsec:evaluation-methodology}

The evaluation methodology is based on two dimension. The first dimension regards the quantity of dataset used for proposed model training. The second dimension regards the results quality of each dataset. As inspiration for quantitative approach, the review state-of-the-art model Tim-Net~\cite{Jiaxin-TIM-NN} was trained on six independent emotion databases. On the other hand, for each dataset is provided only weighted and unweighted accuracy and lacks the of other classification metrics, which causes insufficiency in qualitative dimension.

An opposite case is an approach in the paper that proposed C-RNN~\cite{Kumaran2021-C-RNN} that used only one dataset for evaluation, but on the other hand, it provides not only accuracy metric but also the whole confusion matrix, loss function progress curve and accuracy progress curve. 















% of size (the authors in ~\cite{transformers-NIPS2017_3f5ee243} chose \(h=8\)).
 


%============================================
% \chapter{veronika a martinkoo je saskuuardko alickz chztil babku ya cickz babka mu to vratila a cickz mu ukayala}