\chapter{Results}
\label{ch:results}


The model implementation described in chapter~\ref{ch:implementation} was tested on a set of experiments with the principal intention of determining its performance on a speech emotion recognition task.

Each experiment comprehended model training on the particular emotional dataset with the following evaluation, which concept was proposed in Subsection~\ref{subsec:evaluation-methodology}. The proper description of evaluation methodology with particular goals is part of this chapter. All experiments were run with the following setting: \textit{number of training epochs} = 200, \textit{initial learning rate value} = 0.0001, \textit{mini-batch size} = 32, \textit{mel spectrogram columns count (time frames)} = 512, \textit{mel spectrogram rows count (mel filter bank size)} = 64, \textit{random seed} = 0 (to control random operation outputs), \textit{number of folds} = 10. The training was performed on computer with one GPU of type Nvidia A40 and operation system Linux. 

\section{Results evaluation methodology}
\label{sec:evaluation-methodology}
The evaluation was designed with the aim of testing the AST's ability to learn emotional patterns that are present in the input speech recordings. The methodology was inspired by reviewed state-of-the-art works~\cite{Kumaran2021-C-RNN, Jiaxin-TIM-NN, Zhao-2D-CNN-selfAM-CTC, Xianfeng-CNN-BiLSTM-STL} in this field that is described in Section~\ref{sec:current-state-of-the-art} with additional evaluation components that the thesis author lacked in these works.

The main idea is to evaluate trained models from a 10-fold cross-validation method and according to observed metrics (accuracy, precision, recall, F1 score and specificity) decide, if the model has enough generalization ability and is suitable for this task. This approach should ensure that the results are independent of dataset partitioning to train and test part. 

The evaluation is performed for each dataset exactly once. From each fold, there is selected the best model that obtained the lowest value of loss function during the training. Each model then makes predictions on test data samples and their results are stored in evidence. After the last model finishes its predictions, this evidence is used for confusion matrix construction and metrics calculation. For each dataset, the measured values are provided in a tabular form together with metrics provided by other state-of-the-art solutions (if they used a particular dataset obviously). Moreover, there is provided confusion matrix and the progress curves of the loss function and training accuracy during epochs. 

The originally-intended evaluation of carbon footprint with Codecarbon library and its environment, unfortunately, fails due to its buggy behaviour and incomplete documentation that lacks necessary details. In order to provide some kind of interpretation of measured carbon footprint data, there is a dedicated section on this topic. It comprehends the bar chart for each dataset with the amount of emitted CO\(_2\) mass per each fold. To provide an interpretation that is more "human-friendly", there is a figure, which depicts the frequency or quantity of popular activities that have the same environmental impact as the performed computation during this thesis. The chosen activities \textit{driving the car}, \textit{smartphone charging} and \textit{coal burning}. For this purpose is selected Greenhouse Gas Equivalencies Calculator\footnote{\url{https://www.epa.gov/energy/greenhouse-gas-equivalencies-calculator\#results}}. It is worth noting that this portal is primarily addressed to the USA region and therefore obtained values are just rough estimations but principal goal is to show the environmental costs of deep learning.

The last sections are dedicated to the evaluation of model complexity in terms of time and memory, discussion of obtained results and potential future work.

\section{RAVDESS dataset}
\label{sec:ravdess-dataset}

 \textbf{The Ryerson Audio-Visual Database of Emotional Speech and Song}~\cite{RAVDESS} is one of the most used datasets for developing and testing models for speech emotion recognition tasks according to the number of citations, as shown in Table~\ref{tab:databases}. This freely available dataset contains 1440 English speech recordings labelled by eight discrete emotion classes. The dataset is categorized according to taxonomy, introduced in Section~\ref{sec:databases}, into the acted category because each recording was performed by a professional actor.

Figure~\ref{fig:RAVDESS-confusion-matrix} shows the confusion matrix. It implies that trained models recognize emotion \textit{surprised} with the highest accuracy, while with \textit{sad} and \textit{neutral} emotions struggle. These facts are confirmed by values in Table~\ref{table:RAVDESS-local-metrics}, which contains the per-class metrics. It suggests that \textit{neutral} emotion has more false positive predictions than false negative because precision is smaller than recall. The opposite case is \textit{happy} and \textit{emotions}, where the precision is higher than recall suggesting more false negative than false positive predictions. The metrics for other emotions indicate almost equal values for false positive and false negative predictions. 

 \begin{figure}[h!]
     \centering
     \includegraphics[width=0.63\textwidth]{obrazky-figures/results/RAVDESS_confusion_matrix.pdf}
     \caption{RAVDESS confusion matrix.}
     \label{fig:RAVDESS-confusion-matrix}
 \end{figure}


\begin{table}[h!]
\centering
\begin{tabular}{|l|l|l|l|l|}
\hline
      \textit{Emotion}  & \textit{Precision} & \textit{Recall}  & \textit{F1 score}  & \textit{Specificity} \\ 
     \hline
     neutral & 0.709 & 0.813 & 0.757 & 0.976 \\ 
     \hline
     calm & 0.862 & 0.880 & 0.871 & 0.978 \\ 
     \hline
     happy & 0.836 & 0.770 & 0.802 & 0.976 \\ 
     \hline
     sad & 0.796 & 0.750 & 0.772 & 0.970 \\ 
     \hline
     angry & 0.880 & 0.885 & 0.883 & 0.981 \\ 
     \hline
     fearful & 0.853 & 0.844 & 0.848 & 0.978 \\ 
     \hline
     disgust & 0.873 & 0.896 & 0.884 & 0.984 \\ 
     \hline
     surprised & 0.888 & 0.906 & 0.897 & 0.982 \\ 
     \hline
\end{tabular}
\caption{Class metrics for RAVDESS dataset}
\label{table:RAVDESS-local-metrics}
\end{table}


 Table~\ref{table:ravdess-global-results} contains values of global metrics compared with other SER models that used RAVDESS. This table shows that the AST model reached the second-best accuracy among compared models. Unfortunately, the other metrics are not provided.

\begin{table}[h!]
\centering
\begin{tabular}{|l||l|l|l|l|l|}
 \hline

 & \textit{Accuracy} & \textit{Precision} & \textit{Recall} & \textit{F1 score} & \textit{Specificity} \\ 
 \hline
 C-RNN & ~0.7/- & - & - & - & - \\ 
 \hline
% CNN-BiLSTM & - & - & - & - & - \\ 
% \hline
% PCNSE & - & - & - & - & - \\
% \hline
TIM-Net & 91.93 / 92.08 & - & - & - & - \\
\hline
\hline
AST & 0.845 / 0.851 & 0.837 / 0.846 & 0.843 / 0.845 & 0.839 / 0.845 & 0.978 / 0.978 \\
\hline
\end{tabular}
\captionsetup{justification=centering}
\caption[Global metrics for RAVDESS dataset]{Global metrics for RAVDESS dataset compared with other solutions. The format for each cell is \textit{*unweighted value*} / \textit{*weighted value*}}
\label{table:ravdess-global-results}
\end{table}

Figure~\ref{fig:RAVDESS-training-data} shows statistics observed during the fitting of the models. Figure~\ref{subfig:RAVDESS-loss} shows a chart with the loss function progress, while Figure~\ref{subfig:RAVDESS-accuracy} depicts the accuracy progress for training data. Both figures provide curves for each fold but they are aligned so well due to performance consistency so it is hard to recognize them. This consistency indicates that for AST with initial conditions set in this work, the model is independent of dataset splitting.


\begin{figure}[H]
    \centering
    \begin{subfigure}[b]{0.495\textwidth}
        \includegraphics[width=\textwidth]{obrazky-figures/results/RAVDESS_loss.pdf}
        \caption{Loss function progress}
        \label{subfig:RAVDESS-loss}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.495\textwidth}
        \includegraphics[width=\textwidth]{obrazky-figures/results/RAVDESS_accuracy.pdf}
        \caption{Training accuracy progress}
        \label{subfig:RAVDESS-accuracy}
    \end{subfigure}
    \captionsetup{justification=centering}
    \caption[RAVDESS loss and accuracy progress.]{RAVDESS loss and accuracy progress for 10-fold cross-validation. Both curves aligned so well that it indicates consistent behaviour no matter without the influence of dataset split.}
    \label{fig:RAVDESS-training-data}
\end{figure}

\section{Emo-DB dataset}
\label{sec:emodb-dataset}

Emo-DB~\cite{berlin-database-of-emotional-speech} is the most popular dataset for speech emotion recognition according to the number of citations in academic papers, which demonstrates Table~\ref{tab:databases}. It comprehends 535 utterances in the German language recorded by professional actors, so it is categorized as the acted dataset. It divides recordings into seven discrete emotional classes.

Figure~\ref{fig:EMODB-confusion-matrix} depicts the confusion matrix created from predictions of trained models on their test sets. It suggests that model performance for this dataset is good because of low false positives and false negatives occurrence frequency. The only odd one in false positive predictions is \textit{happiness} emotion.

 \begin{figure}[ht!]
     \centering
     \includegraphics[width=0.63\textwidth]{obrazky-figures/results/EMODB_confusion_matrix.pdf}
     \caption{Emo-DB confusion matrix.}
     \label{fig:EMODB-confusion-matrix}
 \end{figure}

 According to class metrics in Table~\ref{table:EMODB-local-metrics}, for five out of seven classes the model reached more than 90\% for each observed metric. The biggest problem is noticeable with emotion \textit{happiness}. Recall value under 80\% confirms a high ratio of \textit{false negative} predictions for this emotion, as mentioned earlier.


\begin{table}[ht!]
\centering
\begin{tabular}{|l|l|l|l|l|}
\hline
      \textit{Emotion}  & \textit{Precision} & \textit{Recall}  & \textit{F1 score}  & \textit{Specificity} \\ 
     \hline
     anger & 0.904 & 0.969 & 0.936 & 0.968 \\ 
     \hline
     boredom & 0.926 & 0.926 & 0.926 & 0.987 \\ 
     \hline
     disgust & 0.88 & 0.957 & 0.917 & 0.988 \\ 
     \hline
     fear & 0.969 & 0.913 & 0.940 & 0.996 \\ 
     \hline
     happiness & 0.844 & 0.761 & 0.800 & 0.978 \\ 
     \hline
     sadness & 0.966 & 0.935 & 0.950 & 0.996 \\ 
     \hline
     neutral & 0.924 & 0.924 & 0.924 & 0.987 \\ 
     \hline
\end{tabular}
\caption{Class metrics for Emo-DB dataset.}
\label{table:EMODB-local-metrics}
\end{table}

Table~\ref{table:EMODB-global-results} shows global statics compared to other solutions mentioned in the thesis, but only TIM-Net utilized the Emo-DB as well.

\begin{table}[h!]
\centering
\begin{tabular}{|l||l|l|l|l|l|}
 \hline

 & \textit{Accuracy} & \textit{Precision} & \textit{Recall} & \textit{F1 score} & \textit{Specificity} \\ 
 \hline
TIM-Net & 0.952 / 0.957 & - & - & - & - \\
\hline
\hline
AST & 0.912 / 0.916 & 0.916 / 0.916 & 0.912 / 0.916 & 0.913 / 0.915 & 0.990 / 0.983 \\
\hline
\end{tabular}
\captionsetup{justification=centering}
\caption[Global metrics for Emo-DB dataset.]{Global metrics for Emo-DB dataset compared with other solutions. The format for each cell is \textit{*unweighted value*} / \textit{*weighted value*}.}
\label{table:EMODB-global-results}
\end{table}

Figure~\ref{fig:EMODB-training-data} comprehends training statistics for loss function together with accuracy progress. The curves are again well-aligned, but the deviations are more frequent, mainly in the first hundred epochs compared to the RAVDESS data. These figures also imply the fast convergence of the loss function, which mostly decreases during the first thirty epochs. This fact is also noticeable in training accuracy, which reached 0.9 value in the first fifty epochs in each fold.

\begin{figure}[h!]
    \centering
    \begin{subfigure}[b]{0.495\textwidth}
        \includegraphics[width=\textwidth]{obrazky-figures/results/EMODB_loss.pdf}
        \caption{Loss function progress.}
        \label{subfig:EMODB-loss}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.495\textwidth}
        \includegraphics[width=\textwidth]{obrazky-figures/results/EMODB_accuracy.pdf}
        \caption{Training accuracy progress.}
        \label{subfig:EMODB-accuracy}
    \end{subfigure}
    \captionsetup{justification=centering}
    \caption[Emo-DB loss and accuracy progress.]{Emo-DB loss and accuracy progress for 10-fold cross-validation. The curves contain occasional deviations that are stabilized after the first hundred epochs.}
    \label{fig:EMODB-training-data}
\end{figure}

\section{EMOVO dataset}
\label{sec:emovo-dataset}

EMOVO~\cite{emovo} is an Italian emotional corpus, which contains 588 recordings that belong to seven emotional classes. The utterances in EMOVO were recorded by professional actors. Therefore is EMOVO included in the family of acted datasets. The reason for the inclusion of this dataset is to expand the set of languages that were utilized during training and evaluation and observe their influence. 

The confusion matrix depicted in Figure~\ref{fig:EMOVO-confusion-matrix} indicates that despite EMOVO having almost the same number of recordings as Emo-DB, model performance is not the same or similar. The \textit{joy} emotion seems the most problematic because of its high ratio of false positives as well as false negatives. The model predicted \textit{sad} emotion most successfully.

\begin{figure}[H]
     \centering
     \includegraphics[width=0.6\textwidth]{obrazky-figures/results/EMOVO_confusion_matrix.pdf}
     \captionsetup{justification=centering}
     \caption{EMOVO confusion matrix.}
     \label{fig:EMOVO-confusion-matrix}
 \end{figure}


Table~\ref{table:EMOVO-class-metrics} shows that model poorly recognizes \textit{joy} and \textit{surprise} emotions, but on the other hand it, \textit{sad} and \textit{natural} emotions classifies with relatively good results. \textit{Surprise} and \textit{neutral} emotions have a higher ratio of false positive than false negative predictions. The opposite cases are \textit{disgust}, \textit{anger} and \textit{fear} with a smaller number of false positive than false negative predictions. Other emotions have almost the same ratio of observed metrics. 

\begin{table}[h!]
\centering
\begin{tabular}{|l|l|l|l|l|}
\hline
      \textit{Emotion}  & \textit{Precision} & \textit{Recall}  & \textit{F1 score}  & \textit{Specificity} \\ 
     \hline
     disgust & 0.713 & 0.679 & 0.695 & 0.954 \\ 
     \hline
     joy & 0.529 & 0.536 & 0.533 & 0.921 \\ 
     \hline
     fear & 0.695 & 0.679 & 0.687 & 0.950 \\ 
     \hline
     anger & 0.838 & 0.798 & 0.817 & 0.974 \\ 
     \hline
     surprise & 0.652 & 0.690 & 0.671 & 0.938 \\ 
     \hline
     sad & 0.894 & 0.905 & 0.899 & 0.982 \\ 
     \hline
     neutral & 0.850 & 0.881 & 0.865 & 0.974 \\ 
     \hline
\end{tabular}
\caption{Class metrics for EMOVO dataset.}
\label{table:EMOVO-class-metrics}
\end{table}

Table~\ref{table:EMOVO-global-results} contains a comparison of global metrics between AST and Tim-NET. Tim-NET with the same training and test methodology has better accuracy almost by 0.2. The rest of the models described in this thesis as state-of-the-art did not use the EMOVO as their training dataset.


\begin{table}[h!]
\centering
\begin{tabular}{|l||l|l|l|l|l|}
 \hline

 & \textit{Accuracy} & \textit{Precision} & \textit{Recall} & \textit{F1 score} & \textit{Specificity} \\ 
 \hline
TIM-Net & 0.92 / 0.92 & - & - & - & - \\
\hline
\hline
AST & 0.738 / 0.738 & 0.739 / 0.739 & 0.738 / 0.738 & 0.738 / 0.738 & 0.956 / 0.956 \\
\hline
\end{tabular}
\captionsetup{justification=centering}
\caption[Global metrics for EMOVO dataset.]{Global metrics for EMOVO dataset compared with other solutions. The format for each cell is \textit{*unweighted value*} / \textit{*weighted value*}.}
\label{table:EMOVO-global-results}
\end{table}

Loss and accuracy progress depicted~\ref{fig:EMOVO-training-data} shows an interesting paradox in terms of results during evaluation. Despite having the excellent convergence of loss function and training accuracy to 0 respectively to 1, the model performance on test data is not quite well. This fact implies the overfitting problem in spite of the data augmentation utilization. 

The progress curves show the stability of training for each fold without any notable deviations, similar to RAVDESS dataset. The fold curves are aligned, which means the convergence speed is almost the same for each fold, as in the case of previous datasets.

\begin{figure}[h!]
    \centering
    \begin{subfigure}[b]{0.495\textwidth}
        \includegraphics[width=\textwidth]{obrazky-figures/results/EMOVO_loss.pdf}
        \caption{Loss function progress.}
        \label{subfig:EMOVO-loss}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.495\textwidth}
        \includegraphics[width=\textwidth]{obrazky-figures/results/EMOVO_accuracy.pdf}
        \caption{Training accuracy progress.}
        \label{subfig:EMOVO-accuracy}
    \end{subfigure}
    \captionsetup{justification=centering}
    \caption[EMOVO loss and accuracy progress.]{EMOVO loss and accuracy progress for 10-fold cross-validation. The curves indicate steady training without siginificant deviations.}
    \label{fig:EMOVO-training-data}
\end{figure}
 

\section{Carbon footprint evaluation}
\label{sec:carbon-footprint-evaluation}

This section is dedicated to the evaluation of the carbon footprint that was produced during neural network training. The information worth emphasizing is that provided CO\(_2\) data are computed from energy consumed by GPU (detailed explanation of reasons in Section~\ref{sec:training-implementation}) during training measured in kilowatt-hours. This data were according to the used methodology multiplied by the carbon intensity index.

Table~\ref{tab:training-statistics} shows statistics of emitted CO\(_2\) with total training time for each dataset. According to the data in this table, the most-consuming training was with the RAVDESS dataset. This fact follows from the size of this dataset, which is bigger in order of magnitude than other used datasets.

The second dataset in terms of measured emissions as well as training time is EMOVO. Despite a comparable number of recordings to Emo-DB, during training with EMOVO was emitted vastly more CO\(_2\) than training with Emo-DB. The argument for this is the length of recordings, which are longer in EMOVO datasets in the unit of seconds. This implies also the training time difference between these two datasets, which is almost three hours.


\begin{table}[h!]
\centering
\begin{tabular}{|l|l|l|}
\hline
      \textit{Dataset}  & \textit{Emitted CO\(_2\) (grams)} & \textit{Training runtime length} \\
     \hline
     RAVDESS & 521.3366 g & 16h 17m \\ 
     \hline
     Emo-DB & 198.2820 g & 5h 20m \\ 
     \hline
     EMOVO & 338.7516 g & 8h 17m\\
     \hline
     \hline
     \textit{Total} & 1058.3703 g & 29h 54m \\
     \hline
     
\end{tabular}
\caption[Carbon footprint and training time statistics.]{Carbon footprint and training time statistics.}
\label{tab:training-statistics}
\end{table}

Figure~\ref{fig:co2-per-training-folds} shows the amount of emitted CO\(_2\) in grams per fold for each dataset. Training with Emo-DB (Figure \ref{subfig:EMODB-co2-folds}) and EMOVO (Figure~\ref{subfig:EMOVO-co2-folds}) shows relatively steady emission production without any significant variance across folds. The opposite case is in the training with RAVDESS (Figure~\ref{subfig:RAVDESS-co2-folds}), where the variance between fold with the lowest amount of emitted CO\(_2\) (fold 1) and fold with the highest (fold 6) amount is almost ten grams. The source of such variability is unknown.

 \begin{figure}
    \centering
    \begin{subfigure}[b]{0.49\textwidth}
        \includegraphics[width=\textwidth]{obrazky-figures/results/RAVDESS_co2.pdf}
        \captionsetup{justification=centering}
        \caption{Emitted emissions during training on RAVDESS dataset.}
        \label{subfig:RAVDESS-co2-folds}
    \end{subfigure}
    \begin{subfigure}[b]{0.49\textwidth}
        \includegraphics[width=\textwidth]{obrazky-figures/results/EMODB_co2.pdf}
        \captionsetup{justification=centering}
        \caption{Emitted emissions during training on Emo-DB dataset.}
        \label{subfig:EMODB-co2-folds}
    \end{subfigure}
    \begin{subfigure}[b]{0.49\textwidth}
        \includegraphics[width=\textwidth]{obrazky-figures/results/EMOVO_co2.pdf}
        \captionsetup{justification=centering}
        \caption{Emitted emissions during training on EMOVO dataset.}
        \label{subfig:EMOVO-co2-folds}
    \end{subfigure}
    \captionsetup{justification=centering}
    \caption[Emitted CO\(_2\) per training folds.]{Emitted CO\(_2\) per datasets. Each subfigure depicts the amount of emitted CO\(_2\) in grams per folds.}
     \label{fig:co2-per-training-folds}
 \end{figure}

Figure~\ref{fig:comparable-acitivities-co2} depicts activities that have a similar environmental impact as all training sessions performed during the thesis writing. This figure implies that for such training it is needed to produce energy that has the same carbon footprint as driving an average gasoline car for circa 4.34 kilometres, charging a smartphone 129 times and burning roughly 0.5 kilograms of coal with respect to the location of Czechia (dependent on used carbon intensity index).

 \begin{figure}
     \centering
     \includegraphics[width=0.7\textwidth]{obrazky-figures/results/co2_comparison.pdf}
     \captionsetup{justification=centering}
     \caption[Activities with a comparable amount of emitted CO\(_2\).]{Activities with a comparable amount of emitted CO\(_2\) as the produced amount throughout the whole training.}
     \label{fig:comparable-acitivities-co2}
 \end{figure}


\section{Model analysis}
\label{sec:complexity-analysis}
This section is dedicated to the model analysis in terms of its space requirements, steps required to perform one inference and average time to train one fold in 10-fold cross-validation in the used environment. 

The implemented model has \(\approx 5.8\) million trainable parameters and takes 23.1 MiB of memory. To complete one inference, the implemented model needs to perform the following steps:
\begin{enumerate}
    \item divides input mel spectrogram into patches - implemented convolutional layer,
    \item transforms patches to embeddings and add positional encoding - implemented with matrix multiplication,
    \item passes through encoder blocks - implemented as a series of self-attention blocks and fully connected layers,
    \item computes of output in the classification head - implemented as one fully connected with layer normalization.
\end{enumerate}
The required time for model training differs according to the selected computational resources and values of hyperparameters. In the case of this work (exact hyperparameters setting is at the beginning of Chapter~\ref{ch:results}), the average training time to complete one fold in 10-fold cross-validation for RAVDESS was 2 hours and 5 minutes, for Emo-DB 31 minutes and for EMOVO 52 minutes.


\section{Discussion and future work}
\label{sec:discussion-and-future-work}

This section is dedicated to the discussion of obtained results in terms of stated challenges that implemented system should address. The obtained results have several implications for further work.

The first challenge this model addressed was if the model based on Transformer architecture without convolutions or recurrent units is able to classify emotions based on the speech input. The results give answers that it is possible with relative success. Although the model's performance did not reach the highest state-of-the-art score, it is possible to confidently claim, based on the measured metrics, that the model with this architecture is capable of learning emotional patterns in speech recordings and classifying them in a manner that cannot be considered coincidental.

The second addressed challenge was to answer the question if it is possible to transfer knowledge from other domains, in this case, image classification, and use it for emotion classification. The evaluation outcomes indicate that the Transfomer-based model (ViT), pretrained on the image dataset, is able to use obtained "knowledge" in speech emotion recognition when the input recordings are adjusted to the appropriate form.

The last thesis solution challenge was to measure the environmental impact of model training. As pointed out in the summary of challenges, this topic is neglected, despite its importance for the whole deep learning paradigm. This thesis showed that some methods and tools for measuring the environmental impact of deep learning are already present but not sufficiently tuned to be satisfactorily used even for academic-level work. Apart from this fact, throughout the training were tracked carbon footprint as the rough estimates of emitted CO\(_2\). Their interpretation provides insights into how the carbon footprint was developed during the training and which activities have similar environmental impacts.

For future work, the emphasis should be put on three principal goals. The first is to train the AST with different initial settings (mel filter banks size, number of mel spectrogram frames, patch size etc.) to find out if it impacts results. 

The second objective is to train the AST on a more comprehensive and robust dataset in terms of data sample count and speech recordings origin, for example MSP-Podcast~\cite{MSP-Podcast}, which contains more than hundred thousand of natural speech recordings. Utilization of this dataset should bring more insight into the capabilities of AST in speech emotion recognition.

The last goal of future work should be to perform cross-corpus evaluation. The idea of this approach is to train the model on one dataset and evaluate it on completely different (e.g. to train on the RAVDESS dataset and evaluate on the EMOVO dataset). It should address the generalization question that is crucial for speech emotion recognition. There is already an early methodology for this topic in the paper by Cheng et al.~\cite{wen2022ctlmtnet}.

Overall, the implemented Audio Spectrogram Transformer model showed promising results that make it a suitable candidate for future improvement in the speech emotion recognition task. One of the significant benefits of this work is the comprehensive evaluation methodology, which was not present in any of the reviewed works. The last highlight of this thesis is the measurement of the environmental impact of the selected deep learning approach, which was not implemented in any of the reviewed works and therefore bring a novelty with future development potential.
  
