\newglossaryentry{SER}
{
    name={SER},
    description={Speech emotion recognition}
}

% \newacronym{ser}{SER}{Speech emotion recognition}

\chapter{Speech emotion recognition}
\label{ch:ser}
% \todo{Describe theoretical part needed for emotion recognition from speech}\newline
Speech emotion recognition (\Gls{SER}) is a task, where the machine is able to infer emotion from the human voice. This task belongs to the field called \textit{Affective computing}, which joins together computer science, psychology and cognitive science~\cite{affective-computing-book}.

Systems performing SER are complex and they are assembled from components, which consider different parts of the SER process. The general structure of this system is depicted in Figure~\ref{fig:SER-system}, which suggests there are several areas to study in order to know and understand the entire system, in particular, these: an underlying psychological theory of emotions, speech signal processing, and machine learning. This chapter is designated to describe each of these areas at a high-level overview and also in the context of the SER.

\begin{figure}
    \centering
    \includegraphics[width=\textwidth]{obrazky-figures/ser_description/SER_system.pdf}
    \captionsetup{justification=centering}
    \caption[SER system and its components.]{SER system and its components. In order to design such system, it is required knowledge from psychology, signal processing and machine learning. It depicts its overall complexity.}
    \label{fig:SER-system}
\end{figure}


 


\section{Emotion analysis}

According to the Cambridge dictionary, emotion is: "\textit{A strong feeling such as love or anger, or strong feelings in general}"~\cite{emotion-cambridge-dictionary-2022}. However, from a scientific point of view, there is still no clear consensus on what emotion actually is. Scientists have been studying the emotional phenomenon for centuries and have a variety of theories about what causes it.

From the current point of view, the first serious study of emotions was conducted by Charles Darwin in the nineteenth century, when he treated emotions as biological processes~\cite{darwin1998expression}. He laid down the foundations for the next development of emotional research with his theory called \textbf{the Facial feedback hypothesis}, in which he described the connection of facial neurons and emotions. It explains that emotions are influenced by the activity of muscles responsible for facial expressions. Later, it was proven by experiments that the inhibition of facial neurons causes degradation of the brain part responsible for the interpretation of emotions~\cite{facial-feedback-emotion-theory}.

% Nowadays, there is no general agreement on whether emotions have a physiological or cognitive base or in other words if before humans feel emotions, because of physiological arousal or cognition. According to the Jeon~\cite{emotions-and-affect-in-HCI}, there are three fundamental theories explaining the origin of emotions: \textbf{James-Lange theory}, \textbf{Cannon-Bart theory} and \textbf{The Schachter-Singer two-factor theory}.

Nowadays, there is no general agreement on whether emotions have a physiological or cognitive base or in other words, if humans feel emotions, because of physiological arousal or cognition. According to the Jeon~\cite{emotions-and-affect-in-HCI}, there are three fundamental theories explaining the origin of emotions: \textbf{James-Lange theory}, \textbf{Cannon-Bart theory} and \textbf{The Schachter-Singer two-factor theory}.

The James-Lange theory says that the first occurs the physiological manifest to external emotional stimulus. Emotional perception is a result of the cognitive awareness of particular physiological change~\cite{james-what-is-emotion}. It is one of the oldest theories, which is still not fully disproved.

The Cannon-Bard theory explains that physiological change and emotion occur simultaneously and independently~\cite{cannon-theory-of-emotions}. Cannon proposed his ideas as the response to the James-Lange theory, where he argued that the same physiological change can cause different emotions, e.g. crying can lead to sadness emotion as well as to happiness; hence he wanted to show that James-Lange theory is unlikely.

The baseline for the Schachter-Singer two-factor theory is that mental processes such as information processing and the creation of thoughts are the main reason for the formation of emotions. It indicates that emotion is a result of the perceptual processing of emotional stimulus and physiological changes~\cite{Schachter1962}. It explains that when the stimulus comes, the physiological change appears, but only after the cognitive process of mentioned stimulus together with physiological change, the man starts to perceive emotion. 

\bigskip

This thesis aims to propose the SER system that recognizes emotions from a speech in a way that approaches the abilities of humans, and to do so there is a need to study how to categorize emotions like them as the fundamental brick for the following work. Two types of emotional models are interesting from the SER point of view: \textbf{discrete (categorical)} model and \textbf{dimensional (continuous)} model~\cite{Akcay-SER-review}. The following subsections are designated to these models with their definitions and descriptions of advantages and disadvantages.

% we are going to examine both of them and describe their advantages and disadvantages.



% Scientist described two different approach how humans perceive emotions. The first is discrete or categorical way and the second is dimensional or continuous way~\cite{emotion-models-review}. Now, we are going to examine them and describe their advantages and disadvantages. 

% In our work we are going to aim to emotion recognition and for this purpose we need to have proper categorization of emotions as fundamental element, from which we can further built on. Scientist have created framework called \textit{emotion model} that suits for our case. 
% In general, we recognize two emotion models. The first is discrete model and the second is dimensional model~\cite{emotion-models-review}.

\subsection{Discrete model}
The discrete model divides emotions into disjunct categories. Ekman and Oster proposed one of the most popular types of discrete models when they studied the expressions of different cultures. They concluded that there are six emotion categories: \textit{anger}, \textit{fear}, \textit{disgust}, \textit{happiness}, \textit{surprise}, \textit{sadness}~\cite{Ekman-Oster-facial-expressions}. Later, Ekman observed that the proposed categorization could be proved not only by visual factors but also by observing physiological changes when a man perceives emotion~\cite{Ekman-argument-for-basic-emotions}.

The study by Shaver et al. used a prototype-based approach, where they grouped 135 expressive words into prototypes or groups, where one word was characteristic. With this methodology, they found these emotional groups: \textit{joy}, \textit{sadness}, \textit{fear}, \textit{anger}, \textit{love} and \textit{surprise}~\cite{Shaver1987}.

% Another discrete model was proposed by Robert Plutchik. This model is called emotional wheel and it is depicted in the figure~\ref{fig:plutchik-emotional-wheel}. This model represents eight basic emotions joined to pairs, in which each pair contains mutually opposite positive and negative emotion, e.g joy and sadness, fear and anger etc. From figure~\ref{fig:plutchik-emotional-wheel} is also obvious that emotional wheel considers compound emotions like optimism or love, which consist of basic emotions.
Another discrete model was proposed by Robert Plutchik. This model is called the \textbf{emotional wheel} and is depicted in Figure~\ref{fig:plutchik-emotional-wheel}. This model represents eight basic emotions joined in pairs, in which each pair contains mutually opposite positive and negative emotions e.g. joy and sadness, fear and anger etc. It is also apparent from figure \ref{fig:plutchik-emotional-wheel} that the emotional wheel considers compound emotions such as optimism or love, which are made up of basic emotions.

\begin{figure}[h!]
    \centering
    \includegraphics[width=0.6\textwidth]{obrazky-figures/ser_description/Plutchik-wheel.pdf}
    \captionsetup{justification=centering}
    \caption[Plutchik's emotional wheel.]{Plutchik's emotional wheel~\cite{Plutchik2000}. It contains eight atomic emotions grouped in the pair, where each contains mutually exclusive emotions, e.g. joy and sadness. The second-level or compound emotions are made of basic emotions, e.g. submission or disapproval.}
    \label{fig:plutchik-emotional-wheel}
\end{figure}

The advantage of a discrete model is its intuitiveness and comprehensibility because emotions are often described in this way. However, it does not have such expressive power to depict complex emotions which occur in everyday life from a scientific point of view.

In the SER, the discrete models have been predominant since the beginning of its research~\cite{Akcay-SER-review}, but it is not clear why researchers take into account this type of model in most approaches. In the author's opinion, one of the likely explanations is that most of the available databases for this area, examined in Section~\ref{sec:databases}, consider data labelled in a way that follows the discrete model pattern. However, many of them do not strictly follow some of the mentioned categorizations. Mostly, they can be regarded as derivatives of Ekman's model with subtle domain-dependent adjustments.



\subsection{Dimensional model}
For more precise emotion categorization researchers proposed the dimensional model represented as an \(n\)-dimensional continuous space, where each axis denotes some emotional characteristics. With this model, it is possible to express emotion as a point defined by its coordinates.

The most relevant dimensional models in the context of SER~\cite{Akcay-SER-review} are the Circumplex model by Russell, depicted in Figure~\ref{fig:circumplex-model}~\cite{circumplex-model-Russell} that contains the horizontal axis representing the valence domain and on the vertical axis is the arousal domain. Around these axes are organized particular emotion states in a circular shape expressed as a ratio of valence and arousal.

Another famous model is Mehrabian's Pleasure-Arousal-Dominance (PAD) three-dimensional model. Each of these terms represents one axis, as is shown in Figure~\ref{fig:pad-model}~\cite{basic-dimension-for-psychological-theory-Mehrabian}. This model is comparable with Russell's Circumplex model because the only difference between them is that the PAD model has an extra dominance axis. It represents a level of the subject's control above the current state.

The advantage of the dimensional model is that it enables the expression of emotional state as a point in continuous \(n\)-dimensional space. It implies the ability to compute the exact similarity degree between emotions. The disadvantage of this model is its non-intuitive approach to emotions and the fact that some emotions cannot be precisely expressed, e.g. \textit{surprise} in two-dimensional valence-arousal space, because its valence can be either positive or negative. Bakker et al.~\cite{Bakker2014-PAD-revisited} consider as the disadvantage of the dimensional model that there is no unified axis labelling, and hence it is not clarified the interpretation of them. In other words, there is no standardized measurement unit for the dimensions.

According to the reviewed literature~\cite{Akcay-SER-review}, the most commonly used dimensional model is the three-dimensional PAD model.

\begin{figure}[h!]
    \centering
    \begin{subfigure}[b]{0.47\textwidth}
        \includegraphics[width=\textwidth]{obrazky-figures/ser_description/circumplex_model.pdf}
        \caption{Circumplex model~\cite{circumplex-model-Russell}.}
        \label{fig:circumplex-model}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.47\textwidth}
        \includegraphics[width=\textwidth]{obrazky-figures/ser_description/PAD_model.pdf}
        \caption{PAD model~\cite{Bakker2014-PAD-revisited}.}
        \label{fig:pad-model}
    \end{subfigure}
    \caption[Dimensional models.]{Dimensional models.}
\end{figure}

\bigskip
% \todo{ADD COMPARISON TABLE OF EMOTIONAL MODEL}
% \todo{Related works in SER - which emotion model have been chosen}\newline
\begin{table}[H]
\begin{tabularx}{\textwidth} { 
  | >{\centering\arraybackslash}X
  | >{\centering\arraybackslash}X 
  | >{\centering\arraybackslash}X 
  | >{\centering\arraybackslash}X | }
    
\hline
     \textit{Model type} & \textit{Emotions interpretability} & \textit{Emotions comparability} & \textit{Datasets availability} \\ \hline
     Discrete model & similar to human emotions perception & not present any exact metrics & many available databases (freely available, on demand) \\ \hline
     Dimensional model & ratings in given number of dimensions, less intuitive & cosine similarity, Euclidean distance & a few available datasets \\ \hline 
\end{tabularx}
\caption{Emotional models comparison.}
\end{table}


\section{Databases}
\label{sec:databases}
Databases in the context of the SER serve for obtaining data that are necessary for further building a classification model. Database quality is crucial because it directly influences the system's performance. If the database contains low-quality, incomplete or data that are damaged in some way, the results of the system will not be satisfactory, so the choice of the database has to be performed carefully.

The categorization of the databases from the SER perspective includes three types: \textbf{natural speech (spontaneous)}, \textbf{acted}, and \textbf{elicited}~\cite{comprehensive-review-of-SER}. Databases from the spontaneous category contain voice recordings from a natural environment, e.g. dialogues from talk shows or podcasts. This database type is the most valuable because it comprehends emotional expressions from real-life situations, which are hard to reproduce artificially. However, due to privacy laws, it is difficult to build and access these databases.

The acted databases contain recordings produced by professional actors in recording studios, which implies the high quality of such voice recordings. Their advantage is that they are relatively easy to obtain, and most are even free. On the other hand, the expressed emotions may be exaggerated, possibly leading to degradation of recognition performance on real-life recordings~\cite{Akcay-SER-review}.

The elicited databases have voice recordings created in a simulated emotional environment~\cite{comprehensive-review-of-SER}. A speaker is placed into a situation that stimulates him in order to evoke a particular emotion. After this procedure, his speech is recorded.

Most of the available databases are from the category acted and are labeled with discrete emotional categories. Currently, according to the number of citations from the scientific portal Scopus\footnote{\url{https://www.scopus.com/}}, the most popular databases used for SER research are \textbf{Berlin Database of Emotion speech (Emo-DB)}~\cite{berlin-database-of-emotional-speech}, \textbf{The Interactive Emotional Dyadic Motion Capture (IEMOCAP)}~\cite{IEMOCAP} and \textbf{The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDEES)}~\cite{RAVDESS}. We have provided additional information about mentioned databases in the table~\ref{tab:databases}.

\begin{table}[H]
\begin{tabular}{|l|m{2cm}|m{2cm}|m{4cm}|m{2cm}|}
\hline
\textit{Name} & \multicolumn{1}{c|}{\textit{Type}} & \multicolumn{1}{c|}{\textit{Language}} & \multicolumn{1}{c|}{\textit{Emotions}} & \multicolumn{1}{c|}{\textit{Citations count}} \\ \hline
Emo-DB~\cite{berlin-database-of-emotional-speech}  & \multicolumn{1}{c|}{Acted} & \multicolumn{1}{c|}{German}  & Neutral, anger, sadness, fear, boredom, happiness, disgust. & \multicolumn{1}{c|}{1237~\cite{EMODB-citations-cnt}} \\ \hline
IEMOCAP~\cite{IEMOCAP} & \multicolumn{1}{c|}{Elicited}                                    & \multicolumn{1}{c|}{English} & Anger, happiness, sadness, frustration, neutral. & \multicolumn{1}{c|}{1421~\cite{IEMOCAP-citations-cnt}} \\ \hline
RAVDESS~\cite{RAVDESS} & \multicolumn{1}{c|}{Acted} & \multicolumn{1}{c|}{English} & Surprise, anger, fear, disgust, sadness, neutral, calm, happiness. &   \multicolumn{1}{c|}{558~\cite{RAVDESS-citations-cnt}}       \\ \hline
\end{tabular}
\caption{Brief description of databases for SER~\cite{comprehensive-review-of-SER}.}
\label{tab:databases}
\end{table}




\section{Speech and sound analysis}
\label{sec:speech-and-sound-analysis}

A speech is a voice communication method based on the rules of the chosen language. Speech production is a complex process that starts in the human brain, which sends neural signals transmitted into a series of physiological changes in the human body. The vocal cords placed in the larynx have the principal function in speech production. These instruments change the airflow from the lungs out of the human body in a way that results in speech signals propagating through an environment.

The following subsections are designated to the basic speech properties that characterize its perception, the speech processing, the useful speech features in the context of the SER and how to extract them. The end of the section belongs to classification methods and deep learning. The author emphasizes that the focus is given to offline methods that relate to SER systems retrieving recordings from databases, not from a real-time environment.

\subsection{Speech acoustic properties}
\label{subsec:speech-properties}

The acoustic properties of the speech signal influence its information value. The first examined property is \textbf{the fundamental frequency of speech} (\(F_{0}\)), which is the lowest frequency component of a complex sound wave or in other words, the first peak of the signal's power spectrum. In the case of speech, it is the frequency of oscillating vocal cords (number of times that vocal cords open and close per second)~\cite{intro-to-speech-processing-2022}. People perceive this frequency as a pitch of the voice and it is the main property for recognition of voiced and unvoiced parts of the speech.

The following inspected property is \textbf{the resonance of the vocal tract}. It describes how well is the voice from vocal cords amplified in the cavities above them. The resonance structure is described by \textit{formants}. The most discussed formants are \(F_{1}\) and \(F_{2}\), also called the lowest resonance formants~\cite{intro-to-speech-processing-2022}. The whole resonance structure is obtained from the envelope of the speech power spectrum, as illustrated in Figure~\ref{fig:speech-spectrum}. In this figure, the blue curve represents the magnitude of frequencies, which is possible to observe in a speech wave and the red curve represents its spectral envelope. In this case, the fundamental frequency \(F_{0}\) of this signal is around 150 Hz with 17 decibels intensity, formant \(F_{1}\) has the frequency 700 Hz with intensity near 37 decibels and formant \(F_{2}\) has the approximate frequency of 1200 Hz and intensity almost 30 decibels.

\begin{figure}[h!]
    \centering
    \includegraphics[width=\textwidth]{obrazky-figures/ser_description/spectral_envelope.png}
    \caption[Audio signal power spectrum and its spectral envelope.]{Audio signal power spectrum and its spectral envelope~\cite{intro-to-speech-processing-2022}.}
    \label{fig:speech-spectrum}
\end{figure}

The last considered property is \textbf{the audio signal amplitude (intensity)}. It shows the deviation in air pressure in time (during the flow of the speech). The most common way to plot this deviation is a waveform, whose examples are shown in Figure~\ref{fig:waveform-examples}. Subfigure~\ref{subfig:waveform} shows a waveform of the whole recording, but due to the dense amplitude array, the peaks are not well recognizable and thus we provide a detailed portion of the waveform that is depicted in Subfigure~\ref{subfig:waveform-portion}.

\begin{figure}[h!]
    \centering
    \begin{subfigure}[b]{0.495\textwidth}
        \includegraphics[width=\textwidth]{obrazky-figures/ser_description/complex_waveform.pdf}
        \caption{Waveform of whole speech recording.}
        \label{subfig:waveform}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.495\textwidth}
        \includegraphics[width=\textwidth]{obrazky-figures/ser_description/detailed_waveform.pdf}
        \captionsetup{justification=centering}
        \caption{Waveform portion. It is zoomed version of the original waveform.}
        \label{subfig:waveform-portion}
    \end{subfigure}
    \caption[Waveform examples.]{Waveform examples.}
    \label{fig:waveform-examples}
\end{figure}


\subsection{Influence of emotions on the speech properties}
\label{subsec:influence-of-emotions-on-the-speech-properties}

This subsection inspects the impact of emotions on particular speech properties. Nwe et al. compiled a complex overview of this topic and described how emotions such as \textit{anger}, \textit{joy}, \textit{fear}, \textit{sadness}, \textit{surprise} and \textit{disgust} influence speech properties~\cite{NWE-SER-using-HMM}. The following paragraphs provide a summary of this overview. All comparisons are relative to speech considered neutral (without any emotions).

For \textit{anger}, the overview shows that the average pitch is higher and its range is wider. The intensity of the voice is raised, and the energy of the unvoiced speech part is higher.

The emotion of \textit{joy} has similar parameters as anger; hence, it has a higher pitch with a wider frequency range and increased intensity however, in terms of the spectral properties, the energy is distributed into higher frequencies.

In the case of \textit{fear}, the average pitch value with its range are higher, but the intensity of the speech is almost the same as in the neutral state. In terms of the speech spectrum, more energy is present in higher frequencies.

\textit{Sadness} is the opposite case of the aforementioned emotions. The average pitch is lower and the pitch range is narrower. The intensity is significantly decreased and there are often present downward inflexions.

\textit{Surpise} pitch has a wide range and its mean value is neither above nor below the normal state. It is caused by the character of emotion that allows expressing the positive, as well as negative emotions.

The last emotion discussed in this overview is \textit{disgust}. It has a pitch a lot below the average of neutral, but the frequency band is slightly wider. The intensity of speech during the disgust emotion is rather low.

\subsection{Speech preprocessing}
\label{subsec:speech-preprocessing}
Preprocessing includes methods that manipulate audio signals in a way that makes them more suitable for extraction and selection of the features. This subsection examines the preprocessing techniques relevant to the thesis topic.

The first examined method is called \textbf{framing} or \textbf{segmentation}. It divides the speech signal into smaller chunks (frames or segments) with lengths from 20 to 30 milliseconds. Framing is used because the expression of emotions tends to vary during the flow of the speech, which causes the audio signal to be non-stationary, but for this short time period is invariant and thus suitable for examining semi-fixed and local features~\cite{comprehensive-review-of-SER}. This method allows Discrete Fourier Transform (DFT) to be performed and is also convenient for use with some classifiers~\cite{Akcay-SER-review}.

Another processing method related to the above framing is \textbf{windowing}. It multiplies the speech signal with a window function (e.g. Hanning window, Hamming window), which preserves values in the middle of the signal and smooths values at both ends removing discontinuities between neighbouring frames~\cite{comprehensive-review-of-SER}. The main reason for applying windowing is the preparation of frames for transformation from the time to the frequency domain. The negative aspect of windowing is information loss but it is possible to remove this drawback by setting sufficient overlay between frames during signal split.

Subfigure~\ref{subfig:windowing-functions} shows how Hamming (blue curve) and Hanning (orange curve) windows look like for frames with a length of 128 amplitude values (samples). Both functions have a bell shape, and their peak is in the middle of the horizontal axis. From this subfigure follows that the biggest difference between these windowing functions is at both ends. Next, Subfigure~\ref{subfig:windowing-demonstration} depicts the signal before (blue curve) and signal after (orange curve) windowing with Hamming window. The key property is saving the middle values of signals as much unchanged as possible, while completely eliminating marginal values that could cause discontinuities.


\begin{figure}[h!]
    \centering
    \begin{subfigure}[b]{0.495\textwidth}
        \includegraphics[width=\textwidth]{obrazky-figures/ser_description/Hamming_hanning_widows.pdf}
        \caption{Hamming and Hanning windows.}
        \label{subfig:windowing-functions}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.495\textwidth}
        \includegraphics[width=\textwidth]{obrazky-figures/ser_description/Windowing_demonstrations.pdf}
        \caption{Demonstration of windowing effect.}
        \label{subfig:windowing-demonstration}
    \end{subfigure}
    \caption[Windowing examples.]{Windowing examples.}
    \label{fig:windowing}
\end{figure}

Another practical method is \textbf{normalization}, which reduces the influence of speaker and recording variability in audio signal values without loss of their discriminatory properties~\cite{Akcay-SER-review}. It is performed either at the frame or recording level. The most widely used normalization method is Z-normalization, which we compute for each signal value in sequence \textit{x} according to equation~\ref{eq:z-normalization}, where \(\mu\) is the mean and \(\sigma\) is  of the audio signal.

\begin{equation}
    z = \frac{x-\mu}{\sigma}
    \label{eq:z-normalization}
\end{equation}

The last examined preprocessing technique is \textbf{voice activity detection (speech detection)}, which is useful for detecting if there is present some speech activity (either voiced or unvoiced) or if it is a silent part of the recording~\cite{Akcay-SER-review}. The purpose of this method is to eliminate speech frames that contain silent parts, which increase computational complexity and decrease the performance of the SER system. The most used algorithms for implementation of voice activity detection are \textbf{zero-crossing rate} and \textbf{auto-correlation method}. 



\section{Speech features}
\label{subsec:features}
Speech features are a set of values that describe important speech properties. They are one of the key elements that have a big impact on the performance of the SER systems; hence, it is necessary to take care of their proper selection process. Although the SER has been a research-interesting topic for a long time, there is still no consensus about the universal set of features that we can reliably use in our SER system without consideration of their pros and cons~\cite{Akcay-SER-review}.


Currently, a lot of features from different categories were utilized during the SER research. This section studies the most relevant features according to the reviewed literature. The first and most simple categorization is differencing between \textbf{segmental (local)} and \textbf{supra-segmental (global)} features~\cite{comprehensive-review-of-SER}. Supra-segmental features are obtained from the whole speech signal (e.g. minimum, maximum, variance), while segmental are acquired from the individual frames of the partitioned recording. These frames contain the speech temporal dynamics that are important characteristics of speech features. Their study is significant because the expression of emotions is not uniform during the speech mentioned also in subsection~\ref{subsec:speech-preprocessing}.

According to the literature review for this topic, the following speech features categories are considered: \textbf{prosodic features}, \textbf{spectral features}, \textbf{voice quality features} and \textbf{teager energy operator (TEO) based features}~\cite{comprehensive-review-of-SER}, which are discussed in the following subsections.

\subsection{Prosodic features}
This feature type is connected to the prosodic (para-linguistic) speech properties, which man can perceive as intonation, tempo or rhythm. These properties are applied during the speech to units like syllables, words or sentences. 

Speech phenomena suitable for prosodic feature gain are the fundamental frequency \(F_{0}\) (defined in subsection ~\ref{subsec:speech-properties}), the energy of the speech (directly connected to the speech signal amplitude) and the speech duration. 
These features are called time-based and supra-segmental because they are related to the change of speech in time and are examined in the bigger portions of the recording, respectively.   

Prosodic features are the results of the statistical methods such as mean, maximum, minimum and variance applied to the measured values of the above phenomena~\cite{Akcay-SER-review}. The performance level of the SER depends on the level which is for features computation (word level, sentence level, recording level).

\subsection{Spectral features}
On the opposite of the prosodic features, the spectral features are the result of a short-time analysis of the speech signal; hence, extracted features represent local changes in the speech, which are significant for emotion recognition. From the high overview, the spectral features represent the spectral energy distribution across different frequencies. This energy distribution is implied by the shape of the vocal tract~\cite{survey-on-speech-emotion-recognition}.

To obtain spectral features, the Fourier transformation is performed on each frame of the speech signal, which results in frequency domain representation of the signal. It is possible to investigate spectral features after this kind of transformation or with further operations gain cepstral features.~\cite{Swain2018}.

According to reviewed literature~\cite{Swain2018, Akcay-SER-review, comprehensive-review-of-SER, survey-on-speech-emotion-recognition}, the most used spectral features are \textbf{Formants} (described in \ref{subsec:speech-properties}), \textbf{Mel Frequency Cepstral Coefficients (MFCC)}, \textbf{Gammatone Frequency Cepstral Coefficients (GFCC)}, \textbf{Log-Frequency Power Coefficients (LFPC)} and \textbf{Linear Prediction Cepstral Coefficients (LPCC)}.

MFCCs represent the short-term power spectrum of the speech signal expressed in the mel scale. They are defined as the result of the Discrete cosine transformation computed from the log of the power spectrum expressed in the mel scale. The argument for MFCCs utilization in the SER is that they mirror how the human auditory system works. The transition is performed from hertz to mel scale with a set of filters from the mel filter bank, which example is depicted in Figure~\ref{fig:mel-filter-bank}. This bank contains six filters distinguished by colours.

\begin{figure}[h!]
    \centering
    \includegraphics[width=\textwidth]{obrazky-figures/ser_description/mel-filter-bank.pdf}
    \caption[Mel filter bank example.]{Mel filter bank example with 6 filters.}
    % https://siggigue.github.io/pyfilterbank/melbank.html
    \label{fig:mel-filter-bank}
\end{figure}

GFCCs are another cepstrum-based features obtained with an almost similar method as MFCCs but it is used the gamma filter bank, instead of the mel bank. This filter bank represents changes in the inner ear and external middle ear~\cite{gammatone-cepstral-coefficients}.

They are included in the family of cepstrum coefficients as well with the LFPCs, which reflect the frequency filtering property of the human auditory system expressed on the logarithmic scale. In the computational process is used a filter bank that contains log frequency bands~\cite{log-frequency-power-coefficients}.

LPCCs are Linear Prediction Coefficients (LPC) represented in cepstrum. The main idea behind LPC is a mathematical speech production model, where the vocal tract part is modelled with the all-pole filter with coefficients equal to LPC~\cite{Comparison-of-LPCC-and-MFCC}. The reason for the usage of LPCC is they are more robust than LPC.

\subsection{Voice quality features}
\label{subsec:voice-quality-features}

The physical state of the vocal tract (e.g. condition of vocal cords) highly influences this feature type. The defects in speech signal cause effects called \textbf{jitter}, \textbf{shimmer} and \textbf{harmonics to noise ratio (HNR)}.

Jitter and shimmer are related to fundamental frequency \(F_0\), and it is possible to describe them together. They are perceived as roughness, shortness of breath, or hoarseness in speech. In terms of the fundamental frequency, the jitter represents frequency instability (variation in vibrations of the vocal cords), and the shimmer is the amplitude instability~\cite{Akcay-SER-review}. Figure~\ref{fig:jitter-and-Shimmer} depicts the speech signal affected by these effects.

Harmonics to noise ratio feature describes a ratio between periodic and aperiodic in the speech signal or also as a noise that is present in a voice, particularly in the vowels.

\begin{figure}[h!]
    \centering
    \includegraphics[width=\textwidth]{obrazky-figures/ser_description/jitter_shimmer.png}
    \caption[Jitter and shimmer example.]{Jitter and shimmer example~\cite{intro-to-speech-processing-2022}.}
    \label{fig:jitter-and-Shimmer}
\end{figure}

\subsection{Teager energy operator based features}
\label{subsec:teager-energy-operator-features}

This feature category is based on \textbf{Teager energy operator} proposed by Teager and Teager~\cite{Teager1990}. The foundation of their research was that the speech production process is the vortex flow changed by the current shape of the vocal tract. The airflow is responsible for the energy transmission from speakers to listeners. Non-linear Equation~\ref{eq:teager-energy-operator} describes the airflow change during the speech, where \(\Psi\) is the sign of the TEO operator and \(x(n)\) is the sampled speech signal. Kaiser et al. utilized this non-linear property and proposed a set of features based on TEO suitable for emotion recognition, particularly for stress recognition~\cite{kaiser-TEO-features}. This features set consists of \textbf{TEO-decomposed FM variation (TEO-FM-Var)}, \textbf{normalized TEO autocorrelation envelope area (TEO-Auto-Env)} and \textbf{critical band based TEO autocorrelation envelope area (TEO-CB-Auto-Env)}.

\begin{equation}
    \Psi \left [{ {x\left ({n }\right)} }\right]=x^{2}\left ({n }\right)-x\left ({{n-1} }\right)x\left ({{n+1} }\right)
    \label{eq:teager-energy-operator}
\end{equation}



\section{Classification methods}
\label{sec:classification-methods}

The last component of the SER system is classification responsible for assigning an emotion label or continuous value(s) to input recording. Since the start of the SER research, there were used, proposed and utilized different kinds of classifiers but none of them was sufficiently suitable and universal to cover all cases.
 

From a high-level overview of machine learning, classifiers need input in a structured form of the feature vectors, which consist of the values obtained from the feature selection and feature extraction. In the reviewed literature, SER research considers universal classifiers and there is no machine learning algorithm designed specifically for the needs of the SER~\cite{Akcay-SER-review, comprehensive-review-of-SER, Swain2018, Jahangir2021-ra-deep-learning-approaches}. The most used classifiers for solving the SER task are \textbf{Hidden Markov Model (HMM)}, \textbf{Gaussian Mixture Model (GMM)}, \textbf{Naive Bayesian Classifier}, \textbf{Support Vector Machine (SVM)} and \textbf{K-Nearest-Neighbours (KNN)}. The rest of this section contains a description of the machine learning algorithm.

Hidden Markov Model is a supervised classifier that was comprehensively utilized in speech recognition, and the researchers successfully used it in the SER context, too. HMM is a sequential model, which can be drawn as the set of states and edges between them~\cite{Akcay-SER-review}. The transition process between states is based on probability distribution; thus, the choice logic of the following state is hidden from the observer. The current state at time step \(t\) depends only on one at the previous time step \(t-1\).

Gaussian Mixture Model is a supervised generative model which consists of Gaussian components. Each component models the probabilistic distribution of one class, and they are described by their Gaussian parameters optimized during the fitting process. It is the version of HMM with only one state~\cite{survey-on-speech-emotion-recognition}. 

Naive Bayesian supervised classifier is based on the Bayes theorem~\cite{survey-on-speech-emotion-recognition}. It exploits the properties of the joint probability and assumes that each feature is independent of one another. Equation~\ref{eq:Bayes-formula} represents the Bayes formula, where \(c\) represents a class label and \(\textbf{x}\) is a feature vector.

\begin{equation}
    P(c|\textbf{x}) = \frac{P(\textbf{x}|c) p(c)}{p(\textbf{x})}
    \label{eq:Bayes-formula}
\end{equation}

Support Vector Machine is the supervised linear algorithm that looks for a separating hyperplane with the maximal margin among the classes. The hyperplane lies in the middle of this margin and the samples that lie at the margin borders are called support vectors.
If the problem could not be solved with a linear solution in the original representation, data are mapped to the new space with higher dimensionality where the algorithm can find the separating linear hyperplane.

K-Nearest-Neighbours is a parameterless supervised algorithm that assigns a class label to each sample, described by its feature vector \(\textbf{x}\),  according to a majority vote of its \(k\) neighbours~\cite{comprehensive-review-of-SER}.  The distance between samples represented as vectors are measured, for example, with Euclidean distance (Equation~\ref{subeq:euclidean-distance}) or Cosine similarity (Equation~\ref{subeq:cosine-similarity}), where \(\textbf{x}\) and \(\textbf{y}\) are vectors in  \(n\)-dimensional space and \(\|\textbf{x}\|, \|\textbf{y}\|\) are their norms.

\begin{subequations}
  \begin{equation}
      d(\textbf{x}, \textbf{y}) = \sqrt{\sum_{i=0}^{n-1}(x_{i} - y_{i})^2}
      \label{subeq:euclidean-distance}
  \end{equation}  

  \begin{equation}
      \cos{\theta} = \frac{\textbf{x}\cdot\textbf{y}}{\|\textbf{x}\| \|\textbf{y}\|}
      \label{subeq:cosine-similarity}
  \end{equation}
\end{subequations}


\section{Deep learning}
\label{subsec:deep-learning-approach}
The deep learning area is a relatively new approach but gradually begins to be ubiquitous. It is a subfield of machine learning based on the concept of the \textbf{artificial neural network (ANN)}, which tries to imitate human brain functionality~\cite{Jahangir2021-ra-deep-learning-approaches}.

The deep learning power lies in the automatic extraction of features. While traditional machine learning algorithms, described in Section~\ref{sec:classification-methods}, need a set of handcrafted features arranged in the appropriate structure, the deep learning models can process raw unstructured data and a feature extraction process taking place during the computation without the need of human intervention. However, in the SER context, it is a common approach to combine feature extraction methods together with deep learning to obtain better SER system performance.

This section provides a concise description of the models that are according to reviewed literature most popular: \textbf{Convolutional Neural Network (CNN)}, \textbf{Recurrent Neural Network (RNN)}, \textbf{Restricted Boltzmann Machine (RBM)}, \textbf{Deep Belief Network (DBN)}, \textbf{Deep Boltzmann Machine (DBM)}, \textbf{Autoencoders (AE)} and \textbf{Attention Mechanism}~\cite{Khalil-2019-deep-learning-techniques-8805181, Jahangir2021-ra-deep-learning-approaches, comprehensive-review-of-SER, Akcay-SER-review, Lieskovská-deep-learning-techniques-attention-2021}. A model description includes the model type according to machine learning taxonomy (generative/discriminative, supervised/unsupervised), the main idea of the model and its basic structure.   

\subsubsection*{Convolutional Neural Networks}
CNN is a discriminative supervised model, whose main domain is the processing of the raw input data e.g. images and audio signals.~\cite{Khalil-2019-deep-learning-techniques-8805181}. CNN is built on a \textit{feed-forward deep neural network architecture} that uses convolutional layers for extracting features from the input. These layers consist of filters that search for patterns according to their setting. To create a meaningful, but computationally efficient feature set, the downsampling method has to be performed in order to reduce the number of extracted features and at the same time to preserve discriminative information with a reasonable trade-off ratio. CNN uses for downsampling the pooling layers, where this trade-off is ensured. Usually, the last part of the CNN is made of dense layers dedicated to a classification of extracted features.


\subsubsection*{Recurrent Neural Networks}
The RNN is a supervised discriminative model built on the \textit{recurrent architecture}. Its purpose is to process sequential or time series data when there is a need to focus on the relationships and dependencies between them. The RNN mechanism works in a sequential fashion, which means that at each time step is processed one part of an input sequence. The output is computed as a product of the input vector and current context that is updated for further steps. The RNN structure is based on the recurrent unit, which computes output in the current time step and updates context information according to input. The improved versions of recurrent units used in the RNN are \textbf{Long Short-Term Memory (LSTM)} and \textbf{Gated Recurrent Unit (GRU)}. These methods improve the concept of RNNs and provide better performance and a more stable training process~\cite{Jahangir2021-ra-deep-learning-approaches}. 

\subsubsection*{Restricted Boltzmann Machine}
RBM belongs to the group of unsupervised models based on the generative approach. The main idea of the RBM is to model probability distribution over the neurons and by adjusting weights between them to learn the parameters of the distribution. RBM has two-layer architecture, where the input layer is called \textit{visible} and the output layer is called \textit{hidden}. The structure is represented as an undirected bipartite graph.

\subsubsection*{Deep Belief Network, Deep Boltzmann Machine}
DBN and DBM models are built upon the idea of RBM. The DBN is deep learning architecture consisting of a cascade of RBMs, where the first two layers are connected with the undirected edges and the rest with directed edges in one way~\cite{Khalil-2019-deep-learning-techniques-8805181}. On the other hand, DBM is also made of RBMs cascade, but in this case, they are connected with undirected edges. DBM has one visible layer.

\subsubsection*{Autoencoder}
Autoencoder is an unsupervised generative model whose architecture consists of two parts, \textit{encoder} and \textit{decoder}. The encoder realizes function \(f_{\theta}\), which encodes the input vector to a hidden structure or latent space of the model, and the decoder part represents function \(g_{\theta}\), which has the goal to reconstruct an original input from latent space~\cite{Jahangir2021-ra-deep-learning-approaches}. The main aim of the AE is to compress input features with the encoder to as low dimensions as possible while preserving the reconstruction error of the decoder between its output and original input at a reasonable rate. AE is useful for feature extraction and dimensionality reduction because of its ability to compress input data with low information loss. 

\subsubsection*{Attention Mechanism}
The last described deep learning method is the Attention Mechanism. The idea behind this technique is the ability to look for the salient parts of the input data based only on the input data relationships. The model evaluates each input part with a weight according to its significance which is determined either by position in the input or by the input context. The Attention Mechanism was originally incorporated in the sequence-to-sequence type of RNN, where it helps to mark significant input parts in a given context~\cite{Lieskovská-deep-learning-techniques-attention-2021}. Since the first proposal of the Attention Mechanism, researchers created various modifications that improved its original concept and enabled the emergence of other deep learning architectures, for example, the Transformer architecture~\cite{transformers-NIPS2017_3f5ee243}.

\section{Current SER state-of-the-art}
\label{sec:current-state-of-the-art}

Although the SER research is extensive for the past 20 years, there are still a lot of unresolved issues that even the current proposed solutions do not solve at a satisfactory rate~\cite{comprehensive-review-of-SER}. This section analyzes the contemporary SER systems from the author's point of view; hence it focuses on the selected emotional database or corpus, the feature extraction and selection process and the classification model. Besides that, there is a discussion of the results of each system with the metrics used for the evaluation. Each system is reviewed with a critical debate to highlight its benefits and drawbacks.
The final part of this section is dedicated to the challenges mentioned in reviewed literature and also the implications of described SER systems. This part serves as a guideline that should be followed by the SER system proposed in this work.

\subsection{SER systems}
\label{subsec:SER-systems}

This subsection comprehends recent methods that perform at the state-of-the-art level. There is a description of the overall system structure, what are system inputs (features), what is the basic working mechanism and what are the results with the provided evaluation methodology for each of the mentioned systems. After this description, the thesis author expresses his opinion if the results of the works fulfilled their objectives. He states what is missing, what is insufficient and how to make it better from our point of view as well.

% Since there are a lot of suggested solutions, we will focus on the most recent ones with regard  the current trends in the SER field. 

% \todo{ADD METHODOLOGY OF SYSTEMS SELECTION}
\subsubsection*{Convolution-recurrent neural network}
\label{MFCCs-GFCCs}
The first system with state-of-the-art results by Kumaran et al.~\cite{Kumaran2021-C-RNN} introduced a fusion of 40 MFFCs 39 with GFCCs that were fed to the neural network. The architecture of this network combines the advantages of the convolutional layers extracting salient input regions together with the LSTM units for modelling long-term relationships. The authors of this paper chose the RAVDESS dataset for training and testing purposes. In the evaluation, the authors provided total accuracy followed by a confusion matrix with precision, recall and F1 score values for each considered class. The accuracy of the system for the test data is above 70~\%. The rest of the provided evaluation data implies that there is a high probability that the model was sensitive to \textit{surprise} emotion and, on the other hand, the worst resolution shown for \textit{fear} emotion. In the evaluation strategy, the thesis author lacks the comparison with other models, the implementation details of the model and the critical opinion of the proposed architecture from the authors of the paper.

From the author's point of view, the main acquisition of this work is the fusion of MFCCs and GFCCs that provided innovative ideas in the feature extraction step. On the other hand, there is no evidence in the evaluation part that exactly confirms that feature extraction is a key element of successful results. The author considers the disadvantage of this model the application of pure LSTM without any type of Attention mechanism, which enables to process only fixed-size sequences and forces a training process to be performed in a sequential way that implies an inability to use parallel computations to boost the speed of training.

\subsubsection*{CNN-BiLSTM with Stacked Transformers Layer model}
\label{CNN-BiLSTM}
Xianfeng et al.~\cite{Xianfeng-CNN-BiLSTM-STL} introduced a SER system based on the CNN-BiLSTM and  Stacked Transformer Layers (STL). The former-mentioned part is responsible for contextual representation extraction, and the latter part enhances the BiLSTM context vector resulting in emotional embedding. The output of the STL is forwarded to the fully connected neural network that generates discrete emotion labels. The authors used as the preprocessing method length unification of each recording to 7.5 seconds with the following framing, where each frame has a length of 25 milliseconds with a hop size of 10 milliseconds. The feature extraction and selection process consists of 40 MFCCs. The authors used the IEMOCAP emotion corpus for the evaluation of the proposed architecture with WA and UA metrics. According to provided results, the \textit{weighted accuracy (WA)} score was 91.28~\%, which is a 15.28~\% improvement, and the \textit{unweighted accuracy (UA)} score was 92~\% with almost 20~\% improvement compared with the second-best listed models. They also provided the confusion matrix, which implies a problem with a classification of the \textit{neutral} emotion. For comparison, the authors provided WA for an ablated architecture without the STL that was equal to 72.79~\% with which they demonstrated a significant impact of STL.

According to the author's opinion, the results of this work fulfilled stated expectations, and the application of the Transformer architecture played a key role in the proposed architecture. The evaluation process is comprehensive and well-structured. It provides a comparison of the proposed model results with previous state-of-the-art models, an examination of the number of the Transformers layers impact on the model performance, and an ablation study for the STL part together with how the model \textit{grow up} with the number its part in terms of parameters number. Despite all these facts, the evaluation lacks other metrics such as precision, recall and F1 score that would depict which emotions are easily recognizable for the model and in contrast which of them make difficulties. Another drawback of this paper is the usage of only one dataset when the improvement for the IEMOCAP corpus was significant. The enhancement of evaluation with more datasets would bring more insights.

% \hyperref[xiangfeng]{Xianfeng}

\subsubsection*{PCNSE-SADRN-CTC model}
\label{PCNSE-SADRN-CTC}
Zhao et al.~\cite{Zhao-2D-CNN-selfAM-CTC} proposed an architecture of a neural network combining parallel convolutional layers and attention mechanism together with a dilated approach to classify emotions in a discrete fashion. They called it \textbf{PCNSE-SADRN-CTC} where PCNSE stands for \textit{Parallel Convolutional Layers} with the squeeze-and-excitation technique, SADRN is an acronym for \textit{Self-Attention Dilated Residual Network} and CTC is \textit{Connectionist Temporal Classification}. The authors chose 40 MFCCs with delta-deltas values as input, which they obtained from recording partitioned to frames with a length of 25 milliseconds with a hop size of 10 milliseconds. According to the authors, the PCNSE block is responsible for extracting features from the input spectrogram, the SADRN block models relationships from extracted features and the CTC is used as a classifier. The model was trained and tested on datasets IEMOCAP and FAU Aibo Emotion corpus\footnote{\url{https://www5.cs.fau.de/en/our-team/steidl-stefan/fau-aibo-emotion-corpus/}} evaluated with WA and UA metrics. The results were compared against the architecture from previous research papers that authors used at that time state-of-the-art architectures, particularly convolutional bidirectional LSTM (CNN-BLSTM), convolutional GRU (CNN-GRU) and bidirectional LSTM with CTC loss (BiLSTM-CTC). For the IEMOCAP dataset, the proposed architecture achieved WA 73.1~\% and UA 66.3~\%, which is an increase of 1~\%, respectively 2~\% compared to the second-best model (CNN-GRU). The UA of the PCNSE-SADRN-CTC model on FAU Aibo Emotion corpus was 41.4~\%, which is 0.3~\% less than the best-listed model (BiLSTM-CTC).

The novel idea of this architecture is the utilization of CNN with CTC in the SER. Although the results of the proposed model are not better compared with other listed models, the model architecture lacks sequential processing, which means that inference and training time demands are lower positively impacting the power consumption for both operations. 

The evaluation part provides high-level information about the implemented model as a short table with all of the layer types with their parameters. The authors also include the results of the ablation study (e.g. \textit{PCN}, \textit{PCNSE}, \textit{PCNSE-SADRN}) to examine the effect on individual components of the final model. These results prove that the incorporation of each part was reasonable. Another part to emphasize is the usage of the FAU-Aibo speech corpus, which is considered a natural speech database and provides a new point of view on how well could nowadays models perform in the real world. Overall, the author would welcome a more comprehensive and complex evaluation with more metrics and figures about the training process to show how well the model converged under the provided settings.

\subsubsection*{TIM-Net}
\label{TIM-Net}
One of the most novel experimental approaches was proposed by Jiaxin et al.~\cite{Jiaxin-TIM-NN}, who created neural network architecture with the name \textbf{Temporal-aware Bi-direction Multi-scale Network (TIM-Net)}, to remove shortcomings in the long-range dependencies modelling and fixed temporal scale\footnote{The paper is still under review by the time of writing this thesis.}. Its main building block is \textit{Temporal-aware block}, which serves for feature extraction and modelling relationships between the input data. This block is a subnetwork, whose core element is the dilated causal convolutional layer modelling temporal-aware features, which are formed by temporal attention layers. The network contains an arbitrary number of these blocks. The term \textit{Bi-direction} refers to the fact that the network process input from both ends to integrate complementary information from \textit{past} and \textit{future} together. Hence, for each block that processes data in a forward manner exists block processing data in reverse order and their results are connected to produce one temporal-aware feature. The last expression, \textit{Multi-scale}, refers to the part of the network, where all temporal-aware features are fused as their linear combination. This network is designed to predict discrete emotion labels. 

The network input consists of the spectrogram from 39 MFFCs that are obtained from framed recording. Each frame has a length of 50 milliseconds with a shift size between frames of length 12.5 milliseconds. On each frame is applied Hamming window.

The evaluation methodology proposed by the authors involves performance testing on six emotional datasets (Emo-DB, IEMOCAP, RAVDESS, SAVEE, CASIA and EMOVO) where they monitored the weighted and unweighted accuracy. For each dataset, there is a table with mentioned metrics for four models including TIM-Net. For every dataset, the TIM-Net architecture outperforms others. According to the authors, the average gain measured for all datasets in the UA is 2.34~\% and in WA 2.64~\% compared to the second-best listed models. For the datasets CASIA and Emo-DB, the authors provided also the confusion matrix. As part of the evaluation, the model was tested in a cross-corpus manner, or in other words, the authors trained the model on one database and tested it on the other for demonstration of generalization ability. For this purpose, the authors utilized only five emotions (\textit{angry}, \textit{fear}, \textit{happy}, \textit{neutral} and \textit{sad}) that represent the label intersection of mentioned datasets. This model obtained the best results in used metrics (WA, UA) among all compared models. The improvement is around 2~\%.

The authors also conducted the ablation study, where they examined the impact of removing individual components on the watched metrics. The result of this study shows that all components contribute to overall model performance with the almost same value (around 3.5~\% for both metrics).

From the thesis author's point of view, the proposed SER system obtained the most promising results from the reviewed literature. Researchers evaluated the proposed architecture on six emotional databases, showing that the dynamic fusion of multi-scale features performs better than features with fixed scales. It is not clear how the authors were aiming to improve the modelling of long-range dependency since they did not define what consider long-range dependency. There are a few facts worth noting. First, the model testing was realized on more than two emotional databases followed by the cross-database multilingual test method used in this paper. The second is the ablation study, where the authors show the importance of each system component. On the other hand, there is a space for improvement in the selection of evaluation datasets by adding at least one based on natural speech.  

The unanswered question is the complexity of the proposed model in terms of parameter count together with its resource needs for training. This issue arises from the fact that the authors did not provide this kind of implementation detail.

\begin{table}[H]
\centering
\begin{tabular}{|>{\centering\arraybackslash}m{2cm}|>{\centering\arraybackslash}m{2cm}|>{\centering\arraybackslash}m{4cm}|>{\centering\arraybackslash}m{2cm}|>{\centering\arraybackslash}m{3cm}|} \hline
\textit{Model} & \textit{Database(s)} & \textit{Preprocessing} & \textit{Features} & \textit{Results} \\ \hline
C-RNN & RAVDESS & -  & 40 MFCCs, 39 GFCCs & 70~\%~/~- \\ \hline
CNN-BiLSTM & IEMOCAP & Framing with the frame length of 25 ms and hop length of 10 ms & 40 MFCCs. & 92~\%~/~91.28~\% \\ \hline
PCNSE & IEMOCAP, FAU-AEC & Framing with the frame length of 25 ms and hop length of 10 ms, Hamming window applied on each frame & 40 MFCCs enhanced by their delta-deltas. & 66.3~\%~/~ \mbox{73.1~\%} \mbox{for IEMOCAP}, 41.1~\%~/~- for FAU-AEC \\ \hline
TIM-Net & EmoDB, IEMOCAP, RAVDESS, SAVEE, CASIA, EMOVO & Framing with frame length of 50 ms and hop length of 12.5 ms, Hamming window applied on each frame. & 39 MFCCs &  \mbox{2.34~\%~/~2.36~\%} (avg.~improvement) \\ \hline
\end{tabular}
\captionsetup{justification=centering}
\caption{A brief overview of current state-of-the-art SER systems described in subsection~\ref{subsec:SER-systems}. The \textit{Results} column contains values in form of \textit{*unweighted accuracy*}~/~\textit{*weighted accuracy*}.
Because TIM-Net used 6 datasets, there is provided only average improvement from all results.}
\label{table:overview-of-SOTA-SER-systems}
\end{table}


\subsection{Challenges in the SER research}
\label{subsec:challanges-in-the-SER-research}
Although results from the current state-of-the-art SER systems presented in the previous subsection~\ref{subsec:SER-systems} are promising, there is still a lot of work to even approach the human emotion recognition abilities. First, it is discussed the long-term problems that are in SER from almost the beginning of its research, and then there is a focus on the issues that come with deep learning.

\subsubsection*{Size of datasets} 
The size of the current datasets for SER is significantly smaller than for other recognition tasks. For example, ImageNet\footnote{\url{https://www.image-net.org/}}, one of the most used datasets for image classification tasks, contains more than 14 million images. For comparison, the EmoDB dataset contains 550 recordings, and the RAVDESS dataset contains 1440 recordings. After deep learning entered the SER field, this problem became even more recognizable because deep learning models need more training data than traditional machine learning algorithms.

\subsubsection*{Datasets labelling} 
The next challenge related to the previous idea is the creation of new datasets for SER needs. The biggest problem according to the reviewed literature is the human recognition accuracy, which is below 90~\%~\cite{comprehensive-review-of-SER, Jahangir2021-ra-deep-learning-approaches, Akcay-SER-review}. This problem arises mainly from the subjective perception of emotions by humans, who labels data samples in the dataset.

\subsubsection*{Multilingual SER} 
Another unsolved task is overcoming differences in emotional expression in various languages or cultures. Each language has its own patterns in the expression of emotions. One of the discussed SER systems, TIM-Net, was evaluated on multiple databases that contained recordings in a different language, however, the performance was far below the results when only one language was considered.


\subsubsection*{Raw speech utterances as model input}
This topic considers the SER systems that do not include preprocessing stage and therefore make the whole process more straightforward. It is interesting mainly with deep learning algorithms because some are designed to process raw input data. Although in other speech or audio fields, such as speaker recognition or audio sound classification deep learning algorithms reach state-of-the-art results with raw data, this approach is not successful in the SER. 

Any of the listed state-of-the-art solutions did not consider raw speech as the input for the model. After specifically dedicated research to this topic separately, the following solutions were found. Latif et al.~\cite{direct-modelling-of-speech-emotion-from-raw-speech} implemented a deep learning model that processed raw audio speech. They evaluated implementation on the datasets IEMOCAP and MSP-Podcast~\cite{MSP-Podcast} with unweighted accuracy as the metric. The obtained results were 59~\% respectively 52~\%, which we consider as average compared to models from the previous section. Mustaqeem et al.~\cite{MLT-DNet} created a model that processed normalized speech signals. They used databases IEMOCAP and Emo-DB for benchmarking and obtained 73~\% and 92~\% of unweighted accuracy. Even though the results are promising, this research did not find any other recent similar solutions with such results.


\subsubsection*{Transfer learning} 
This challenge relates to the deep learning paradigm and it is a way how to \textit{transfer knowledge} from the previously trained large models to the contemporary ones~\cite{Jahangir2021-ra-deep-learning-approaches}. There are good examples of transfer learning in other fields, where deep learning is dominant right now, e.g. image or speech recognition, where big models were trained on a huge amount of data and their training took weeks or months and further they were \textit{fine-tuned} for more specific tasks. Again, this topic is also connected to the size of current datasets, which does not allow a pretraining general model for speech emotion recognition. 

\subsubsection*{Deep learning models size} 
This topic is pragmatic because most of the SER use cases include real-time processing on a device with limited resources. The deep learning models are usually large and need a lot of space and computation power to make an inference. Due to this fact, the SER models based on this paradigm should consider the ratio between model resources necessary for proper functioning and model performance.

\subsubsection*{Deep learning environmental impact} 
This deep learning issue arises from the introduction of larger and more complicated models. With the increasing size of the models, the requirements for the training resources grow as well, which has an ultimate impact on power consumption and the production of greenhouse gases. According to Strubell et al.~\cite{energy-and-policy-consideration-for-DL}, the complete training of the large natural language processing model produces more greenhouse gases than the average car during its lifetime. Lacoste et al.~\cite{quantifying-the-carbon-emissions-of-ml} proposed a method for calculating that takes into account the hardware used for training, the number of hours needed to perform training, the cloud provider and world region, where is the training device. Anthony et al.~\cite{Carbontracker} created the tool with the name \textit{Carbontracker}, especially for deep learning purposes to track and predict the carbon footprint of the models.


\section{Chapter summary and its implications}
\label{sec:summary}

This chapter extensively analyzed all the necessary components to build a SER system. It included the emotion theory necessary for a complete grasp of the field followed by the taxonomy of emotions. Further sections were dedicated to speech signal overview, methods suitable for its preprocessing and features utilized for SER purposes. Then, the focus was redirected to machine learning and deep learning algorithms used in SER. Each algorithm was described to show its principal functionality.

After the overview of SER components, the chapter continued with the section about current state-of-the-art systems from this field to review the methodology of how are they designed, implemented and evaluated. For each system, the author provided his opinion, highlighted innovative approaches and ideas and mentioned drawbacks, which he considered as 
resolvable. This review implies current trends connect cepstral features with the chosen deep learning algorithm. The very popular system that combined convolutional and recurrent neural networks, however, there was also mentioned cases with the Attention mechanism and the Transformer neural network. The trend in the evaluation of SER systems is to choose one or two emotional corpora, perform the model fitting process and provide weighted or unweighted accuracy, sometimes enhanced with the confusion matrix.

The last part of this chapter stated a list of the contemporary challenges in SER. It indicates that this topic is still not entirely solved despite progress in the fields of psychology, signal processing and machine learning; hence there are unresolved problems and untested technologies that need future investigation. The next part of this thesis tries to target some of the mentioned challenges and make a contribution to their solution.
  

