% This file should be replaced with your file with thesis content.
%=========================================================================
% Authors: Michal Bidlo, Bohuslav Křena, Jaroslav Dytrych, Petr Veigend and Adam Herout 2019

% For compilation piecewise (see projekt.tex), it is necessary to uncomment it and change
% \documentclass[../projekt.tex]{subfiles}

\chapter{Introduction}

Since the beginning of humanity's written history people were separated by the so-called curse of Babel. People of different cultures and ethnics group speaking different languages were incapable of communicating with each other on a somewhat decent level without a translator. With the invention of computing machines came the idea of delegating this work, or at least part of it, to machines. So Machine translation was born.

The quality of translation produced by MT systems was substantially improving over the last decades. Firstly, with the introduction of Statistical machine translation systems \cite{brown1990statistical} which, unlike the preceding rule-based approach, were able to generate translations based on statistical models whose parameters are derived from the analysis of bilingual text corpora. The second improvement came with the introduction of end-to-end neural encoder-decoder MT systems in 2013, which marked the dawn of neural machine translation (NMT).

Modern-day NMT systems show a translation quality that is comparable to that provided by professional translators. But, to achieve such results NMT system should be trained on huge parallel corpora (also known as bitexts). Not meeting this condition results in subpar performance, comparable to, or even inferior to, the performance of SMT systems.

Data augmentation is one of the most commonly employed methods for enhancing the performance of MT systems. Augmentation techniques such as Back-translation \cite{sennrich2016improving}, and synthetic data generation have been shown to effectively increase the amount and diversity of training data, leading to improvements in translation quality. In addition to data augmentation, other strategies such as the fine-tuning of pre-trained language models, and incorporating domain-specific knowledge \cite{moslem-etal-2022-domain} have also been explored to enhance the performance of MT systems.

The novel NMT approach -- Bilingual dictionary based neural machine translation (BDBNMT) \cite{duan-etal-2020-bilingual} -- utilizes a synthetic data generation technique that leverages the bilingual dictionary. This approach considers the absence of parallel corpora, while it is possible to utilize large-scale monolingual corpora and ground-truth bilingual dictionary to close the gap between two languages by establishing the anchoring points via using the mappings provided by the dictionary.

This work investigates the capabilities of the following bilingual dictionary based techniques: \textit{Anchored Cross-lingual pretraining} (ACP) and \textit{Anchored training} (AT) described in \cite{duan-etal-2020-bilingual}, and the training of language model on pseudo-parallel data generated by the substitutions provided by a bilingual dictionary.

The main goal of this work was to examine the capabilities of these methods to provide performance improvement for models for different language pairs under various initial conditions:
\begin{enumerate}[label=\roman*.]
    \item Training the models from scratch with various amounts of additional monolingual data.
    \item Training the models from scratch with a limited amount of monolingual data.
    \item Fine-tuning the pre-trained \textit{XLM 17} language model with a limited amount of both parallel and non-parallel data.
\end{enumerate}

\chapter{Machine translation background}

This chapter focuses on providing the reader with information concerning machine translation, its history of development, recent Neural machine translation developments and the translation quality evaluation metrics that are usually utilized.

\section{Pioneers of machine translation}

Machine translation is a task of natural language processing that involves the automatic translation of text from one language to another using computer algorithms. The history of machine translation is traced from the early systems of the 1950s when the first experiments were carried out. One of these experiments, the Georgetown–IBM experiment, despite the facts that this experiment was of demonstrative nature and the product of it had almost no real practical application (the system had only 6 grammar rules and a very restricted vocabulary of 250 words to translate sentences in Russian that were prepared in advance into English), the demonstration had great success. This, along with excessively optimistic estimations that automatic translation would be a solved problem within a decade, resulted in a significant increase in funding and garnered substantial attention from the press and scientific community.

However, in the next few years, these first-generation MT systems had very limited success. They consisted of a large bilingual dictionary where the entry for each word in the source language was provided with equivalents in other languages and syntactic rules that were used to place the output words in the right order. After years of research, it became evident that systems of this kind are not capable of doing a high-quality translation. Mainly because the solution was not systematic: syntactic rules and a bilingual dictionary just weren't enough to make the system solve the problem consistently in various domains because of languages' semantic ambiguity and overly complex rules.

In 1960 Yehoshua Bar-Hillel, a linguist and a pioneer in the field of MT, argued that MT systems back then were not capable of fully automatic high-quality translation \cite{BarHillel2003ADO}. He stated that in different kinds of documents, there could be sentences "whose ambiguity is resolvable only on the basis of extra-linguistic knowledge", those sentences, in opposite to scientific documents or reports, are more prone to misinterpretation by MT systems without aforementioned knowledge. His statements turned out to be right when six years after the Automatic Language Processing Advisory Committee (ALPAC), following the examination of the current state and prospects of MT, concluded in its report that the products of research in the MT field are disappointing and that "there is no immediate or predictable prospect of useful machine translation". This report justified a reduction in government funding for MT research and it was abandoned for years to come.

\section{Years of quiet}

Despite the impact that the ALPAC report had in the US, active research in the MT field continued in Europe and Canada to satisfy the needs of local governments and companies. For the next 20 years, there was a development of the Rule-based machine translation (RBMT) approach. RBMT systems are based on linguistic information of source and target languages provided by dictionaries and grammars covering syntactic, morphological and semantic aspects of languages. Although there was progress in translation quality, coverage of different languages and introduction of new techniques -- Transfer based machine translation, Interlingual machine translation -- systems still had crucial shortcomings: to create one, a whole staff of linguists was required in order to build new rules and dictionaries. Also, they still were lacking the ability to generalize across different domains and were mainly geared up towards the translation of scientific papers and reports.

Some of the many successful systems of the time:
\begin{itemize}
    \item \textbf{SYSTRAN} Russian-to-English MT system made for the United States Air Force, was also used as assistance in the translation of scientific documents written in Russian.
    \item \textbf{METEO} MT system designed for the translation of weather forecasts between English and French languages issued by the Department of environmental policies of Canada.
\end{itemize}

\section{Statistical machine translation}

The introduction of the groundbreaking Statistical Machine Translation (SMT) approach in 1990 \cite{brown1990statistical} marked a significant turning point, leading to rapid changes in the MT field. The proliferation of the Internet and the accumulation of parallel corpora, i.e. collections of aligned sentences in two languages (also known as bitext), in previous years, created conditions that enabled MT systems to move beyond predefined rules and bilingual dictionaries but instead to be created in a process of training on large parallel corpora. It was at this point that the approach to MT began to leverage machine learning methods.

In practice, this meant that although the objective of translating a sequence of tokens (sentence) from a source language $L_{src}$ with vocabulary $V_{L_{src}}$ to the most semantically similar sequence of tokens in the target language $L_{tgt}$ with vocabulary $V_{L_{tgt}}$ remained the same, the method for achieving it shifted from the application of rules that constitute an MT model to a different approach. Now, instead of the manual creation of these rules, they are extracted from large parallel corpora via training.

SMT model's translation of a sequence from the source language to the target one is determined by the estimation of the probability that some sequence in the target language is an accurate translation of this source sequence. The search for the best translation is based on Bayes' rule:

\begin{equation}
    \underset{y}{argmax}\ P(y|x) = \underset{y}{argmax}\ \frac{P(x|y)P(y)}{P(x)} = \underset{y}{argmax}\ P(x|y)P(y),
\end{equation}    

\noindent where 
\begin{description}
    \item $P(y|x)$ is a probability, that sequence $y$ is a translation of sequence $x$.
    \item $P(x|y)$ is a the probability of how often we see a $x$ given that $y$ is seen. It is defined by the translation model.
    \item $P(x)$ and $P(y)$ are the probabilities of some sentences in source/target languages defined by language models.
\end{description}

The parameters of these models are estimated during the training. Model training itself can be described as a process of improving the performance of a model for a specific task (such as MT) by leveraging the training data. In the case of SMT, one of the most widely used training procedures was the Minimum error rate training \cite{och-2003-minimum}. 

\section{Neural machine translation}

 The Neural machine translation (NMT) approach was explored and evaluated back in the 1990s \cite{jain-etal-1991-connectionist,castano-etal-1997-machine} and despite the fact that the approach was very similar to the methods that we use today, its results were unsatisfactory and neural method was considered subpar and ineffective compared to SMT, that was dominant in those years. From today's perspective, fail of early NMT systems seems predestined: none of these models was trained on corpora of size large enough.

The modern incarnation of NMT started taking effect in 2007 with the integration of neural language models into SMT systems \cite{schwenk-etal-2006-continuous} and, after some time, the major breakthrough for MT came with the introduction of pure NMT with a sequence-to-sequence model in 2014 \cite{DBLP:journals/corr/SutskeverVL14,DBLP:journals/corr/ChoMGBSB14} which, after some additions -- attention mechanism \cite{s2s_attention} and byte pair encodings \cite{DBLP:journals/corr/SennrichHB15}, was able to outperform its SMT competitors at shared task for MT at WMT16. NMT continued its development and the most recent architectural development emerged in 2017 with the introduction of the Transformer in the "Attention is all you need" paper by Vaswani et al. \cite{vaswani2017attention}.

\subsection{Natural language representation}

Before we move on to NMT architectures, I would like to review the modern techniques that are being used to facilitate the machine's understanding of natural language.

\paragraph{Byte pair encoding}

Before we feed the input text to the model, it should be tokenized. One of the most efficient and most popular methods (used in BERT \cite{devlin2019bert}, GPT \cite{gpt} and many other systems) to tokenize the input text is byte pair encoding (BPE) \cite{sennrich2016neural}. With this subword-based method, we obtain the vocabulary which consists of tokens during the training. A token can either be a full word or a subword from a training corpus. To indicate the end of the word, the algorithm adds the special token (e.g. "\textit{</w>}") to the end of the token.

Having some limitations regarding vocabulary size, the BPE algorithm ensures that only the most frequent words and subwords extracted from rare words are present in the vocabulary. For example, the morphologically complex German word \textit{Sonnensystem} ("Solar system") may be split into separate subwords \textit{Sonn}, \textit{en} and \textit{system</w>} which can be used for the tokenization of other words that are compound of at least one of these subwords. 

This is a far better approach than, for example, word-based tokenization because it allows us to reduce the size of the vocabulary and the number of OOV (out-of-vocabulary) word encounters in test data. Moreover, the nature of the BPE algorithm, that it tries to catch the most frequent patterns from the training corpus, implies that it will capture the semantically relevant language units (prefixes, suffixes, root words, etc.).

\paragraph{Word Embeddings} 

\textit{Word embeddings} is a fundamental concept in NMT. They are continuous vector representations of words in a high-dimensional space, where each dimension of a vector corresponds to a specific feature or attribute of the word. The main idea behind word embeddings is to capture the semantic and syntactic relationships between words in a way that natural language processing systems (NMT systems in particular) can easily process.

\begin{figure}[H]
    \centering
    \subfloat[One-hot word vector representations. The dimensionality of word vectors is equal to the size of the vocabulary. Vectors are extremely sparse, each has a value of 1 at the index of the corresponding word and 0 at other positions. Vectors bear no semantic information.]{
        \includegraphics[scale=0.7]{obrazky-figures/one-hot_example.pdf}
    } 
    \qquad \qquad
    \subfloat[Word embedding vectors. This type of word representation offers us a drastic reduction in word vector dimensionality and a contextual similarity between word vectors.]{
        \includegraphics[scale=0.7]{obrazky-figures/embeddings_example.pdf}
    }
    \caption{Illustration of different word representations.}
    \label{fig:word_representations}
\end{figure}

Since semantics and syntax are concepts that are hardly understandable by the computer, embedding algorithms adopt the approach that is best described by the famous quote of J.R. Firth \cite{Firth_linguistic-theory}:  

\begin{quote}
    \textit{You shall know a word by the company it keeps.}
\end{quote}

And thus, to install the aforementioned word dependencies, we define the meaning of a word by the context in which it occurs. Words that occur in similar contexts (e.g. words "fox" and "wolf") will have similar embedding representations.

\subsection{Sequence-to-sequence models}

At a high level, the sequence-to-sequence (Seq2Seq) model is an end-to-end model which consists of two components: an encoder and a decoder. Both components traditionally utilise the RNN architecture (being stacked layers of either long short-term memory cells (LSTM) or gated recurrent unit cells (GRU)) and, unlike all non-neural translation systems before, are trained jointly, so they can learn the same context vectors to maximize the translation performance.

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.9]{obrazky-figures/seq2seq_scheme.pdf}
    \caption{Example of Seq2Seq model translating from English to Basque. Illustration is inspired by \cite{s2s_scheme_inspiration}.}
\end{figure}

\paragraph{Encoder}

The purpose of an encoder is to read the variable-length input sentence and to produce a fixed-length representation called the \textit{context vector}. To do so, the encoder first of all maps words in a sentence to vectors of ones and zeros (one-hot representation). Then, using the embedding layer, the encoder maps each one-hot encoded input token to a dense vector representation -- embedding. Once the input tokens are transformed into dense vector representations, the context vector \textbf{c} can be computed step by step for each input token using the deep RNN (LSTM or GRU models)

\begin{equation}
    \textbf{h}_i = f(\textbf{W}_{hx}\textbf{x}_i + \textbf{W}_{hh}\textbf{h}_{i-1} + \textbf{b}_h),
\end{equation}

\noindent where 
\begin{description}
    \item $\textbf{h}_i$ is the hidden state of RNN at timestep $i$. The hidden state of the last RNN layer is the context vector.
    \item $\textbf{x}_i$ is the input embedding vector at timestep $i$.
    \item $\textbf{W}_{hx}$ is a learnt weight matrix, that integrates an input vector.
    \item $\textbf{W}_{hh}$ is a learnt weight matrix, that integrates a vector from the previous timestep.
    \item $\textbf{b}_h$ is a bias term.
\end{description}

\paragraph{Decoder}

Decoder, having the same architecture as an encoder, has a different task: it is responsible for generating the output sentence based on the context vector provided by an encoder. Its first layer's hidden state is being initialised with context vector \textbf{c} provided by the encoder. Then decoder at each timestep, given the context vector \textbf{c} and all the previously predicted words (at the first step decoder only has the <BOS> token), runs all layers of LSTM/GRU and applies the softmax function after that to generate the next word. If a newly generated word is <EOS> (end of sequence) token generation ends. It is important to note that due to the way that output is being generated there is no relation between the length of the input sequence and the length of the output sequence.

\paragraph{Revealed problems}

Although the new Seq2Seq approach has shown promising results, after analysing the NMT systems' performance, Cho et al. \cite{DBLP:journals/corr/ChoMBB14} discovered several drawbacks. They found that, while being on par with traditional SMT systems under favouring conditions, the Seq2Seq systems' performance degrades rapidly: first, with the increase of the number of unknown words, and second, with the increased length of source sentences (see Figure \ref{fig:cho_sentence_length}). 

\begin{figure}[h]
    \centering
    \subfloat[RNNenc]{\includegraphics[scale=0.69]{obrazky-figures/cho_sentence_length_graph_RNNenc.pdf}}
    \subfloat[grConv]{\includegraphics[scale=0.69]{obrazky-figures/cho_sentence_length_graph_grConv.pdf}}
    \caption{The BLEU scores achieved by the RNN Encoder–Decoder \cite{DBLP:journals/corr/ChoMGBSB14} (a) and by the Gated recursive convolutional neural network (b) depending on the length of the sentence obtained in the study of \cite{DBLP:journals/corr/ChoMBB14}. Degradation of performance with sentence length increase is evident.}
    \label{fig:cho_sentence_length}
\end{figure}

And while the first problem may be potentially solved by increasing the size of vocabularies used by NMT systems, the second one, caused by the fact that the fixed-length vector representation does not have enough capacity to encode a long sentence with complicated structure and meaning, required the redesign of the system's architecture itself.

\subsection{Sequence-to-Sequence models with attention mechanism}

To address the described problem of deteriorating performance for longer sentences, \cite{s2s_attention} proposed an extension to the encoder-decoder model which learns to align and translate jointly. 

\begin{figure}[h]
    \centering
    \includegraphics[scale=0.8]{obrazky-figures/seq2seq.pdf}
    \caption{Illustration of RNN encoder-decoder with the attention mechanism.}
\end{figure}

\paragraph{Encoder}

Encoder, unlike the one in the vanilla encoder-decoder architecture, now utilises a bidirectional recurrent neural network (BiRNN), which consists of \textit{forward} and \textit{backward} RNNs: the former one reads the input sequence as it is ordered (from $x_1$ to $x_N$) and computes a sequence of \textit{forward} hidden states ($\overrightarrow{h_1}$, ..., $\overrightarrow{h_N}$) for every input word, the latter one does the same computation except it reads the input sequence in reverse order and thus provides the \textit{backward} hidden states ($\overleftarrow{h_1}$, ..., $\overleftarrow{h_N}$). Then we obtain \textit{word representation} $h_i$ for each word $x_i$ by concatenating $i$-th forward and backward hidden states $h_i = \begin{bmatrix} \overrightarrow{h_i} \\ \overleftarrow{h_i} \end{bmatrix}$. This way each word representation $h_i$ will contain information about both the preceding and the following words with a strong emphasis on the $i$-th word. And thereby encoder is now able to capture dependencies between the $i$-th token and the tokens that are from both sides of it.

\paragraph{Decoder}

In the proposed architecture a decoder is almost identical to the one from the original Seq2Seq model's architecture, except now for each step $i$, in addition to the previously predicted target words ($y_1$, ..., $y_{i-1}$) and the previous hidden state $s_{i-1}$, it should also take in account the \textit{attention}. That is, the probability of the target word is:

\begin{equation}
    p(y_i|y_1,...,y_{i-1},\textbf{x})=g(y_{i-1}, s_i, c_i) \quad \text{with} \quad s_i=f(s_{i-1},y_{i-1},c_i)
\end{equation}

\noindent  The attention has the same role as context vectors and is also denoted as $c_i$ for each step $i$. But now, instead of providing a single representation for the entire sentence for every step of the output generation, we, having a single vector of \textit{word representations}, are evaluating it differently depending on the current target position $i$ with the help of distinct weights. In essence, the attention is a weighted sum of word representations $\textbf{h}$:

\begin{equation}
    c_i = \sum^{N}_{j=1}\alpha_{ij}h_{j}
\end{equation}
\noindent where $\alpha_{ij}$ is a specific weight that represents the probability that target word $y_i$ is aligned with source word $x_j$. It is a product of \textit{softmax} function over all other alignments:

\begin{equation}
    \alpha_{ij} = \frac{\exp{e_{ij}}}{\sum^{N}_{k=1}\exp{e_{ik}}} \quad \text{with} \quad e_{ij} = a(s_{i-1}, h_j)
\end{equation}

\noindent where $e_{ij}$ is an alignment model which estimates how well the inputs around the $j$-th position match the $i$-th output word.

This approach removed the need to encode the entire source sentence into a single context vector by allowing the decoder to look through all the encoder states and evaluate the importance of each source word for a current step. As a result, the model achieved superior performance when translating longer sentences, thereby solving the problem that motivated the creation of extension in the first place, and improving the overall translation quality establishing the Seq2Seq approach (and thus NMT as a whole) as the dominant paradigm.

\subsection{Transformer}

The next breakthrough in NMT came along with the introduction of a novel model architecture in the famous "Attention Is All You Need" paper from Google Research in 2017 \cite{vaswani2017attention}. This architecture, the Transformer, in order to overcome the constraint of sequential computation was designed purely around the attention mechanism which is used to extract dependencies between input and output. This new method permits parallel computing, which drastically speeds up the model's training since GPUs that are used for its training are specifically designed for parallel processing.

\begin{figure}[h]
    \centering
    \includegraphics[scale=0.5]{obrazky-figures/transformer_orig.png}
    \caption{The Transformer model architecture. Taken from the original paper \cite{vaswani2017attention}.}
    \label{graph}
\end{figure}

Transformer's architecture follows the general idea of encoder-decoder structure: it consists of two same "macro" blocks -- an encoder and a decoder, the former one maps the input words ($x_1$, ..., $x_n$) to their continuous representations ($z_1$, ..., $z_n$). The latter one generates an output sequence ($y_1$, ..., $y_m$) based on these representations and previously predicted words. Both of these blocks consist of $N$ ($N=6$ in the original paper) encoder/decoder layers.

\paragraph{Encoder layers}
 
Each encoder's layer consists of two sublayers -- \textit{multi-head self-attention mechanism} and \textit{position-wise fully connected feed-forward neural network} (FFNN). The first one relates different positions of a sentence to compute a representation of it. The second one applies non-linear transformations to its input features.

In addition to this, encoder layers also include \textit{residual connections} around both sublayers followed by \textit{layer normalization}. Residual connections allow the model to learn more complex functions by adding the input to the layer's output, and layer normalization helps stabilize the training process. That is, each sublayer has the output 

\begin{equation}
    \textbf{y}=LayerNorm(\textbf{x}+Sublayer(\textbf{x}))
\end{equation}    
    
\noindent where $SubLayer(\textbf{})$ is the sublayer's function.

\paragraph{Decoder layers}

Decoder layers are structured in the same way as the encoder layers -- they also utilize residual connections and the normalization step after each sublayer. However, the decoder layers also contain another sublayer that is placed between the two sublayers that are included in the encoder layer -- the \textit{Encoder-Decoder attention} sublayer -- which performs multi-head attention over the output of the encoder stack. Additionally, the self-attention sublayer is masked which restricts the decoder from attending to the positions that were not yet generated during the training.

\paragraph{Attention}

The Transformer's attention can be described as a function with 3 arguments, all of them being vectors: query $\textbf{q}$, key $\textbf{k}$ and value $\textbf{v}$ of dimensions $d_k$, $d_k$, $d_v$, respectively; the output of this function is computed as a weighted sum of values $\textbf{v}$, where the weight assigned to each element in it is computed by a compatibility function of the query with the corresponding key.

The attention used in the original Transformer is called "Scaled Dot-Product Attention", it is computed with the above-described query, key and value as arguments, with the difference that in practice these are the matrices of queries/keys/values packed together:

\begin{equation}
    Attention(\textbf{Q}, \textbf{K}, \textbf{V})=softmax(\frac{\textbf{Q}\textbf{K}^T}{\sqrt{d_k}})\textbf{V} \quad \text{with} \quad d_k=d_v=\frac{d_{model}}{h}
\end{equation}

\noindent In addition to that, the attention used in the Transformer is \textit{multi-headed}, which provides the attention with different representation subspaces at different positions:

\begin{equation}
    MultiHead(\textbf{Q}, \textbf{K}, \textbf{V})=concat(head_1,...,head_h)\textbf{W}^O,
\end{equation}
\noindent with
\begin{equation}
    head_i=Attention(\textbf{QW}^Q_i,\textbf{KW}^K_i,\textbf{VW}^V_i)
\end{equation}

\noindent where $h$ is the number of heads ($h=8$ in the original Transformer), $\textbf{W}^Q_i \in \mathbb{R}^{d_{model}\times d_k}$, $\textbf{W}^K_i \in \mathbb{R}^{d_{model}\times d_k}$ and $\textbf{W}^V_i \in \mathbb{R}^{d_{model} \times d_v}$ are the projection matrices for queries, keys and values, which are used to project the input embeddings into a different representation subspace; the $\textbf{W}^O \in \mathbb{R}^{hd_v\times d_{model}}$ is a weight matrix that is used to condense concatenated attention heads into a single matrix. All three matrices are learnt jointly during the training.

\paragraph{Position-wise feed-forward neural network}

Each layer in both the encoder and decoder also contains a feed-forward neural network  which is applied to each position separately. It consists of two linear transformations with a ReLU activation function in between. It transforms the output of the self-attention layer and produces the final output of the encoder layer.

\begin{equation}
    FFNN(x)=ReLU(0, \textbf{xW}_1+\textbf{b}_1)\textbf{W}_2+\textbf{b}_2
\end{equation}

\paragraph{Positional Encoding} 

Positional encodings (PE) are used to provide the model with information about the relative or absolute positions of the words in the sentence. PE have the same dimensionality as the embeddings so that both can be summed. They are added to the input embeddings before they are passed to the encoder/decoder stack of the Transformer. The positional encodings are obtained from an encoding function that should be periodic, thus, the position of the token can be correctly represented, despite the length of the sentence. In the original Transformer Vaswani et al. utilized two sinusoidal functions of different frequencies:

\begin{equation}
\renewcommand*{\arraystretch}{2}
\begin{array}{l}
    PE_{(pos,2i)}=\cos(pos/10000^{2i/d_{model}}) \\
    PE_{(pos,2i)}=\sin(pos/10000^{2i/d_{model}})
\end{array}
\end{equation}

\noindent where $pos$ is the position of given token and $i$ is the embeddings dimension.

The positional encodings along with the attention-centric nature of the Transformer itself have changed the way in which the input is being processed. Now instead of presenting input sequentially, we are able to pass all tokens at once and get the information about their position from the embeddings.
    
\section{Evaluation metrics}

The purpose of automatic evaluation metrics for MT is to provide a quantitative measure of how accurate are the translations provided by an MT system without any assessment from a human translator. These metrics usually involve comparing the output generated by the MT system (\textit{hypothesis} or \textit{candidate} translation) to a \textit{reference} translation. Two metrics that will be used for automatic evaluation in this work are \textit{BLEU} and \textit{chrF}.

\subsection{BLEU score}

The BLEU score \cite{papineni-etal-2002-bleu} is a commonly used metric for evaluating the quality of the machine-generated text, particularly in the context of machine translation. The BLEU score compares a generated candidate translation to one or more reference translations.

The cornerstone of the BLEU metric is the \textit{precision} measure, which is the proportion of \textit{n-grams} (contiguous sequences of $n$ words), that appear in both the candidate translation and the reference translation to n-grams that appear in candidate translation. The more n-grams that are shared between the candidate and reference translations, the higher the BLEU score.

\begin{equation}
    p_n = \frac{\sum_{\textit{n-gram}\in Cdt \cap Ref}count(\textit{n-gram})}{\sum_{\textit{n-gram}\in Cdt}count(\textit{n-gram})}
\end{equation}

It is computed as the geometric mean of the n-gram precision scores, where the n-gram precision for a given $n$ is the $count$ of n-grams in candidate translation that also appear in reference translation, divided by the total number of n-grams in candidate translation. Finally, the BLEU score is \textit{brevity-penalized} ($BP$) if the candidate translation $c$ is shorter than the reference translation $r$ that has the closest length to $c$.

\begin{equation}
    \textsc{Bleu}=\textsc{BP}\cdot (\prod_{n=1}^{N} p_n)^{\frac{1}{N}}
\quad \text{with} \quad
    \textsc{BP}= 
\begin{cases}
    \exp(1-r/c), & \text{if } c<r  \\[0.2cm]
    1,           & \text{otherwise} 
\end{cases}
\end{equation}

\subsection{ChrF score}

The chrF score proposed by M. Popović \cite{chrf} is another commonly used MT evaluation metric. This metric was created to offset the problems identified during the period of using the BLEU metric. One of which is that the BLEU score measures the translation quality from a word-level perspective which makes it less accurate when it comes to the translation quality evaluation of morphologically rich languages.

The chrF metric is an F-score-based metric. To address the described issue the chrF compares hypothesis and reference on a character level instead.

\begin{equation}
    \textsc{chrF}\beta = (1+\beta^2)\frac{\textsc{chrP}\cdot\textsc{chrR}}{\beta ^2 \cdot \textsc{chrP}+\textsc{chrR}}
\end{equation}
\noindent where 
\begin{description}
    \item \textsc{chrP} is a character n-gram precision, the percentage of character n-grams in the hypothesis translation that are also present in the reference.
    \item \textsc{chrR} is a character n-gram recall, the percentage of character n-grams in the reference which are also present in the hypothesis translation.
    \item $\beta$ is a parameter that assigns $\beta$ times more importance to recall than to precission.
\end{description}

\noindent In recent years, the chrF metric, along with its subsequent iterations -- as chrF+(+) -- has demonstrated a strong correlation with human translation quality estimation, particularly in the evaluation of translation quality for morphologically rich languages. To better illustrate this, let's consider the following example:

\begin{table*}[h]
\centering
\captionsetup{labelformat=empty}
\begin{tabular}{lll}
    \midrule 
    \textbf{Source}         & \textbf{en} & Tired student was preparing for the exam. \\ [0.2cm]
    \textbf{Reference}      & \textbf{cs} & Unavený student se připravoval na zkoušku. \\ [0.2cm]
    \textbf{Hypothesis 1}   & \textbf{cs} & Unavená studentka se připravovala na zkoušku. \\ [0.2cm]
    \textbf{Hypothesis 2}   & \textbf{cs} & Unavený student se připravoval na zkoušku. \\ 
    \midrule
\end{tabular}
    \caption{With no additional context provided both hypotheses listed above can be considered correct translations of the source sentence, the difference between them is that the first one contains the feminine forms of words and the second one - the masculine ones. Despite the fact that both translations are correct, the first one will reach a comparatively low BLEU score of 26.27 whereas the second one will reach a BLEU score of 100. ChrF metric offsets this problem by comparing the reference and hypothesis from a character-level perspective, the chrF score with $\beta=3$ will be 0.74 and 1.0 for the first and the second hypotheses respectively.}
\end{table*}


\chapter{Approaches for low-resource neural MT}

In this chapter, I will cover the contemporary techniques employed in NMT that are used when it comes to low-resource Machine translation.

\section{Back-translation}
\label{section:back_translation}

\textit{Back-translation} (BT) is the unsupervised training technique first proposed for SMT and then rediscovered for NMT by Sennrich et al. \cite{sennrich2016improving}. The idea behind this technique is that we can leverage the model's decoding ability and monolingual corpus during its training in a way of iteratively, for each observed training entry, first providing the perhaps incorrect translation of the source sentence by decoding and then learning to reconstruct the original sentence from this translated sentence. The inconsistency between these two sentences provides the error signal to train the model in the target-to-source direction.

\section{Cross-lingual Language Model Pretraining}
\label{section:XLM}

\textit{Cross-lingual language modelling} (XLM) is a task of training language models that can predict tokens in multiple languages: language models trained with Masked language modelling or Translation language modelling (MLM and TLM) are able to predict the masked token at any position given the surrounding tokens (context), CLM -- Causal language model -- on the other hand, is able to predict a sequence of tokens after the preceding words. XLM is particularly useful in situations when the parallel data for a specific language pair is limited, or when the model's application involves the use of many languages.

Lample et al. \cite{lample2019crosslingual} surveyed the capabilities of cross-lingual modelling to improve models' performance for various tasks -- Cross-lingual classification, Cross-lingual language inference and Machine translation. In particular, to leverage XLM for MT, the Transformer's encoder is pre-trained using a causal, translation, or masked language modelling objective prior to being fine-tuned with the Machine translation (for parallel data) or Back-translation (for un-parallel data) objectives.

\subsection{Language modelling}


\textit{Language modelling} is the NLP task that involves the modelling sequences of tokens in a given language. More specifically, the task involves training a model that predicts the probability distribution of the next word given the context of the preceding words. For models of this kind, the input will be the sequence of words and the output will be the probability distribution over the model's vocabulary for the next word. For example, for the English sentence

\begin{quote}
    \textit{He was sitting in front of the \_\_\_}
\end{quote}

\noindent as the input, the output may be the distribution \{..., "computer": 0.3, ..., "window": 0.4, ..., "charger": 0.0, ...\} which signifies that the most probable next word is "window".

\subsection{Masked language modelling}

\textit{Masked language modelling} (MLM) that was introduced in the BERT paper by Devlin et al. \cite{devlin2019bert} is a type of language modelling that, unlike the previously described approach, has the ability to predict the word at any position. As it can be seen in Figure \ref{fig:mlm}, training of the masked language model consists of randomly masking a few tokens in the input sentence (15\% of tokens in the original paper) and then predicting the word that was in the original unmasked sentence based on surrounding tokens. The masking itself is simply a replacement of a chosen word with $[MASK]$ token 80\% of the time, a random token from the vocabulary 10\% of the time and an unchanged token 10\% of the time.

\begin{figure}[h]
    \centering
    \includegraphics[scale=0.6]{obrazky-figures/mlm.pdf}
    \caption{XLM variant of MLM training, $[\backslash s]$ tokens denotes the boundaries of text streams. Replicates the scheme from \cite{lample2019crosslingual}.}
    \label{fig:mlm}
\end{figure}

\noindent Lample et al. modified the original training algorithm by using the input streams of an arbitrary number of sentences instead of pairs of sentences. Also, in order to counter the imbalance between rare and common tokens they discarded the frequent words in the training set with the probability given by the formula

\begin{equation}
  P(w_i) = 1-\sqrt{\frac{t}{f(w_i)}}  
\end{equation}

\noindent where $f(w_i)$ is the frequency of the word $w_i$ and $t$ is a threshold parameter.

\subsection{Translation language modelling}

In order to further improve the cross-lingual pretraining Lample et al. proposed a novel language modelling objective called \textit{Translation language modelling}. TLM, being an extension of MLM, tries to leverage parallel data. 

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.6]{obrazky-figures/tlm.pdf}
    \caption{Training the English-Basque LM with translation language modelling objective.}
    \label{fig:tlm}
\end{figure}

\noindent As this can be seen in Figure \ref{fig:tlm}, the training with such an objective consists of a concatenation of parallel sentences, masking random tokens (with no regard to tokens' language) in the new concatenated text and then predicting the original tokens that were masked based on tokens surrounding masked tokens in both source and target languages. It is also worth mentioning that the positions of tokens in the target sentences are reset. Training model this way encourages it to learn the alignments between the representations of parallel sentences.

The TLM pretraining proved to be useful for the Cross-lingual classification task, the aforementioned research shows that leveraging data through the TLM+MLM objective provides a boost in performance of 3.6\% accuracy.

\section{Leveraging bilingual dictionaries for cross-lingual pretraining}
\label{section:BDBNMT}

"The unsupervised training techniques, based on a bilingual dictionary, as presented by Duan et al. in \cite{duan-etal-2020-bilingual}, rely on the use of unparallel corpora in at least two languages, along with a bilingual dictionary that provides word mappings. In this section, I will describe these training methods.

\subsection{Bilingual dictionaries}

Bilingual dictionaries (or lexicons) are collections of pairs of mutual translations of expressions (single words or phrases) in two languages. The main benefit that bilingual dictionaries bring is that they may provide multiple ground-truth translations of a single expression that could not usually appear in datasets. However, bilingual dictionaries often provide translations without any additional information regarding the translation pair: word class (part of speech) and the case- or tense-related information may be omitted. Another downside is that we can't extract the context in which a given word is usually used neither on the source nor the target side.

\subsection{Anchored training}

\paragraph{Target view Anchored training} To bridge the gap between semantic spaces of languages Duan et al, considering the fact that the cross-lingual language model is being trained on monolingual corpora and that word embeddings are highly dependent on a word's surrounding context,  proposed a method called Anchored training (AT) which leverages the bilingual dictionary by injecting \textit{anchors} on the monolingual corpora. 

\begin{figure}[h]
    \centering
    \includegraphics[scale=0.65]{obrazky-figures/anchored_training.pdf}
    \caption{Example of Back-translation used for AT made in both directions for the same sentence pair, anchor words are written in bold.}
    \label{fig:AT.pdf}
\end{figure}

\noindent As it is evident from Figure \ref{fig:AT.pdf}, the essence of the AT method is that we, having a monolingual corpus and bilingual lexicon, replace the words covered by this lexicon with \textit{anchors} -- translations of these words provided by the lexicon. Through these anchors, words with similar meanings in different languages will share the same word embeddings. 
The training process consists of the back-translation steps in both -- source-to-target (s2t) and target-to-source (t2s) -- directions:
\begin{enumerate}
    \item The anchored source sentence is translated via s2t NMT decoding into the target sentence and then two of them are used as a pair for t2s translation training.
    \item The target sentence is translated into the anchored source sentence via t2s decoding and then both sentences are used to train the s2t translation model.
\end{enumerate}
During the inference, the source sentence is also in advanced translated into the anchored sentence by looking up the dictionary and only then the s2t decoding is used.

\paragraph{Bi-view Anchored training} The modification to AT, the Bi-view AT, extends the way the model looks at the sentence pair by adding the source language view to the target language view that was described previously. This extension provides two additions: first, we add steps similar to those previously described for AT, with the exception that now substitutions are applied to target sentences; second, usage of a dictionary in a bidirectional manner allows us to generate additional anchored sentences from true denoised sentences produced by decoding in either of 2 directions, which will be also utilized during the training.

\subsection{Anchored Cross-lingual pretraining}

Another method presented in this article is \textit{Anchored Cross-lingual pretraining} (ACP). It assumes the absence of parallel corpora. ACP involves the same noising step that was previously described for AT -- we first generate the pseudo-parallel sentence based on the source sentence by substitutions of words covered by a bilingual dictionary.

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.6]{obrazky-figures/acp.pdf}
    \caption{Anchored Cross-lingual Pretraining -- leveraging noisy data through MLM objective.}
    \label{fig:mlm_sub}
\end{figure}

\noindent As can be seen in Figure \ref{fig:mlm_sub}, the source words are replaced with their target language counterparts, and the resulting noised sentence, treated as if it was in the source language, is leveraged through the Masked language modelling objective.

\section{XLM with pseudo-parallel data}

Another method that is covered in this work includes the use of pseudo-parallel data. As can be seen in the scheme from Figure \ref{fig:tlm_sub}, we, having only the monolingual data in one language are generating the noisy version of it. Then we are, treating the true source corpus and the noisy version of it as the source data and the target data respectively, leverage them through the language modelling objective (either TLM or MLM).

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.57]{obrazky-figures/tlm_sub.pdf}
    \caption{XLM on pseudo-parallel data with TLM objective.}
    \label{fig:tlm_sub}
\end{figure}

\chapter{Implementation}

The XLM framework\footnote{\href{https://github.com/facebookresearch/XLM}{https://github.com/facebookresearch/XLM}} provides training options for MLM and TLM pretraining as well as training options for Machine translation and Back-translation training out-of-the-box.

However, the ACP and AT techniques are not directly embedded in the XLM framework. In order to leverage the data the way these methods expect to, we have to modify the data used during the training in the following manner for each of the methods and then leverage them through corresponding training objectives:
\begin{description}
    \item[ACP] assumes the unsupervised training on non-parallel corpora, where the source side corpus is noised with anchoring points and the target side corpus remained clean. Data are leveraged through the MLM objective.
    \item[AT] assumes the unsupervised training through the BT objective in both directions (source-to-target and target-to-source) on monolingual corpora with only the source side corpus noised by placing the anchors provided by the bilingual lexicon.
    \item[XLM on pseudo-parallel data] assumes that the target language data is generated by the BD-based substitutions applied to true source data. The data will be leveraged through MLM or TLM objectives as it is depicted in Figure \ref{fig:tlm_sub}.
\end{description}

The script that applies the substitutions is named \texttt{sub\_by\_dict.py}, the substitution algorithm itself is described in Appendix \ref{app:dictionary_utilization}. All the training arguments used for all the experiments described further are listed in Appendix \ref{app:training_args}.

\chapter{Experiments}

In this chapter, I will cover the subjects closely related to the experiments conducted for this work: used data, data preprocessing pipeline and rationalization of the choice of languages.

Later parts of this chapter analyse the effectiveness of the application of described bilingual dictionary based techniques under different conditions. This work investigates the settings that are not covered by the research of \cite{duan-etal-2020-bilingual} -- first, I surveyed the situations which involve the use of a small amount of parallel data and various amounts of non-parallel data; second, a case somewhat similar to the one discussed in the above-mentioned study, with the exception that there will only be a limited amount of monolingual data for both languages.

\section{Data overview}

In this section, I would like to list and briefly describe all data sources that were used to conduct the experiments. In subsections \ref{subsection:Training_data} and \ref{subsection:Valid_data} I will review in terms of data quality, domain and size the corpora that were used as training, validation and testing sets during the models' training and evaluation. In subsection \ref{subsection:Vocabs} used bilingual dictionaries are covered. Unless otherwise specified, all listed corpora were downloaded from OPUS \cite{TIEDEMANN12.463}.

\begin{table}[h]
\renewcommand*{\arraystretch}{1.15}
\centering
\begin{tabular}{|l|c|c|c|c|c|}
    \hline
    & \multicolumn{4}{c|}{Size of datasets (in sentences)} &  \multirow{2}{*}{Domain} \\
    \cline{2-5}
    & en-fr & en-cs & en-vi & en-eu & \\
    \hline
    CCMatrix & - & - & - & 7.8M & Various \\
    Europarl & 2.1M & 647K & - & - & Politics \\
    NewsCommentary & - & 212K & - & - & News \\
    Wikipedia & 818K & - & 58K & - & Wiki articles \\
    Wikimedia & 1.0M & - & - & - & Wiki articles \\
    OpenSubtitles & - & - & 350K & - & Movie subtitles \\
    ElhWebCorp & - & - & - & 12M & Various \\
    EhuHac & - & - & - & 585K & Various \\
    TED2020 & - & - & 326K & - & Various \\
    NeuLab\_TedTalks & - & - & 183K & - & Various  \\
    EUBookshop & 10.8M & - & - & - & Legal \\
    QED & - & - & 338K & - & Various \\
    flores-200\_dev & 997 & 997 & 997 & 997 & Various \\
    flores-200\_devtest & 1012 & 1012 & 1012 & 1012 & Various \\
    Tatoeba & 268K & 31K & 5694 & 2066 & Various \\
    \hline
\end{tabular}
    \caption{Overview of all datasets that are used in this work.}
\end{table}

\subsection{Training datasets}
\label{subsection:Training_data}

\begin{description}
    

\item[CCMatrix]\cite{CCMatrix} is the giant corpus containing parallel sentences in 90 languages. The content of it was extracted by data mining from web crawls obtained from the entire web. The implication of that is the fact that the content may contain many noisy entities: hyperlinks, random punctuation, markup elements, and misalignments.
%In this work, the Basque-English pair of CCMatrix corpus is used.

\item[Europarl] \cite{koehn-2005-europarl} is a parallel corpus containing 21 European languages. It consists of transcripts of European Parlament sessions translated into these languages by professional translators.
%In this work, English-French and English-Czech pairs of Europarl are utilized.

 \item[NewsCommentary] is a parallel corpus that was created by WMT. It consists of news commentaries and is available in 15 languages.

 \item[Wikimedia] is a large collection of parallel sentences in 322 languages. It includes the dumps of articles from various Wikimedia projects (Wikipedia, Wiktionary and others). 
 %The French-English part of this dataset is used in this work. 

\item[OpenSubtitles] is a parallel corpus of translated movie subtitles from OpenSubtitles web site\footnote{\href{http://www.opensubtitles.org/}{http://www.opensubtitles.org/}} it is available in 62 languages.

\item[ElhWebCorp] (Elhuyar Basque Web Corpus) is a product of Igor Leturia's Ph.D. thesis \cite{Azkarate2014TheWA}. It was collected using both search engines and crawling. It is a large monolingual Basque general corpus containing around 186M raw tokens. As well as the CCMatrix corpus, ElhWebCorp includes a lot of noisy elements.

\item[EhuHac] is a dataset created by Basque Country University. The corpus is built on translations of 171 books, it provides parallel data with Basque on one side and French/English/Spanish on the other. 

\item[TEDTalks] NeuLab-TedTalks\footnote{\href{https://github.com/UKPLab/sentence-transformers/blob/master/docs/datasets/TED2020.md}{https://github.com/UKPLab/sentence-transformers/blob/master/docs/datasets/TED2020.md}} and TED2020 are both corpora that are based on transcripts of TED Talks presentations. TED2020  was made by Reimers and Gurevych \cite{reimers2020making}, it is based on translated subtitles of 4000 TED talks in 100 languages. NeuLab-TedTalks is based on translated subtitles that were made by volunteers, the dataset is available in 59 languages. 

\item[EUBookshop] is a parallel corpus that was created by crawling the EU Bookshop, the archive of various publications from EU institutions. The dataset is available in 48 languages. 

\item[QED] (QCRI Educational Domain Corpus) is a parallel corpus that consists of translated subtitles of educational videos and lectures developed by Qatar Computing Research Institute \cite{abdelali-etal-2014-amara}. The corpus supports 225 languages.
\end{description}

\subsection{Validation and test datasets}
\label{subsection:Valid_data}

\begin{description}
\item[Flores-200]\cite{flores-200} is a high-quality parallel multilingual benchmark for low-resource and multilingual machine translation. Flores-200 consists of translations of 842 distinct web articles for 200 languages made by professional translators, the dataset is divided into two splits: \textit{dev} and \textit{devtest} that were used as validation and test sets respectively for each language pair in this research. Flores-200 benchmark can be downloaded from the flores200 GitHub repository\footnote{\href{https://github.com/facebookresearch/flores/blob/main/flores200/README.md\#download}{https://github.com/facebookresearch/flores/blob/main/flores200/README.md\#download}}.

\item[Tatoeba]\cite{tatoeba} is a large corpus that is based on the Association Tatoeba database of sentences and their translations that are being contributed by the community. The main product of the project is the toolkit that provides the examples of usage of given words. And thus the parallel corpus consists mostly of simple "example sentences" in different languages. Tatoeba is available in 380 languages. Tatoeba corpora were used for model testing for all declared language pairs. 
\end{description}

\subsection{Bilingual dictionaries} 
\label{subsection:Vocabs}

\begin{description}
\item[MUSE dictionaries] are the high-quality bilingual dictionaries for 110 languages that were presented alongside a state-of-the-art approach for unsupervised MT in the work of Conneau et al. \cite{conneau2017word}. Authors took into account the problem of words' polysemy and instead of using some online translation tools to generate expressions' translations they used an internal translation tool designed specifically for the task of lexicon induction. In this work, Vietnamese-, French-, and Czech-English MUSE dictionaries are utilized. Bilingual lexicons are publicly available as part of the MUSE library\footnote{\href{https://github.com/facebookresearch/MUSE}{https://github.com/facebookresearch/MUSE}}.
\item[ELRC English-Basque dictionary] is an automatically created dictionary that was inferred during the training of an unsupervised MT system as a part of MT4ALL project\footnote{\href{https://ixa2.si.ehu.eus/mt4all/project.html}{https://ixa2.si.ehu.eus/mt4all/project.html}}. 

\item[Apertium]\cite{10.2307/41487458} is an open-source platform for rule-based non-neural machine translation. One of the modules of this system, namely, lexical transfer, relies heavily on a bilingual dictionary for looking up lexical forms in them. Unlike other bilingual lexicons, the ones used in Apertium systems provide additional information like the word's part of speech, case, and tense to further improve the language translation. However, in this research dictionaries will be only used for word mapping. English-Basque bilingual lexicon from the research of O'Regan and Forcada \cite{PLN4867} is utilized in this work.
\end{description}

It is worth noting that the resulting English-Basque lexicon has lower quality compared to the ones provided by the MUSE library, it includes many entries with translation of phrases which may substantially reduce the dictionary coverage of the dataset. Also, the vocabulary was pre-processed by cleaning from noisy elements (redundant punctuation, unrelated symbols).

\section{Languages}
\label{section:language_pairs}
In order to better explore the capabilities of bilingual dictionary based techniques it was decided to train the MT models for several language pairs. All of them consider English as the source language. The choice of target languages is based on their morphological richness and  linguistic distance from English.

Here is the list of these target languages and some of their linguistic features that are relevant to this research:
\renewcommand{\thefootnote}{\fnsymbol{footnote}}
\begin{description}

    \item[French] is a Romance language that is very closely related to English. French and English share about 30\% of words that are either derived from Latin or borrowed by one language from another. Just like English, French has the SVO word order. French was chosen as a language with a small linguistic distance from English and as a language of a comparably simple morphology.
    
    \item[Vietnamese] is an isolating language, which means that words are made up of morphemes that cannot be further divided into meaningful smaller units. Similar to English, it has an SVO word order. Vietnamese can be characterized as a language with low morphological complexity: it lacks any direct modifications of words. Instead, to indicate any grammatical inflection, additional words are added before or after the word we want to inflect. Because of the combination of all these properties, it is expected that the performance boost provided by BDBNMT for English-Vietnamese MT will be the highest among all considered language pairs.
    
    \begin{figure}[H]
        \centering
        \captionsetup{labelformat=empty,belowskip=0pt}
        \includegraphics[width=0.88\linewidth]{obrazky-figures/vietnamese_example.pdf}
    \end{figure}
    \footnotetext[1]{Translation generated by Google Translate}
    
    As one can see in this example, the translation generated by substitution is not very accurate: it is verbose and English-centric due to the word mapping in the English-to-Vietnamese direction. However, it allows the model to learn better representations of Vietnamese words during the pretraining phase: in the case of this sentence, having the vocabulary entry \{"paper": ["trát", "gi\^{a}y"]\} with two Vietnamese translations, we, using the substitution algorithm, get \textit{trát} ("warrant") as a translation, which is not accurate in this situation but these two words have a somewhat close meaning and thus it is desirable that they are located close to each other in the embedding space.

    \item[Basque] is an under-resourced language which is considered a language isolate (there are no other languages related to Basque). Basque is known for its complex morphology and agglutinative nature. It has a relatively flexible word order, which means that words in a sentence can be rearranged for emphasis or stylistic purposes without really changing the meaning of the sentences. However, the existing rules imply that the basic word order in Basque is SOV (subject-object-verb). To better understand the morphological richness and agglutinative nature of the Basque language let's consider the example\footnotemark[1]: the verb \textit{dakartzat} ("I bring them") is formed as follows: \textit{da} indicates present tense, \textit{kar} is a root word of infinitive \textit{ekkari} ("to bring"), \textit{tza} indicates plural and \textit{t} indicates subject ("I"). Such high morphological complexity causes doubts about the applicability of bilingual dictionary based word substitutions: lexicons simply can't provide mappings to all the forms that are inflected from the word. However, considering the fact that Basque is an under-resourced language, BD-based can still be effective for improving the model's performance by the provision of ground-truth expressions' translations. 
    
    \footnotetext[1]{Taken from \href{https://en.wikipedia.org/wiki/Agglutination\#Eurasia_and_Oceania}{Wikipedia}}

    \item[Czech] is a fusional language which implies that the meaning of a word is often changed by adding suffixes, prefixes, or inflections to the base word. Czech nouns, adjectives, pronouns, and verbs are inflected to reflect their grammatical role in a sentence, as well as their gender, number, and case. Even though Czech, like many other Slavic languages, has a relatively free word order, in general, it follows the SVO word order.

\end{description}
\renewcommand{\thefootnote}{\arabic{footnote}}
\setcounter{footnote}{5}

\section{Data preprocessing}
\label{section:data_prep}

The preprocessing of corpora in general follows the same steps that are shown as the example in the \href{https://github.com/facebookresearch/XLM}{XLM GitHub repository}:
\begin{enumerate}
    \item Tokenize the data using the rule-based tokenizer which will split words into subwords and separate the interruption.
    \item Learn BPE codes from both source and target tokenized data.
    \item Apply the learned BPE codes on tokenized data.
    \item Extract the vocabulary from BPE tokenized data.
    \item Divide the data into parallel and non-parallel subsets if we expect to train the model with TLM objective and there are non-parallel segments.
    \item Binarize the data.
\end{enumerate}

\section{Tools}

\paragraph{Preprocessing} To perform data preprocessing described in Section \ref{section:data_prep} the following tools were used:
\begin{description}
    \item Moses tokenizer\footnote{\href{https://github.com/moses-smt/mosesdecoder}{https://github.com/moses-smt/mosesdecoder}} is used for subword tokenization of raw data.
    \item fastBPE\footnote{\href{https://github.com/glample/fastBPE}{https://github.com/glample/fastBPE}} is used for all BPE-related operations (BPE codes learning, vocabulary extraction, BPE codes application)
\end{description}

\paragraph{Training} XLM\footnote{\href{https://github.com/facebookresearch/XLM}{https://github.com/facebookresearch/XLM}} framework was used for model training with both Language modelling and Machine translation objectives.

\paragraph{Evaluation} In order to perform models' performance evaluation during both validation and testing the following tools were used:
\begin{description}
    \item Since the XLM framework provides no option for the training with the chrF metric, the evaluation module was modified by adding a chrF score implementation provided by nltk \cite{Bird_Natural_Language_Processing_2009} and a corresponding option for a training script.
    \item SacreBLEU implementation of chrF and BLEU scores is used for the evaluation of converged MT models, sacreBLEU was introduced by M. Post \cite{post2018clarity}.
\end{description}


\section{Basic sets: parallel data for training MT systems}

In order to provide some parallel data for MT training, subsets of 130k sentences dubbed \textit{basic sets} were extracted from the following corpora: Europarl, NeuLab-TedTalks, Europarl, CCMatrix for English-French, -Vietnamese, -Basque and -Czech language pairs respectively. Basic sets are the only parallel datasets used in this research. They are used in all experiments that assume the use of parallel data.

\section{Notation}

Throughout the following experiments' descriptions and discussions, some specific abbreviations and terms will be used. The list is as follows:

\begin{description}
    \item[Training objectives] (MLM, TLM, MT, ACP, AT) will be denoted as an abbreviation with a subscript indicating the number of training sentences from both source and target sides, the prime sign will mean that data used for training are pseudo-parallel, stages of training are separated by a comma. E.g. \(\text{TLM100}+\text{MLM130}^{\prime}\text{,} \text{MT130}\) means that the model was first pre-trained: with TLM objective on first 100k sentences and MLM objective on the next 130k pseudo-parallel sentences; next, the model was MT fine-tuned leveraging only 130k sentences.
    \item[MT] Machine Translation.
    \item[BD] Bilingual Dictionary.
    \item[T-/M-LM] Translation/Masked Language Modelling.
    \item[ACP] Anchored Cross-lingual Pretraining.
    \item[AT] Anchored training.
\end{description}

\section{Training models from scratch with a small amount of additional data}
\label{section:training_small}
This set of experiments emulates the situation where one doesn't possess the large-scale corpora to train an NMT model. The training was conducted for all 4 language pairs that were listed previously. These experiments should explore the efficiency of models' pretraining  applicability of the BD-based approach to languages of different morphological complexity. It is expected that the performance boost may be less evident for more morphologically rich languages.

\subsection{Determining the optimal hyperparameters}

Since it is expected that the performance of MT models will be subpar because of data scarcity, it was decided to find the embedding dimensionality that will be optimal for these models. Related experiments are described in Appendix \ref{app:find_emb}.

\subsection{Baseline}

The baseline case considers the absence of additional monolingual data, so baseline models are leveraging only the basic sets for training. Experiments conducted will also serve as a showcase of how efficient both TLM and MLM pretraining with following MT fine-tuning are compared to the pure MT approach and to one another.

\begin{figure}[H]
    \centering
    \includegraphics[width=0.47\linewidth]{obrazky-figures/baseline_mt-vs-mlm+mt-vs-tlm+mt_bleu.pdf}
    \includegraphics[width=0.47\linewidth]{obrazky-figures/baseline_mt-vs-mlm+mt-vs-tlm+mt_chrf.pdf}
    \caption{Comparison of performance of models trained without pretraining and models trained with different pretraining objectives.}
    \label{fig:baseline_mt-vs-mlm+mt-vs-tlm+mt}
\end{figure}

First, in order to better understand the significance of LM pretraining it was decided to train models with the MT objective without any pretraining to provide a point of reference.  As it is apparent from both graphs from Figure \ref{fig:baseline_mt-vs-mlm+mt-vs-tlm+mt}, MLM pretraining provides a notable performance boost in the BLEU score (30\%) and in the chrF3 score (10\%) on average among all the test sets.

Second, to explore the expediency of leveraging parallel data through the TLM objective was trained the MT model that was pre-trained with TLM. Graphs from Figure \ref{fig:baseline_mt-vs-mlm+mt-vs-tlm+mt} demonstrate that MT models that were first TLM pre-trained outperform models that had MLM pretraining for each language pair. To enhance the performance of future models, it was decided to always consider the basic set suitable for the TLM objective.

\subsubsection{Test sets' complexity}

From the graphs in Figure \ref{fig:baseline_mt-vs-mlm+mt-vs-tlm+mt} it is evident that the Tatoeba set is different from flores200 dev/devtest sets in terms of complexity -- as mentioned before, Tatoeba set consists of simple short example sentence whereas flores200 is based on web articles and consists of sentences that are more complex, so it is worth considering that results for flores200 sets are in general more representative. For all subsequent experiments, graphs will only depict the average performance for a particular model over all datasets. The summary of the performance of every model trained during this research for each dataset can be found in Appendix \ref{app:models_performance}.

\subsection{XLM with the use of pseudo-parallel data}

In order to provide a set of English monolingual data to be used for BD-based substitutions to generate the pseudo-parallel data, 100k sentences were extracted from the corpora listed in Table \ref{tab:small_additional_corpora}.

\begin{table}[H]
    \renewcommand*{\arraystretch}{1.3}
    \centering
    \scalebox{0.88}{
    \begin{tabular}{|l|c|c|c|}
        \hline
        Language pair & Corpora & BD entries & BD coverage  \\
        \hline
        English-French & \multirow{3}{*}{English-French Europarl and Wikipedia} & 113286 & 54.6\% \\
        \cline{1-1}\cline{3-4}
        English-Vietnamese & & 76364 & 49.8\%  \\
        \cline{1-1}\cline{3-4}
        English-Basque &  & 29282 & 35.5\%  \\
        \hline
        English-Czech & English-Czech Europral and NewsCommentary & 64211 & 54.7\% \\
        \hline
    \end{tabular}}
    \caption{Information about corpora that were utilized as additional data as well as the percentage of words covered by bilingual dictionary.}
    \label{tab:small_additional_corpora}
\end{table}

After the application of BD-based substitutions on monolingual corpora we obtain 100k pseudo-parallel sentences. After the merge with the true parallel basic set, we obtain a parallel corpus of 230k sentences. At the pretraining stage, the first 130k will be utilized for the TLM objective, the remaining 100k will be utilized for the MLM objective, and then the model will be fine-tuned with the MT objective on the true parallel part.

\subsection{Topline}

To train models that suppose to provide better (topline) results compared to models that were trained on pseudo-parallel data, this time non-synthetic data were utilized as the target language data and were labelled as non-parallel.

The additional data for the target side of training sets for each language were taken from the following corpora (50k sentences from each listed corpus): English-French -- Europarl, Wikipedia; English-Vietnameses -- NeuLab-TedTalks, OpenSubtitles, English-Basque -- ElhWebCorp, EhuHac, English-Czech -- Europral, NewsCommentary.


\subsection{Results analysis}

From the performance graphs in Figure \ref{fig:small_comparison} it can be clearly seen that the addition of pseudo-parallel data leads to the substantial improvement of models' performance for language pairs with a high BD data coverage of around 50\% (English-Frech, -Vietnamese, -Czech). Models that were pre-trained using XLM with the use of pseudo-parallel data demonstrate either a similar or superior performance compared to the models marked as Topline which were leveraging the true monolingual additional data for pretraining.

Another observation is the fact that for the English-Basque model, the lower BD data coverage and lower quality of BD lead to limited improvement in the resulting MT model's performance.

In conclusion, it can be said that the use of pseudo-parallel data generated by a large BD for language modelling can be extremely beneficial for the resulting MT models' performance under the described setting -- up to the point that models pre-trained on this kind of data outperform the ones that were trained on true non-parallel data.

\begin{figure}[H]
    \centering
    \includegraphics[width=0.47\linewidth]{obrazky-figures/small_comparison_bleu.pdf}
    \includegraphics[width=0.47\linewidth]{obrazky-figures/small_comparison_chrf.pdf}
    \caption{Performance comparison of Baseline/XLM with pseudo-parallel data/Topline models.}
    \label{fig:small_comparison}
\end{figure}

\section{Training models from scratch on a small amount of monolingual data}
\label{section:mono}

Unsupervised methods of training such as the BT require an enormously big amount of unparallel data, usually several millions of sentences on source and target sides. BD-based methods -- ACP and AT -- proposed by Duan et al. \cite{duan-etal-2020-bilingual} demonstrated their efficiency in the setting of an abundance of monolingual data.

This series of experiments suppose to examine the effectiveness of the proposed training method in the setting of a scarce amount of monolingual data.

\subsection{Baseline}

Since it is expected that the performance of the resulting model trained with ACP+AT training steps will be extremely low. The translation made by the application of a bilingual dictionary will be considered a Baseline performance (basically, the noised versions of original corpora are used as the hypothesis translations).

\subsection{Anchored training}

For the ACP+AT training were used small-scale non-parallel corpora of 230k sentences. The source language corpus consists of pseudo sentences generated from the true source language data.

\subsection{Results analysis}

As one can see from the resulting performance depicted in the graph from Figure \ref{fig:mono_comparison}, ACP+AT training on a small-scale monolingual corpora shows translation quality that is inferior to the one provided by the word mappings from bilingual dictionaries.

\begin{figure}[H]
    \centering
    \includegraphics[width=0.47\linewidth]{obrazky-figures/mono_comparison_bleu.pdf}
    \includegraphics[width=0.47\linewidth]{obrazky-figures/mono_comparison_chrf.pdf}
    \caption{Transaltion quality comparison of translations provided by bilingual dictionary and by the ACP+AT trained model.}
    \label{fig:mono_comparison}
\end{figure}

The deteriorating performance of the trained model can be explained by the fact that unsupervised methods of training require large amounts of data, ACP and AT methods proved to be efficient exclusively under the setting of the abundance of non-parallel data.

\section{Fine-tuning the ACP+AT pre-trained model}

Since all other experiments for model pretraining only consider the tuning of the Transformer's encoder, it was decided to test the performance of the MT fine-tuned model the pretraining of which includes the training of a decoder. Models described in Section \ref{section:mono} trained solely on monolingual corpora suit this setting since they had a stage of AT training, which includes the optimization of parameters of a decoder.

\subsection{Baseline}

Since this series of experiments consider the usage of parallel data for MT fine-tuning, we can utilize results provided by MT-tuned models from previous experiments for translation quality comparison. As the Baseline model will be considered the model that was MLM pre-trained on the same data as the model described in Section \ref{section:mono} and then fine-tuned on a basic set. 

\subsection{MT fine-tuning}

Since the preceding ACP and AT training procedures were done on source data noised with BD-based substitutions, it is not obvious whether the models should be fine-tuned with the MT objective on data with the noised or the clean source side.

\begin{figure}[H]
    \centering
    \includegraphics[width=0.47\linewidth]{obrazky-figures/noisymt-vs-cleanmt_bleu.pdf}
    \includegraphics[width=0.47\linewidth]{obrazky-figures/noisymt-vs-cleanmt_chrf.pdf}
    \caption{Comparison of performance of models which were fine-tuned with MT objective on noised source data (yellow) and clean source data (orange).}
    \label{fig:noisymt-vs-cleanmt_comparison}
\end{figure}

Performance graph from Figure \ref{fig:noisymt-vs-cleanmt_comparison} shows that the models that were MT fine-tuned on parallel data with clean source side outperform ones that considered noised corpus as a source language data. The possible explanation for such degrading performance may lie in the fact that the substitutions that were observed during the training didn't appear in the test data. 

\subsection{Topline}

The performance of TLM pre-trained Topline models described in \ref{section:training_small} will be considered a Topline performance.

\subsection{Results analysis}

From the testing results depicted in Figure \ref{fig:mono-ft_comparison} it can be seen that the ACP+AT pre-training yields results that are either similar or superior to the ones delivered by the systems which were first pre-trained with MLM objective. 

It can be said that tuning the Transformer's decoder parameters during the pre-training via ACP+AT training may be beneficial in some cases -- there is a significant improvement for the model for the English-French language pair, however, the improvement yielded by such pre-training is not persistent for different language pairs, so this method of pre-training cannot be considered superior to the MLM.

\begin{figure}[H]
    \centering
    \includegraphics[width=0.47\linewidth]{obrazky-figures/mono-ft_comparison_bleu.pdf}
    \includegraphics[width=0.47\linewidth]{obrazky-figures/mono-ft_comparison_chrf.pdf}
    \caption{Performance comparison of Baseline/ACP+AT pre-trained/Topline models.}
    \label{fig:mono-ft_comparison}
\end{figure}

\section{Training models from scratch with a large amount of additional data}
\label{section:training_big}

This series of experiments suppose to investigate the applicability of the BD-based methods to the situation where we add a large amount of monolingual data for the model to leverage in addition to previously described parallel basic sets.

Models will be trained for two language pairs with comparatively low morphological complexity: English-French -- suppose to show how efficient pretraining on pseudo-parallel data is for linguistically close languages, English-Vietnamese -- suppose to show how efficient it is efficient for distant languages.

Since the amount of training data was increased, the size of embedding vectors utilized by models trained for this series was increased to 512 to enhance the models' capacity to represent the language's features.

\subsection{Baseline}

The Baseline case considers the presence of large-scale monolingual corpus only for the source side. This way the model will have a lot of data to generalize on which may lead to the emergence of bias towards the source language. However, due to the fact that the additional data are provided for the source side, it may not be as noticeable since the model will have a good ability to represent the source language sentences and thus the ability to translate them to sentences in the target language.

\subsection{XLM with the use of pseudo-parallel data}

Having the same initial conditions as the ones described for the Baseline, we now extend the training corpus by generating the noisy data that will serve as target language data.

\begin{table}[H]
\renewcommand*{\arraystretch}{1.3}
    \centering
    \begin{tabular}{|l|c|c|c|}
        \hline
         & Number of words & Number of substitutions & Coverage \\
         \hline
         English-French & 24713580 & 13408303 & 54.2 \% \\
         English-Vietnamese & 18869018 & 9472170 & 50.2 \% \\
         \hline
    \end{tabular}
    \caption{Dictionary coverage for each language pair.}
    \label{tab:big_bdb_coverage}
\end{table}


\subsection{Topline}

The Topline case resembles a situation somewhat close to the one investigated by Duan et al. in \cite{duan-etal-2020-bilingual}. This case assumes the presence of large-scale non-parallel corpora -- 1M sentences for each language, so the language modelling training is done on true data.

\subsection{Results analysis}

\begin{figure}[H]
    \centering
    \includegraphics[width=0.47\linewidth]{obrazky-figures/big_comparison_bleu.pdf}
    \includegraphics[width=0.47\linewidth]{obrazky-figures/big_comparison_chrf.pdf}
    \caption{Performance comparison of Baseline/XLM with pseudo-parallel data/Topline models. Scores marked with an asterisk correspond to the performance of models from Section \ref{section:comparison_TLM-vs-FullTLM} that are best for a particular language pair.}
    \label{fig:big_comparison}
\end{figure}

From the performance evaluation depicted in graphs from Figure \ref{fig:big_comparison} it is evident that XLM models trained with the use of pseudo-parallel data yield a significant boost in performance compared to the models that were utilizing solely additional source data.

The initial assumption that the performance of the improvement provided by the addition of (pseudo-)parallel additional data models will depend on the linguistic distance proved to be wrong: the difference between the improvement yielded by Topline models is not significant between two language pairs -- 10.7\% and 7.76\% BLEU score improvements for English-French and English-Vietnamese respectively. What is observable, however, is the fact that the boost provided by the inclusion of additional pseudo-parallel sentences is more significant for the English-Vietnamese language pair than for the English-French language pair which may indicate that the target language's morphological complexity plays a significant role in whether this method will be useful for a particular language pair or not.

In conclusion, it can be stated that XLM with the use of pseudo-parallel data is a viable approach for enhancing the models' performance in this setting. However, unlike for the experiments with a small amount of additional monolingual data (Section \ref{section:training_small}), it cannot be stated that this method provides results that are similar to ones obtained from training on true monolingual data since there is a noticeable performance gap.

\section{Applicability of TLM pretraining on pseudo-parallel data}
\label{section:comparison_TLM-vs-FullTLM}

As previously mentioned, when it comes to the utilization of parallel corpus through some language modelling objective for further MT fine-tuning the TLM yields better results than the MLM. However, it is debatable whether we should consider the pseudo-parallel data generated by the BD-based substitutions parallel or not.

For the experiments described in Sections \ref{section:training_small} and \ref{section:training_big}, there were two groups of models that were trained with the use of additional pseudo-parallel data. Models dubbed \textit{TLMcat} were considering pseudo-parallel data as parallel and thus were leveraging them through the TLM training objective. Models dubbed \textit{TLMsep}, on the contrary, treated pseudo-parallel data as available exclusively for the MLM training objective.

Displayed in the graph in Figure \ref{fig:comparison_TLM-vs-FullTLM} (b), the evaluation of the performance of models described in Section \ref{section:training_big} shows that the LMcat models outperform the LMsep models for the English-French language pair. At the same time, the former ones show inferior performance for the English-Vietnamese language pair.

The same, even though not as clear, can be said for the chrF3 score evaluation of LMcat models described in Section \ref{section:training_small} for English-French and -Czech language pairs: they slightly outperform their counterparts. The opposite observation is also valid for the evaluation of the models for English-Vietnamese and -Basque pairs: LMsep models provide better results for each test set.

Both sets of experiments with different sizes of training sets, share the same properties that are relevant to this research: the coverage of words by bilingual dictionaries and the domain of data remained unchanged. The only difference was the amount of pseudo-parallel data. The increase in the size of which emphasized the tendency which was only barely observable with the models trained on small datasets: it appears that leveraging pseudo-parallel data with TLM objective is beneficial only for the target languages that are linguistically close to the source language (French, Czech) and harmful when the target language is distant from the source one (Vietnamese, Basque).

\begin{figure}[H]
    \centering
    \subfloat[Models trained on a limited amount of additional monolingual data]{
        \includegraphics[width=0.46\linewidth]{obrazky-figures/small_comparison_TLM-vs-FullTLM_bleu.pdf}
        \includegraphics[width=0.46\linewidth]{obrazky-figures/small_comparison_TLM-vs-FullTLM_chrf.pdf}
    }
    
    \subfloat[Models trained on a large amount of additional monolingual data]{
        \includegraphics[width=0.46\linewidth]{obrazky-figures/big_comparison_TLM-vs-FullTLM_bleu.pdf}
        \includegraphics[width=0.46\linewidth]{obrazky-figures/big_comparison_TLM-vs-FullTLM_chrf.pdf}
    }
    \caption{Performance comparison of models that were treating the pseudo-parallel data differently: yellow -- leveraging it through MLM objective (will be referred to as \textit{LMsep}), orange -- leveraging it through the TLM objective (will be referred to as \textit{LMcat}).}
    \label{fig:comparison_TLM-vs-FullTLM}
\end{figure}

\section{Fine-tuning the XLM-R model}
\label{section:fine-tuning_XLM-R}

The last series of experiments and, perhaps, the one that is the closest to the real situation when it comes to training an MT model for low-resource language pair, resembles the case where there exists a large publicly available pre-trained model which is supposed to be fine-tuned for a concrete language pair for the MT task.

In order to fine-tune the XLM-R model, it is required to download the model itself, codes and vocabulary listed in the XLM repository and train the model in the way described before on data processed using downloaded codes and vocabulary. 

\subsection{XLM 17}

For this set of experiments, \textit{XLM 17} \cite{conneau-etal-2020-unsupervised} is utilized as such a pre-trained model. XLM 17 is a multilingual model that was pre-trained with the MLM objective on corpora in 17 languages (en, fr, es, de, it, pt, nl, sv, pl, ru, ar, tr, zh, ja, ko, hi, vi). The model has 16 layers and 16 attention heads, it utilizes embedding vectors of size 1280 and a vocabulary size of 200k tokens.

\subsection{Fine-tuning with MT objective}

XLM 17 models fine-tuned with the MT objective suppose to serve as a Baseline. Tuning the model like this is the traditional way of adapting a language model for the MT task.

\subsection{Two-stage fine-tuning}

Two-stage fine-tuning of XLM 17 consists of first fine-tuning the model with TLM+MLM objectives on set that consists of the basic set and pseudo-parallel data and, second, of again fine-tuning it but with MT objective instead. It is expected that the performance of models for language pairs which have the target language covered during the XLM 17 training (French, Vietnamese) will be similar to the one of previously described MT fine-tuned models. It is so because the model will fail to optimize on provided data since the data was either already observed (true data from basic sets) or the data will have less quality (pseudo-parallel data) compared to the data provided for the initial XLM 17 LM training.

\subsection{Results analysis}

As is apparent from the performance graph from Figure \ref{fig:XLM-R_comparison}, there is no visible improvement provided by the two-stage fine-tuning.

Like it was expected, the performance of MT-tuned models for language pairs with the target language covered by the initial XLM 17 training does differ from the performance provided by the models that were two-stage fine-tuned.

In addition to that 2-stage fine-tuned models also failed to provide performance improvement for 2 other language pairs. For the English-Czech language pair, it may be explained by the fact that the linguistic features that can be extracted from the provided corpora could have been already learned from the languages with similar linguistic features -- XLM 17 was trained, among other languages, on Polish data. For English-Basque there are two possible explanations of such behaviour: first, similar to English-Czech, the linguistic features could have been learned from another agglutinative language with SOV (subject-object-verb) word order -- XLM 17 was trained on Turkish data; second, shared sub-word vocabulary that is provided along with XLM 17 model is not optimized for the BPE encoding of the Basque language data and thus it is hard to learn the features specific to the Basque through the language modelling.

\begin{figure}[H]
    \centering
    \includegraphics[width=0.47\linewidth]{obrazky-figures/XLM-R_comparison_bleu.pdf}
    \includegraphics[width=0.47\linewidth]{obrazky-figures/XLM-R_comparison_chrf.pdf}
    \caption{Performance comparison of MT fine-tuning/2-stage fine-tuning.}
    \label{fig:XLM-R_comparison}
\end{figure}

\section{Findings}

I will provide a brief overview of the findings of the BDBNMT derived from the conducted experiments.

\begin{itemize}
    \item The higher bilingual dictionary coverage the bigger the yield the bilingual dictionary methods bring to the models' performance. 
    \item Target languages with weak inflection (e.g. Vietnamese) get a bigger performance boost from the utilization of pseudo-parallel data.
    \item Leveraging the pseudo-parallel data through the TLM objective is justified only when the languages that make up a language pair are linguistically close (e.g. English-French).
\end{itemize}

\chapter{Future work}

Even though the experiments conducted for this work demonstrate that the BDBNMT approach is applicable to many of the investigated cases, there exist a few additions to this approach that may positively affect the results which were not implemented and tested.

\section{Lemmatization}

It is obvious that the extension of bilingual dictionaries should increase the coverage of data and thus will be beneficial for the performance of models that adopt this approach. However, the induction of a ground-truth bilingual lexicon is not an easy task. Another way to improve the data coverage without any additions of ground-truth entries to the original dictionary is to lemmatize both the tokens in a monolingual corpus that were not covered by the initial dictionary and the keys of the bilingual dictionary itself. Consider the initial English-Czech dictionary \{"fox": "liška", "lazy": "liný", "jumped": "přeskočil"\} and the following pair of clean and noised versions of the same sentence:
\begin{quote}
    Brown fox jumps over the lazy dog. $\rightarrow$ Brown liška jumps over the liný dog.
\end{quote}
\noindent by the lemmatization of the initial dictionary we will obtain the following extension: \{"jump": ["přeskočil", "přeskočit"]\}, and after its application on the initial sentence we will obtain the sentence with the increased quantity of words translated by a bilingual dictionary:
\begin{quote}
    Brown fox jump over the lazy dog. $\rightarrow$ Brown liška přeskočit over the liný dog.
\end{quote}
Even though the form of the target word may not be correct, the greater bilingual dictionary coverage of the source monolingual dataset forces the model to drive the embeddings of similar words in different languages even closer during the pretraining.
\section{Synonyms utilization}

Another improvement that is even more reliant on a bilingual dictionary is that can be added in addition to previously described pseudo-parallel data generation and may be useful for extremely low-resource language pairs. This technique includes the usage of bilingual dictionaries in both directions: in opposition to the previously described process of generation of pseudo-parallel data we, instead of sampling the random ground-truth translation, now apply all of them by the creation of new target pseudo sentences for each available word translation. In addition to that we apply word mappings in the opposite direction thus also enriching the source language data. Consider the following vocabulary \{"country": ["země", "stát"], "land": "země", "ground": "země", "earth": "země",\}. Since the word "has "country" has two corresponding words in the target language, by the application of substitutions we will generate two new target language pseudo sentences, then with the application of a dictionary in the opposite direction three new source sentences will be created that will cover the words in the source language that, perhaps, are not present in corpus. This way we will make the embedding vector representation of words involved in substitutions extremely close to each other (up to the point where embeddings of these words can be the same), which may be harmful to the performance of a model trained on a sufficient amount of data -- even though the covered words may have the similar meaning they may be used in different contexts. However, as stated before, this technique may be useful for extremely low-resource languages, the corpus of which may not even include the words covered by a dictionary.

\chapter{Conclusion}

The goal of this work was to investigate how NMT models' performance can be improved by the application of bilingual dictionary based methods. In this work were considered two methods of bilingual dictionary utilization that can be applied in certain situations.

The first one assumes the usage of \textit{Anchored cross-lingual pretraining} (ACP) in conjunction with \textit{Anchored training} (AT). Both methods expect the usage of non-parallel corpora for training. This research surveyed the setting that was not covered by the work which introduced these methods -- the case when there are small-scale monolingual corpora. The research shows that there is no improvement that can ACP+AT training provide in this setting even when compared to the translations based solely on word mappings provided by the bilingual lexicon. 

The second method -- \textit{Cross-lingual language modelling (XLM) with pseudo-parallel data} -- assumes that there is a little amount of true parallel data available for Machine translation (MT) training and some amount of monolingual data available only in one of two languages. This technique consists of first generating the pseudo-parallel data based on a monolingual corpus and bilingual dictionary, and then leveraging this data through the language modelling objective. Conducted experiments proved that the addition of pseudo-parallel data to true parallel data is an effective method of boosting the model's performance: with the inclusion of a relatively small amount of extra monolingual data (100k source language sentences) it yielded improvement comparable to this achieved by the addition of true non-parallel data for both languages; with the bigger amount of additional monolingual data (1M source language sentences) it also led to performance improvement, however, it was more modest.

In addition to this survey, there was conducted the experiment that considers models that were trained with the ACP and AT stages as models for further fine-tuning with the MT objective. Pre-training with such objectives allows the model to also tune a decoder during the pre-training. The resulting performance of such models is comparable to this of the MT models pre-trained with the MLM objective on the same non-parallel data.

Another survey that was conducted for the method of XLM with the pseudo-parallel data was examining whether it is beneficial to leverage the pseudo-parallel data through the TLM objective. For the models trained in different settings, results show that it is advantageous to leverage the pseudo-parallel data this way when the linguistic distance between the source and target languages is relatively small -- the improvement was observable for the models for English-French and English-Czech language pairs.


% \bibliographystyle{unsrt}

%=========================================================================

% For compilation piecewise (see projekt.tex), it is necessary to uncomment it
