# Bachelor's thesis

This repository contains source parts for thesis paper compilation. Also, it contains tools that were utilized during the research: the modified XLM framework, utilized bilingual vocabularies and a few scripts.

### Compiling the thesis paper

```
cd doc/src
make
# compiled document will be named projekt.pdf
```

### Installation

Assuming that Python and Pip are installed, in order to install XLM package and all its dependencies, it is required to execute the following:

```
cd XLM
pip install numpy torch nltk safe_gpu
pip install -e .
./install-tools.sh
cd ..
```

To install the `pyonmttok` tokenizer required for bilingual dictionary based substitutions:

```
pip install pyonmttok
```

### Scripts

#### Data pre-processing

Data preprocessing (tokenization, extraction of BPE codes, application of BPE codes, data binarization) is done by `prepare_data.sh` and `prepare_data_separate.sh` scripts. Both scripts should be executed from this directory since they assume that tools required are located in `./XLM/tools` directory.

```
# data will be stored in data/bdbnmt/mlm/mlm/processed
# process the data considering that the entire training set is parallel
./scripts/prepare_data.sh --src en --tgt fr --data data/bdbnmt/mlm/mlm --codes 5000
# or
# process the data considering that only the first 130k sentences are parallel
./scripts/prepare_data_separate.sh --src en --tgt fr --data data/bdbnmt/mlm/mlm --codes 5000 --n_para 130000
```

Use `sub_by_dict.py` to apply the dictionary-based substitutions:

```
python3 scripts/sub_by_dict.py --dict_path <dictionary path> --src_path <source set path> --tgt_path <noisy set path>
```

Will search for the raw data in data/bdbnmt/mlm/mlm/en-fr and output the prepared data in data/bdbnmt/mlm/mlm/en-fr/processed


#### Executing (pre-)training

Scripts that suppose to execute the training are expecting that user will follow some hierarchical structure within the root `data` and `models` folders.
`run_pretrain.sh` script executes the model T/MLM pre-training, `run_tune_mt.sh` and `run_tune_bt.sh` execute MT training and BT training respectively.

```
# Example of TLM pre-training
# the experiment will be stored in models/bdbnmt/tlm/mlm/en-fr/e256_h6_h8_b64/test/
./scripts/run_pretrain.sh --src en --tgt fr --emb_dim 256 --n_layers 6 --n_heads 8 --batch_size 64 --data data/bdbnmt/mlm/mlm/processed --type bdbnmt --exp test --tlm
```

### XLM framework

Slightly modified version of XLM framework is stored in `XLM` directory. Modifications include the following:

- Addition of chrF evaluation metric for MT/BT training.
- Addition of bilingual dictionary based substitution preprocessing before the translation.

### Vocabularies

All used vocabularies are stored in `data/vocabs/en-{cs,vi,fr,eu}` directories.

### Evaluation

`sacreBLEU` chrF3/BLEU implementation was used for the models' performance evaluation.
