# Integration of Amazon Computational Cloud into k-Dispatch

## Introduction

This repository contains the bachelor thesis focusing on the integration of the Amazon Computational Cloud services into k-Dispatch.

This repository's main component is a CLI application created in Python using the prompt_toolkit library. The application employs the AWS ParallelCluster Python library, which is designed to facilitate the creation and management of high-performance computing (HPC) clusters within the AWS environment. The app implements its functionality, and uses it to simplify the processes of creating, listing, updating, and deleting HPC clusters. 

The functionality is driven by a configuration file, in which the details of the cluster setup are specified, making it easier than ever to manage your HPC resources. The app also provides functionality to generate these configuration files, based on the configuration proces of the AWS ParallelCluster CLI counterpart to the Pythonic library used in this application. It expands its functionality to allow user to create clusters with shared storage with either the AWS Elastic Block Storage (EBS), or Elastic File System (EFS), and other minor functionality, such as generation of SSH key pairs for the AWS EC2 service, upon which the individual nodes of the cluster operate.

In the "examples" directory is a example-config.yaml file, which is capable of generating an HPC Cluster capable of running jobs submitted by the k-Dispatch system. It uses the EBS storage volumes for storing the Matlab Runtime installation, and the specific binaries necessary for its operability. This configuration file was not generated by the CLI app itself, however the app provides the full suite of functionality to generate such file for yourself. It is also possible to generate a cluster from this file with full operability necessary for the k-Dispatch system from within the App itself.

## Repository structure

    +--Data             - Example data, Measured results, etc.
    +--Literature       - Publications, references, manuals, etc.
    +--Sources          - Root folder for the sources.
       +--config        - Modules which provide dictionaries, which are concated together to create cluster configs.
       +--menu         - Provide options for navigating functionality of cluster operiatons/config creation
       +--wrapper      - Functionality of spedific parts of the AWS system necessary for cluster operations.
    +--Thesis           - Latex / MS Word sources of the thesis.
    +--Misc             - Other auxiliary materials.
    Readme.md           - Read me file


## Pre-build requirements

Before the install it is necessary for the user to have AWS credentials to have configured on their device. This is easily achievable with the AWS CLI tool. The tutorial on how to install this tool can be found here:

https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html

The credentials configuration is provided from within the files that are maintained by the AWS CLI app. These include individual profile settings, where each profile represents one set of credentials for access to the AWS services. More on how to configure these profiles can be found here:

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html

## Build instruction

1. (Optional) It is often good practice to create a virtual environment for your Python projects to manage dependencies separately. This is possible using the venv module of the Python programming language. To do so, the user needs to choose a path, in which this environment will reside, and then enter the following command:

```
python -m venv /path/to/environment/.venv
```

Afterwards it is necessary to activate this environment using:

Windows:

```
.\env\Scripts\activate
```

MacOS/Linux:

```
source env/bin/activate
```

2. In the requirements.txt file, there are a number of dependencies required for this application to run. These can be installed using the command: 

```
pip install -r requirements.txt
```

3. Afterwards, the application can be run by using:

```
python /path/to/Sources/main.py
```

## Usage instruction

After the start of the application, the following options appear for the user:

```
[1] List clusters
[2] Create new cluster
[3] Find cluster
[4] Delete cluster
[5] Create new cluster config
[6] Quit
```

The user is now asked to input one of the options, designated by the number next to it in brackets, to choose the function they want to use. The first 4 options deal with cluster management. The user may be asked for additional info, such as cluster name, or cluster config file, depending on the chosen option

With the 5th option, the user has the capability to create a brand new configuration file. The application starts asking questions surrounding the configuration necessary to create a new cluster such as queue configuration, shared storage configuration, network configuration, etc. 

There is a possibility of some questions needing specific answers based on the current configuration part. In this case, there will be a tip as to what are the possible options within rounded brackets like these () next to the question.

Many of the options also have default parameters, when the user does not input any answer. In that case the default option shall be shown within square brackets like these [].

## Author information

 * Name: Patrik Kupčo
 * Email: kupco.patrik.16@gmail.com
 * Data: 2022/2023

