\documentclass[../xlapes02]{subfiles}
\begin{document}

    \chapter{Introduction}\label{ch:introduction}


    \section{Background}\label{sec:background}
% Portfolio Allocation
    The \emph{Portfolio allocation problem} is to spread appropriate finite cash budget into financial instruments~\cite{Model-Free-Reinforcement-Learning-for-Asset-Allocation}.\ Under the financial instruments, we can imagine Stocks, Bonds, Mutual Funds, Commodities, Derivatives, Real Estate Investments Trusts (REITs), Exchange-Traded Funds (ETFs), and many more.\ The outcome should be to increase the initial capital over the course of a selected investing horizon, which can vary from a few days to decades.\ Portfolio management is essential for investors, particularly those who manage large sums of money such as institutional investors, pension funds, and wealthy individuals.\ While allocating assets instead of cash one must think about minimizing risk and maximizing the expected return on the investment.\ For that, the key considered strategy is \emph{diversification}, which involves spreading investments across different instrument classes and markets in order to reduce the overall risk of the portfolio.\ A~portfolio full of different assets can change over time due to market conditions, where the value of other assets may increase or decrease, which may cause the portfolio to become imbalanced. \emph{Re-balancing} ensures that the portfolio remains aligned with the investor's goals and risk tolerance.

    Among other portfolio allocation strategies could be mentioned \emph{Modern portfolio theory (MPT)} to optimally allocate assets in a portfolio~\cite{enwiki:1043516653}.\ MPT uses statistical tools to determine the efficient frontier, which is the set of optimal portfolios that offer the highest expected return for a given level of risk, or the lowest risk for a given level of expected return~\cite{sirucek-2015}.\ Another approach is \emph{Mean-variance optimization}, which uses mathematical models to determine the optimal portfolio based on an investor's risk tolerance and expected returns~\cite{meanvarianceportfoliooptimazation}.

    These approaches are not too appropriate for portfolio management, because the stock market is stochastic, volatile, quickly changing, and uncertain environment.\ These strategies are not flexible enough to adapt to the changing environment like the stock market, because they assume the future will be similar to the past, which may not always be accurate.

% Reinforcement learning
    So, the most recent state-of-the-art portfolio management strategies are based on machine learning techniques. \emph{Reinforcement learning (RL)} is a type of machine learning that is well-suited for solving problems involving decision-making and control~\cite{sutton2018reinforcement}.\ In the context of portfolio allocation, RL can be used to optimize the allocation of assets in a portfolio in order to maximize returns or minimize risk.\ RL algorithms can learn from historical data and adapt to changing market conditions, which can lead to more efficient and profitable portfolio management.\ The benefits of RL have been used in many different fields, such as robotics, games, and finances.

    In the last decade, RL has become popular, because of its ability to learn difficult tasks in a variety of domains without knowing the environment model~\cite{sutton2018reinforcement}.\ RL has advantages, such as flexibility, adaptability, and utilization of various information like e.q.\ experience gained from the environment under certain conditions.\ The agent is trained under a certain policy in a particular environment, which is modeled using \emph{Markov Decision Process (MDP)}.\ MDP is a mathematical framework for modeling sequential decision-making problems~\cite{rao2022foundations}.\ MDP can be used to model the fully observable environment, where the agent can observe the state of the environment.\ If the environment is not fully observable, then the agent can observe only a part of the state of the environment, which is called \emph{partially observable Markov decision process (POMDP)}~\cite{kurniawati2021partially}.\ In finances, the environment is usually fully observable, because the agent can observe the state of the environment.\ MDP is composed of the following elements:
    \begin{enumerate}[label=\textbf{\arabic*}., ref=\arabic*]
        \item \textbf{State:} The state is the current situation of the environment.
        \item \textbf{Action:} The action is the decision that the agent can take.
        \item \textbf{Reward:} The reward is the feedback that the agent receives after taking an action.
        \item \textbf{Transition:} The transition is the change of the state after taking an action.
    \end{enumerate}
    In agent training we handle the following problems:
    \begin{itemize}
        \item \textbf{State space}\\ The state space is a finite set of all possible configurations of the environment.\ In the context of portfolio allocation, the state space can be defined as the finite set of all possible instrument features (fundamental and technical analysis) and their weights in the portfolio.
        \item \textbf{Action space} \\ Action space should be designed so that the agent weights the assets in the portfolio.\ Here the question is: Should be this asset in the portfolio and if yes, what is the weight of this asset in the portfolio?\ These decisions are crucial for the performance of the agent.\ It is really difficult to find the optimal policy for the portfolio allocation because the agent has to choose between multiple assets with various differences in information about the assets.\ Also, actions should be considered profitable and safe in the long term, which means that the agent usually has to make decisions based on long-term rewards or on the defined investment horizon.
        \item \textbf{Reward function}\\ The reward should reflect the agent's performance in the environment.\ Is the current portfolio value increasing or decreasing after the agent takes actions proposed by the policy?
    \end{itemize}

    When the state space is too large, then is merely impossible to be explored with the limited computational resources \emph{Deep Reinforcement Learning (DRL)} can be used.\ DRL is a subfield of Reinforcement Learning (RL) that combines the use of deep neural networks with RL algorithms.\ In traditional RL, the agent's policy and value functions are typically represented by simple, hand-designed features or a small number of parameters.\ In contrast, DRL uses deep neural networks to represent these functions, allowing the agent to learn from high-dimensional and complex inputs.\ DRL algorithms are used to train agents to perform a wide range of tasks, such as playing video games, controlling robotic arms, and driving cars.\ There are several popular algorithms in DRL, such as: \emph{Deep Q-Network (DQN)}, \emph{Deep Deterministic Policy Gradient (DDPG)}, \emph{Proximal Policy Optimization (PPO)}, \emph{Soft Actor-Critic (SAC)}, and \emph{Twin Delayed Deep Deterministic Policy Gradient (TD3)}.


    \section{Limitations}\label{sec:limitations}

    \begin{enumerate}
        \item \textbf{Data availability:} DRL models require large amounts of historical data to train effectively, which may be difficult to obtain for certain assets or markets.
        \item \textbf{Model Over-fitting:} DRL models can easily over-fit to the training data, leading to poor performance on unseen data.
        \item \textbf{High computational cost:} DRL models can require significant computational resources to train agents.
        \item \textbf{Risk management:} DRL models may not be able to effectively handle risk management, such as different market situations (Market sentiment, Bull and Bear markets).
    \end{enumerate}


    \section{Aim of the Thesis}\label{sec:aim-of-the-thesis}
    We will evaluate the performance of portfolio allocation methods based on DRL and compare them to traditional portfolio optimization techniques (MPT, Mean-Variance).\ Our goal is to determine the potential of DRL for portfolio allocation and identify the limitations of DRL-based portfolio allocation methods for future research.

    The thesis objectives are:
    \begin{itemize}
        \item \textbf{Experimental evaluation \& Benchmarks}\\ Compare existing portfolio allocation agents.\ Evaluate the performance of the RL agents by comparing them with the baseline portfolio management strategies, such as MPT, Mean-Variance Optimization, and indexes (DJI, Nasdaq-100, S\&P500, RUSSEL2000).
        \item \textbf{Dataset}\\ Create a suitable datasets for the portfolio allocation problem.\ Datasets will be focused on the company's financial data, such as fundamental and technical analysis data.
        \item \textbf{Reimplementation}\\ Try to improve current agents (Portfolio Allocation agent from \textbf{FinRL}~\cite{finrl-portfolio-allocation-2020}) with new datasets, focusing on Data Engineering and different DRL algorithms.
    \end{itemize}

    The thesis is be implemented using the programming language Python3 and open-source libraries such as NumPy, Pandas, Stable Baselines3, and OpenAI Gym.
\end{document}
