Deep Reinforcement Learning in Multiple UAV-and-RIS Assisted Cognitive Radio System

Abstract

Cognitive radio (CR) systems enable dynamic spectrum sharing but face substantial challenges in optimizing the rates of secondary users (SUs), particularly in scenarios where multiple SUs compete for the limited resources of the primary user (PU). To address this issue, we propose a multi-unmanned aerial vehicle (UAV)-assisted CR system in which reconfigurable intelligent surfaces (RISs) are mounted on UAVs to enhance spectral efficiency. Furthermore, we cast this challenge as a multi-agent Markov decision process (MDP), providing a formal framework to explore the critical trade-off between independent decision-making and centralized coordination. Consequently, we leverage established deep reinforcement learning algorithms to probe this trade-off. To provide a comprehensive performance evaluation, we adopt a Multi-Agent Proximal Policy Optimization (MAPPO) algorithm to maximize the sum rate of the proposed system. Numerical results demonstrate that the developed UAV-RIS-assisted system adopting the MAPPO algorithm can achieve a faster convergent speed and higher sum rate when compared with that adopting Independent Proximal Policy Optimization (IPPO) and MAPPO with a clipping scheme. In addition, for the MAPPO with a clipping scheme, a selected moderate clipping parameter can effectively balance the trade-off between training stability and learning efficiency.