Abstract:This paper proposes a reinforcement learning-based method to address the flight control problem for multi-UAV formation. The proposed approach integrates curriculum learning, the leader-follower model, and the proximal policy optimization (PPO) method. Firstly, based on curriculum learning, the complex formation control task is decomposed into two learning stages. In the first stage, the leader UAV is trained using the PPO method to fly along a preset trajectory. Subsequently, the follower UAVs are trained in the second stage. During this stage, the control policy of the leader UAV is fixed as the neural network obtained in the first stage, and the delayed position of the leader is used as the tracking target for the followers. Based on partially observable information, the reward function was elaborately designed to guide the UAVs to maintain a stable linear formation during flight. To validate the effectiveness of the proposed method, we conducted simulations of a four-UAV formation performing a complex “8” shaped flight pattern in a three-dimensional space using the Unity software. The results demonstrate that, compared to traditional control methods, our approach enables agents to learn effective strategies through interaction with the environment via a relatively simple training process, without the need to establish precise mathematical models. This method simplifies the complexity of formation control and provides a novel solution for UAV formation control.