Abstract:This paper addresses the configuration control problem of multi-UAV systems by proposing a control method based on the Stackelberg game model and hierarchical reinforcement learning. First, the system dynamics are modeled within a leader-follower framework, and the task is analyzed using the Stackelberg game model. A game-theoretic dynamic system with N+1 participants is constructed, transforming the problem into optimal control and differential game theory. Subsequently, optimal control and dynamic programming techniques are employed to derive the optimal strategies for each participant, along with the corresponding coupled Hamilton-Jacobi-Bellman (HJB) equations. Building on this, a two-stage hierarchical value iteration reinforcement learning algorithm is proposed to iteratively compute the optimal solution. During the value function approximation process, a neural network activation function is incorporated for fitting, thereby enhancing the computational efficiency and accuracy of the algorithm. Compared to traditional methods, the proposed algorithm effectively solves the Stackelberg-Nash equilibrium between the leader and multiple followers. Finally, numerical simulation experiments are conducted to validate the feasibility and effectiveness of the proposed approach in real-world tasks.