Actor-Critic Reinforcement Learning for Helicopter Trim Actuators: An Optimal Control Approach with Hamilton-Jacobi-Bellman Principles

Temiz B., Bayram O., Koroglu O., GÜR E., İŞCAN M.

7th International Congress on Human-Computer Interaction, Optimization and Robotic Applications, ICHORA 2025, Ankara, Türkiye, 23 - 24 Mayıs 2025, (Tam Metin Bildiri) identifier

  • Yayın Türü: Bildiri / Tam Metin Bildiri
  • Doi Numarası: 10.1109/ichora65333.2025.11017124
  • Basıldığı Şehir: Ankara
  • Basıldığı Ülke: Türkiye
  • Anahtar Kelimeler: actor-critic neural network, Hamilton-Jacobi-Bellman optimality, helicopter flight stability, reinforcement learning control, trim actuator control
  • Yıldız Teknik Üniversitesi Adresli: Evet

Özet

This study presents a novel hybrid control strategy for helicopter trim actuators that fuses an actor- critic reinforcement learning (AC-RL) framework, based on Hamilton-Jacobi-Bellman (HJB) optimality principles, with unscented Kalman filtering (UKF) for robust state estimation. The proposed approach is designed to overcome the inherent challenges of nonlinear dynamics, disturbances, and parameter uncertainties that commonly affect rotorcraft trim systems. Utilizing comprehensive reference data from the flight data of the T625 Gökbey Helicopter, the proposed method achieves a remarkable 75% reduction in tracking error and a 59% decrease in control effort compared to conventional feedback-linearization-based PID controllers. Under constant parameter conditions using AC-RL-HJB framework, the tracking error decreases from 0.12 ± 0.17 to 0.03 ± 0.09, while the control input norm is reduced from 0.45 to 0.23. The proposed AC-RL-HJB utilizes constant default DC motor parameters to ensure reliable performance under realistic operating conditions in the training phase. Besides, the AC-RL-HJB architecture is designed to learn optimal control policies from real-time data, while the UKF effectively attenuates measurement noise and improves state estimation accuracy. This integration not only enhances tracking precision but also significantly reduces pilot workload under critical flight conditions. The contributions of this study include: (1) the development of a computationally efficient and robust AC-RL-HJB controller, (2) quantitative performance improvements that validate the approach's superiority over classical methods, and (3) a comprehensive experimental framework that demonstrates its practical viability in addressing real-world uncertainties. These results have profound implications for improving flight safety and operational efficiency in modern rotorcraft systems.