Technical Articles

Customer Insights

Training and Deploying a Real-Time Deep Reinforcement Learning Controller for an Interleaved Buck Converter

Deep Reinforcement Learning for Real-Time Interleaved Buck Converter Control


Authors:

  • Guangchen Wu, Xi’an Jiaotong University
  • Yufei Li, Xi’an Jiaotong University
  • Yu Fu, Xi’an Jiaotong University
  • Xirui Ding, Xi’an Jiaotong University
  • Hongji Wei, Xi’an Jiaotong University
  • Runtian Li, Xi’an Jiaotong University

16-minute read

Summary

  • Researchers at Xi’an Jiaotong University used Simulink to develop deep reinforcement learning controllers that improve the transient response of an interleaved buck converter under changing load conditions.
  • They compared DQN and PPO controllers, used transfer learning to adapt the DQN controller to changes in converter inductance, and deployed a lightweight version on an STM32H723 microcontroller.
  • The team’s embedded DQN controller met the converter’s real-time control constraints, and hardware-in-the-loop testing showed lower voltage overshoot and shorter settling time than a conventional PI controller.

Power converters used in applications such as electric vehicles, data centers, and communications infrastructure must maintain stable output voltage while responding to rapidly changing loads. Interleaved buck converters are commonly used in high-current, low-voltage applications because multiple converter phases can share the load current while reducing output-current ripple. However, controlling these converters becomes challenging when the load changes abruptly or component parameters drift from their nominal values.

For example, a conventional dual-loop proportional-integral (PI) controller can provide effective voltage regulation under nominal operating conditions. However, its performance can degrade during large load transients or when converter parameters change. Maintaining the desired transient response may then require retuning the controller.

Our research group, PRIME Lab (Power Electronics & Reinforcement Intelligence for Multi-scale Energy Conversion Lab) at Xi’an Jiaotong University (XJTU), led by Professor Yufei Li, found that deep reinforcement learning (DRL) can provide a more adaptable alternative. Instead of relying on a fixed control law, the DRL controller learns to map converter states to duty-cycle actions through interaction with the converter model.

In MATLAB® and Simulink®, we developed and compared two DRL approaches—Deep Q-Network (DQN) and Proximal Policy Optimization (PPO)—and trained controllers for each under changing load conditions (see Figure 1 for the DQN control architecture). In simulation, both controllers settled in about 0.7 ms after a 2 kW to 1 kW load step, compared with about 6.5 ms for a conventional PI controller. We investigated transfer learning to adapt a trained DQN controller to different inductor values before deploying a lightweight version of the controller on an STM32H723 microcontroller. The resulting controller averaged 35 μs per inference, less than the 50 μs control period of the 20 kHz converter, demonstrating that the DRL-based controller could execute within the converter’s real-time control constraints. We also used hardware-in-the-loop (HIL) testing to validate the embedded DQN controller in closed-loop operation and compare its transient response with that of a conventional PI controller.

Interleaved buck converter and dual DQN controller architecture with online and target Q-networks.

Figure 1. Interleaved buck converter and DQN control architecture. The DQN architecture used in this work has separate online and target Q-networks to improve training stability.

We modeled a two-phase interleaved buck converter in Simulink using Simscape Electrical™, with a 48 V input and a 12 V output reference. The two converter phases use the same duty cycle, with their switching signals offset by 180° to share the load current and reduce output ripple. We then used Reinforcement Learning Toolbox™ to define the environment (the observations, actions, and reward exchanged between the agent and the converter model) and to create the DQN and PPO controllers. The environment provides four observations used by the agent to select an action:

  • The output voltage
  • Its error from the 12 V reference
  • The integral of that error
  • The inductor current in the first phase

The DQN controller chooses from a discrete set of duty-cycle actions using a Q-network, a neural network that estimates the expected cumulative reward for each available action given the current state. PPO uses an actor-critic architecture, in which one network selects the control action, and another evaluates the resulting behavior. Unlike DQN, which selects from a predefined set of duty-cycle values, PPO can generate a continuous duty-cycle action, essentially any value within the allowed duty-cycle range. We trained both reinforcement learning agents over 200 episodes through repeated interaction with the Simulink converter model (Figure 2), allowing them to learn a control policy without explicitly deriving a control law for each operating condition.

A Simulink model connecting the RL agent, reward function, PWM generator, and interleaved buck converter.

Figure 2. Simulink model for training the DRL controller. The RL Agent block (center) receives observations from the converter model and a reward based on voltage error, outputting a duty-cycle action to the PWM generator that drives the interleaved buck converter.

Designing the reward function was one of the most challenging aspects of the training process because it translates the control objective into the signal the agent learns from. We based the reward primarily on the deviation of the output voltage from its 12 V reference. We used threshold-based reward shaping to provide a progressively larger reward as the voltage error fell within tighter bands, while applying a penalty proportional to the error. This approach enabled the agent to distinguish between larger transient errors and smaller steady-state errors. This encouraged the controllers to respond quickly to load changes while maintaining accurate voltage regulation after the transient. After the controllers were trained, we evaluated the two approaches under identical operating conditions, comparing their training behavior and transient response (Figure 3). Although PPO trained faster and showed slightly lower overshoot, we ultimately selected DQN for further development because its discrete-action approach offered a simpler implementation for the resource-constrained STM32H723 microcontroller.

A line chart comparing episode reward during DQN and PPO training over 200 episodes.

Figure 3. Training performance of the DQN and PPO controllers in Simulink. The training process required approximately 2 hours for DQN and 0.5 hours for PPO, with PPO reaching a high episode reward in fewer episodes.

Adapting the Controller with Transfer Learning

A key challenge in applying a learned controller to power electronics is adapting it when converter parameters change. To address this issue, we investigated whether a pretrained controller could be adapted to a new operating condition through transfer learning. We started with a DQN trained for the nominal converter and used its learned network as the starting point for training under different inductance values. This allowed the controller to retain useful control behavior from the original task while adapting to the changed converter characteristics.

We found that the pretrained controller could adapt with substantially less additional training. Specifically, training a DQN from scratch required more than 150 episodes to reach a relatively high reward level, while the transfer-learning approach reached a comparable level in approximately 40 episodes (Figure 4). More importantly, the adapted controller also demonstrated improved transient performance. Simulations in Simulink showed that, for a load transient from 2 kW to 1 kW with inductance reduced by 40% from its nominal value, voltage overshoot decreased from 2.364 V with training from scratch to 2.033 V with transfer learning, while settling time decreased from 0.513 ms to 0.469 ms.

A line chart comparing DQN episode reward for training from scratch and transfer learning.

Figure 4. Episode reward for a DQN controller adapting to a new inductance value, comparing training from scratch with transfer learning from a pretrained agent.

Deploying the DQN Controller on the Microcontroller

After demonstrating that the DQN controller could adapt to changes in converter inductance, we moved from simulation toward embedded implementation. We selected a lightweight version of the trained DQN controller for deployment on an STM32H723 microcontroller. We used Deep Learning Toolbox Converter for TensorFlow Models to export the trained Q-network from MATLAB in TensorFlow™ format, converted it to a Keras H5 model using Python®, and then imported it into STM32CubeMX through X-CUBE-AI, which generated the inference code for integration into the STM32 project (Figure 5). The resulting network required roughly 5,400 multiply-accumulate operations per inference, about 20 KB of flash memory, and 512 bytes of RAM, well within the STM32H723’s resource constraints.

The STM32CubeMX X-CUBE-AI configuration for deploying the trained network to an STM32H723 microcontroller.

Figure 5. The STM32CubeMX environment showing the configuration for X-CUBE-AI, which converts trained neural networks into optimized C code and libraries that can run inference on an STM32 microcontroller.

Deploying the controller also required integrating the DQN’s output with the converter’s two-phase PWM control. At each control step, the DQN generated a new duty-cycle command that had to be applied to both phases while preserving their required 180° phase shift. We implemented the embedded control logic to coordinate the DQN inference with the PWM updates, allowing the trained controller to operate as part of the converter’s real-time control loop. The PWM timer peripherals were configured using STM32CubeMX, while the coordination logic was manually implemented in C. Specifically, the STM32H723 TIM1 peripheral was used, with interrupt-driven CCR1 updates for duty-cycle adjustment and DMA-triggered CCR2 updates to maintain the required 180° phase shift.

Integrating the DQN inference with the PWM updates provided a practical bridge between the controller developed and validated in Simulink and its embedded implementation on the STM32H723.

Validating the Embedded Controller with HIL

With the DQN controller running on the STM32H723, we set out to evaluate its behavior in a real-time closed-loop environment via HIL testing. To do this, we used MATLAB Coder™ and Simulink Coder™ to generate code from the Simulink converter model. We compiled the generated code into an executable model and ran it on a ModelingTech MT 6020 real-time simulator, which was linked to the STM32 running the embedded DQN controller (Figure 6).

The HIL test setup with an MT 6020 simulator, STM32H723 board, host PC, interface board, and oscilloscope.

Figure 6. The HIL testing setup includes the ModelingTech MT 6020 real-time simulator, an interface board connecting it to the STM32H723 running the embedded DQN controller, a host PC for configuration and monitoring, and an oscilloscope for capturing the converter’s transient response.

We compared the embedded DQN controller with the conventional dual-loop PI controller under a variety of load transients (Figure 7). The DQN controller achieved lower voltage overshoot and shorter settling time than the conventional PI controller, further validating the practicality of the DRL approach.

Four HIL plots comparing PI and DQN output-voltage responses to different load transients.

Figure 7. Output voltage as a function of time for the PI and DQN controllers in HIL tests of four different load transients.

Looking Ahead

Our results demonstrated that DRL can be applied to power-electronics control from initial modeling and training through embedded implementation and real-time HIL validation. Using MATLAB and Simulink, we developed DQN and PPO controllers, evaluated their performance, and used transfer learning to adapt a trained DQN controller to changes in converter inductance. We then deployed and ran a lightweight DQN on an STM32H723. HIL testing showed that the embedded controller operated within the real-time closed-loop environment and delivered better transient performance than a conventional PI controller. The results helped our team earn First Prize in the “Delta Cup” AI and Power Electronics sector of the 11th University Power Electronics Application Design Competition.

Our next step is to extend our unified MATLAB and Simulink workflow to include embedded deployment. We plan to use Embedded Coder® and STM32 hardware support to generate embedded code directly from the trained controller, reducing the intermediate model conversion steps required by our current TensorFlow- and Keras-based deployment path. We also plan to move beyond HIL testing to experiments with the physical power converter, where we can evaluate the complete system, including sensing, PWM generation, gate driving, and the power stage. In addition, we will investigate network compression techniques to reduce inference time and extend the approach to converters operating at higher switching frequencies. With these capabilities in place, we can move from HIL validation to testing DRL-based control on a complete power converter and under a broader range of changing operating conditions.

Wu, Guangchen, Yufei Li, and Yu Fu. “A Real-Time MCU-Embedded Deep Reinforcement Learning-Based Control Method for Interleaved Buck Converters.” 2026 Energy Conversion Congress & Expo Europe (ECCE Europe), September 2026, 1–6. https://doi.org/10.1109/ecceeurope67962.2026.11738027.

Learn More

Published 2026


Keep Exploring