Deep Reinforcement Learning-Based Control Strategy for Boost Converter

doi:10.16182/j.issn1004731x.joss.22-0101

Abstract

Abstract:

In view of the problems of model uncertainty and nonlinearity in bus voltage stability control of Boost converter, an intelligent control strategy based on model-free deep reinforcement learning(RL) is proposed. RL double DQN(DDQN) algorithm and deep deterministic policy gradient(DDPG) algorithm are used, and the Boost converter controller is designed. The state, action space, reward function, and neural network are also designed to improve the dynamic performance of the controller. The joint simulation of the Boost converter model and RL agent is realized by RL modelica(RLM), a toolkit developed based on ModelicaGym. The proposed controller is compared with the double-loop PI controller, and the simulation shows that the bus voltage stability control based on the RL controller has better dynamic performance under three working conditions.

Key words: Boost converter, deep reinforcement learning, DDQN (double DQN) algorithm, DDPG (deep deterministic policy gradient) algorithm, joint simulation

CLC Number:

TP391.9

Yuxuan Dai, Chenggang Cui. Deep Reinforcement Learning-Based Control Strategy for Boost Converter[J]. Journal of System Simulation, 2023, 35(5): 1109-1119.

Figures/Tables 16

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Table 1

Table 2

RL training hyperparameters

参数	数值
学习率 $α$	0.001
DDQN折扣率 $γ 1$	0.9
DDPG折扣率 $γ 2$	0.98
批采样大小b	256
经验池大小B	2×10⁵

Table 2

Table 3

Fig. 8

Fig. 9

Fig. 10

Fig. 11

Fig. 12

Table 4

References 19

1	李洪恩, 胡健, 唐志扬, 等. 孤岛直流微电网DC-DC变换器的非线性控制技术[J]. 电源学报, 2020, 18(2): 4-14.
	Li Hongen, Hu Jian, Tang Zhiyang, et al. Nonlinear Control Technology for DC-DC Converter in Islanded DC Microgrid[J]. Journal of Power Supply, 2020, 18(2): 4-14.
2	Ghosh S K, Roy T K, Pramanik A H, et al. Design of Nonlinear Backstepping Double-Integral Sliding Mode Controllers to Stabilize the DC-Bus Voltage for DC-DC Converters Feeding CPLs[J]. Energies (S1996-1073), 2021, 14(20): 6753.
3	Boutchich N, Moufid A, Bennis N, et al. A Constrained MPC Approach Applied to Buck DC-DC Converter for Greenhouse Powered by Photovoltaic Source[C]// 2020 International Conference on Electrical and Information Technologies (ICEIT). Piscataway, NJ, USA: IEEE, 2020: 1-6.
4	陶彩霞, 王伟斌, 祝曦, 等. 带恒功率负载Buck变换器的模型预测控制[J]. 湖南大学学报(自然科学版), 2021, 48(8): 96-102.
	Tao Caixia, Wang Weibin, Zhu Xi, et al. A Model Predictive Control for Buck Converter with Constant Power Load[J]. Journal of Hunan University Natural Sciences, 2021, 48(8): 96-102.
5	周雪松, 李康, 马幼捷. DC-DC变换器滑模变结构控制研究[J]. 电力系统及其自动化学报, 2021, 33(3): 11-17.
	Zhou Xuesong, Li Kang, Ma Youjie. Research on Sliding-Mode Variable-Structure Control of DC-DC Converter[J]. Proceedings of the CSU-EPSA, 2021, 33(3): 11-17.
6	Fan Jingwen, Li Shihua, Wang Junxiao, et al. A GPI Based Sliding Mode Control Method for Boost DC-DC Converter[C]// 2016 IEEE International Conference on Industrial Technology (ICIT). Piscataway, NJ, USA: IEEE, 2016: 1826-1831.
7	Yfoulis C, Papadopoulou S, Voutetakis S. Robust Linear Control of Boost and Buck-Boost DC-DC Converters in Micro-grids with Constant Power Loads[J]. Energies (S1996-1073), 2020, 13(18): 4829.
8	杨航, 刘凌, 阎治安, 等. 双闭环Buck变换器系统模糊PID控制[J]. 西安交通大学学报, 2016, 50(4): 35-40, 67.
	Yang Hang, Liu Ling, Yan Zhian, et al. A Fuzzy PID Control Strategy for Buck Converter System of Double Closed Loop Circuits[J]. Journal of Xi'an Jiaotong University, 2016, 50(4): 35-40, 67.
9	Farsizadeh H, Gheisarnejad M, Mosayebi M, et al. An Intelligent and Fast Controller for DC/DC Converter Feeding CPL in a DC Microgrid[J]. IEEE Transactions on Circuits and Systems II: Express Briefs (S1549-7747), 2020, 67(6): 1104-1108.
10	Raj R N, Purushothaman K V, Singh N A. Adaptive TSK-Type Neural Fuzzy Controller for Boost DC-DC Converter[C]// 2017 IEEE International Conference on Circuits and Systems (ICCS). Piscataway, NJ, USA: IEEE, 2017: 441-446.
11	Khorashadizadeh S, Mahdian M. Voltage Tracking Control of DC-DC Boost Converter Using Brain Emotional Learning[C]// 2016 4th International Conference on Control, Instrumentation, and Automation (ICCIA). Piscataway, NJ, USA: IEEE, 2016: 268-272.
12	贾振宇, 刘子龙. 一种通过强化学习的四旋翼姿态控制算法[J]. 小型微型计算机系统, 2021, 42(10): 2074-2078.
	Jia Zhenyu, Liu Zilong. Quadrotor Attitude Control Algorithm Based on Reinforcement Learning[J]. Journal of Chinese Computer Systems, 2021, 42(10): 2074-2078.
13	董豪, 杨静, 李少波, 等. 基于深度强化学习的机器人运动控制研究进展[J]. 控制与决策, 2022, 37(2): 278-292.
	Dong Hao, Yang Jing, Li Shaobo, et al. Research Progress of Robot Motion Control Based on Deep Reinforcement Learning[J]. Control and Decision, 2022, 37(2): 278-292.
14	张耀中, 许佳林, 姚康佳, 等. 基于DDPG算法的无人机集群追击任务[J]. 航空学报, 2020, 41(10): 314-326.
	Zhang Yaozhong, Xu Jialin, Yao Kangjia, et al. Pursuit Missions for UAV Swarms Based on DDPG Algorithm[J]. Acta Aeronautica et Astronautica Sinica, 2020, 41(10): 314-326.
15	熊珞琳, 毛帅, 唐漾, 等. 基于强化学习的综合能源系统管理综述[J]. 自动化学报, 2021, 47(10): 2321-2340.
	Xiong Luolin, Mao Shuai, Tang Yang, et al. Reinforcement Learning Based Integrated Energy System Management: A Survey[J]. Acta Automatica Sinica, 2021, 47(10): 2321-2340.
16	周毅, 周良才, 丁佳立, 等. 基于深度强化学习的电网拓扑优化及潮流控制[J]. 上海交通大学学报, 2021, 55(增2): 7-14.
	Zhou Yi, Zhou Liangcai, Ding Jiali, et al. Power Network Topology Optimization and Power Flow Control Based on Deep Reinforcement Learning[J]. Journal of Shanghai Jiaotong University, 2021, 55(S2): 7-14.
17	Hajihosseini M, Andalibi M, Gheisarnejad M, et al. DC/DC Power Converter Control-Based Deep Machine Learning Techniques: Real-Time Implementation[J]. IEEE Transactions on Power Electronics (S0885-8993), 2020, 35(10): 9971-9977.
18	Gheisarnejad M, Farsizadeh H, Tavana M R, et al. A Novel Deep Learning Controller for DC-DC Buck-Boost Converters in Wireless Power Transfer Feeding CPLs[J]. IEEE Transactions on Industrial Electronics (S0278-0046), 2021, 68(7): 6379-6384.
19	Lukianykhin O, Bogodorova T. ModelicaGym: Applying Reinforcement Learning to Modelica Models[C]// 9th International Workshop on Equation-Based Object-Oriented Modeling Languages and Tools. New York, NY, USA: Association for Computing Machinery, 2019: 27-36.

参数	数值
电感L/mH	2
电容C/μF	1500
输入电压E/V	85
母线电压V_ref/V	170

工况	k_vp	k_vi	K_cp	K_ci
微源电压变化	100	30	0.005	2
阻性负载变化	30	500	0.01	10
恒功率负载变化	1	300	0.02	100

工况	指标	PI	DDQN	DDPG
一	MOVR	0.119 8	0.034 3	0.013 7
一	MOVD	0.296 1	0.016 7	0.014 1
二	MOVR	0.150 5	0.051 3	0.011 8
二	MOVD	0.304 8	0.057 3	0.059 5
三	MOVR	1.368 0	0.886 5	0.246 8
三	MOVD	4.731 0	1.396 0	1.001 5

[1]	Jiang Ming, He Tao. Solving the Vehicle Routing Problem Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(9): 2177-2187.
[2]	Li Junhui, Sun Songtao, Liu Fei. Research on Strong Real-time Synchronisation Algorithm for LVC Co-simulation [J]. Journal of System Simulation, 2025, 37(9): 2301-2314.
[3]	Ni Peilong, Mao Pengjun, Wang Ning, Yang Mengjie. Robot Path Planning Based on Improved A-DDQN Algorithm [J]. Journal of System Simulation, 2025, 37(9): 2420-2430.
[4]	Xie Xueyuan, Lin Chen, Wu Han, Zhao Qinglan, Gao Junfei, Hao Qiangguo, Zheng Xinqian. Research on Joint Simulation of Special Vehicle Engine Operation Characteristics Based on Virtual Driving Scenarios [J]. Journal of System Simulation, 2025, 37(8): 2163-2175.
[5]	Chen Zhen, Wu Zhuoyi, Zhang Lin. Research on Policy Representation in Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(7): 1753-1769.
[6]	Wu Guohua, Zeng Jiaheng, Wang Dezhi, Zheng Long, Zou Wei. A Quadrotor Trajectory Tracking Control Method Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(5): 1169-1187.
[7]	Bai Zhenzu, Hou Yizhi, He Zhangming, Wei Juhui, Zhou Haiyin, Wang Jiongqi. Optimization of Dynamic Weapon Target Assignment Considering Random Disturbances [J]. Journal of System Simulation, 2025, 37(12): 2967-2980.
[8]	Zheng Jiayu, Mai Zhuxue, Chen Zheyi. Optimization of Service Caching and Computation Offloading in Digital Twin Cloud-edge Networks [J]. Journal of System Simulation, 2025, 37(11): 2741-2753.
[9]	Di Jian, Wan Xue, Jiang Limei. Evolutionary Reinforcement Learning Based on Elite Instruction and Random Search [J]. Journal of System Simulation, 2025, 37(11): 2877-2887.
[10]	Liu Bingli, Wu Jiarong, Yang Lin, Yan Dinglin. Simulation of Three-degree-of-freedom Internal Mode Sliding Mode Control for Non-ideal Single-inductor Dual-output Boost Converter [J]. Journal of System Simulation, 2025, 37(10): 2500-2510.
[11]	Xu Zhongkai, Chu Chenyang, Xie Kai, Zhao Ruizhuo, Ke Wenjun. Optimization Dispatch Method for High-proportion Renewable Energy Power Systems Based on SC-PPO [J]. Journal of System Simulation, 2025, 37(10): 2511-2521.
[12]	Liang Xiuman, Liu Ziliang, Liu Zhendong. Path Planning of Improved RRT Algorithm Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(10): 2578-2593.
[13]	Jiang Jiachen, Jia Zhengxuan, Xu Zhao, Lin Tingyu, Zhao Pengpeng, Ou Yiming. Decision Modeling and Solution Based on Game Adversarial Complex Systems [J]. Journal of System Simulation, 2025, 37(1): 66-78.
[14]	Qin Baoxin, Zhang Yuxiao, Wu Sirui, Cao Weichong, Li Zhan. Intelligent Optimization of Coal Terminal Unloading Scheduling Based on Improved D3QN Algorithm [J]. Journal of System Simulation, 2024, 36(3): 770-781.
[15]	Li Ming, Ye Wangzhong, Yan Jiehua. Path Planning of Desert Robot Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2024, 36(12): 2917-2925.