Deep Reinforcement Learning-Based Control Strategy for Boost Converter

doi:10.16182/j.issn1004731x.joss.22-0101

Abstract

Abstract:

In view of the problems of model uncertainty and nonlinearity in bus voltage stability control of Boost converter, an intelligent control strategy based on model-free deep reinforcement learning(RL) is proposed. RL double DQN(DDQN) algorithm and deep deterministic policy gradient(DDPG) algorithm are used, and the Boost converter controller is designed. The state, action space, reward function, and neural network are also designed to improve the dynamic performance of the controller. The joint simulation of the Boost converter model and RL agent is realized by RL modelica(RLM), a toolkit developed based on ModelicaGym. The proposed controller is compared with the double-loop PI controller, and the simulation shows that the bus voltage stability control based on the RL controller has better dynamic performance under three working conditions.

Key words: Boost converter, deep reinforcement learning, DDQN (double DQN) algorithm, DDPG (deep deterministic policy gradient) algorithm, joint simulation

CLC Number:

TP391.9

Yuxuan Dai, Chenggang Cui. Deep Reinforcement Learning-Based Control Strategy for Boost Converter[J]. Journal of System Simulation, 2023, 35(5): 1109-1119.

Figures/Tables 16

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Table 1

Table 2

RL training hyperparameters

参数	数值
学习率 $α$	0.001
DDQN折扣率 $γ 1$	0.9
DDPG折扣率 $γ 2$	0.98
批采样大小b	256
经验池大小B	2×10⁵

Table 2

Table 3

Fig. 8

Fig. 9

Fig. 10

Fig. 11

Fig. 12

Table 4

References 19

1	李洪恩, 胡健, 唐志扬, 等. 孤岛直流微电网DC-DC变换器的非线性控制技术[J]. 电源学报, 2020, 18(2): 4-14.
	Li Hongen, Hu Jian, Tang Zhiyang, et al. Nonlinear Control Technology for DC-DC Converter in Islanded DC Microgrid[J]. Journal of Power Supply, 2020, 18(2): 4-14.
2	Ghosh S K, Roy T K, Pramanik A H, et al. Design of Nonlinear Backstepping Double-Integral Sliding Mode Controllers to Stabilize the DC-Bus Voltage for DC-DC Converters Feeding CPLs[J]. Energies (S1996-1073), 2021, 14(20): 6753.
3	Boutchich N, Moufid A, Bennis N, et al. A Constrained MPC Approach Applied to Buck DC-DC Converter for Greenhouse Powered by Photovoltaic Source[C]// 2020 International Conference on Electrical and Information Technologies (ICEIT). Piscataway, NJ, USA: IEEE, 2020: 1-6.
4	陶彩霞, 王伟斌, 祝曦, 等. 带恒功率负载Buck变换器的模型预测控制[J]. 湖南大学学报(自然科学版), 2021, 48(8): 96-102.
	Tao Caixia, Wang Weibin, Zhu Xi, et al. A Model Predictive Control for Buck Converter with Constant Power Load[J]. Journal of Hunan University Natural Sciences, 2021, 48(8): 96-102.
5	周雪松, 李康, 马幼捷. DC-DC变换器滑模变结构控制研究[J]. 电力系统及其自动化学报, 2021, 33(3): 11-17.
	Zhou Xuesong, Li Kang, Ma Youjie. Research on Sliding-Mode Variable-Structure Control of DC-DC Converter[J]. Proceedings of the CSU-EPSA, 2021, 33(3): 11-17.
6	Fan Jingwen, Li Shihua, Wang Junxiao, et al. A GPI Based Sliding Mode Control Method for Boost DC-DC Converter[C]// 2016 IEEE International Conference on Industrial Technology (ICIT). Piscataway, NJ, USA: IEEE, 2016: 1826-1831.
7	Yfoulis C, Papadopoulou S, Voutetakis S. Robust Linear Control of Boost and Buck-Boost DC-DC Converters in Micro-grids with Constant Power Loads[J]. Energies (S1996-1073), 2020, 13(18): 4829.
8	杨航, 刘凌, 阎治安, 等. 双闭环Buck变换器系统模糊PID控制[J]. 西安交通大学学报, 2016, 50(4): 35-40, 67.
	Yang Hang, Liu Ling, Yan Zhian, et al. A Fuzzy PID Control Strategy for Buck Converter System of Double Closed Loop Circuits[J]. Journal of Xi'an Jiaotong University, 2016, 50(4): 35-40, 67.
9	Farsizadeh H, Gheisarnejad M, Mosayebi M, et al. An Intelligent and Fast Controller for DC/DC Converter Feeding CPL in a DC Microgrid[J]. IEEE Transactions on Circuits and Systems II: Express Briefs (S1549-7747), 2020, 67(6): 1104-1108.
10	Raj R N, Purushothaman K V, Singh N A. Adaptive TSK-Type Neural Fuzzy Controller for Boost DC-DC Converter[C]// 2017 IEEE International Conference on Circuits and Systems (ICCS). Piscataway, NJ, USA: IEEE, 2017: 441-446.
11	Khorashadizadeh S, Mahdian M. Voltage Tracking Control of DC-DC Boost Converter Using Brain Emotional Learning[C]// 2016 4th International Conference on Control, Instrumentation, and Automation (ICCIA). Piscataway, NJ, USA: IEEE, 2016: 268-272.
12	贾振宇, 刘子龙. 一种通过强化学习的四旋翼姿态控制算法[J]. 小型微型计算机系统, 2021, 42(10): 2074-2078.
	Jia Zhenyu, Liu Zilong. Quadrotor Attitude Control Algorithm Based on Reinforcement Learning[J]. Journal of Chinese Computer Systems, 2021, 42(10): 2074-2078.
13	董豪, 杨静, 李少波, 等. 基于深度强化学习的机器人运动控制研究进展[J]. 控制与决策, 2022, 37(2): 278-292.
	Dong Hao, Yang Jing, Li Shaobo, et al. Research Progress of Robot Motion Control Based on Deep Reinforcement Learning[J]. Control and Decision, 2022, 37(2): 278-292.
14	张耀中, 许佳林, 姚康佳, 等. 基于DDPG算法的无人机集群追击任务[J]. 航空学报, 2020, 41(10): 314-326.
	Zhang Yaozhong, Xu Jialin, Yao Kangjia, et al. Pursuit Missions for UAV Swarms Based on DDPG Algorithm[J]. Acta Aeronautica et Astronautica Sinica, 2020, 41(10): 314-326.
15	熊珞琳, 毛帅, 唐漾, 等. 基于强化学习的综合能源系统管理综述[J]. 自动化学报, 2021, 47(10): 2321-2340.
	Xiong Luolin, Mao Shuai, Tang Yang, et al. Reinforcement Learning Based Integrated Energy System Management: A Survey[J]. Acta Automatica Sinica, 2021, 47(10): 2321-2340.
16	周毅, 周良才, 丁佳立, 等. 基于深度强化学习的电网拓扑优化及潮流控制[J]. 上海交通大学学报, 2021, 55(增2): 7-14.
	Zhou Yi, Zhou Liangcai, Ding Jiali, et al. Power Network Topology Optimization and Power Flow Control Based on Deep Reinforcement Learning[J]. Journal of Shanghai Jiaotong University, 2021, 55(S2): 7-14.
17	Hajihosseini M, Andalibi M, Gheisarnejad M, et al. DC/DC Power Converter Control-Based Deep Machine Learning Techniques: Real-Time Implementation[J]. IEEE Transactions on Power Electronics (S0885-8993), 2020, 35(10): 9971-9977.
18	Gheisarnejad M, Farsizadeh H, Tavana M R, et al. A Novel Deep Learning Controller for DC-DC Buck-Boost Converters in Wireless Power Transfer Feeding CPLs[J]. IEEE Transactions on Industrial Electronics (S0278-0046), 2021, 68(7): 6379-6384.
19	Lukianykhin O, Bogodorova T. ModelicaGym: Applying Reinforcement Learning to Modelica Models[C]// 9th International Workshop on Equation-Based Object-Oriented Modeling Languages and Tools. New York, NY, USA: Association for Computing Machinery, 2019: 27-36.

参数	数值
电感L/mH	2
电容C/μF	1500
输入电压E/V	85
母线电压V_ref/V	170

工况	k_vp	k_vi	K_cp	K_ci
微源电压变化	100	30	0.005	2
阻性负载变化	30	500	0.01	10
恒功率负载变化	1	300	0.02	100

工况	指标	PI	DDQN	DDPG
一	MOVR	0.119 8	0.034 3	0.013 7
一	MOVD	0.296 1	0.016 7	0.014 1
二	MOVR	0.150 5	0.051 3	0.011 8
二	MOVD	0.304 8	0.057 3	0.059 5
三	MOVR	1.368 0	0.886 5	0.246 8
三	MOVD	4.731 0	1.396 0	1.001 5

[1]	Haotian Xu, Long Qin, Junjie Zeng, Yue Hu, Qi Zhang. Research Progress of Opponent Modeling Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2023, 35(4): 671-694.
[2]	Jiajie Shi, Peng Yang, Yannan Pi. Machine Learning-based Simulation Research of On-line Subway Pedestrian Flow Control [J]. Journal of System Simulation, 2023, 35(2): 386-395.
[3]	Sen Zhang, Mengyan Zhang, Jingping Shao, Jiexin Pu. Multi-UAVs 3D Path Planning Method Based on Random Strategy Search [J]. Journal of System Simulation, 2022, 34(6): 1286-1295.
[4]	Lingjia Ni, Xiaoxia Huang, Hongga Li, Zibo Zhang. Research on Fire Emergency Evacuation Simulation Based on Cooperative Deep Reinforcement Learning [J]. Journal of System Simulation, 2022, 34(6): 1353-1366.
[5]	Hongwei Wang, Peng Yang. Research on Optimization of Airport Cargo Business Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2022, 34(3): 651-660.
[6]	Qirui Li, Xinyi Peng. Job Scheduling and Simulation in Cloud Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2022, 34(2): 258-268.
[7]	Gao Ang, Dong Zhiming, Zhang Guohui, Liang Tao, Guo Qisheng. Research on Generation Technology of Computer Generated Force in LVC Training System [J]. Journal of System Simulation, 2021, 33(3): 745-752.
[8]	Zeng Bi, Fang Xiao, Kong Deshuai, Song Xiangxiang, Jia Zhengxuan, Lin Tingyu. A Data-Driven Modeling Method for Game Adversity Agent [J]. Journal of System Simulation, 2021, 33(12): 2838-2845.
[9]	Liu Hui, Pan Tinglong. Design of Multi -port Converter for DC Micro-grid System with Wind-Photovoltaic Storage [J]. Journal of System Simulation, 2019, 31(9): 1916-1924.
[10]	Li Yueqin, Xiao Fei, Fan Ming, Xu Xiaodong. Aerodynamic Noise Generating Mechanism of Corbula Type Throttle Valves [J]. Journal of System Simulation, 2018, 30(4): 1615-1621.
[11]	Song Zhongyue, Zhen Dong, Zhang Hao, Shi Zhanqun, Ma Jiaojiao. Research on Wind Turbine Test System Based on JointSimulation [J]. Journal of System Simulation, 2018, 30(2): 740-746.
[12]	Li Xiaoqiu, Zhang Xiaoping. Study on Nonlinear Characteristics of Buck-Boost Converter in Pulse Input Voltage [J]. Journal of System Simulation, 2017, 29(5): 1021-1027.
[13]	Xiao Yanjun, Feng Hua, Wang Zhao, Wang Yonggeng. Joint Dynamics Simulation Analysis and Optimization in Carrying Manipulator of Sound-absorbing Board Packing Device [J]. Journal of System Simulation, 2016, 28(9): 2118-2125.
[14]	Xia Kun, Zeng Yanneng, Ge Yue, Li Xiaodi. Simulation Study of Soft Switching Technology for Quasi-Z-source Boost Converter [J]. Journal of System Simulation, 2015, 27(3): 656-661.
[15]	Cai Jihong, Qing Duzheng, Xie Baodi. Research of Joint Simulation Platform Supporting Interoperability of LVC [J]. Journal of System Simulation, 2015, 27(1): 93-97.