基于深度强化学习的Boost变换器控制策略

doi:10.16182/j.issn1004731x.joss.22-0101

摘要/Abstract

摘要：

针对Boost变换器母线电压稳定控制存在模型不确定性和非线性的问题，提出了一种基于无模型深度强化学习的智能控制策略。结合强化学习DDQN(double DQN)算法与DDPG算法设计了Boost变换器控制器，包括了状态、动作空间、奖励函数以及神经网络的设计以提高控制器动态性能；基于ModelicaGym库开发工具包reinforment learning modelica(RLM)实现了Boost变换器模型与强化学习智能体的联合仿真。通过与双环PI控制器的对比仿真表明：强化学习控制器在三种工况下的母线电压稳定控制结果具有更好的动态性能。

关键词: Boost变换器, 深度强化学习, DDQN算法, DDPG算法, 协同仿真

Abstract:

In view of the problems of model uncertainty and nonlinearity in bus voltage stability control of Boost converter, an intelligent control strategy based on model-free deep reinforcement learning(RL) is proposed. RL double DQN(DDQN) algorithm and deep deterministic policy gradient(DDPG) algorithm are used, and the Boost converter controller is designed. The state, action space, reward function, and neural network are also designed to improve the dynamic performance of the controller. The joint simulation of the Boost converter model and RL agent is realized by RL modelica(RLM), a toolkit developed based on ModelicaGym. The proposed controller is compared with the double-loop PI controller, and the simulation shows that the bus voltage stability control based on the RL controller has better dynamic performance under three working conditions.

Key words: Boost converter, deep reinforcement learning, DDQN (double DQN) algorithm, DDPG (deep deterministic policy gradient) algorithm, joint simulation

中图分类号:

TP391.9

戴宇轩, 崔承刚. 基于深度强化学习的Boost变换器控制策略[J]. 系统仿真学报, 2023, 35(5): 1109-1119.

Yuxuan Dai, Chenggang Cui. Deep Reinforcement Learning-Based Control Strategy for Boost Converter[J]. Journal of System Simulation, 2023, 35(5): 1109-1119.

图/表 16

图1

图2

图3

图4

图5

图6

图7

表1

表2

RL训练超参数

参数	数值
学习率 $α$	0.001
DDQN折扣率 $γ 1$	0.9
DDPG折扣率 $γ 2$	0.98
批采样大小b	256
经验池大小B	2×10⁵

表2

表3

图8

图9

图10

图11

图12

表4

参考文献 19

1	李洪恩, 胡健, 唐志扬, 等. 孤岛直流微电网DC-DC变换器的非线性控制技术[J]. 电源学报, 2020, 18(2): 4-14.
	Li Hongen, Hu Jian, Tang Zhiyang, et al. Nonlinear Control Technology for DC-DC Converter in Islanded DC Microgrid[J]. Journal of Power Supply, 2020, 18(2): 4-14.
2	Ghosh S K, Roy T K, Pramanik A H, et al. Design of Nonlinear Backstepping Double-Integral Sliding Mode Controllers to Stabilize the DC-Bus Voltage for DC-DC Converters Feeding CPLs[J]. Energies (S1996-1073), 2021, 14(20): 6753.
3	Boutchich N, Moufid A, Bennis N, et al. A Constrained MPC Approach Applied to Buck DC-DC Converter for Greenhouse Powered by Photovoltaic Source[C]// 2020 International Conference on Electrical and Information Technologies (ICEIT). Piscataway, NJ, USA: IEEE, 2020: 1-6.
4	陶彩霞, 王伟斌, 祝曦, 等. 带恒功率负载Buck变换器的模型预测控制[J]. 湖南大学学报(自然科学版), 2021, 48(8): 96-102.
	Tao Caixia, Wang Weibin, Zhu Xi, et al. A Model Predictive Control for Buck Converter with Constant Power Load[J]. Journal of Hunan University Natural Sciences, 2021, 48(8): 96-102.
5	周雪松, 李康, 马幼捷. DC-DC变换器滑模变结构控制研究[J]. 电力系统及其自动化学报, 2021, 33(3): 11-17.
	Zhou Xuesong, Li Kang, Ma Youjie. Research on Sliding-Mode Variable-Structure Control of DC-DC Converter[J]. Proceedings of the CSU-EPSA, 2021, 33(3): 11-17.
6	Fan Jingwen, Li Shihua, Wang Junxiao, et al. A GPI Based Sliding Mode Control Method for Boost DC-DC Converter[C]// 2016 IEEE International Conference on Industrial Technology (ICIT). Piscataway, NJ, USA: IEEE, 2016: 1826-1831.
7	Yfoulis C, Papadopoulou S, Voutetakis S. Robust Linear Control of Boost and Buck-Boost DC-DC Converters in Micro-grids with Constant Power Loads[J]. Energies (S1996-1073), 2020, 13(18): 4829.
8	杨航, 刘凌, 阎治安, 等. 双闭环Buck变换器系统模糊PID控制[J]. 西安交通大学学报, 2016, 50(4): 35-40, 67.
	Yang Hang, Liu Ling, Yan Zhian, et al. A Fuzzy PID Control Strategy for Buck Converter System of Double Closed Loop Circuits[J]. Journal of Xi'an Jiaotong University, 2016, 50(4): 35-40, 67.
9	Farsizadeh H, Gheisarnejad M, Mosayebi M, et al. An Intelligent and Fast Controller for DC/DC Converter Feeding CPL in a DC Microgrid[J]. IEEE Transactions on Circuits and Systems II: Express Briefs (S1549-7747), 2020, 67(6): 1104-1108.
10	Raj R N, Purushothaman K V, Singh N A. Adaptive TSK-Type Neural Fuzzy Controller for Boost DC-DC Converter[C]// 2017 IEEE International Conference on Circuits and Systems (ICCS). Piscataway, NJ, USA: IEEE, 2017: 441-446.
11	Khorashadizadeh S, Mahdian M. Voltage Tracking Control of DC-DC Boost Converter Using Brain Emotional Learning[C]// 2016 4th International Conference on Control, Instrumentation, and Automation (ICCIA). Piscataway, NJ, USA: IEEE, 2016: 268-272.
12	贾振宇, 刘子龙. 一种通过强化学习的四旋翼姿态控制算法[J]. 小型微型计算机系统, 2021, 42(10): 2074-2078.
	Jia Zhenyu, Liu Zilong. Quadrotor Attitude Control Algorithm Based on Reinforcement Learning[J]. Journal of Chinese Computer Systems, 2021, 42(10): 2074-2078.
13	董豪, 杨静, 李少波, 等. 基于深度强化学习的机器人运动控制研究进展[J]. 控制与决策, 2022, 37(2): 278-292.
	Dong Hao, Yang Jing, Li Shaobo, et al. Research Progress of Robot Motion Control Based on Deep Reinforcement Learning[J]. Control and Decision, 2022, 37(2): 278-292.
14	张耀中, 许佳林, 姚康佳, 等. 基于DDPG算法的无人机集群追击任务[J]. 航空学报, 2020, 41(10): 314-326.
	Zhang Yaozhong, Xu Jialin, Yao Kangjia, et al. Pursuit Missions for UAV Swarms Based on DDPG Algorithm[J]. Acta Aeronautica et Astronautica Sinica, 2020, 41(10): 314-326.
15	熊珞琳, 毛帅, 唐漾, 等. 基于强化学习的综合能源系统管理综述[J]. 自动化学报, 2021, 47(10): 2321-2340.
	Xiong Luolin, Mao Shuai, Tang Yang, et al. Reinforcement Learning Based Integrated Energy System Management: A Survey[J]. Acta Automatica Sinica, 2021, 47(10): 2321-2340.
16	周毅, 周良才, 丁佳立, 等. 基于深度强化学习的电网拓扑优化及潮流控制[J]. 上海交通大学学报, 2021, 55(增2): 7-14.
	Zhou Yi, Zhou Liangcai, Ding Jiali, et al. Power Network Topology Optimization and Power Flow Control Based on Deep Reinforcement Learning[J]. Journal of Shanghai Jiaotong University, 2021, 55(S2): 7-14.
17	Hajihosseini M, Andalibi M, Gheisarnejad M, et al. DC/DC Power Converter Control-Based Deep Machine Learning Techniques: Real-Time Implementation[J]. IEEE Transactions on Power Electronics (S0885-8993), 2020, 35(10): 9971-9977.
18	Gheisarnejad M, Farsizadeh H, Tavana M R, et al. A Novel Deep Learning Controller for DC-DC Buck-Boost Converters in Wireless Power Transfer Feeding CPLs[J]. IEEE Transactions on Industrial Electronics (S0278-0046), 2021, 68(7): 6379-6384.
19	Lukianykhin O, Bogodorova T. ModelicaGym: Applying Reinforcement Learning to Modelica Models[C]// 9th International Workshop on Equation-Based Object-Oriented Modeling Languages and Tools. New York, NY, USA: Association for Computing Machinery, 2019: 27-36.

参数	数值
电感L/mH	2
电容C/μF	1500
输入电压E/V	85
母线电压V_ref/V	170

工况	k_vp	k_vi	K_cp	K_ci
微源电压变化	100	30	0.005	2
阻性负载变化	30	500	0.01	10
恒功率负载变化	1	300	0.02	100

工况	指标	PI	DDQN	DDPG
一	MOVR	0.119 8	0.034 3	0.013 7
一	MOVD	0.296 1	0.016 7	0.014 1
二	MOVR	0.150 5	0.051 3	0.011 8
二	MOVD	0.304 8	0.057 3	0.059 5
三	MOVR	1.368 0	0.886 5	0.246 8
三	MOVD	4.731 0	1.396 0	1.001 5

[1]	徐浩添, 秦龙, 曾俊杰, 胡越, 张琪. 基于深度强化学习的对手建模方法研究综述[J]. 系统仿真学报, 2023, 35(4): 671-694.
[2]	史佳洁, 杨鹏, 皮雁南. 基于机器学习的地铁行人流在线优化控制研究[J]. 系统仿真学报, 2023, 35(2): 386-395.
[3]	张森, 张孟炎, 邵敬平, 普杰信. 基于随机策略搜索的多机三维路径规划方法[J]. 系统仿真学报, 2022, 34(6): 1286-1295.
[4]	倪凌佳, 黄晓霞, 李红旮, 张子博. 基于协作式深度强化学习的火灾应急疏散仿真研究[J]. 系统仿真学报, 2022, 34(6): 1353-1366.
[5]	王红微, 杨鹏. 基于深度强化学习的机场货运业务优化研究[J]. 系统仿真学报, 2022, 34(3): 651-660.
[6]	李启锐, 彭心怡. 基于深度强化学习的云作业调度及仿真研究[J]. 系统仿真学报, 2022, 34(2): 258-268.
[7]	尹长青, 谭天然, 王建民. 车载人因协同仿真系统研究及实现[J]. 系统仿真学报, 2022, 34(1): 134-144.
[8]	高昂, 董志明, 张国辉, 梁涛, 郭齐胜. LVC训练系统中计算机生成兵力生成技术研究[J]. 系统仿真学报, 2021, 33(3): 745-752.
[9]	曾贲, 房霄, 孔德帅, 宋祥祥, 贾政轩, 林廷宇. 一种数据驱动的对抗博弈智能体建模方法[J]. 系统仿真学报, 2021, 33(12): 2838-2845.
[10]	刘慧, 潘庭龙. 面向风光储直流微网系统的多端口变换器设计[J]. 系统仿真学报, 2019, 31(9): 1916-1924.
[11]	张高峰, 吉玉洁. 多分辨率模型的协同仿真方法研究[J]. 系统仿真学报, 2018, 30(7): 2525-2532.
[12]	李小秋, 张小平. 基于脉冲输入电压的Buck-Boost变换器非线性特性研究[J]. 系统仿真学报, 2017, 29(5): 1021-1027.
[13]	田赤军, 沈胜兵, 李艳雷, 王宪宗, 郭卓锋. 基于光纤反射内存网实现远程协同仿真方法[J]. 系统仿真学报, 2015, 27(8): 1766-1773.