基于Q-learning算法的多智能体微电网能量管理策略

doi:10.16182/j.issn1004731x.joss.22-0378

摘要/Abstract

摘要：

针对微电网电力市场的能量交易和利益分配问题，提出了一种基于Q-learning算法的多智能体微电网能量管理方法。构建基于电力市场的微电网系统及交易过程，明确各单元的职责；考虑风速、光照强度和环境温度的变化情况，以及各发电单元的输出功率上下限约束，建立分布式电源数学模型。在此基础上，将分布式电源和用户负载视为智能体，基于Q-learning算法设计它们的马尔可夫决策过程，以最大化分布式电源收益和最小化用户负载成本为目标，提出了基于Q-learning算法的微电网能量管理方案。研究结果表明，在不同场景下所提方法既可以增加分布式电源的收益，还可以降低用户负载的成本。

关键词: 能量管理, 微电网, 多智能体, 收益与成本, 分布式电源

Abstract:

This paper proposes a multi-agent microgrid energy management method for the energy trading and benefit distribution in the microgrid power market based on the Q-learning algorithm. Based on the electricity market, microgrid system and transaction process are constructed to clarify the responsibilities of each unit. The mathematical models of distributed power generations are established by considering the changes in wind speed, light intensity and ambient temperature, as well as the upper and lower limit constraints of the output power of each power generation unit. On this basis, the distributed power generations and user loads are regarded as agents, and the Markov decision-making process is designed based on the Q-learning algorithm. Aiming at maximizing the benefits of distributed power generations and minimizing the costs of user loads, a microgrid energy management scheme based on Q-learning algorithm is proposed. The results show that the proposed method can not only increase the benefits of distributed power generations but also reduce the costs of user loads in different scenarios.

Key words: energy management, microgrid, multi-agents, benefit and cost, distributed power generation

中图分类号:

TP391.9

马苗苗, 董利鹏, 刘向杰. 基于Q-learning算法的多智能体微电网能量管理策略[J]. 系统仿真学报, 2023, 35(7): 1487-1496.

Miaomiao Ma, Lipeng Dong, Xiangjie Liu. Energy Management Strategy of Multi-agent Microgrid Based on Q-learning Algorithm[J]. Journal of System Simulation, 2023, 35(7): 1487-1496.

图/表 18

图1

图2

表1

风电机组特征参数

参数名称	参数值
切入风速 $v c i$ /( $m ⋅ s - 1$ )	4
额定风速 $v r$ /( $m ⋅ s - 1$ )	24
切出风速 $v c o$ /( $m ⋅ s - 1$ )	14
额定功率 $P W T r$ /kW	8

表1

表2

光伏机组特征参数

参数名称	参数值
最大功率点电压 $V M P P$ /V	17.32
最大功率点电流 $I M P P$ /A	4.76
开路电压 $V O C$ /V	21.98
短路电流 $I S C$ /A	5.32
电池标称工作温度 $N O T$ /(℃)	43
电压系数 $K V$	14.4
电流系数 $K I$	1.22

表2

图3

图4

表3

图5

图6

表4

各智能体的最佳学习参数

智能体	$α$	$γ$
可再生能源发电单元	0.7	0.4
用户负载	0.5	0.5
化石燃料发电单元	0.3	0.9

表4

图7

图8

图9

图10

图11

图12

图13

图14

参考文献 22

1	Hao Ran, Jiang Ziqing, Ai Qian, et al. Hierarchical Optimisation Strategy in Microgrid Based on the Consensus of Multi-agent System[J]. IET Generation, Transmission & Distribution, 2018, 12(10): 2444-2451.
2	李利明, 李征. 基于分层多目标优化的微电网能量管理算法[J]. 电力建设, 2018, 39(4): 75-82.
	Li Liming, Li Zheng. Microgrid Energy Management Based on Hierarchical Multi-objective Optimization Algorithm[J]. Electric Power Construction, 2018, 39(4): 75-82.
3	Shah P, Mehta B. Microgrid Optimal Scheduling with Renewable Energy Sources Considering Islanding Constraints[J]. Iranian Journal of Science and Technology, Transactions of Electrical Engineering, 2020, 44(2): 805-819.
4	郭骏. 风光储互补微电网的优化调度与能量管理策略研究[J]. 电气工程, 2020, 8(1): 36-47.
	Guo Jun. Research on Optimal Scheduling and Energy Management Strategy of Wind-PV-ES Complementary Micro-grid[J]. Journal of Electrical Engineering, 2020, 8(1): 36-47.
5	李华, 于潇. 基于DDPG的光储微电网系统能量决策优化[J]. 可再生能源, 2021, 39(5): 687-693.
	Li Hua, Yu Xiao. Energy Decision Optimization of Photovoltaic Energy Storage System Based on Deep Deterministic Strategy Gradient Algorithm[J]. Renewable Energy Resources, 2021, 39(5): 687-693.
6	郭国栋, 龚雁峰. 电力市场环境下基于深度强化学习的微网能量管理系统实时自动控制算法[J]. 电测与仪表, 2021, 58(9): 78-88.
	Guo Guodong, Gong Yanfeng. Real-time Automatic Control Algorithm of Micro-grid Energy Management System Based on Deep Reinforcement Learning in Electricity Market Environment[J]. Electrical Measurement & Instrumentation, 2021, 58(9): 78-88.
7	Li Zhengmao, Xu Yan. Temporally-coordinated Optimal Operation of a Multi-energy Microgrid Under Diverse Uncertainties[J]. Applied Energy, 2019, 240: 719-729.
8	刘俊峰, 陈剑龙, 王晓生, 等. 基于深度强化学习的微能源网能量管理与优化策略研究[J]. 电网技术, 2020, 44(10): 3794-3803.
	Liu Junfeng, Chen Jianlong, Wang Xiaosheng, et al. Energy Management and Optimization of Multi-energy Grid Based on Deep Reinforcement Learning[J]. Power System Technology, 2020, 44(10): 3794-3803.
9	Lee S, Seon J, Kyeong C, et al. Novel Energy Trading System Based on Deep-reinforcement Learning in Microgrids[J]. Energies, 2021, 14(17): 5515.
10	Mojica-Nava E, Barreto C, Quijano N. Population Games Methods for Distributed Control of Microgrids[J]. IEEE Transactions on Smart Grid, 2015, 6(6): 2586-2595.
11	刘帅. 基于智能算法的微电网能量协调管理研究[D]. 上海: 东华大学, 2019.
	Liu Shuai. Research on Energy Coordination Management of Microgrid Based on Intelligent Algorithms[D]. Shanghai: Donghua University, 2019.
12	黄代政, 龚仁喜, 阎昌国, 等. 免疫约束多目标算法在微电网能量优化的应用[J]. 系统仿真学报, 2014, 26(1): 225-229.
	Huang Daizheng, Gong Renxi, Yan Changguo, et al. Immune Constrained Multi-objective Optimization for Microgrids Energy Management[J]. Journal of System Simulation, 2014, 26(1): 225-229.
13	Foo Eddy Y S, Gooi H B, Chen S X. Multi-agent System for Distributed Management of Microgrids[J]. IEEE Transactions on Power Systems, 2015, 30(1): 24-34.
14	高雪莹, 唐昊, 苗刚中, 等. 储能系统能量调度与需求响应联合优化控制[J]. 系统仿真学报, 2016, 28(5): 1165-1172.
	Gao Xueying, Tang Hao, Miao Gangzhong, et al. Joint Optimization Control of Energy Storage System Management and Demand Response[J]. Journal of System Simulation, 2016, 28(5): 1165-1172.
15	Ju Liwei, Huang Liling, Tan Qinliang, et al. Three-level Energy Flexible Management Strategy for Micro Energy Grids Considering Multiple Uncertainties at Different Time Scales[J]. International Journal of Energy Research, 2020, 45(1): 316-341.
16	Cintuglu M H, Martin H, Mohammed O A. Real-time Implementation of Multiagent-based Game Theory Reverse Auction Model for Microgrid Market Operation[J]. IEEE Transactions on Smart Grid, 2015, 6(2): 1064-1072.
17	Mocanu E, Mocanu D C, Nguyen P H, et al. On-line Building Energy Optimization Using Deep Reinforcement Learning[J]. IEEE Transactions on Smart Grid, 2019, 10(4): 3698-3708.
18	Zhang Yuxiang, Ma Rui, Zhao Dongdong, et al. A Novel Energy Management Strategy Based on Dual Reward Function Q-learning for Fuel Cell Hybrid Electric Vehicle[J]. IEEE Transactions on Industrial Electronics, 2022, 69(2): 1537-1547.
19	黄星源, 李岩屹. 基于双Q学习算法的干扰资源分配策略[J]. 系统仿真学报, 2021, 33(8): 1801-1808.
	Huang Xingyuan, Li Yanyi. The Allocation of Jamming Resources Based on Double Q-learning Algorithm[J]. Journal of System Simulation, 2021, 33(8): 1801-1808.
20	朱志斌, 王付永, 尹艳辉, 等. 基于Q-learning的离散时间多智能体系统一致性[J]. 控制理论与应用, 2021, 38(7): 997-1005.
	Zhu Zhibin, Wang Fuyong, Yin Yanhui, et al. Consensus of Discrete-time Multi-agent System Based on Q-learning[J]. Control Theory & Applications, 2021, 38(7): 997-1005.
21	Zhang Qingchen, Lin Man, Yang L T, et al. Energy-efficient Scheduling for Real-time Systems Based on Deep Q-learning Model[J]. IEEE Transactions on Sustainable Computing, 2017, 4(1): 132-141.
22	Kofinas P, Dounis A I, Vouros G A. Fuzzy Q-learning for Multi-agent Decentralized Energy Management in Microgrids[J]. Applied Energy, 2018, 219: 53-67.

[1]	李成兵, 李云飞, 吴鹏. 基于多智能体的城市群客运网络脆弱性动态仿真[J]. 系统仿真学报, 2023, 35(6): 1183-1190.
[2]	卢自宝, 丁浩, 孙方云, 丁梓琼, 龚莉, 郑睿. 直流微电网中能量协调控制策略的研究[J]. 系统仿真学报, 2023, 35(6): 1215-1225.
[3]	魏利胜, 杨奔奔, 孙瑞霞. 基于新型BBO算法的微电网优化调度研究[J]. 系统仿真学报, 2023, 35(5): 1075-1085.
[4]	刘延东, 黄高翔, 陈文. 基于增强心理行为异质性的改进社会力模型[J]. 系统仿真学报, 2023, 35(5): 1120-1130.
[5]	袁国栋, 何明, 马子玉, 张伟士, 刘学达, 李伟. 基于K-means聚类的多智能体跟随多领导者算法[J]. 系统仿真学报, 2023, 35(3): 616-622.
[6]	罗俊仁, 张万鹏, 袁唯淋, 胡振震, 陈少飞, 陈璟. 面向多智能体博弈对抗的对手建模框架[J]. 系统仿真学报, 2022, 34(9): 1941-1955.
[7]	陆淼嘉, 黄承媛, 滕靖. 基于多智能体的网购生鲜无人车配送调度仿真[J]. 系统仿真学报, 2022, 34(6): 1185-1195.
[8]	倪凌佳, 黄晓霞, 李红旮, 张子博. 基于协作式深度强化学习的火灾应急疏散仿真研究[J]. 系统仿真学报, 2022, 34(6): 1353-1366.
[9]	李锋, 魏莹. 社会学习和参照点效应对企业产品决策的影响[J]. 系统仿真学报, 2022, 34(2): 234-246.
[10]	王磊, 孙晋海, 李拓键. 基于SIS模型的锻炼获益感知传播建模与仿真[J]. 系统仿真学报, 2022, 34(12): 2566-2574.
[11]	林永君, 陈鑫, 杨凯, 周珊珊, 白青飞. 含多微网的主动配电网双层分布式优化调度[J]. 系统仿真学报, 2022, 34(11): 2323-2336.
[12]	赵毓, 郭继峰, 颜鹏, 白成超. 稀疏奖励下多航天器规避决策自学习仿真[J]. 系统仿真学报, 2021, 33(8): 1766-1774.
[13]	桂欣冬, 吉鸿江, 范玲玲, 刘世达. 信任驱动的自适应协调控制算法及应用[J]. 系统仿真学报, 2021, 33(8): 1809-1817.
[14]	韩金刚, 李旭. 基于模糊控制的海流发电系统能量管理[J]. 系统仿真学报, 2021, 33(6): 1406-1414.
[15]	李锋, 魏莹. 复杂网络对羊群效应现象影响的仿真研究[J]. 系统仿真学报, 2021, 33(3): 539-553.