Energy Management Strategy of Multi-agent Microgrid Based on Q-learning Algorithm

doi:10.16182/j.issn1004731x.joss.22-0378

Abstract

Abstract:

This paper proposes a multi-agent microgrid energy management method for the energy trading and benefit distribution in the microgrid power market based on the Q-learning algorithm. Based on the electricity market, microgrid system and transaction process are constructed to clarify the responsibilities of each unit. The mathematical models of distributed power generations are established by considering the changes in wind speed, light intensity and ambient temperature, as well as the upper and lower limit constraints of the output power of each power generation unit. On this basis, the distributed power generations and user loads are regarded as agents, and the Markov decision-making process is designed based on the Q-learning algorithm. Aiming at maximizing the benefits of distributed power generations and minimizing the costs of user loads, a microgrid energy management scheme based on Q-learning algorithm is proposed. The results show that the proposed method can not only increase the benefits of distributed power generations but also reduce the costs of user loads in different scenarios.

Key words: energy management, microgrid, multi-agents, benefit and cost, distributed power generation

CLC Number:

TP391.9

Miaomiao Ma, Lipeng Dong, Xiangjie Liu. Energy Management Strategy of Multi-agent Microgrid Based on Q-learning Algorithm[J]. Journal of System Simulation, 2023, 35(7): 1487-1496.

Figures/Tables 18

Fig. 1

Fig. 2

Table 1

Characteristic parameters of wind turbine

参数名称	参数值
切入风速 $v c i$ /( $m ⋅ s - 1$ )	4
额定风速 $v r$ /( $m ⋅ s - 1$ )	24
切出风速 $v c o$ /( $m ⋅ s - 1$ )	14
额定功率 $P W T r$ /kW	8

Table 1

Table 2

Characteristic parameters of photovoltaic unit

参数名称	参数值
最大功率点电压 $V M P P$ /V	17.32
最大功率点电流 $I M P P$ /A	4.76
开路电压 $V O C$ /V	21.98
短路电流 $I S C$ /A	5.32
电池标称工作温度 $N O T$ /(℃)	43
电压系数 $K V$	14.4
电流系数 $K I$	1.22

Table 2

Fig. 3

Fig. 4

Table 3

Fig. 5

Fig. 6

Table 4

Optimal learning parameters for each agent

智能体	$α$	$γ$
可再生能源发电单元	0.7	0.4
用户负载	0.5	0.5
化石燃料发电单元	0.3	0.9

Table 4

Fig.7

Fig. 8

Fig. 9

Fig. 10

Fig. 11

Fig. 12

Fig. 13

Fig. 14

References 22

1	Hao Ran, Jiang Ziqing, Ai Qian, et al. Hierarchical Optimisation Strategy in Microgrid Based on the Consensus of Multi-agent System[J]. IET Generation, Transmission & Distribution, 2018, 12(10): 2444-2451.
2	李利明, 李征. 基于分层多目标优化的微电网能量管理算法[J]. 电力建设, 2018, 39(4): 75-82.
	Li Liming, Li Zheng. Microgrid Energy Management Based on Hierarchical Multi-objective Optimization Algorithm[J]. Electric Power Construction, 2018, 39(4): 75-82.
3	Shah P, Mehta B. Microgrid Optimal Scheduling with Renewable Energy Sources Considering Islanding Constraints[J]. Iranian Journal of Science and Technology, Transactions of Electrical Engineering, 2020, 44(2): 805-819.
4	郭骏. 风光储互补微电网的优化调度与能量管理策略研究[J]. 电气工程, 2020, 8(1): 36-47.
	Guo Jun. Research on Optimal Scheduling and Energy Management Strategy of Wind-PV-ES Complementary Micro-grid[J]. Journal of Electrical Engineering, 2020, 8(1): 36-47.
5	李华, 于潇. 基于DDPG的光储微电网系统能量决策优化[J]. 可再生能源, 2021, 39(5): 687-693.
	Li Hua, Yu Xiao. Energy Decision Optimization of Photovoltaic Energy Storage System Based on Deep Deterministic Strategy Gradient Algorithm[J]. Renewable Energy Resources, 2021, 39(5): 687-693.
6	郭国栋, 龚雁峰. 电力市场环境下基于深度强化学习的微网能量管理系统实时自动控制算法[J]. 电测与仪表, 2021, 58(9): 78-88.
	Guo Guodong, Gong Yanfeng. Real-time Automatic Control Algorithm of Micro-grid Energy Management System Based on Deep Reinforcement Learning in Electricity Market Environment[J]. Electrical Measurement & Instrumentation, 2021, 58(9): 78-88.
7	Li Zhengmao, Xu Yan. Temporally-coordinated Optimal Operation of a Multi-energy Microgrid Under Diverse Uncertainties[J]. Applied Energy, 2019, 240: 719-729.
8	刘俊峰, 陈剑龙, 王晓生, 等. 基于深度强化学习的微能源网能量管理与优化策略研究[J]. 电网技术, 2020, 44(10): 3794-3803.
	Liu Junfeng, Chen Jianlong, Wang Xiaosheng, et al. Energy Management and Optimization of Multi-energy Grid Based on Deep Reinforcement Learning[J]. Power System Technology, 2020, 44(10): 3794-3803.
9	Lee S, Seon J, Kyeong C, et al. Novel Energy Trading System Based on Deep-reinforcement Learning in Microgrids[J]. Energies, 2021, 14(17): 5515.
10	Mojica-Nava E, Barreto C, Quijano N. Population Games Methods for Distributed Control of Microgrids[J]. IEEE Transactions on Smart Grid, 2015, 6(6): 2586-2595.
11	刘帅. 基于智能算法的微电网能量协调管理研究[D]. 上海: 东华大学, 2019.
	Liu Shuai. Research on Energy Coordination Management of Microgrid Based on Intelligent Algorithms[D]. Shanghai: Donghua University, 2019.
12	黄代政, 龚仁喜, 阎昌国, 等. 免疫约束多目标算法在微电网能量优化的应用[J]. 系统仿真学报, 2014, 26(1): 225-229.
	Huang Daizheng, Gong Renxi, Yan Changguo, et al. Immune Constrained Multi-objective Optimization for Microgrids Energy Management[J]. Journal of System Simulation, 2014, 26(1): 225-229.
13	Foo Eddy Y S, Gooi H B, Chen S X. Multi-agent System for Distributed Management of Microgrids[J]. IEEE Transactions on Power Systems, 2015, 30(1): 24-34.
14	高雪莹, 唐昊, 苗刚中, 等. 储能系统能量调度与需求响应联合优化控制[J]. 系统仿真学报, 2016, 28(5): 1165-1172.
	Gao Xueying, Tang Hao, Miao Gangzhong, et al. Joint Optimization Control of Energy Storage System Management and Demand Response[J]. Journal of System Simulation, 2016, 28(5): 1165-1172.
15	Ju Liwei, Huang Liling, Tan Qinliang, et al. Three-level Energy Flexible Management Strategy for Micro Energy Grids Considering Multiple Uncertainties at Different Time Scales[J]. International Journal of Energy Research, 2020, 45(1): 316-341.
16	Cintuglu M H, Martin H, Mohammed O A. Real-time Implementation of Multiagent-based Game Theory Reverse Auction Model for Microgrid Market Operation[J]. IEEE Transactions on Smart Grid, 2015, 6(2): 1064-1072.
17	Mocanu E, Mocanu D C, Nguyen P H, et al. On-line Building Energy Optimization Using Deep Reinforcement Learning[J]. IEEE Transactions on Smart Grid, 2019, 10(4): 3698-3708.
18	Zhang Yuxiang, Ma Rui, Zhao Dongdong, et al. A Novel Energy Management Strategy Based on Dual Reward Function Q-learning for Fuel Cell Hybrid Electric Vehicle[J]. IEEE Transactions on Industrial Electronics, 2022, 69(2): 1537-1547.
19	黄星源, 李岩屹. 基于双Q学习算法的干扰资源分配策略[J]. 系统仿真学报, 2021, 33(8): 1801-1808.
	Huang Xingyuan, Li Yanyi. The Allocation of Jamming Resources Based on Double Q-learning Algorithm[J]. Journal of System Simulation, 2021, 33(8): 1801-1808.
20	朱志斌, 王付永, 尹艳辉, 等. 基于Q-learning的离散时间多智能体系统一致性[J]. 控制理论与应用, 2021, 38(7): 997-1005.
	Zhu Zhibin, Wang Fuyong, Yin Yanhui, et al. Consensus of Discrete-time Multi-agent System Based on Q-learning[J]. Control Theory & Applications, 2021, 38(7): 997-1005.
21	Zhang Qingchen, Lin Man, Yang L T, et al. Energy-efficient Scheduling for Real-time Systems Based on Deep Q-learning Model[J]. IEEE Transactions on Sustainable Computing, 2017, 4(1): 132-141.
22	Kofinas P, Dounis A I, Vouros G A. Fuzzy Q-learning for Multi-agent Decentralized Energy Management in Microgrids[J]. Applied Energy, 2018, 219: 53-67.

电源	额定功率
风电机组	8
光伏机组	6
微型燃气轮机	10
柴油发电机	3
燃料电池	8
储能系统	3

[1]	Zibao Lu, Hao Ding, Fangyun Sun, Ziqiong Ding, Li Gong, Rui Zheng. Research on Energy Coordination Control Strategy in DC Microgrid [J]. Journal of System Simulation, 2023, 35(6): 1215-1225.
[2]	Lisheng Wei, Benben Yang, Ruixia Sun. Research on Optimal Scheduling of Microgrid Based on NBBO Algorithm [J]. Journal of System Simulation, 2023, 35(5): 1075-1085.
[3]	Yongjun Lin, Xin Chen, Kai Yang, Shanshan Zhou, Qingfei Bai. Bilevel Distributed Optimal Dispatch of Active Distribution Network with Multi-microgrids [J]. Journal of System Simulation, 2022, 34(11): 2323-2336.
[4]	Yan Xiuying, Dang Miaomiao. Optimization of Household Electricity Consumption Period Based on Improved Multi-objective Particle Swarm Optimization [J]. Journal of System Simulation, 2022, 34(1): 70-78.
[5]	Han Jingang, Li Xu. Energy Management of Marine Current Power Generation System Based on Fuzzy Logical Control [J]. Journal of System Simulation, 2021, 33(6): 1406-1414.
[6]	Guo Ai, Chen Chao, Shi Junjie, Liu Zhengjie, Chen Weirong, Liang Jiayi, Liu Nan. Comparative Study of Energy Management Strategy Based on Value Loss of Tram Hybrid Power System [J]. Journal of System Simulation, 2021, 33(3): 572-580.
[7]	Zhang Kunping, Hao Lin. Frequency Coordinated Control Strategy of Microgrid Based on Fuzzy Prediction [J]. Journal of System Simulation, 2021, 33(3): 581-590.
[8]	Zheng Wen, Dou Chenxi, Zhang Zhe. Simulation of Benefit Distribution in Platform Delivery Mode [J]. Journal of System Simulation, 2021, 33(3): 631-644.
[9]	Deng Tao, Yu Haoyuan, Xu Bin. Control Optimization of Hybrid System with Hybrid Power Based on DP Algorithm [J]. Journal of System Simulation, 2019, 31(9): 1860-1867.
[10]	Sun Weiqing, Liu Tong, Liu Bo, Wang Youjia. Network Mode and Economy Analysis of Photovoltaic-Battery System in Microgrid [J]. Journal of System Simulation, 2018, 30(5): 1812-1818.
[11]	Gao Cong, Wu Dinghui, Pan Tinglong, Ji Zhicheng. Optimal Economic Dispatch of Microgrid Based on Immune Particle Swarm Optimization Algorithm [J]. Journal of System Simulation, 2018, 30(2): 636-646.
[12]	Liang Hongtao, Kang Fengju, Fu Yanfang. Overview of Multi-agents Modeling and Simulation for Underwater Unmanned Combat Systems [J]. Journal of System Simulation, 2018, 30(11): 4053-4066.
[13]	Wu Xiaolan, Bai Zhifeng, Shi Xiaohui, Cao Binggang. Optimization Design of Fuzzy Energy Management for Plug-in Hybrid Electric Vehicles [J]. Journal of System Simulation, 2018, 30(1): 242-248.
[14]	Zhang Xuefeng, Bai Chenxi. Study of Crowd Evacuation in Two-Story Teaching Building [J]. Journal of System Simulation, 2017, 29(5): 1070-1076.