基于安全强化学习的热电联产机组经济调度策略研究

doi:10.16182/j.issn1004731x.joss.23-1472

摘要/Abstract

摘要：

针对DRL算法在热电联产(combined heat and power，CHP)机组优化中缺乏安全性和稳定性保证的问题，提出了一种基于安全强化学习(SRL)的调度优化方法。在Dymola平台以CHP机组为热源建立了区域供热系统模型。设计了CHP机组经济调度的MDP模型，并通过控制障碍函数(control barrier functions，CBF)指导DRL安全探索。仿真结果表明：CBF-DRL方法在复杂且非线性的区域供热系统中，不仅能够提升DRL算法的收敛速度，还能够有效利用供热管道的热惯性提高CHP机组的经济效益，并在安全性方面表现出优势。

关键词: 热电联产, 区域供热系统, 安全强化学习, 控制障碍函数, 经济调度, 协同仿真

Abstract:

In addressing the challenge of the DRL algorithm in the optimization of combined heat and power (CHP) units, lacking safety and stability guarantees, a scheduling optimization method based on SRL is proposed. Utilizing Dymola platform, a district heating system model is constructed with the CHP unit as the heat source. A MDP model for the economic dispatching of CHP units is designed, incorporating control barrier functions (CBF) to guide safe exploration in DRL. Simulation results show that the CBF-DRL method, in complex and nonlinear district heating systems, not only accelerates the convergence of DRL algorithms but also efficiently utilizes the thermal inertia of heating pipelines to enhance the economic performance of CHP units. In particular, it has significant safety benefits.

Key words: combined heat and power(CHP), district heating system, safe reinforcement learning, control barrier function(CBF), economic dispatch, co-simulation

中图分类号:

TP391.9

王欣,崔承刚,王想想等 . 基于安全强化学习的热电联产机组经济调度策略研究[J]. 系统仿真学报, 2025, 37(4): 968-981.

Wang Xin,Cui Chenggang,Wang Xiangxiang,et al . Research on Economic Dispatching Strategy of CHP Units Based on SRL[J]. Journal of System Simulation, 2025, 37(4): 968-981.

图/表 18

图1

图2

图3

图4

图5

图6

图7

图8

图9

图10

图11

图12

表1

供热系统参数

参数	数值
[h_t, p_t ]/MW	(0,10), (10,5), (70,35), (0,50)
a₀	58.908 2
a₁	274.899 6
$T m i n S$ /℃	70
$τ m a x S$ /℃	110
$m ˙ m a x$ /(m·s^-1)	3
τ^R_min/℃	45
L/m	5 000, 15 000
d/m	0.595 8

表1

图13

图14

图15

图16

图17

参考文献 23

1	Franco Alessandro, Bellina Francesco. Methods for Optimized Design and Management of CHP Systems for District Heating Networks (DHN)[J]. Energy Conversion and Management, 2018, 172: 21-31.
2	Rezaie B, Rosen M A. District Heating and Cooling: Review of Technology and Potential Enhancements[J]. Applied Energy, 2012, 93: 2-10.
3	Vandermeulen Annelies, Bram van der Heijde, Helsen Lieve. Controlling District Heating and Cooling Networks to Unlock Flexibility: A Review[J]. Energy, 2018, 151: 103-115.
4	Buffa Simone, Mohammad Hossein Fouladfar, Franchini Giuseppe, et al. Advanced Control and Fault Detection Strategies for District Heating and Cooling Systems—a Review[J]. Applied Sciences, 2021, 11(1): 455.
5	Xu Fei, Hao Ling, Chen Lei, et al. Integrated Heat and Power Optimal Dispatch Method Considering the District Heating Networks Flow Rate Regulation for Wind Power Accommodation[J]. Energy, 2023, 263, Part A: 125656.
6	Verrilli Francesca, Srinivasan S, Gambino Giovanni, et al. Model Predictive Control-based Optimal Operations of District Heating System with Thermal Energy Storage and Flexible Loads[J]. IEEE Transactions on Automation Science and Engineering, 2017, 14(2): 547-557.
7	Grosswindhager S, Voigt A, Kozek M. Predictive Control of District Heating Network Using Fuzzy DMC[C]//2012 Proceedings of International Conference on Modelling, Identification and Control. Piscataway: IEEE, 2012: 241-246.
8	Yang Dong, Yan Siyun, Zhou Dengji, et al. Reinforcement Learning Methods on Optimization Problems of Natural Gas Pipeline Networks[C]//2020 4th International Conference on Smart Grid and Smart Cities (ICSGSC). Piscataway: IEEE, 2020: 29-34.
9	Zhang Bin, Hu Weihao, Cao Di, et al. Deep Reinforcement Learning-based Approach for Optimizing Energy Conversion in Integrated Electrical and Heating System with Renewable Energy[J]. Energy Conversion and Management, 2019, 202: 112199.
10	Claessens B J, Vanhoudt D, Desmedt J, et al. Model-free Control of Thermostatically Controlled Loads Connected to a District Heating Network[J]. Energy and Buildings, 2018, 159: 1-10.
11	Wan Yanni, Qin Jiahu, Ma Qichao, et al. Multi-agent DRL-based Data-driven Approach for PEVs Charging/Discharging Scheduling in Smart Grid[J]. Journal of the Franklin Institute, 2022, 359(4): 1747-1767.
12	Gangopadhyay Briti, Dasgupta Pallab, Dey Soumyajit. Safe and Stable RL (S²RL) Driving Policies Using Control Barrier and Control Lyapunov Functions[J]. IEEE Transactions on Intelligent Vehicles, 2023, 8(2): 1889-1899.
13	Cheng R, Orosz Gábor, Murray R M, et al. End-to-end Safe Reinforcement Learning Through Barrier Functions for Safety-critical Continuous Control Tasks[C]//Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence. Palo Alto: AAAI Press, 2019: 3387-3395.
14	Marvi Z, Kiumarsi B. Safe Reinforcement Learning: A Control Barrier Function Optimization Approach[J]. International Journal of Robust and Nonlinear Control, 2021, 31(6): 1923-1940.
15	Cohen M H, Belta C. Safe Exploration in Model-based Reinforcement Learning Using Control Barrier Functions[J]. Automatica, 2023, 147: 110684.
16	Gräber M, Kosowski K, Richter C, et al. Modelling of Heat Pumps with an Object-oriented Model Library for Thermodynamic Systems[J]. Mathematical and Computer Modelling of Dynamical Systems, 2010, 16(3): 195-209.
17	Leitner B, Widl E, Gawlik W, et al. A Method for Technical Assessment of Power-to-heat Use Cases to Couple Local District Heating and Electrical Distribution Grids[J]. Energy, 2019, 182: 729-738.
18	Wetter M, Zuo Wangda, Nouidui T S, et al. Modelica Buildings Library[J]. Journal of Building Performance Simulation, 2014, 7(4): 253-270.
19	van der Heijde B, Fuchs M, Ribas Tugores C, et al. Dynamic Equation-based Thermo-hydraulic Pipe Model for District Heating and Cooling Systems[J]. Energy Conversion and Management, 2017, 151: 158-169.
20	Zhou Suyang, Hu Zijian, Gu Wei, et al. Combined Heat and Power System Intelligent Economic Dispatch: A Deep Reinforcement Learning Approach[J]. International Journal of Electrical Power & Energy Systems, 2020, 120: 106016.
21	Haghrah A, Nazari-Heris M, Mohammadi-Ivatloo B. Solving Combined Heat and Power Economic Dispatch Problem Using Real Coded Genetic Algorithm with Improved Mühlenbein Mutation[J]. Applied Thermal Engineering, 2016, 99: 465-475.
22	庞力平, 梁其缘, 梁惠勋, 等. 基于Dymola平台的超超临界二次再热机组切除高压加热器动态仿真[J]. 热能动力工程, 2023, 38(1): 120-128.
	Pang Liping, Liang Qiyuan, Liang Huixun, et al. Dynamic Simulation of Ultra-supercritical Double Reheat Unit Based on Dymola Platform After HP Feedwater Heater Bypass[J]. Journal of Engineering for Thermal Energy and Power, 2023, 38(1): 120-128.
23	Li Zhigang, Wu Wenchuan, Shahidehpour M, et al. Combined Heat and Power Dispatch Considering Pipeline Energy Storage of District Heating Network[J]. IEEE Transactions on Sustainable Energy, 2016, 7(1): 12-22.

[1]	任旭阳, 卜旭辉, 尹艳玲, 刘静滑. 基于多策略多目标差分进化算法的风光储系统协调优化调度[J]. 系统仿真学报, 2025, 37(2): 450-461.
[2]	余洋, 夏雨星, 陆文韬, 刘霡, 高世轩, 陈东阳. 基于荷-储碳流模型的电力系统双层优化调度[J]. 系统仿真学报, 2024, 36(10): 2288-2299.
[3]	刘金朋, 彭锦淳, 邓嘉明, 刘胡诗涵. 考虑电热联产的虚拟电厂两阶段鲁棒优化经济调度[J]. 系统仿真学报, 2023, 35(8): 1799-1813.
[4]	范雅倩, 于松源, 房方. 热电联产虚拟电厂两阶段分布鲁棒优化调度[J]. 系统仿真学报, 2023, 35(5): 1046-1058.
[5]	戴宇轩, 崔承刚. 基于深度强化学习的Boost变换器控制策略[J]. 系统仿真学报, 2023, 35(5): 1109-1119.
[6]	刘兴华, 耿晨, 谢胜寒, 田佳强, 曹晖. 考虑光伏发电不确定性的日前火电-光伏经济调度[J]. 系统仿真学报, 2022, 34(8): 1874-1884.
[7]	马立新, 程颍. 计及可中断负荷的园区综合能源系统优化调度[J]. 系统仿真学报, 2022, 34(4): 817-825.
[8]	尹长青, 谭天然, 王建民. 车载人因协同仿真系统研究及实现[J]. 系统仿真学报, 2022, 34(1): 134-144.
[9]	魏乐, 李希金. 基于多目标的含风电场动态环境经济调度[J]. 系统仿真学报, 2020, 32(9): 1825-1830.
[10]	梅盼盼, 吴亮红, 张红强, 王慧莹. 含风电电力系统动态经济调度的差分纵横交叉优化[J]. 系统仿真学报, 2020, 32(6): 1179-1187.
[11]	孙伟卿, 叶磊, 涂轶昀. 基于GAMS的电力系统有功-无功综合优化[J]. 系统仿真学报, 2018, 30(8): 3082-3091.
[12]	赵文清, 覃智补. 采用改进型智能水滴算法的电力环境经济调度[J]. 系统仿真学报, 2018, 30(8): 3213-3218.
[13]	张高峰, 吉玉洁. 多分辨率模型的协同仿真方法研究[J]. 系统仿真学报, 2018, 30(7): 2525-2532.
[14]	田赤军, 沈胜兵, 李艳雷, 王宪宗, 郭卓锋. 基于光纤反射内存网实现远程协同仿真方法[J]. 系统仿真学报, 2015, 27(8): 1766-1773.