建筑空调系统迁移模仿学习仿真调度策略研究

doi:10.16182/j.issn1004731x.joss.24-FZ0761

摘要/Abstract

摘要：

为解决空调调度在线部署初期，低质量数据工况存在性能不稳定与训练过程效率低下的问题，提出一种基于迁移模仿学习的空调仿真调度策略制定方法。通过强化学习方法获得建筑运行策略，建立标准建筑仿真模型作为源域部署迁移学习，模仿学习损失函数被应用于智能体损失函数中以增强算法性能。结果表明：相比未采用迁移学习的方法，运行效益提升了16.2%，有效解决了强化学习训练初期的运行不稳定问题；相比未采用模仿学习的方法，运行效益提升了11.5%，有效提高了强化学习的训练效率。

关键词: 迁移学习, 强化学习, 模仿学习, 空调调控方法, 室温控制

Abstract:

To solve the problem of unstable performance and inefficient training process of low-quality data conditions at the initial stage of online deployment of air conditioner scheduling, we propose a migration-imitation learning-based air conditioning scheduling strategy simulation method. Reinforcement learning methods are used to generate building operation strategies. A standard building simulation model serves as the source domain, upon which migration learning is applied. An imitation learning loss function is incorporated into the intelligent loss function to enhance algorithm performance. The results indicate that, compared with the non-use of migration learning, the proposed method can improve the operational efficiency by 16.2%, effectively resolving the operational instability issues at the initial stage of reinforcement learning training. Compared to methods without imitation learning, operational efficiency is enhanced by 11.5%, significantly improving the training efficiency of reinforcement learning.

Key words: transfer learning, reinforcement learning, imitation learning, air conditioning control, room temperature control

中图分类号:

TP29

王翘楚,丁研,梁传志等 . 建筑空调系统迁移模仿学习仿真调度策略研究[J]. 系统仿真学报, 2024, 36(12): 2884-2893.

Wang Qiaochu,Ding Yan,Liang Chuanzhi,et al . Research on Scheduling Strategies Simulation for Building Air-conditioning Systems Based on Transfer Imitation Learning[J]. Journal of System Simulation, 2024, 36(12): 2884-2893.

图/表 9

图1

图2

表1

表2

强化学习算法超参数设置

参数	设定值
$γ$	0.99
$ε$	0.8
$α$	0.01
经验回放池大小	869
批量采样样本数量	128
目标网络更新间隔	8
神经网络层数	5
神经网络隐藏层节点数	32~128
神经网络输入节点数	DQN状态空间维度
神经网络输出节点数	DQN动作空间维度
优化器	Adam
激活函数	ReLU

表2

表3

图3

图4

图5

图6

参考文献 24

1	中国建筑节能协会, 重庆大学城乡建设与发展研究院. 中国建筑能耗与碳排放研究报告(2023年)[J]. 建筑, 2024(2): 46-59.
2	宋梦. 空调负荷参与电力系统调控关键技术研究[D]. 南京: 东南大学, 2018.
	Song Meng. Research on Key Technologies of Air Conditioning Loads Participating in Power System Dispatch and Control[D]. Nanjing: Southeast University, 2018.
3	孟庆龙, 张志博, 熊成. 变风量空调系统变水温策略运行效果仿真研究[J]. 系统仿真学报, 2019, 31(8): 1617-1626.
	Meng Qinglong, Zhang Zhibo, Xiong Cheng. Simulation Research on Operation Effect of VAV Air-conditioning System with Variable Water Temperature Strategy[J]. Journal of System Simulation, 2019, 31(8): 1617-1626.
4	Alghamdi Sultan M, Ajour Mohammed N, Abu-Hamdeh Nidal H, et al. Using PCM for Building Energy Management to Postpone the Electricity Demand Peak Load and Approving a New PID Controller to Activate Alternative Chiller[J]. Journal of Building Engineering, 2022, 57: 104884.
5	Wang Shengwei, Tang Rui. Supply-based Feedback Control Strategy of Air-conditioning Systems for Direct Load Control of Buildings Responding to Urgent Requests of Smart Grids[J]. Applied Energy, 2017, 201: 419-432.
6	张青. 中央空调系统节能运行控制方法研究[D]. 南京: 东南大学, 2016.
	Zhang Qing. Study on Energy Saving Control Stratedies of Central Air Conditioning System[D]. Nanjing: Southeast University, 2016.
7	Wang Wenyi, Zhou Qun, Pan Chao, et al. Energy-efficient Operation of a Complete Chiller-air Handing Unit System via Model Predictive Control[J]. Applied Thermal Engineering, 2022, 201, Part B: 117809.
8	Chen Sihao, Ding Puxian, Zhou Guang, et al. A Novel Machine Learning-based Model Predictive Control Framework for Improving the Energy Efficiency of Air-conditioning Systems[J]. Energy and Buildings, 2023, 294: 113258.
9	Kim D, Wang Zhe, Brugger J, et al. Site Demonstration and Performance Evaluation of MPC for a Large Chiller Plant with TES for Renewable Energy Integration and Grid Decarbonization[J]. Applied Energy, 2022, 321: 119343.
10	王霄汉, 张霖, 任磊, 等. 基于强化学习的车间调度问题研究简述[J]. 系统仿真学报, 2021, 33(12): 2782-2791.
	Wang Xiaohan, Zhang Lin, Ren Lei, et al. Brief Review on Applying Reinforcement Learning to Job Shop Scheduling Problems[J]. Journal of System Simulation, 2021, 33(12): 2782-2791.
11	段勇, 刘兴刚, 徐心和. 基于强化学习的机器人模糊控制系统设计[J]. 系统仿真学报, 2006, 18(6): 1597-1600.
	Duan Yong, Liu Xinggang, Xu Xinhe. Design of Robot Fuzzy Logic Controller Based on Reinforcement Learning[J]. Journal of System Simulation, 2006, 18(6): 1597-1600.
12	江达, 蔡志勤, 刘忠振, 等. 基于强化学习的连续型机械臂自适应跟踪控制[J]. 系统仿真学报, 2022, 34(10): 2264-2271.
	Jiang Da, Cai Zhiqin, Liu Zhongzhen, et al. Reinforcement-learning-based Adaptive Tracking Control for a Space Continuum Robot Based on Reinforcement Learning[J]. Journal of System Simulation, 2022, 34(10): 2264-2271.
13	李启锐, 彭心怡. 基于深度强化学习的云作业调度及仿真研究[J]. 系统仿真学报, 2022, 34(2): 258-268.
	Li Qirui, Peng Xinyi. Job Scheduling and Simulation in Cloud Based on Deep Reinforcement Learning[J]. Journal of System Simulation, 2022, 34(2): 258-268.
14	Zhang Xinfang, Li Zhenhai, Li Zhengwei, et al. Differential Pressure Reset Strategy Based on Reinforcement Learning for Chilled Water Systems[J]. Building Simulation, 2022, 15(2): 233-248.
15	Wang Dan, Gao Cheng, Sun Yuying, et al. Reinforcement Learning Control Strategy for Differential Pressure Setpoint in Large-scale Multi-source Looped District Cooling System[J]. Energy and Buildings, 2023, 282: 112778.
16	付玉, 张垚, 赵萌, 等. 基于仿真数据迁移学习的固定翼无人机检测[J]. 系统仿真学报, 2023, 35(5): 998-1007.
	Fu Yu, Zhang Yao, Zhao Meng, et al. Fixed-wing UAV Detection Based on Simulated Data Transfer Learning[J]. Journal of System Simulation, 2023, 35(5): 998-1007.
17	Tao Yuechuan, Qiu Jing, Lai Shuying. A Hybrid Cloud and Edge Control Strategy for Demand Responses Using Deep Reinforcement Learning and Transfer Learning[J]. IEEE Transactions on Cloud Computing, 2022, 10(1): 56-71.
18	Coraci Davide, Brandi Silvio, Hong Tianzhen, et al. Online Transfer Learning Strategy for Enhancing the Scalability and Deployment of Deep Reinforcement Learning Control in Smart Buildings[J]. Applied Energy, 2023, 333: 120598.
19	Fang Xi, Gong Guangcai, Li Guannan, et al. Cross Temporal-spatial Transferability Investigation of Deep Reinforcement Learning Control Strategy in the Building HVAC System Level[J]. Energy, 2023, 263, Part B: 125679.
20	Lee Z E, Zhang K. Generalized Reinforcement Learning for Building Control Using Behavioral Cloning[J]. Applied Energy, 2021, 304: 117602.
21	陈武, 周兴禧, 夏凊, 等. 双联变频空调系统建模与控制方法仿真研究[J]. 系统仿真学报, 2002, 14(5): 643-646.
	Chen Wu, Zhou Xingxi, Xia Qing, et al. Simulation Research on Control Strategies and Modeling of the Double-evaporator Air-conditioner with Inverter[J]. Journal of System Simulation, 2002, 14(5): 643-646.
22	Luo Sha, Schomaker Lambert. Reinforcement Learning in Robotic Motion Planning by Combined Experience-based Planning and Self-imitation Learning[J]. Robotics and Autonomous Systems, 2023, 170: 104545.
23	Bojarski M, Del Testa D, Dworakowski D, et al. End to End Learning for Self-driving Cars[EB/OL]. (2016-04-25) [2023-12-17]. .
24	Hester T, Vecerik M, Pietquin O, et al. Deep Q-learning from Demonstrations[C]//Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence. Palo Alto: AAAI Press, 2018: 3223-3230.

参数	目标建筑	源域
气候分区	寒冷	寒冷
气象文件	天津	北京
人员活动时间表	测试获得	与目标域相同
建筑面积/m²	3 970.5	3 582.2

模型	方法	源域	模仿学习损失函数	重复训练
A	迁移模仿学习	源域A	式(6)与式(7)	×
B	模仿学习	×	式(6)与式(7)	×
C	迁移学习	源域A	×	×
D	仅在线训练	×	×	×
E	离线训练	×	×	200轮

[1]	徐丽霞, 钟季龙, 伍劭实, 丁一珊, 翟小玉, 陈世钊, 王鹥喆, 温雪, 曾隽芳, 侯新文. 基于云模型和最大均值差异的指标迁移学习[J]. 系统仿真学报, 2024, 36(9): 2004-2015.
[2]	李超, 李贾宝, 丁才昌, 叶志伟, 左方威. 基于DRL的边缘监控任务卸载与资源分配算法[J]. 系统仿真学报, 2024, 36(9): 2113-2126.
[3]	钱殿伟, 齐红敏, 刘振, 周志明, 易建强. 基于改进近端策略优化的空战自主决策研究[J]. 系统仿真学报, 2024, 36(9): 2208-2218.
[4]	孙怡峰, 李智, 吴疆, 王玉宾. 作战方案驱动的可学习兵棋推演智能体研究[J]. 系统仿真学报, 2024, 36(7): 1525-1535.
[5]	蒋权, 魏静萱. 用于动态柔性作业车间调度的实时调度方法[J]. 系统仿真学报, 2024, 36(7): 1609-1620.
[6]	龚雪, 彭鹏菲, 荣里, 郑雅莲, 姜俊. 基于深度强化学习的任务分析方法[J]. 系统仿真学报, 2024, 36(7): 1670-1681.
[7]	周志勇, 莫非, 赵凯, 郝云波, 钱宇峰. 基于PPO的自适应PID控制算法研究[J]. 系统仿真学报, 2024, 36(6): 1425-1432.
[8]	王红军, 林俊强, 邹湘军, 张坡, 周铭轩, 邹伟锐, 唐昀超, 罗陆锋. 基于数字孪生的果园虚拟交互系统构建[J]. 系统仿真学报, 2024, 36(6): 1493-1508.
[9]	王远, 徐琳, 宫小泽, 张永亮, 王永利. 基于梯度的深度强化学习解释方法[J]. 系统仿真学报, 2024, 36(5): 1130-1140.
[10]	秦保新, 张羽霄, 吴思锐, 曹卫冲, 李湛. 基于改进D3QN的煤炭码头卸车排产智能优化方法[J]. 系统仿真学报, 2024, 36(3): 770-781.
[11]	赵莹莹, 董普森, 朱天晨, 李凡, 苏运, 邰振赢, 孙庆赟, 凡航. 面向电网拓扑调度仿真的采样效率优化方法研究[J]. 系统仿真学报, 2024, 36(2): 283-295.
[12]	王鑫鹏, 傅汇乔, 邓归洲, 唐开强, 陈春林, 留沧海. 基于DRL和自由步态的六足机器人运动规划研究[J]. 系统仿真学报, 2024, 36(2): 373-384.
[13]	潘海南, 陈柏良, 黄开宏, 任君凯, 程创, 卢惠民, 张辉. 基于深度强化学习的履带机器人摆臂控制方法[J]. 系统仿真学报, 2024, 36(2): 405-414.
[14]	丁开源, 艾斯卡尔·艾木都拉, 朱斌, 伊克萨尼·普尔凯提, 马正堂. 基于多模态深度强化学习的端到端无人车运动规划[J]. 系统仿真学报, 2024, 36(11): 2631-2643.
[15]	张永夫, 刘洋, 袁禾. 一种基于兵棋推演的作战目标体系关键节点识别方法[J]. 系统仿真学报, 2024, 36(11): 2654-2661.