Research on Scheduling Strategies Simulation for Building Air-conditioning Systems Based on Transfer Imitation Learning

doi:10.16182/j.issn1004731x.joss.24-FZ0761

Abstract

Abstract:

To solve the problem of unstable performance and inefficient training process of low-quality data conditions at the initial stage of online deployment of air conditioner scheduling, we propose a migration-imitation learning-based air conditioning scheduling strategy simulation method. Reinforcement learning methods are used to generate building operation strategies. A standard building simulation model serves as the source domain, upon which migration learning is applied. An imitation learning loss function is incorporated into the intelligent loss function to enhance algorithm performance. The results indicate that, compared with the non-use of migration learning, the proposed method can improve the operational efficiency by 16.2%, effectively resolving the operational instability issues at the initial stage of reinforcement learning training. Compared to methods without imitation learning, operational efficiency is enhanced by 11.5%, significantly improving the training efficiency of reinforcement learning.

Key words: transfer learning, reinforcement learning, imitation learning, air conditioning control, room temperature control

CLC Number:

TP29

Wang Qiaochu, Ding Yan, Liang Chuanzhi, Zhang Haozheng, Huang Chen. Research on Scheduling Strategies Simulation for Building Air-conditioning Systems Based on Transfer Imitation Learning[J]. Journal of System Simulation, 2024, 36(12): 2884-2893.

Figures/Tables 9

Fig. 1

Fig. 2

Table 1

Table 2

RL algorithm hyperparameter setting

参数	设定值
$γ$	0.99
$ε$	0.8
$α$	0.01
经验回放池大小	869
批量采样样本数量	128
目标网络更新间隔	8
神经网络层数	5
神经网络隐藏层节点数	32~128
神经网络输入节点数	DQN状态空间维度
神经网络输出节点数	DQN动作空间维度
优化器	Adam
激活函数	ReLU

Table 2

Table 3

Fig. 3

Fig. 4

Fig. 5

Fig. 6

References 24

1	中国建筑节能协会, 重庆大学城乡建设与发展研究院. 中国建筑能耗与碳排放研究报告(2023年)[J]. 建筑, 2024(2): 46-59.
2	宋梦. 空调负荷参与电力系统调控关键技术研究[D]. 南京: 东南大学, 2018.
	Song Meng. Research on Key Technologies of Air Conditioning Loads Participating in Power System Dispatch and Control[D]. Nanjing: Southeast University, 2018.
3	孟庆龙, 张志博, 熊成. 变风量空调系统变水温策略运行效果仿真研究[J]. 系统仿真学报, 2019, 31(8): 1617-1626.
	Meng Qinglong, Zhang Zhibo, Xiong Cheng. Simulation Research on Operation Effect of VAV Air-conditioning System with Variable Water Temperature Strategy[J]. Journal of System Simulation, 2019, 31(8): 1617-1626.
4	Alghamdi Sultan M, Ajour Mohammed N, Abu-Hamdeh Nidal H, et al. Using PCM for Building Energy Management to Postpone the Electricity Demand Peak Load and Approving a New PID Controller to Activate Alternative Chiller[J]. Journal of Building Engineering, 2022, 57: 104884.
5	Wang Shengwei, Tang Rui. Supply-based Feedback Control Strategy of Air-conditioning Systems for Direct Load Control of Buildings Responding to Urgent Requests of Smart Grids[J]. Applied Energy, 2017, 201: 419-432.
6	张青. 中央空调系统节能运行控制方法研究[D]. 南京: 东南大学, 2016.
	Zhang Qing. Study on Energy Saving Control Stratedies of Central Air Conditioning System[D]. Nanjing: Southeast University, 2016.
7	Wang Wenyi, Zhou Qun, Pan Chao, et al. Energy-efficient Operation of a Complete Chiller-air Handing Unit System via Model Predictive Control[J]. Applied Thermal Engineering, 2022, 201, Part B: 117809.
8	Chen Sihao, Ding Puxian, Zhou Guang, et al. A Novel Machine Learning-based Model Predictive Control Framework for Improving the Energy Efficiency of Air-conditioning Systems[J]. Energy and Buildings, 2023, 294: 113258.
9	Kim D, Wang Zhe, Brugger J, et al. Site Demonstration and Performance Evaluation of MPC for a Large Chiller Plant with TES for Renewable Energy Integration and Grid Decarbonization[J]. Applied Energy, 2022, 321: 119343.
10	王霄汉, 张霖, 任磊, 等. 基于强化学习的车间调度问题研究简述[J]. 系统仿真学报, 2021, 33(12): 2782-2791.
	Wang Xiaohan, Zhang Lin, Ren Lei, et al. Brief Review on Applying Reinforcement Learning to Job Shop Scheduling Problems[J]. Journal of System Simulation, 2021, 33(12): 2782-2791.
11	段勇, 刘兴刚, 徐心和. 基于强化学习的机器人模糊控制系统设计[J]. 系统仿真学报, 2006, 18(6): 1597-1600.
	Duan Yong, Liu Xinggang, Xu Xinhe. Design of Robot Fuzzy Logic Controller Based on Reinforcement Learning[J]. Journal of System Simulation, 2006, 18(6): 1597-1600.
12	江达, 蔡志勤, 刘忠振, 等. 基于强化学习的连续型机械臂自适应跟踪控制[J]. 系统仿真学报, 2022, 34(10): 2264-2271.
	Jiang Da, Cai Zhiqin, Liu Zhongzhen, et al. Reinforcement-learning-based Adaptive Tracking Control for a Space Continuum Robot Based on Reinforcement Learning[J]. Journal of System Simulation, 2022, 34(10): 2264-2271.
13	李启锐, 彭心怡. 基于深度强化学习的云作业调度及仿真研究[J]. 系统仿真学报, 2022, 34(2): 258-268.
	Li Qirui, Peng Xinyi. Job Scheduling and Simulation in Cloud Based on Deep Reinforcement Learning[J]. Journal of System Simulation, 2022, 34(2): 258-268.
14	Zhang Xinfang, Li Zhenhai, Li Zhengwei, et al. Differential Pressure Reset Strategy Based on Reinforcement Learning for Chilled Water Systems[J]. Building Simulation, 2022, 15(2): 233-248.
15	Wang Dan, Gao Cheng, Sun Yuying, et al. Reinforcement Learning Control Strategy for Differential Pressure Setpoint in Large-scale Multi-source Looped District Cooling System[J]. Energy and Buildings, 2023, 282: 112778.
16	付玉, 张垚, 赵萌, 等. 基于仿真数据迁移学习的固定翼无人机检测[J]. 系统仿真学报, 2023, 35(5): 998-1007.
	Fu Yu, Zhang Yao, Zhao Meng, et al. Fixed-wing UAV Detection Based on Simulated Data Transfer Learning[J]. Journal of System Simulation, 2023, 35(5): 998-1007.
17	Tao Yuechuan, Qiu Jing, Lai Shuying. A Hybrid Cloud and Edge Control Strategy for Demand Responses Using Deep Reinforcement Learning and Transfer Learning[J]. IEEE Transactions on Cloud Computing, 2022, 10(1): 56-71.
18	Coraci Davide, Brandi Silvio, Hong Tianzhen, et al. Online Transfer Learning Strategy for Enhancing the Scalability and Deployment of Deep Reinforcement Learning Control in Smart Buildings[J]. Applied Energy, 2023, 333: 120598.
19	Fang Xi, Gong Guangcai, Li Guannan, et al. Cross Temporal-spatial Transferability Investigation of Deep Reinforcement Learning Control Strategy in the Building HVAC System Level[J]. Energy, 2023, 263, Part B: 125679.
20	Lee Z E, Zhang K. Generalized Reinforcement Learning for Building Control Using Behavioral Cloning[J]. Applied Energy, 2021, 304: 117602.
21	陈武, 周兴禧, 夏凊, 等. 双联变频空调系统建模与控制方法仿真研究[J]. 系统仿真学报, 2002, 14(5): 643-646.
	Chen Wu, Zhou Xingxi, Xia Qing, et al. Simulation Research on Control Strategies and Modeling of the Double-evaporator Air-conditioner with Inverter[J]. Journal of System Simulation, 2002, 14(5): 643-646.
22	Luo Sha, Schomaker Lambert. Reinforcement Learning in Robotic Motion Planning by Combined Experience-based Planning and Self-imitation Learning[J]. Robotics and Autonomous Systems, 2023, 170: 104545.
23	Bojarski M, Del Testa D, Dworakowski D, et al. End to End Learning for Self-driving Cars[EB/OL]. (2016-04-25) [2023-12-17]. .
24	Hester T, Vecerik M, Pietquin O, et al. Deep Q-learning from Demonstrations[C]//Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence. Palo Alto: AAAI Press, 2018: 3223-3230.

参数	目标建筑	源域
气候分区	寒冷	寒冷
气象文件	天津	北京
人员活动时间表	测试获得	与目标域相同
建筑面积/m²	3 970.5	3 582.2

模型	方法	源域	模仿学习损失函数	重复训练
A	迁移模仿学习	源域A	式(6)与式(7)	×
B	模仿学习	×	式(6)与式(7)	×
C	迁移学习	源域A	×	×
D	仅在线训练	×	×	×
E	离线训练	×	×	200轮

[1]	Xu Lixia, Zhong Jilong, Wu Shaoshi, Ding Yishan, Zhai Xiaoyu, Chen Shizhao, Wang Yizhe, Wen Xue, Zeng Juanfang, Hou Xinwen. Indicator Transfer Learning Based on Cloud Model and Maximum Mean Discrepancy [J]. Journal of System Simulation, 2024, 36(9): 2004-2015.
[2]	Sun Yifeng, Li Zhi, Wu Jiang, Wang Yubin. Research on Learnable Wargame Agent Driven by Battle Scheme [J]. Journal of System Simulation, 2024, 36(7): 1525-1535.
[3]	Jiang Quan, Wei Jingxuan. Real-time Scheduling Method for Dynamic Flexible Job Shop Scheduling [J]. Journal of System Simulation, 2024, 36(7): 1609-1620.
[4]	Gong Xue, Peng Pengfei, Rong Li, Zheng Yalian, Jiang Jun. Task Analysis Methods Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2024, 36(7): 1670-1681.
[5]	Qin Baoxin, Zhang Yuxiao, Wu Sirui, Cao Weichong, Li Zhan. Intelligent Optimization of Coal Terminal Unloading Scheduling Based on Improved D3QN Algorithm [J]. Journal of System Simulation, 2024, 36(3): 770-781.
[6]	Ding Kaiyuan, Hamdulla Askar, Zhu Bin, Firkat Eksan, Ma Zhengtang. End-to-end Motion Planning of Unmanned Vehicles Based on Multimodal Deep Reinforcement Learning [J]. Journal of System Simulation, 2024, 36(11): 2631-2643.
[7]	Zhang Yongfu, Liu Yang, Yuan He. A Method for Key Node Identification in Operational Target System Based on War Gaming [J]. Journal of System Simulation, 2024, 36(11): 2654-2661.
[8]	Li Jian, Li Huankun, He Pengbo, Wang Huabei, Xu Liping, He Kui. Flexible Job Shop Scheduling Method Based on Collaborative Agent Reinforcement Learning Algorithm [J]. Journal of System Simulation, 2024, 36(11): 2699-2711.
[9]	Zhang Guohui, Gao Ang, Zhang Ya'nan. Combat Effectiveness Evaluation Method of Homogeneous Cluster Equipment System Based on RLoMAG+EAS [J]. Journal of System Simulation, 2024, 36(1): 160-169.
[10]	An Jing, Si Guangya, Zhang Lei. Strategy Optimization Method of Multi-dimension Projection Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2024, 36(1): 39-49.
[11]	Guo Runxia, Wang Yifu. Aircraft Assignment Method for Optimal Utilization of Maintenance Intervals [J]. Journal of System Simulation, 2023, 35(9): 1985-1999.
[12]	Junqiang Lin, Hongjun Wang, Xiangjun Zou, Po Zhang, Chengen Li, Yipeng Zhou, Shujie Yao. Obstacle Avoidance Path Planning and Simulation of Mobile Picking Robot Based on DPPO [J]. Journal of System Simulation, 2023, 35(8): 1692-1704.
[13]	Jiayi Liu, Gang Wang, Qiang Fu, Xiangke Guo, Siyuan Wang. Intelligent Air Defense Task Assignment Based on Assignment Strategy Optimization Algorithm [J]. Journal of System Simulation, 2023, 35(8): 1705-1716.
[14]	Laiyi Yang, Jing Bi, Haitao Yuan. Intelligent Path Planning for Mobile Robots Based on SAC Algorithm [J]. Journal of System Simulation, 2023, 35(8): 1726-1736.
[15]	Fei Ding, Yuchen Sha, Ying Hong, Xiao Kuai, Dengyin Zhang. Joint Optimization Strategy of Computing Offloading and Edge Caching for Intelligent Connected Vehicles [J]. Journal of System Simulation, 2023, 35(6): 1203-1214.