Research on Scheduling Strategies Simulation for Building Air-conditioning Systems Based on Transfer Imitation Learning

doi:10.16182/j.issn1004731x.joss.24-FZ0761

Abstract

Abstract:

To solve the problem of unstable performance and inefficient training process of low-quality data conditions at the initial stage of online deployment of air conditioner scheduling, we propose a migration-imitation learning-based air conditioning scheduling strategy simulation method. Reinforcement learning methods are used to generate building operation strategies. A standard building simulation model serves as the source domain, upon which migration learning is applied. An imitation learning loss function is incorporated into the intelligent loss function to enhance algorithm performance. The results indicate that, compared with the non-use of migration learning, the proposed method can improve the operational efficiency by 16.2%, effectively resolving the operational instability issues at the initial stage of reinforcement learning training. Compared to methods without imitation learning, operational efficiency is enhanced by 11.5%, significantly improving the training efficiency of reinforcement learning.

Key words: transfer learning, reinforcement learning, imitation learning, air conditioning control, room temperature control

CLC Number:

TP29

Wang Qiaochu, Ding Yan, Liang Chuanzhi, Zhang Haozheng, Huang Chen. Research on Scheduling Strategies Simulation for Building Air-conditioning Systems Based on Transfer Imitation Learning[J]. Journal of System Simulation, 2024, 36(12): 2884-2893.

Figures/Tables 9

Fig. 1

Fig. 2

Table 1

Table 2

RL algorithm hyperparameter setting

参数	设定值
$γ$	0.99
$ε$	0.8
$α$	0.01
经验回放池大小	869
批量采样样本数量	128
目标网络更新间隔	8
神经网络层数	5
神经网络隐藏层节点数	32~128
神经网络输入节点数	DQN状态空间维度
神经网络输出节点数	DQN动作空间维度
优化器	Adam
激活函数	ReLU

Table 2

Table 3

Fig. 3

Fig. 4

Fig. 5

Fig. 6

References 24

1	中国建筑节能协会, 重庆大学城乡建设与发展研究院. 中国建筑能耗与碳排放研究报告(2023年)[J]. 建筑, 2024(2): 46-59.
2	宋梦. 空调负荷参与电力系统调控关键技术研究[D]. 南京: 东南大学, 2018.
	Song Meng. Research on Key Technologies of Air Conditioning Loads Participating in Power System Dispatch and Control[D]. Nanjing: Southeast University, 2018.
3	孟庆龙, 张志博, 熊成. 变风量空调系统变水温策略运行效果仿真研究[J]. 系统仿真学报, 2019, 31(8): 1617-1626.
	Meng Qinglong, Zhang Zhibo, Xiong Cheng. Simulation Research on Operation Effect of VAV Air-conditioning System with Variable Water Temperature Strategy[J]. Journal of System Simulation, 2019, 31(8): 1617-1626.
4	Alghamdi Sultan M, Ajour Mohammed N, Abu-Hamdeh Nidal H, et al. Using PCM for Building Energy Management to Postpone the Electricity Demand Peak Load and Approving a New PID Controller to Activate Alternative Chiller[J]. Journal of Building Engineering, 2022, 57: 104884.
5	Wang Shengwei, Tang Rui. Supply-based Feedback Control Strategy of Air-conditioning Systems for Direct Load Control of Buildings Responding to Urgent Requests of Smart Grids[J]. Applied Energy, 2017, 201: 419-432.
6	张青. 中央空调系统节能运行控制方法研究[D]. 南京: 东南大学, 2016.
	Zhang Qing. Study on Energy Saving Control Stratedies of Central Air Conditioning System[D]. Nanjing: Southeast University, 2016.
7	Wang Wenyi, Zhou Qun, Pan Chao, et al. Energy-efficient Operation of a Complete Chiller-air Handing Unit System via Model Predictive Control[J]. Applied Thermal Engineering, 2022, 201, Part B: 117809.
8	Chen Sihao, Ding Puxian, Zhou Guang, et al. A Novel Machine Learning-based Model Predictive Control Framework for Improving the Energy Efficiency of Air-conditioning Systems[J]. Energy and Buildings, 2023, 294: 113258.
9	Kim D, Wang Zhe, Brugger J, et al. Site Demonstration and Performance Evaluation of MPC for a Large Chiller Plant with TES for Renewable Energy Integration and Grid Decarbonization[J]. Applied Energy, 2022, 321: 119343.
10	王霄汉, 张霖, 任磊, 等. 基于强化学习的车间调度问题研究简述[J]. 系统仿真学报, 2021, 33(12): 2782-2791.
	Wang Xiaohan, Zhang Lin, Ren Lei, et al. Brief Review on Applying Reinforcement Learning to Job Shop Scheduling Problems[J]. Journal of System Simulation, 2021, 33(12): 2782-2791.
11	段勇, 刘兴刚, 徐心和. 基于强化学习的机器人模糊控制系统设计[J]. 系统仿真学报, 2006, 18(6): 1597-1600.
	Duan Yong, Liu Xinggang, Xu Xinhe. Design of Robot Fuzzy Logic Controller Based on Reinforcement Learning[J]. Journal of System Simulation, 2006, 18(6): 1597-1600.
12	江达, 蔡志勤, 刘忠振, 等. 基于强化学习的连续型机械臂自适应跟踪控制[J]. 系统仿真学报, 2022, 34(10): 2264-2271.
	Jiang Da, Cai Zhiqin, Liu Zhongzhen, et al. Reinforcement-learning-based Adaptive Tracking Control for a Space Continuum Robot Based on Reinforcement Learning[J]. Journal of System Simulation, 2022, 34(10): 2264-2271.
13	李启锐, 彭心怡. 基于深度强化学习的云作业调度及仿真研究[J]. 系统仿真学报, 2022, 34(2): 258-268.
	Li Qirui, Peng Xinyi. Job Scheduling and Simulation in Cloud Based on Deep Reinforcement Learning[J]. Journal of System Simulation, 2022, 34(2): 258-268.
14	Zhang Xinfang, Li Zhenhai, Li Zhengwei, et al. Differential Pressure Reset Strategy Based on Reinforcement Learning for Chilled Water Systems[J]. Building Simulation, 2022, 15(2): 233-248.
15	Wang Dan, Gao Cheng, Sun Yuying, et al. Reinforcement Learning Control Strategy for Differential Pressure Setpoint in Large-scale Multi-source Looped District Cooling System[J]. Energy and Buildings, 2023, 282: 112778.
16	付玉, 张垚, 赵萌, 等. 基于仿真数据迁移学习的固定翼无人机检测[J]. 系统仿真学报, 2023, 35(5): 998-1007.
	Fu Yu, Zhang Yao, Zhao Meng, et al. Fixed-wing UAV Detection Based on Simulated Data Transfer Learning[J]. Journal of System Simulation, 2023, 35(5): 998-1007.
17	Tao Yuechuan, Qiu Jing, Lai Shuying. A Hybrid Cloud and Edge Control Strategy for Demand Responses Using Deep Reinforcement Learning and Transfer Learning[J]. IEEE Transactions on Cloud Computing, 2022, 10(1): 56-71.
18	Coraci Davide, Brandi Silvio, Hong Tianzhen, et al. Online Transfer Learning Strategy for Enhancing the Scalability and Deployment of Deep Reinforcement Learning Control in Smart Buildings[J]. Applied Energy, 2023, 333: 120598.
19	Fang Xi, Gong Guangcai, Li Guannan, et al. Cross Temporal-spatial Transferability Investigation of Deep Reinforcement Learning Control Strategy in the Building HVAC System Level[J]. Energy, 2023, 263, Part B: 125679.
20	Lee Z E, Zhang K. Generalized Reinforcement Learning for Building Control Using Behavioral Cloning[J]. Applied Energy, 2021, 304: 117602.
21	陈武, 周兴禧, 夏凊, 等. 双联变频空调系统建模与控制方法仿真研究[J]. 系统仿真学报, 2002, 14(5): 643-646.
	Chen Wu, Zhou Xingxi, Xia Qing, et al. Simulation Research on Control Strategies and Modeling of the Double-evaporator Air-conditioner with Inverter[J]. Journal of System Simulation, 2002, 14(5): 643-646.
22	Luo Sha, Schomaker Lambert. Reinforcement Learning in Robotic Motion Planning by Combined Experience-based Planning and Self-imitation Learning[J]. Robotics and Autonomous Systems, 2023, 170: 104545.
23	Bojarski M, Del Testa D, Dworakowski D, et al. End to End Learning for Self-driving Cars[EB/OL]. (2016-04-25) [2023-12-17]. .
24	Hester T, Vecerik M, Pietquin O, et al. Deep Q-learning from Demonstrations[C]//Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence. Palo Alto: AAAI Press, 2018: 3223-3230.

参数	目标建筑	源域
气候分区	寒冷	寒冷
气象文件	天津	北京
人员活动时间表	测试获得	与目标域相同
建筑面积/m²	3 970.5	3 582.2

模型	方法	源域	模仿学习损失函数	重复训练
A	迁移模仿学习	源域A	式(6)与式(7)	×
B	模仿学习	×	式(6)与式(7)	×
C	迁移学习	源域A	×	×
D	仅在线训练	×	×	×
E	离线训练	×	×	200轮

[1]	Jiang Ming, He Tao. Solving the Vehicle Routing Problem Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(9): 2177-2187.
[2]	Ni Peilong, Mao Pengjun, Wang Ning, Yang Mengjie. Robot Path Planning Based on Improved A-DDQN Algorithm [J]. Journal of System Simulation, 2025, 37(9): 2420-2430.
[3]	Chen Zhen, Wu Zhuoyi, Zhang Lin. Research on Policy Representation in Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(7): 1753-1769.
[4]	Gu Xueqiang, Luo Junren, Zhou Yanzhong, Zhang Wanpeng. Survey on Large Language Agent Technologies for Intelligent Game Theoretic Decision-making [J]. Journal of System Simulation, 2025, 37(5): 1142-1157.
[5]	Wu Guohua, Zeng Jiaheng, Wang Dezhi, Zheng Long, Zou Wei. A Quadrotor Trajectory Tracking Control Method Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(5): 1169-1187.
[6]	Xu Ming, Li Jinye, Zuo Dongyu, Zhang Jing. Signal Timing Optimization via Reinforcement Learning with Traffic Flow Prediction [J]. Journal of System Simulation, 2025, 37(4): 1051-1062.
[7]	Lu Xinbiao, Ye Chunlin, Chen Yisen, Wu Wen, Chen Yudan. A Transfer Learning-based Hybrid Model for PM_2.5Concentration Prediction [J]. Journal of System Simulation, 2025, 37(4): 882-894.
[8]	Wang Xin, Cui Chenggang, Wang Xiangxiang, Zhu Ping. Research on Economic Dispatching Strategy of CHP Units Based on SRL [J]. Journal of System Simulation, 2025, 37(4): 968-981.
[9]	Zhang Lei, Zhang Xuechao, Wang Chao, Bo Xianglei. An Intelligent Ambulance Regulation Model Based on Online Reinforcement Learning Algorithm [J]. Journal of System Simulation, 2025, 37(3): 584-594.
[10]	Bai Zhenzu, Hou Yizhi, He Zhangming, Wei Juhui, Zhou Haiyin, Wang Jiongqi. Optimization of Dynamic Weapon Target Assignment Considering Random Disturbances [J]. Journal of System Simulation, 2025, 37(12): 2967-2980.
[11]	Liu Xiang, Jin Qiankun. Research on PAC-Bayes-Based A2C Algorithm for Multi-objective Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(12): 3212-3223.
[12]	Zheng Jiayu, Mai Zhuxue, Chen Zheyi. Optimization of Service Caching and Computation Offloading in Digital Twin Cloud-edge Networks [J]. Journal of System Simulation, 2025, 37(11): 2741-2753.
[13]	Xing Lijing, Li Min, Zeng Xiangguang, Zhang Ping, Peng Bei. AUV Path Planning Based on Behavior Cloning and Improved DQN in Partially Unknown Environments [J]. Journal of System Simulation, 2025, 37(11): 2754-2767.
[14]	Di Jian, Wan Xue, Jiang Limei. Evolutionary Reinforcement Learning Based on Elite Instruction and Random Search [J]. Journal of System Simulation, 2025, 37(11): 2877-2887.
[15]	Xu Zhongkai, Chu Chenyang, Xie Kai, Zhao Ruizhuo, Ke Wenjun. Optimization Dispatch Method for High-proportion Renewable Energy Power Systems Based on SC-PPO [J]. Journal of System Simulation, 2025, 37(10): 2511-2521.