Chemical Plant Control Based on Dynamic Simulation and Reinforcement Learning

doi:10.16182/j.issn1004731x.joss.26-0140

Abstract

Abstract:

An intelligent control strategy framework combining offline training and on-site deployment for regenerative thermal oxidizer (RTO) operation control based on RL is proposed to solve the problem that fluctuations in exhaust gas concentration and flow, as well as changes in working conditions, make it difficult for RTO to maintain the system operating parameters within the economically optimal range in practical applications. In the offline phase, RL is employed to enable autonomous policy learning and global optimization. In the online phase, an adaptive policy correction mechanism based on real-time data is introduced. RL control strategy is integrated with the operation monitoring and visualization system to intuitively present operational status and control effects. Simulation results demonstrate that the CO₂ emissions per unit of steam are reduced by approximately 1.4%, and the cumulative economic benefit over a six-month operation period is increased by 0.83%. The steam production is increased by 5.18% in actual industrial conditions.

Key words: regenerative thermal oxidizer, reinforcement learning, dynamic simulation, intelligent control, offline training and on-site deployment

CLC Number:

TP391

Li Yuxin, Li Haijie, Han Shangbo, Gao Kai, Zhang Qingyuan, Xiao Yanwen, Liu Zhengcai. Chemical Plant Control Based on Dynamic Simulation and Reinforcement Learning[J]. Journal of System Simulation, 2026, 38(10): 2802-2818.

Figures/Tables 15

Fig. 1

Fig. 2

Fig. 3

Table 1

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Fig. 11

Fig. 12

Fig. 13

Fig. 14

References 36

[1]	Wang Jin, He Qinghua. Methane Removal from Air: Challenges and Opportunities[J]. Methane, 2023, 2(4): 404-414.
[2]	班鹏. 旋转蓄热氧化装置蓄热室的研究及优化[D]. 哈尔滨: 哈尔滨工业大学, 2022.
	Ban Peng. Study and Optimization on the Regenerator of a Regenerative Rotary Thermal Oxidizer Furnace[D]. Harbin: Harbin Institute of Technology, 2022.
[3]	宋宜凯, 申炜杰, 曾博, 等. 计及[火用]分析的矿山综合能源系统多目标运行方法[J]. 太阳能学报, 2025, 46(9): 650-660.
	Song Yikai, Shen Weijie, Zeng Bo, et al. Multi-objective Operation Method for Coal Mine Integrated Energy Systems Considering Exergy Analysis[J]. Acta Energiae Solaris Sinica, 2025, 46(9): 650-660.
[4]	赵旭生, 马国龙. 煤矿瓦斯智能抽采关键技术研究进展及展望[J]. 煤炭科学技术, 2021, 49(5): 27-34.
	Zhao Xusheng, Ma Guolong. Research Progress and Prospect of Key Technology of Intelligent Gas Drainage in Coal Mine[J]. Coal Science and Technology, 2021, 49(5): 27-34.
[5]	Procacci Alberto, Amaduzzi Ruggero, Coussement Axel, et al. Adaptive Digital Twins of Combustion Systems Using Sparse Sensing Strategies[J]. Proceedings of the Combustion Institute, 2023, 39(4): 4257-4266.
[6]	Bottari Agustín, Braccia Lautaro. Optimal Control Structure Selection Based on Economics for Continuous Cross-flow Grain Dryer[J]. Drying Technology, 2023, 41(10): 1605-1619.
[7]	黄代存. 蓄热式氧化炉在炼油污水处理厂中的应用[J]. 化工环保, 2024, 44(4): 589-594.
	Huang Daicun. Application of Regenerative Thermal Oxidizer in Refinery Wastewater Treatment Plant[J]. Environmental Protection of Chemical Industry, 2024, 44(4): 589-594.
[8]	张磊. 同步Buck DC-DC变换器的模糊控制研究[D]. 贵阳: 贵州大学, 2024.
[9]	Milojkovic Marko T, Djordjevic Andjela D, Stanisa Lj Peric, et al. Model Predictive Control of Nonlinear MIMO Systems Based on Adaptive Orthogonal Polynomial Networks[J]. Elektronika Ir Elektrotechnika, 2021, 27(2): 4-10.
[10]	Borase Rakesh P, Maghade D K, Sondkar S Y, et al. A Review of PID Control, Tuning Methods and Applications[J]. International Journal of Dynamics and Control, 2021, 9(2): 818-827.
[11]	Schwenzer Max, Ay Muzaffer, Bergs Thomas, et al. Review on Model Predictive Control: an Engineering Perspective[J]. The International Journal of Advanced Manufacturing Technology, 2021, 117(5): 1327-1349.
[12]	Darby M L, Nikolaou M, Jones J, et al. RTO: An Overview and Assessment of Current Practice[J]. Journal of Process Control, 2011, 21(6): 874-884.
[13]	Glauce De Souza, Odloak Darci, Zanin Antônio C. Real Time Optimization (RTO) with Model Predictive Control (MPC)[J]. Computers & Chemical Engineering, 2010, 34(12): 1999-2006.
[14]	Eduardo A Graciano José, Jäschke Johannes, Galo A C Le Roux, et al. Integrating Self-optimizing Control and Real-time Optimization Using zone Control MPC[J]. Journal of Process Control, 2015, 34: 35-48.
[15]	Adetola V, Guay M. Integration of Real-time Optimization & and Model Predictive Control[J]. IFAC Proceedings Volumes, 2009, 42(11): 63-68.
[16]	孙雁伯, 袁兴海, 林琳, 等. 三室蓄热式热力氧化炉热工性能数值模拟研究[J]. 力学与实践, 2024, 46(5): 1023-1036.
	Sun Yanbo, Yuan Xinghai, Lin Lin, et al. Numerical Simulation of Thermal Performance of Three-canister Regenerative Thermal Oxidizers[J]. Mechanics in Engineering, 2024, 46(5): 1023-1036.
[17]	陈相煜, 王波, 周秋平, 等. 煤矿乏风蓄热氧化过程数值模拟[J]. 化学工程, 2023, 51(1): 78-83.
	Chen Xiangyu, Wang Bo, Zhou Qiuping, et al. Simulation on Regenerative Thermal Oxidation of Coal Mine Ventilation Air Methane[J]. Chemical Engineering, 2023, 51(1): 78-83.
[18]	张睿智, 罗永浩, 赵善辉, 等. 低浓度VOCs气体漩涡燃烧分解过程的数值模拟研究[J]. 工业锅炉, 2023(5): 7-11.
	Zhang Ruizhi, Luo Yonghao, Zhao Shanhui, et al. Numerical Simulation Study on the Decomposition Process of Low-concentration VOCs by Vortex Combustion[J]. Industrial Boilers, 2023(5): 7-11.
[19]	萧琦, 姜泽毅, 张欣欣. 蓄热式有机废气焚烧炉的数值模拟和应用[J]. 北京科技大学学报, 2011, 33(5): 632-635.
	Xiao Qi, Jiang Zeyi, Zhang Xinxin. Numerical Simulation and Application of a Regenerative Thermal Oxidizer[J]. Journal of University of Science and Technology Beijing, 2011, 33(5): 632-635.
[20]	Sun Bo, Zhao Tong, Li Lu, et al. Optimization of Regenerator Operating Parameters and Thermal Insulation Construction for Rotary Regenerative Thermal Oxidizer (r-RTO) Based on Thermal-fluid Coupling Method and Quadratic Regression Model[J]. Case Studies in Thermal Engineering, 2022, 37: 102314.
[21]	刘丽娜, 尹浩东. 延误场景下基于强化学习的城轨列车运行调度优化[C]//2025世界交通运输大会(WTC2025). 北京: 人民交通出版社, 2025: 35-40.
	Liu Lina, Yin Haodong. Reinforcement Learning-based Optimization of Urban Rail Transit Train Rescheduling Under Delay Scenarios[C]//2025 World Transport Convention (WTC2025). Beijing: China Communications Press, 2025: 35-40.
[22]	姜嘉成, 贾政轩, 徐钊, 等. 基于博弈对抗复杂系统的决策建模与求解[J]. 系统仿真学报, 2025, 37(1): 66-78.
	Jiang Jiacheng, Jia Zhengxuan, Xu Zhao, et al. Decision Modeling and Solution Based on Game Adversarial Complex Systems[J]. Journal of System Simulation, 2025, 37(1): 66-78.
[23]	Park Joonsoo, Jung Hyein, Woo Kim Jong, et al. Reinforcement Learning for Process Control: Review and Benchmark Problems[J]. International Journal of Control Automation and Systems, 2025, 23(1): 1-40.
[24]	李文礼, 徐洋, 王云卿. 基于强化学习的路径跟踪控制方法[J]. 重庆理工大学学报(自然科学), 2025, 39(12): 19-28.
	Li Wenli, Xu Yang, Wang Yunqing. Path Tracking Control Method Based on Reinforcement Learning[J]. Journal of Chongqing University of Technology(Natural Science), 2025, 39(12): 19-28.
[25]	Yang Taoran, Guo Hongwei, Liang Helan, et al. Intelligent Combustion Control of the Hot Blast Stove: A Reinforcement Learning Approach[J]. Processes, 2023, 11(11): 3140.
[26]	Mathew Mithra Noel, Jaganatha Pandian B. Control of a Nonlinear Liquid Level System Using a New Artificial Neural Network Based Reinforcement Learning Approach[J]. Applied Soft Computing, 2014, 23: 444-451.
[27]	Dutta Debaprasad, Upreti Simant R. A Reinforcement Learning-based Transformed Inverse Model Strategy for Nonlinear Process Control[J]. Computers & Chemical Engineering, 2023, 178: 108386.
[28]	Mendiola-Rodriguez Tannia A, Ricardez-Sandoval Luis A. Robust Control for Anaerobic Digestion Systems of Tequila Vinasses Under Uncertainty: A Deep Deterministic Policy Gradient Algorithm[J]. Digital Chemical Engineering, 2022, 3: 100023.
[29]	Syauqi Ahmad, Kim Heehyang, Lim Hankwon. Optimizing Olefin Purification: An Artificial Intelligence-based Process-conscious PI Controller Tuning for Double Dividing Wall Column Distillation[J]. Chemical Engineering Journal, 2024, 500: 156645.
[30]	Zhang Yang, Wang Peng, Yu Liying, et al. Adaptive Tuning of Dynamic Matrix Control for Uncertain Industrial Systems with Deep Reinforcement Learning[J]. IEEE Transactions on Automation Science and Engineering, 2025, 22: 8695-8708.
[31]	Petsagkourakis P, Sandoval I O, Bradford E, et al. Constrained Reinforcement Learning for Dynamic Optimization under Uncertainty[J]. IFAC-PapersOnLine, 2020, 53(2): 11264-11270.
[32]	Xia Kaishu, Sacco C, Kirkpatrick M, et al. A Digital Twin to Train Deep Reinforcement Learning Agent for Smart Manufacturing Plants: Environment, Interfaces and Intelligence[J]. Journal of Manufacturing Systems, 2021, 58: 210-230.
[33]	Liu Chao, Ding Jinliang, Sun Jiyuan. Reinforcement Learning Based Decision Making of Operational Indices in Process Industry Under Changing Environment[J]. IEEE Transactions on Industrial Informatics, 2021, 17(4): 2727-2736.
[34]	Zhou Ping, Wang Xuan, Chai Tianyou. Multiobjective Operation Optimization of Wastewater Treatment Process Based on Reinforcement Self-learning and Knowledge Guidance[J]. IEEE Transactions on Cybernetics, 2023, 53(11): 6896-6909.
[35]	Zhao Mingming, Wang Ding, Ren Jin, et al. Integrated Online Q-learning Design for Wastewater Treatment Processes[J]. IEEE Transactions on Industrial Informatics, 2025, 21(2): 1833-1842.
[36]	东方仿真科技(北京)有限公司. AI+仿真实验平台[EB/OL]. (2026-01-26) [2026-02-01]. .

参数描述	变量名	空间
甲烷入口浓度/%	VCH4.OP	状态
主风机入口压力/Pa	PTC101.OP	状态
引风机入口压力/Pa	PTC301.OP	状态
主风机频率/Hz	B201HZ.OP	动作
引风机频率/Hz	B301HZ.OP	动作
热旁通阀门开度/%	FV202.OP	动作