Soft Sensor Modeling Based on Improved Transformer in Dual-stream Framework

doi:10.16182/j.issn1004731x.joss.24-0490

Abstract

Abstract:

Industrial process information is highly nonlinear and dynamic, with long-term dependencies between data, making it difficult to adequately extract time-series features. To address this issue, an improved Transformer-based soft sensor model in a dual-stream framework was proposed. The data were segmented and expanded. The features were extracted in parallel using a dual-stream structure combining a convolutional neural network with a self-attention mechanism and the improved Transformer model. The dual-stream features were fused for soft sensor regression. Residual connections were further introduced to accelerate the convergence speed of the model, and an orthogonal random features-based improved multi-head attention mechanism was adopted to solve the vanishing gradient and non-parallelization issues faced by traditional recurrent neural networks. Numerical simulations have verified the effectiveness and superiority of the proposed method.

Key words: soft sensor, deep learning, convolutional neural network, Transformer, residual connection, debutanizer

CLC Number:

TP274

Gu Hao, Wang Jiayu, Xiong Weili. Soft Sensor Modeling Based on Improved Transformer in Dual-stream Framework[J]. Journal of System Simulation, 2025, 37(10): 2594-2604.

Figures/Tables 14

Fig. 1

Fig. 2

Fig. 3

Table 1

Table 2

Fig. 4

Fig. 5

Table 3

Fig. 6

Table 4

Selection of auxiliary variables

变量	描述	变量	描述
$x 1$	顶部温度	$x 5$	第六塔板温度
$x 2$	顶部压力	$x 6$	底部温度1
$x 3$	回流流量	$x 7$	底部温度2
$x 4$	流向下一过程流量

Table 4

Table 5

Fig. 7

Fig. 8

Table 6

References 26

[1]	Chiplunkar R, Huang Biao. Siamese Neural Network-based Supervised Slow Feature Extraction for Soft Sensor Application[J]. IEEE Transactions on Industrial Electronics, 2021, 68(9): 8953-8962.
[2]	Fortuna L, Graziani S, Rizzo A, et al. Soft Sensors for Monitoring and Control of Industrial Processes[M]. London: Springer, 2007: 27-51.
[3]	熊伟丽, 孙文心, 马君霞. 带自相关约束的NARX动态软测量模型[J]. 控制与决策, 2020, 35(4): 816-822.
	Xiong Weili, Sun Wenxin, Ma Junxia. Autocorrelation Constrained NARX Dynamic Soft Sensing Model[J]. Control and Decision, 2020, 35(4): 816-822.
[4]	代学志, 熊伟丽. 基于概率选择的主动学习智能软测量建模[J]. 系统仿真学报, 2021, 33(6): 1350-1357.
	Dai Xuezhi, Xiong Weili. Active Learning Intelligent Soft Sensor Based on Probability Selection[J]. Journal of System Simulation, 2021, 33(6): 1350-1357.
[5]	Guo Runyuan, Liu Han. A Hybrid Mechanism-and Data-driven Soft Sensor Based on the Generative Adversarial Network and Gated Recurrent Unit[J]. IEEE Sensors Journal, 2021, 21(22): 25901-25911.
[6]	周平, 张丽, 李温鹏, 等. 集成自编码与PCA的高炉多元铁水质量随机权神经网络建模[J]. 自动化学报, 2018, 44(10): 1799-1811.
	Zhou Ping, Zhang Li, Li Wenpeng, et al. Autoencoder and PCA Based RVFLNs Modeling for Multivariate Molten Iron Quality in Blast Furnace Ironmaking[J]. Acta Automatica Sinica, 2018, 44(10): 1799-1811.
[7]	Xie Xiaochen, Sun Wei, Cheung K C. An Advanced PLS Approach for Key Performance Indicator-related Prediction and Diagnosis in Case of Outliers[J]. IEEE Transactions on Industrial Electronics, 2016, 63(4): 2587-2594.
[8]	耿增显, 柴天佑. 基于LS-SVM的浮选过程工艺技术指标软测量[J]. 系统仿真学报, 2008, 20(23): 6321-6324.
	Geng Zengxian, Chai Tianyou. Soft Sensor of Technical Indices Based on LS-SVM for Flotation Process[J]. Journal of System Simulation, 2008, 20(23): 6321-6324.
[9]	Yan Weiwu, Tang Di, Lin Yujun. A Data-driven Soft Sensor Modeling Method Based on Deep Learning and Its Application[J]. IEEE Transactions on Industrial Electronics, 2017, 64(5): 4237-4245.
[10]	Yao Le, Ge Zhiqiang. Deep Learning of Semisupervised Process Data with Hierarchical Extreme Learning Machine and Soft Sensor Application[J]. IEEE Transactions on Industrial Electronics, 2018, 65(2): 1490-1498.
[11]	徐芳萍, 杨辉, 陈俊, 等. 基于迁移学习与残差注意力卷积网络的稀土元素组分含量软测量[J]. 化工学报, 2025, 76(4): 1647-1660.
	Xu Fangping, Yang Hui, Chen Jun, et al. Soft Sensor of Rare Earth Element Content with Transfer Learning and Residual Attention Convolutional Neural Network[J]. CIESC Journal, 2025, 76(4): 1647-1660.
[12]	Ma Fangyuan, Ji Cheng, Wang Jingde, et al. Early Identification of Process Deviation Based on Convolutional Neural Network[J]. Chinese Journal of Chemical Engineering, 2023, 56: 104-118.
[13]	Yuan Xiaofeng, Qi Shuaibin, A W Shardt Yuri, et al. Soft Sensor Model for Dynamic Processes Based on Multichannel Convolutional Neural Network[J]. Chemometrics and Intelligent Laboratory Systems, 2020, 203: 104050.
[14]	Zhao Yingying, He Yigang, Xing Zhikai, et al. Multibranch 1-D CNN Based on Attention Mechanism for the DAB Converter Fault Diagnosis[J]. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 1-12.
[15]	Fang Chenyu, He Dakuo, Li Kang, et al. Image-based Thickener Mud Layer Height Prediction with Attention Mechanism-based CNN[J]. ISA Transactions, 2022, 128, Part B: 677-689.
[16]	黄婉蓉, 何凯, 刘坤, 等. 基于注意力机制的手写体中文字符识别[J]. 激光与光电子学进展, 2020, 57(8): 29-34.
	Huang Wanrong, He Kai, Liu Kun, et al. Handwritten Chinese Character Recognition Based on Attention Mechanism[J]. Laser & Optoelectronics Progress, 2020, 57(8): 29-34.
[17]	Vaswani A, Shazeer N, Parmar N, et al. Attention is All You Need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2017: 6000-6010.
[18]	Geng Zhiqiang, Chen Zhiwei, Meng Qingchao, et al. Novel Transformer Based on Gated Convolutional Neural Network for Dynamic Soft Sensor Modeling of Industrial Processes[J]. IEEE Transactions on Industrial Informatics, 2022, 18(3): 1521-1529.
[19]	Chang Peng, Zhang Shirao, Wang Zichen. Soft Sensor of the Key Effluent Index in the Municipal Wastewater Treatment Process Based on Transformer[J]. IEEE Transactions on Industrial Informatics, 2024, 20(3): 4021-4028.
[20]	Tay Y, Dehghani Mostafa, Bahri D. Efficient Transformers: A Survey[J]. ACM Computing Surveys, 2022, 55(6): 109.
[21]	Zha Wenshu, Liu Yuping, Wan Yujin, et al. Forecasting Monthly Gas Field Production Based on the CNN-LSTM Model[J]. Energy, 2022, 260: 124889.
[22]	Li Chang, Huang Xiaoyang, Song Rencheng, et al. EEG-based Seizure Prediction via Transformer Guided CNN[J]. Measurement, 2022, 203: 111948.
[23]	Friedman J H. Multivariate Adaptive Regression Splines[J]. The Annals of Statistics, 1991, 19(1): 1-67.
[24]	Meng Yanmei, Lan Qiliang, Qin J, et al. Data-driven Soft Sensor Modeling Based on Twin Support Vector Regression for Cane Sugar Crystallization[J]. Journal of Food Engineering, 2019, 241: 159-165.
[25]	何罗苏阳, 熊伟丽. 助训练策略下的多模型软测量建模[J]. 系统仿真学报, 2024, 36(1): 249-259.
	He Luosuyang, Xiong Weili. Multi-model Soft Sensor Modeling Under Help-training Strategy[J]. Journal of System Simulation, 2024, 36(1): 249-259.
[26]	Fortuna L, Graziani S, Xibilia M G. Soft Sensors for Product Quality Monitoring in Debutanizer Distillation Columns[J]. Control Engineering Practice, 2005, 13(4): 499-508.

模型	层数	隐藏层节点数	学习率	迭代次数
CNN	2	32，64	0.001	100
SAT-CNN	2	72, /	0.001	100
Transformer	1	72	0.001	100
F-Transformer	1	72	0.005	100
CNN-LSTM	2	32, 64	0.005	100
CNN-Transformer	2	72, 72	0.001	100
DSF-Transformer	2	72, 72	0.005	100

模型	RMSE	MAE	R²
CNN	2.070 9	1.677 6	0.706 6
SAT-CNN	1.974 8	1.598 2	0.733 2
Transformer	1.974 1	1.548 9	0.733 4
F-Transformer	1.886 2	1.493 1	0.756 6
CNN-LSTM	1.813 5	1.432 0	0.775 0
CNN-Transformer	1.771 7	1.391 6	0.785 3
DSF-Transformer	1.642 1	1.283 5	0.815 5

模型	运行时间/s	内存占用/MB
CNN	0.07	367.60
SAT-CNN	0.06	366.72
Transformer	0.37	401.16
F-Transformer	0.16	382.20
CNN-LSTM	0.13	404.22
CNN-Transformer	0.25	405.66
DSF-Transformer	0.17	387.94

模型	层数	隐藏层节点数	学习率	迭代次数
CNN	2	32, 64	0.001	100
SAT-CNN	2	72, /	0.001	100
Transformer	1	72	0.005	50
F-Transformer	1	72	0.005	50
CNN-LSTM	2	64, 128	0.001	100
CNN-Transformer	2	72, 72	0.001	50
DSF-Transformer	2	72, 72	0.005	50

模型	RMSE	MAE	R²
CNN	0.049	0.040	0.942
SAT-CNN	0.042	0.034	0.956
Transformer	0.059	0.048	0.915
F-Transformer	0.053	0.040	0.932
CNN-LSTM	0.045	0.036	0.950
CNN-Transformer	0.042	0.031	0.957
DSF-Transformer	0.028	0.021	0.981