Operation System for Simulation Roadheader Based on Visual Motion Capture

doi:10.16182/j.issn1004731x.joss.24-0516

Abstract

Abstract:

To enhance the natural human-machine interaction in simulation roadheader environment, a vision-based simulation roadheader operation system is proposed. The visual motion capture unit is based on the MediaPipe framework, which captures hand gestures through cameras and creates a correspondence between the physical world and virtual space. An improved Kalman filter algorithm is proposed by setting a weighted centroid to address the issue of unreasonable jumps in hand keypoint data during large-scale movements. The operator's gestures are discerned and the corresponding commands are conveyed. The results show that the improved method has significant advantages over the control group in terms of mean square error, signal-to-noise ratio, and approximate entropy parameters. The gesture recognition system is developed with an accuracy rate exceeding 92%. This interface enables the operator to efficiently control the simulated tunneling machine.

Key words: human-machine interaction, motion capture, Kalman filter, simulated operation system, approximate entropy

CLC Number:

TP391.9

Li Yongling, Liu Lingzhi, Zhou Baishun, Lei Jingfa, Zhang Miao, Zhao Ruhai. Operation System for Simulation Roadheader Based on Visual Motion Capture[J]. Journal of System Simulation, 2025, 37(6): 1531-1541.

Figures/Tables 13

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Table 1

Table 2

Table 3

Table 4

Fig. 8

Table5

References 28

1	董泽宇, 刘政, 李勇, 等. 基于混合现实的可移动机械臂操作控制系统[J]. 激光与光电子学进展, 2024, 61(4): 271-279.
	Dong Zeyu, Liu Zheng, Li Yong, et al. Operation Control System for Movable Robotic Arm Based on Mixed Reality[J]. Laser & Optoelectronics Progress, 2024, 61(4): 271-279.
2	邵绪强, 张浩伟, 冯小华. 面向电力变压器虚拟装配的多感官融合交互方法[J]. 系统仿真学报, 2022, 34(10): 2244-2254.
	Shao Xuqiang, Zhang Haowei, Feng Xiaohua. Multi-sensory Fusion Method for Power Transformer Virtual Assembly[J]. Journal of System Simulation, 2022, 34(10): 2244-2254.
3	方维, 许澍虹, 韩磊, 等. AR增强装配中的跟踪注册方法研究与应用进展[J]. 系统仿真学报, 2023, 35(7): 1438-1454.
	Fang Wei, Xu Shuhong, Han Lei, et al. Research and Application Progress of Tracking Registration Methods in AR Assembly[J]. Journal of System Simulation, 2023, 35(7): 1438-1454.
4	张旭辉, 王甜, 张超, 等. 数字孪生驱动的悬臂式掘进机虚拟示教记忆截割方法[J]. 煤炭学报, 2023, 48(11): 4247-4260.
	Zhang Xuhui, Wang Tian, Zhang Chao, et al. DT-driven Memory Cutting Control Method Using VR Instruction of Boom-type Roadheader[J]. Journal of China Coal Society, 2023, 48(11): 4247-4260.
5	张旭辉, 张超, 王妙云, 等. 数字孪生驱动的悬臂式掘进机虚拟操控技术[J]. 计算机集成制造系统, 2021, 27(6): 1617-1628.
	Zhang Xuhui, Zhang Chao, Wang Miaoyun, et al. Digital Twin-driven Virtual Control Technology of Cantilever Roadheader[J]. Computer Integrated Manufacturing Systems, 2021, 27(6): 1617-1628.
6	蔡兴旺, 裴耀强, 杨继华, 等. 煤矿掘进机系统数字孪生技术研究[J]. 系统仿真学报, 2024, 36(10): 2455-2468.
	Cai Xingwang, Pei Yaoqiang, Yang Jihua, et al. Research on Digital Twin Technology of Coal Mine Tunneling Machine System[J]. Journal of System Simulation, 2024, 36(10): 2455-2468.
7	王涛, 吴迎年, 杨睿, 等. 基于IMU与sEMG混合信号的实时手势分类算法研究[J]. 系统仿真学报, 2023, 35(2): 359-371.
	Wang Tao, Wu Yingnian, Yang Rui, et al. Research on Real-time Gesture Classification Algorithm Based on IMU and sEMG Mixed Signals[J]. Journal of System Simulation, 2023, 35(2): 359-371.
8	朱银龙, 沈宏骏, 吴杰, 等. 基于柔性应变传感器的数据手套手势识别研究[J]. 农业机械学报, 2024, 55(6): 451-458.
	Zhu Yinlong, Shen Hongjun, Wu Jie, et al. Data Glove Gesture Recognition Based on Flexible Strain Sensors[J]. Transactions of the Chinese Society for Agricultural Machinery, 2024, 55(6): 451-458.
9	吴常铖, 曹青青, 费飞, 等. 基于数据手套和神经网络的数字手势识别方法[J]. 东南大学学报(自然科学版), 2020, 50(3): 563-569.
	Wu Changcheng, Cao Qingqing, Fei Fei, et al. Digital Gesture Recognition Method Based on Data Glove and Neural Networks[J]. Journal of Southeast University(Natural Science Edition), 2020, 50(3): 563-569.
10	叶素芬, 赖际舟, 吕品, 等. 穿戴式钢琴弹奏手套智能感知与手势识别技术[J]. 仪器仪表学报, 2019, 40(5): 187-194.
	Ye Sufen, Lai Jizhou, Pin Lü, et al. An Intelligent Perception and Gesture Recognition Technology for Wearable Piano-playing Glove[J]. Chinese Journal of Scientific Instrument, 2019, 40(5): 187-194.
11	蔡嘉怡, 初萍, 庄伦涛, 等. 基于空间属性特征的毫米波雷达身体干扰识别[J]. 系统工程与电子技术, 2024, 46(10): 3365-3374.
	Cai Jiayi, Chu Ping, Zhuang Luntao, et al. Millimeter-wave Radar Body Interference Recognition Based on Spatial Attribute Features[J]. Systems Engineering and Electronics, 2024, 46(10): 3365-3374.
12	冯翔, 刘涛, 崔文卿, 等. 基于双视角时序特征融合的毫米波雷达手势数字识别研究[J]. 电子与信息学报, 2023, 45(6): 2134-2143.
	Feng Xiang, Liu Tao, Cui Wenqing, et al. Handwriting Number Recognition Based on Millimeter-wave Radar with Dual-view Feature Fusion Network[J]. Journal of Electronics & Information Technology, 2023, 45(6): 2134-2143.
13	康来, 张亚坤. 基于多维投影时空事件帧的动态视觉传感手势识别[J]. 系统仿真学报, 2024, 36(3): 649-658.
	Kang Lai, Zhang Yakun. Gesture Recognition for Dynamic Vision Sensor Based on Multi-dimensional Projection Spatiotemporal Event Frame[J]. Journal of System Simulation, 2024, 36(3): 649-658.
14	张美娜, 王潇, 梁万杰, 等. 温室远程监控系统人机交互与番茄识别研究[J]. 农业机械学报, 2022, 53(10): 363-370.
	Zhang Meina, Wang Xiao, Liang Wanjie, et al. Human-computer Interaction and Tomato Recognition in Greenhouse Remote Monitoring System[J]. Transactions of the Chinese Society for Agricultural Machinery, 2022, 53(10): 363-370.
15	张旭辉, 张雨萌, 王岩, 等. 融合数字孪生与混合现实技术的机电设备辅助维修方法[J]. 计算机集成制造系统, 2021, 27(8): 2187-2195.
	Zhang Xuhui, Zhang Yumeng, Wang Yan, et al. Auxiliary Maintenance Method for Electromechanical Equipment Integrating Digital Twin and Mixed Reality Technology[J]. Computer Integrated Manufacturing Systems, 2021, 27(8): 2187-2195.
16	Lugaresi C, Tang Jiuqiang, Nash H, et al. MediaPipe: A Framework for Building Perception Pipelines[EB/OL]. (2019-06-14) [2024-04-19]. .
17	张云佐, 董旭, 蔡昭权. 拟合下肢几何特征的多视角步态周期检测[J]. 吉林大学学报(工学版), 2023, 53(9): 2611-2619.
	Zhang Yunzuo, Dong Xu, Cai Zhaoquan. Multi View Gait Cycle Detection by Fitting Geometric Features of Lower Limbs[J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(9): 2611-2619.
18	Hasanov Jamaladdin, Alishzade Nigar, Nazimzade A, et al. Development of a Hybrid Word Recognition System and Dataset for the Azerbaijani Sign Language Dactyl Alphabet[J]. Speech Communication, 2023, 153: 102960.
19	Dill Sebastian, Rösch Andreas, Rohr Maurice, et al. Accuracy Evaluation of 3D Pose Estimation with MediaPipe Pose for Physical Exercises[J]. Current Directions in Biomedical Engineering, 2023, 9(1): 563-566.
20	许杰, 戚大伟. 基于改进加权质心和UKF的移动目标定位算法[J]. 吉林大学学报(工学版), 2016, 46(4): 1354-1359.
	Xu Jie, Qi Dawei. Moving Target Localization Based on Improved Weighted Centroid and UKF Algorithm[J]. Journal of Jilin University(Engineering and Technology Edition), 2016, 46(4): 1354-1359.
21	张鋆豪, 何百岳, 杨旭升, 等. 基于可穿戴式惯性传感器的人体运动跟踪方法综述[J]. 自动化学报, 2019, 45(8): 1439-1454.
	Zhang Junhao, He Baiyue, Yang Xusheng, et al. A Review on Wearable Inertial Sensor Based Human Motion Tracking[J]. Acta Automatica Sinica, 2019, 45(8): 1439-1454.
22	黄伟, 付红坡, 李煜, 等. 一种高斯-重尾切换分布鲁棒卡尔曼滤波器[J]. 哈尔滨工业大学学报, 2024, 56(4): 12-23.
	Huang Wei, Fu Hongpo, Li Yu, et al. A Gaussian-heavy-tailed Switching Distribution Robust Kalman Filter[J]. Journal of Harbin Institute of Technology, 2024, 56(4): 12-23.
23	孟杰, 杨鹏程, 杨朝, 等. 基于Mediapipe的幻影成像装置自然手势交互系统设计[J]. 国外电子测量技术, 2023, 42(3): 116-122.
	Meng Jie, Yang Pengcheng, Yang Zhao, et al. Design of Natural Gesture Interaction System for Phantom Imaging Device Based on Mediapipe[J]. Foreign Electronic Measurement Technology, 2023, 42(3): 116-122.
24	Benitez-Garcia Gibran, Olivares-Mercado Jesus, Sanchez-Perez Gabriel, et al. IPN Hand: A Video Dataset and Benchmark for Real-time Continuous Hand Gesture Recognition[C]//2020 25th International Conference on Pattern Recognition (ICPR). Piscataway, NJ, USA: IEEE, 2021: 4340-4347.
25	杨旭升, 王雪儿, 汪鹏君, 等. 基于渐进无迹卡尔曼滤波网络的人体肢体运动估计[J]. 自动化学报, 2023, 49(8): 1723-1731.
	Yang Xusheng, Wang Xueer, Wang Pengjun, et al. Estimation of Human Limb Motion Based on Progressive Unscented Kalman Filter Network[J]. Acta Automatica Sinica, 2023, 49(8): 1723-1731.
26	仇大伟, 曹慧, 刘静. 中医正骨仿真训练系统中噪声数据的处理方法[J]. 激光与光电子学进展, 2020, 57(3): 122-131.
	Qiu Dawei, Cao Hui, Liu Jing. Noise Data Processing Methods in Simulation and Training System of Bone-setting Manipulations in Traditional Chinese Medicine[J]. Laser & Optoelectronics Progress, 2020, 57(3): 122-131.
27	唐洁, 刘晓琴. 基于近似熵的斯隆数字化巡天中类星体光变复杂性分析[J]. 物理学报, 2019, 68(14): 382-389.
	Tang Jie, Liu Xiaoqin. Analysis on Complexity of Optical Variability Based on Approximate Entropy in Sloan Digital Sky Survey Quasars[J]. Acta Physica Sinica, 2019, 68(14): 382-389.
28	王洪雁, 张海坤, 罗宇华, 等. 复杂动态背景下基于群稀疏的运动目标检测方法[J]. 电子学报, 2021, 49(12): 2330-2338.
	Wang Hongyan, Zhang Haikun, Luo Yuhua, et al. Moving Object Detection Method Based on Group Sparseness Under Complex Dynamic Background[J]. Acta Electronica Sinica, 2021, 49(12): 2330-2338.

P	x	y
500 I	11.503 258 5	19.774 344 7
300 I	11.503 258 6	19.774 344 8
100 I	11.503 258 6	19.774 344 8
I	11.503 259 8	19.774 352 1
0.01 I	11.617 156 5	19.970 157 2

Q	x	y
2 I	1.547	2.664
I	3.144	5.414
0.1 I	11.503	19.774
0.05 I	13.999	23.825
0.01 I	21.889	32.118

R	x	y
10 I	21.889	32.119
5 I	17.701	28.534
1 I	11.503	19.774
0.5 I	8.876	15.282
0.1 I	3.144	5.414

参数	原视频	常规卡尔曼滤波	容积卡尔曼滤波	扩展卡尔曼滤波	本文方法
MSE-x		265.048	434.770	2 701.726	272.835
MSE-y		205.727	310.543	1 450.733	211.761
SNR-x/dB		14.399	11.911	4.086	14.275
SNR-y/dB		13.145	10.931	3.540	13.021
ApEn-x	0.459	0.371	0.374	0.677	0.352
ApEn-y	0.670	0.560	0.541	0.742	0.558

手势动作	N_TP	N_FP	N_TN	N_FN	P_r/%	R_e/%	A_c/%
A₁	188	14	186	12	93	94	94
A₂	189	23	177	11	89	95	92
A₃	185	14	186	15	93	93	93