基于视觉动捕的仿真掘进机操作系统

doi:10.16182/j.issn1004731x.joss.24-0516

系统仿真学报 ›› 2025, Vol. 37 ›› Issue (6): 1531-1541.doi: 10.16182/j.issn1004731x.joss.24-0516

• 论文 • 上一篇

基于视觉动捕的仿真掘进机操作系统

李永玲¹^,³^,⁴, 刘凌志¹^,³, 周百顺², 雷经发¹^,³^,⁴, 张淼¹^,³, 赵汝海¹^,³

^1.安徽建筑大学机械与电气工程学院，安徽合肥 230601
^2.中国劳动关系学院计算机学院，北京 100048
^3.工程机械智能制造安徽省教育厅重点实验室，安徽合肥 230601
^4.过程装备与控制工程四川省高校重点实验室，四川自贡 643000

收稿日期:2024-05-15 修回日期:2024-07-24 出版日期:2025-06-20 发布日期:2025-06-18
通讯作者: 周百顺
第一作者简介:李永玲(1985-)，女，讲师，博士，研究方向为机器视觉。
基金资助:
安徽省重点研究与开发计划(1804a09020009);安徽省高校自然科学研究重大项目与重点项目(J2021ZA0068);安徽省高校协同创新项目(GXXT-2022-019);过程装备与控制工程四川省高校重点实验室开放基金(GK202101)

Operation System for Simulation Roadheader Based on Visual Motion Capture

Li Yongling¹^,³^,⁴, Liu Lingzhi¹^,³, Zhou Baishun², Lei Jingfa¹^,³^,⁴, Zhang Miao¹^,³, Zhao Ruhai¹^,³

^1.School of Mechanical and Electrical Engineering, Anhui Jianzhu University, Hefei 236001, China
^2.School of Computer Science, China University of Labor Relations, Beijing 100048, China
^3.Key Laboratory of Intelligent Manufacturing of Construction Machinery, Anhui Education Department, Hefei 230601, China
^4.Sichuan Provincial Key Laboratory of Process Equipment and Control Engineering, Zigong 643000, China

Received:2024-05-15 Revised:2024-07-24 Online:2025-06-20 Published:2025-06-18
Contact: Zhou Baishun

摘要/Abstract

摘要：

为实现仿真掘进环境中的人机自然交互，设计了一种基于视觉动捕的仿真掘进机操作系统。视觉动捕单元基于Mediapipe框架，通过相机捕捉手势动作，实现真实世界与虚拟空间的映射；针对手部关键点数据在大范围运动中的不合理跳变问题，通过设置加权质心提出一种改进的卡尔曼滤波算法，并依据操作者手势输出对应指令。实验结果表明：该方法在均方误差、信噪比、近似熵参数上更具有优势，所需的特定手势识别正确率可达92%以上，操作者可实现对仿真掘进机的高效控制。

关键词: 人机交互, 手势捕捉, 卡尔曼滤波, 仿真操作系统, 近似熵

Abstract:

To enhance the natural human-machine interaction in simulation roadheader environment, a vision-based simulation roadheader operation system is proposed. The visual motion capture unit is based on the MediaPipe framework, which captures hand gestures through cameras and creates a correspondence between the physical world and virtual space. An improved Kalman filter algorithm is proposed by setting a weighted centroid to address the issue of unreasonable jumps in hand keypoint data during large-scale movements. The operator's gestures are discerned and the corresponding commands are conveyed. The results show that the improved method has significant advantages over the control group in terms of mean square error, signal-to-noise ratio, and approximate entropy parameters. The gesture recognition system is developed with an accuracy rate exceeding 92%. This interface enables the operator to efficiently control the simulated tunneling machine.

Key words: human-machine interaction, motion capture, Kalman filter, simulated operation system, approximate entropy

中图分类号:

TP391.9

李永玲,刘凌志,周百顺等 . 基于视觉动捕的仿真掘进机操作系统[J]. 系统仿真学报, 2025, 37(6): 1531-1541.

Li Yongling,Liu Lingzhi,Zhou Baishun,et al . Operation System for Simulation Roadheader Based on Visual Motion Capture[J]. Journal of System Simulation, 2025, 37(6): 1531-1541.

图/表 13

图1

图2

图3

图4

图5

图6

图7

表1

表2

表3

表4

图8

表5

参考文献 28

1	董泽宇, 刘政, 李勇, 等. 基于混合现实的可移动机械臂操作控制系统[J]. 激光与光电子学进展, 2024, 61(4): 271-279.
	Dong Zeyu, Liu Zheng, Li Yong, et al. Operation Control System for Movable Robotic Arm Based on Mixed Reality[J]. Laser & Optoelectronics Progress, 2024, 61(4): 271-279.
2	邵绪强, 张浩伟, 冯小华. 面向电力变压器虚拟装配的多感官融合交互方法[J]. 系统仿真学报, 2022, 34(10): 2244-2254.
	Shao Xuqiang, Zhang Haowei, Feng Xiaohua. Multi-sensory Fusion Method for Power Transformer Virtual Assembly[J]. Journal of System Simulation, 2022, 34(10): 2244-2254.
3	方维, 许澍虹, 韩磊, 等. AR增强装配中的跟踪注册方法研究与应用进展[J]. 系统仿真学报, 2023, 35(7): 1438-1454.
	Fang Wei, Xu Shuhong, Han Lei, et al. Research and Application Progress of Tracking Registration Methods in AR Assembly[J]. Journal of System Simulation, 2023, 35(7): 1438-1454.
4	张旭辉, 王甜, 张超, 等. 数字孪生驱动的悬臂式掘进机虚拟示教记忆截割方法[J]. 煤炭学报, 2023, 48(11): 4247-4260.
	Zhang Xuhui, Wang Tian, Zhang Chao, et al. DT-driven Memory Cutting Control Method Using VR Instruction of Boom-type Roadheader[J]. Journal of China Coal Society, 2023, 48(11): 4247-4260.
5	张旭辉, 张超, 王妙云, 等. 数字孪生驱动的悬臂式掘进机虚拟操控技术[J]. 计算机集成制造系统, 2021, 27(6): 1617-1628.
	Zhang Xuhui, Zhang Chao, Wang Miaoyun, et al. Digital Twin-driven Virtual Control Technology of Cantilever Roadheader[J]. Computer Integrated Manufacturing Systems, 2021, 27(6): 1617-1628.
6	蔡兴旺, 裴耀强, 杨继华, 等. 煤矿掘进机系统数字孪生技术研究[J]. 系统仿真学报, 2024, 36(10): 2455-2468.
	Cai Xingwang, Pei Yaoqiang, Yang Jihua, et al. Research on Digital Twin Technology of Coal Mine Tunneling Machine System[J]. Journal of System Simulation, 2024, 36(10): 2455-2468.
7	王涛, 吴迎年, 杨睿, 等. 基于IMU与sEMG混合信号的实时手势分类算法研究[J]. 系统仿真学报, 2023, 35(2): 359-371.
	Wang Tao, Wu Yingnian, Yang Rui, et al. Research on Real-time Gesture Classification Algorithm Based on IMU and sEMG Mixed Signals[J]. Journal of System Simulation, 2023, 35(2): 359-371.
8	朱银龙, 沈宏骏, 吴杰, 等. 基于柔性应变传感器的数据手套手势识别研究[J]. 农业机械学报, 2024, 55(6): 451-458.
	Zhu Yinlong, Shen Hongjun, Wu Jie, et al. Data Glove Gesture Recognition Based on Flexible Strain Sensors[J]. Transactions of the Chinese Society for Agricultural Machinery, 2024, 55(6): 451-458.
9	吴常铖, 曹青青, 费飞, 等. 基于数据手套和神经网络的数字手势识别方法[J]. 东南大学学报(自然科学版), 2020, 50(3): 563-569.
	Wu Changcheng, Cao Qingqing, Fei Fei, et al. Digital Gesture Recognition Method Based on Data Glove and Neural Networks[J]. Journal of Southeast University(Natural Science Edition), 2020, 50(3): 563-569.
10	叶素芬, 赖际舟, 吕品, 等. 穿戴式钢琴弹奏手套智能感知与手势识别技术[J]. 仪器仪表学报, 2019, 40(5): 187-194.
	Ye Sufen, Lai Jizhou, Pin Lü, et al. An Intelligent Perception and Gesture Recognition Technology for Wearable Piano-playing Glove[J]. Chinese Journal of Scientific Instrument, 2019, 40(5): 187-194.
11	蔡嘉怡, 初萍, 庄伦涛, 等. 基于空间属性特征的毫米波雷达身体干扰识别[J]. 系统工程与电子技术, 2024, 46(10): 3365-3374.
	Cai Jiayi, Chu Ping, Zhuang Luntao, et al. Millimeter-wave Radar Body Interference Recognition Based on Spatial Attribute Features[J]. Systems Engineering and Electronics, 2024, 46(10): 3365-3374.
12	冯翔, 刘涛, 崔文卿, 等. 基于双视角时序特征融合的毫米波雷达手势数字识别研究[J]. 电子与信息学报, 2023, 45(6): 2134-2143.
	Feng Xiang, Liu Tao, Cui Wenqing, et al. Handwriting Number Recognition Based on Millimeter-wave Radar with Dual-view Feature Fusion Network[J]. Journal of Electronics & Information Technology, 2023, 45(6): 2134-2143.
13	康来, 张亚坤. 基于多维投影时空事件帧的动态视觉传感手势识别[J]. 系统仿真学报, 2024, 36(3): 649-658.
	Kang Lai, Zhang Yakun. Gesture Recognition for Dynamic Vision Sensor Based on Multi-dimensional Projection Spatiotemporal Event Frame[J]. Journal of System Simulation, 2024, 36(3): 649-658.
14	张美娜, 王潇, 梁万杰, 等. 温室远程监控系统人机交互与番茄识别研究[J]. 农业机械学报, 2022, 53(10): 363-370.
	Zhang Meina, Wang Xiao, Liang Wanjie, et al. Human-computer Interaction and Tomato Recognition in Greenhouse Remote Monitoring System[J]. Transactions of the Chinese Society for Agricultural Machinery, 2022, 53(10): 363-370.
15	张旭辉, 张雨萌, 王岩, 等. 融合数字孪生与混合现实技术的机电设备辅助维修方法[J]. 计算机集成制造系统, 2021, 27(8): 2187-2195.
	Zhang Xuhui, Zhang Yumeng, Wang Yan, et al. Auxiliary Maintenance Method for Electromechanical Equipment Integrating Digital Twin and Mixed Reality Technology[J]. Computer Integrated Manufacturing Systems, 2021, 27(8): 2187-2195.
16	Lugaresi C, Tang Jiuqiang, Nash H, et al. MediaPipe: A Framework for Building Perception Pipelines[EB/OL]. (2019-06-14) [2024-04-19]. .
17	张云佐, 董旭, 蔡昭权. 拟合下肢几何特征的多视角步态周期检测[J]. 吉林大学学报(工学版), 2023, 53(9): 2611-2619.
	Zhang Yunzuo, Dong Xu, Cai Zhaoquan. Multi View Gait Cycle Detection by Fitting Geometric Features of Lower Limbs[J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(9): 2611-2619.
18	Hasanov Jamaladdin, Alishzade Nigar, Nazimzade A, et al. Development of a Hybrid Word Recognition System and Dataset for the Azerbaijani Sign Language Dactyl Alphabet[J]. Speech Communication, 2023, 153: 102960.
19	Dill Sebastian, Rösch Andreas, Rohr Maurice, et al. Accuracy Evaluation of 3D Pose Estimation with MediaPipe Pose for Physical Exercises[J]. Current Directions in Biomedical Engineering, 2023, 9(1): 563-566.
20	许杰, 戚大伟. 基于改进加权质心和UKF的移动目标定位算法[J]. 吉林大学学报(工学版), 2016, 46(4): 1354-1359.
	Xu Jie, Qi Dawei. Moving Target Localization Based on Improved Weighted Centroid and UKF Algorithm[J]. Journal of Jilin University(Engineering and Technology Edition), 2016, 46(4): 1354-1359.
21	张鋆豪, 何百岳, 杨旭升, 等. 基于可穿戴式惯性传感器的人体运动跟踪方法综述[J]. 自动化学报, 2019, 45(8): 1439-1454.
	Zhang Junhao, He Baiyue, Yang Xusheng, et al. A Review on Wearable Inertial Sensor Based Human Motion Tracking[J]. Acta Automatica Sinica, 2019, 45(8): 1439-1454.
22	黄伟, 付红坡, 李煜, 等. 一种高斯-重尾切换分布鲁棒卡尔曼滤波器[J]. 哈尔滨工业大学学报, 2024, 56(4): 12-23.
	Huang Wei, Fu Hongpo, Li Yu, et al. A Gaussian-heavy-tailed Switching Distribution Robust Kalman Filter[J]. Journal of Harbin Institute of Technology, 2024, 56(4): 12-23.
23	孟杰, 杨鹏程, 杨朝, 等. 基于Mediapipe的幻影成像装置自然手势交互系统设计[J]. 国外电子测量技术, 2023, 42(3): 116-122.
	Meng Jie, Yang Pengcheng, Yang Zhao, et al. Design of Natural Gesture Interaction System for Phantom Imaging Device Based on Mediapipe[J]. Foreign Electronic Measurement Technology, 2023, 42(3): 116-122.
24	Benitez-Garcia Gibran, Olivares-Mercado Jesus, Sanchez-Perez Gabriel, et al. IPN Hand: A Video Dataset and Benchmark for Real-time Continuous Hand Gesture Recognition[C]//2020 25th International Conference on Pattern Recognition (ICPR). Piscataway, NJ, USA: IEEE, 2021: 4340-4347.
25	杨旭升, 王雪儿, 汪鹏君, 等. 基于渐进无迹卡尔曼滤波网络的人体肢体运动估计[J]. 自动化学报, 2023, 49(8): 1723-1731.
	Yang Xusheng, Wang Xueer, Wang Pengjun, et al. Estimation of Human Limb Motion Based on Progressive Unscented Kalman Filter Network[J]. Acta Automatica Sinica, 2023, 49(8): 1723-1731.
26	仇大伟, 曹慧, 刘静. 中医正骨仿真训练系统中噪声数据的处理方法[J]. 激光与光电子学进展, 2020, 57(3): 122-131.
	Qiu Dawei, Cao Hui, Liu Jing. Noise Data Processing Methods in Simulation and Training System of Bone-setting Manipulations in Traditional Chinese Medicine[J]. Laser & Optoelectronics Progress, 2020, 57(3): 122-131.
27	唐洁, 刘晓琴. 基于近似熵的斯隆数字化巡天中类星体光变复杂性分析[J]. 物理学报, 2019, 68(14): 382-389.
	Tang Jie, Liu Xiaoqin. Analysis on Complexity of Optical Variability Based on Approximate Entropy in Sloan Digital Sky Survey Quasars[J]. Acta Physica Sinica, 2019, 68(14): 382-389.
28	王洪雁, 张海坤, 罗宇华, 等. 复杂动态背景下基于群稀疏的运动目标检测方法[J]. 电子学报, 2021, 49(12): 2330-2338.
	Wang Hongyan, Zhang Haikun, Luo Yuhua, et al. Moving Object Detection Method Based on Group Sparseness Under Complex Dynamic Background[J]. Acta Electronica Sinica, 2021, 49(12): 2330-2338.

P	x	y
500 I	11.503 258 5	19.774 344 7
300 I	11.503 258 6	19.774 344 8
100 I	11.503 258 6	19.774 344 8
I	11.503 259 8	19.774 352 1
0.01 I	11.617 156 5	19.970 157 2

Q	x	y
2 I	1.547	2.664
I	3.144	5.414
0.1 I	11.503	19.774
0.05 I	13.999	23.825
0.01 I	21.889	32.118

R	x	y
10 I	21.889	32.119
5 I	17.701	28.534
1 I	11.503	19.774
0.5 I	8.876	15.282
0.1 I	3.144	5.414

参数	原视频	常规卡尔曼滤波	容积卡尔曼滤波	扩展卡尔曼滤波	本文方法
MSE-x		265.048	434.770	2 701.726	272.835
MSE-y		205.727	310.543	1 450.733	211.761
SNR-x/dB		14.399	11.911	4.086	14.275
SNR-y/dB		13.145	10.931	3.540	13.021
ApEn-x	0.459	0.371	0.374	0.677	0.352
ApEn-y	0.670	0.560	0.541	0.742	0.558

手势动作	N_TP	N_FP	N_TN	N_FN	P_r/%	R_e/%	A_c/%
A₁	188	14	186	12	93	94	94
A₂	189	23	177	11	89	95	92
A₃	185	14	186	15	93	93	93

基于视觉动捕的仿真掘进机操作系统

Operation System for Simulation Roadheader Based on Visual Motion Capture

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 13

参考文献 28

相关文章 15

编辑推荐

Metrics

本文评价

[1]	李宗刚, 李彦博, 焦建军, 杜亚江. 一种基于AEKF的铆接件视觉伺服精确装配方法[J]. 系统仿真学报, 2025, 37(1): 107-118.
[2]	邓文浪, 吴明海, 谢海鹏, 胡英杰. 不平衡电网下AC-DC矩阵变换器的新型模型预测控制[J]. 系统仿真学报, 2024, 36(9): 2159-2170.
[3]	刘兴华, 董思文, 田佳强. 基于卡尔曼滤波分解的配电网安全状态估计[J]. 系统仿真学报, 2023, 35(6): 1191-1202.
[4]	焦嵩鸣, 丁辉, 钟宇飞, 姚鑫, 郑家豪. 一种基于SiamRPN的无人机目标跟踪及控制算法[J]. 系统仿真学报, 2023, 35(6): 1372-1380.
[5]	陈佳昕, 周国辉, 杨建柏. 基于虚拟仿真方法的手部姿态数据集构建技术[J]. 系统仿真学报, 2023, 35(4): 862-870.
[6]	李智杰, 马爽予, 李昌华, 梁霄, 张颉. 基于BIM与MR的楼层疏散仿真[J]. 系统仿真学报, 2023, 35(3): 658-669.
[7]	刘俊, 高阳, 徐涛, 赵青, 单桂华, 迟学斌. 基于过程挖掘的科学工作流交互式构建[J]. 系统仿真学报, 2022, 34(7): 1547-1558.
[8]	王志伟, 胡继宗, 王风杰, 黄杰. 基于非线性观测器的导航过程仿真研究[J]. 系统仿真学报, 2022, 34(5): 1127-1139.
[9]	王启明, 蒋江月, 吕志超, 张汉祖. 基于改进MPC的协同自适应巡航控制策略研究[J]. 系统仿真学报, 2022, 34(09): 2087-2097.
[10]	卢柏宏, 赵建军, 刘戈三. 电影虚拟摄影半实物仿真的研究与实现[J]. 系统仿真学报, 2021, 33(8): 1938-1946.
[11]	马立新, 朱勇杰, 季乐延. 神经网络优化的无感永磁同步电机控制系统[J]. 系统仿真学报, 2021, 33(3): 622-630.
[12]	王一静, 苏中, 李擎, 李磊. 地下空间单兵定位零速修正算法仿真研究[J]. 系统仿真学报, 2021, 33(11): 2545-2551.
[13]	王可, 刘立刚, 周斌, 卜智勇. 基于SINS/GNSS/轮速的列车组合导航算法研究[J]. 系统仿真学报, 2021, 33(1): 189-195.
[14]	李琪, 王向东, 李华. 基于双Kinect传感器的三维人体姿态跟踪方法[J]. 系统仿真学报, 2020, 32(8): 1446-1454.
[15]	郭园, 童倩倩, 郑宇铠, 王子琦, 张玉茹, 王党校. 融合多元触觉和沉浸式视觉的可移植VR软件框架[J]. 系统仿真学报, 2020, 32(7): 1385-1392.