基于改进MobileNet网络的AR辅助手语字母识别方法

doi:10.16182/j.issn1004731x.joss.22-0216

系统仿真学报 ›› 2023, Vol. 35 ›› Issue (6): 1308-1321.doi: 10.16182/j.issn1004731x.joss.22-0216

基于改进MobileNet网络的AR辅助手语字母识别方法

刘春宏¹(), 王松¹(), 王赋攀¹, 唐文生¹, 裴云强¹, 田东生¹, 吴亚东²

^1.西南科技大学计算机科学与技术学院，四川绵阳 621010
^2.四川轻化工大学计算机科学与工程学院，四川自贡 643002

收稿日期:2022-03-15 修回日期:2022-05-25 出版日期:2023-06-29 发布日期:2023-06-20
通讯作者: 王松 E-mail:2501649391@qq.com;wangsong@swust.edu.cn
作者简介:刘春宏(1995-)，女，硕士生，研究方向为移动增强现实、深度学习、人机交互。E-mail：2501649391@qq.com
基金资助:
国家自然科学基金面上项目(61872304);四川省自然科学基金(2022NSFSC0961);西南科技大学博士基金(19zx7144);西南科技大学素质类教改(青年发展研究)专项(20szjg17)

AR-assisted Sign Language Letter Recognition Method Based on Improved MobileNet Network

Chunhong Liu¹(), Song Wang¹(), Fupan Wang¹, Wensheng Tang¹, Yunqiang Pei¹, Dongsheng Tian¹, Yadong Wu²

^1.School of Computer Science & Technology, Southwest University of Science and Technology, Mianyang 621010, China
^2.School of Computer Science & Engineering, Sichuan University of Science and Engineering, Zigong 643002, China

Received:2022-03-15 Revised:2022-05-25 Online:2023-06-29 Published:2023-06-20
Contact: Song Wang E-mail:2501649391@qq.com;wangsong@swust.edu.cn

摘要/Abstract

摘要：

针对手语手势姿态待规范、识别率低的问题，提出一种AR辅助手语字母识别算法MS-MobileNet。设计多尺度卷积模块提取底层特征，增强网络的特征提取能力；利用ELU激活函数来保留更全面的负值特征信息；结合适用于Web的轻量级MobileNet模型，提高面向移动AR应用的手语字母识别准确率和实时性。实验结果表明：MS-MobileNet在数据集ASL-M、NUS-II和Creative Senz3D上识别准确率较原模型分别提高了2.58%、5.32%和3.04%。基于MS-MobileNet网络设计一套WebAR辅助的手语字母协同交互系统，经评估测试，用户平均参与度达到8.2分，单次识别耗时低于0.115 s，能较好地满足用户沉浸式的实时手语字母交互需求。

关键词: 手语字母识别, MobileNet, 多尺度卷积, WebAR, 协同交互

Abstract:

An AR-assisted sign language letter recognition algorithm MS-MobileNet is proposed for the problems of sign language gestures needing to be standardized and low recognition rate. A multi-scale convolution module is designed to extract the low-level features and enhance the feature extraction ability. ELU activation function is used to retain the negative feature information, which combined with a lightweight MobileNet model for the web to improve the recognition accuracy and real-time performance for mobile AR applications. Test results show that compared with the original model, the recognition accuracy of MS-MobileNet on the datasets ASL-M, NUS-II and Creative Senz3D is improved by 2.58%, 5.32% and 3.04%, respectively. Based on improved network, a WebAR-assisted sign language collaborative interaction system is designed. After the evaluation test, the average user participation rate reached 8.2 points, and the single recognition time is less than 0.115 s. User's needs for immersive real-time sign language letter interaction is better met.

Key words: sign language letter recognition, MobileNet, multi-scale convolution, WebAR, collaborative interaction

中图分类号:

TP391

刘春宏, 王松, 王赋攀, 唐文生, 裴云强, 田东生, 吴亚东. 基于改进MobileNet网络的AR辅助手语字母识别方法[J]. 系统仿真学报, 2023, 35(6): 1308-1321.

Chunhong Liu, Song Wang, Fupan Wang, Wensheng Tang, Yunqiang Pei, Dongsheng Tian, Yadong Wu. AR-assisted Sign Language Letter Recognition Method Based on Improved MobileNet Network[J]. Journal of System Simulation, 2023, 35(6): 1308-1321.

图/表 20

图1

图2

图3

表1

图4

图5

图6

图7

图8

图9

图10

图11

表2

表3

表4

表5

表6

图12

图13

图14

参考文献 33

1	郝子煜, 阿里甫·库尔班, 李晓红, 等. 基于CapsNet的中国手指语识别[J]. 计算机应用研究, 2019, 36(10): 3157-3159.
	Hao Ziyu, Kuerban Alifu, Li Xiaohong, et al. Chinese Finger Language Recognition Using CapsNet[J]. Application Research of Computers, 2019, 36(10): 3157-3159.
2	杨淑莹, 赵敏, 郭杨杨, 等. 基于改进的EfficientDet的手语识别算法[J]. 微电子学与计算机, 2022, 39(2): 84-91.
	Yang Shuying, Zhao Min, Guo Yangyang, et al. Sign Language Recognition Algorithm Based on Improved EfficientDet[J]. Microelectronics & Computer, 2022, 39(2): 84-91.
3	Köpüklü O, Gunduz A, Kose N, et al. Real-time Hand Gesture Detection and Classification Using Convolutional Neural Networks[C]//2019 14th IEEE International Conference on Automatic Face & Gesture Recognition(FG 2019). Piscataway, NJ, USA: IEEE, 2019: 1-8.
4	Amaro N M, Piñero Pérez P Y, Montesinos D F P, et al. Tendencies in Augmented Reality[M]//Piñero Pérez P Y, Bello Pérez R E, Kacprzyk J. Artificial Intelligence in Project Management and Making Decisions. Cham: Springer International Publishing, 2022: 345-362.
5	梁欢, 陈一民, 李德旭, 等. 面向移动增强现实的手势交互方法[J]. 微型电脑应用, 2018, 34(5): 9-13.
	Liang Huan, Chen Yimin, Li Dexu, et al. The Method of Hand Gesture Interaction Based on Depth Learning and Hidden Markov Model[J]. Microcomputer Applications, 2018, 34(5): 9-13.
6	裴云强, 吴亚东, 王赋攀, 等. 基于改进L-K光流的WebAR信息可视分析方法[J]. 图学学报, 2020, 41(6): 962-969.
	Pei Yunqiang, Wu Yadong, Wang Fupan, et al. IV LKWA: An Information Visual Analysis Tool with Advanced L-K Optical Flow Based WebAR[J]. Journal of Graphics, 2020, 41(6): 962-969.
7	Qiao Xiuquan, Ren Pei, Guoshun Nan, et al. Mobile Web Augmented Reality in 5G and Beyond: Challenges, Opportunities, and Future Directions[J]. China Communications, 2019, 16(9): 141-154.
8	Lian K Y, Chiu C C, Hong Yongjie, et al. Wearable Armband for Real Time Hand Gesture Recognition[C]//2017 IEEE International Conference on Systems, Man, and Cybernetics(SMC). Piscataway, NJ, USA: IEEE, 2017: 2992-2995.
9	蔡兴泉, 郭天航, 臧坤, 等. 基于无线数据手套的手势识别方法研究[J]. 系统仿真学报, 2014, 26(1): 72-75, 84.
	Cai Xingquan, Guo Tianhang, Zang Kun, et al. Gesture Recognition Method Based on Wireless Data Glove[J]. Journal of System Simulation, 2014, 26(1): 72-75, 84.
10	Tarvekar M P. Hand Gesture Recognition System for Touch-less Car Interface Using Multiclass Support Vector Machine[C]//2018 Second International Conference on Intelligent Computing and Control Systems(ICICCS). Piscataway, NJ, USA: IEEE, 2018: 1929-1932.
11	Barbhuiya A A, Karsh R K, Dutta S. AlexNet-CNN Based Feature Extraction and Classification of Multiclass ASL Hand Gestures[C]//Proceeding of Fifth International Conference on Microelectronics, Computing and Communication Systems. Singapore: Springer Singapore, 2021: 77-89.
12	Hamid M F A, Zaman F H K. Hand Gesture Recognition Using Movidius Neural Compute Stick[C]//2019 IEEE 9th International Conference on System Engineering and Technology(ICSET). Piscataway, NJ, USA: IEEE, 2019: 510-514.
13	Jain A, Sethi A, Vishwakarma D K, et al. Ensembled Neural Network for Static Hand Gesture Recognition[C]//2021 12th International Conference on Computing Communication and Networking Technologies(ICCCNT). Piscataway, NJ, USA: IEEE, 2021: 1-5.
14	Howard A G, Zhu Menglong, Chen Bo, et al. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications[EB/OL]. (2017-04-17)[2022-02-26]. .
15	Liu Peiyu, Guo Jian, Lu Jiemin, et al. An sEMG Gesture Recognition Algorithm Based on Improved MobileNet[C]//2021 6th International Conference on Computational Intelligence and Applications(ICCIA). Piscataway, NJ, USA: IEEE, 2021: 169-173.
16	He Wangpeng, Liao Nannan, Liu Wei, et al. An Ultra-high Speed Gesture Recognition Algorithm Based on MobileNetV2[C]//Neural Computing for Advanced Applications. Singapore: Springer Singapore, 2021: 590-602.
17	Qian Siying, Ning Chenran, Hu Yuepeng. MobileNetV3 for Image Classification[C]//2021 IEEE 2nd International Conference on Big Data, Artificial Intelligence and Internet of Things Engineering(ICBAIE). Piscataway, NJ, USA: IEEE, 2021: 490-497.
18	Gupta N. Sign Language Recognition Using Diverse Deep Learning Models[C]//International Conference on Artificial Intelligence and Sustainable Engineering. Singapore: Springer Singapore, 2022: 463-475.
19	Zhang Kang, Lan Shengchang, Zhang Guiyuan. On the Effect of Training Convolution Neural Network for Millimeter-wave Radar-based Hand Gesture Recognition[J]. Sensors, 2021, 21(1): 259.
20	Qiang Baohua, Zhai Yijie, Zhou Mingliang, et al. SqueezeNet and Fusion Network-based Accurate Fast Fully Convolutional Network for Hand Detection and Gesture Recognition[J]. IEEE Access, 2021, 9: 77661-77674.
21	Gu Lingyun, Zhang Lin, Wang Zhaokui. Hierarchical Attention-based Astronaut Gesture Recognition: A Dataset and CNN Model[J]. IEEE Access, 2020, 8: 68787-68798.
22	Antoshchuk S, Kovalenko M, Sieck J. Gesture Recognition-based Human-computer Interaction Interface for Multimedia Applications[M]//Jat D S, Sieck J, Muyingi H N N, et al. Digitisation of Culture: Namibian and International Perspectives. Singapore: Springer Singapore, 2018: 269-286.
23	Xiao Mengting, Feng Zhiquan, Yang Xiaohui, et al. Multimodal Interaction Design and Application in Augmented Reality for Chemical Experiment[J]. Virtual Reality & Intelligent Hardware, 2020, 2(4): 291-304.
24	Yusnita L, Hadisukmana N, Wahyu R B, et al. Implementation of Real-time Static Hand Gesture Recognition Using Artificial Neural Network[C]//2017 4th International Conference on Computer Applications and Information Processing Technology(CAIPT). Piscataway, NJ, USA: IEEE, 2017: 1-6.
25	Otsu N. A Threshold Selection Method from Gray-level Histograms[J]. IEEE Transactions on Systems, Man, and Cybernetics, 1979, 9(1): 62-66.
26	刘文达. 基于单目摄像头的静态手势识别方法研究[D]. 武汉: 湖北工业大学, 2020.
	Liu Wenda. Research of Static Gesture Recognition Methods Based on Monocular Camera[D]. Wuhan: Hubei University of Technology, 2020.
27	景雨, 祁瑞华, 刘建鑫, 等. 基于改进多尺度深度卷积网络的手势识别算法[J]. 计算机科学, 2020, 47(6): 180-183.
	Jing Yu, Qi Ruihua, Liu Jianxin, et al. Gesture Recognition Algorithm Based on Improved Multiscale Deep Convolutional Neural Network[J]. Computer Science, 2020, 47(6): 180-183.
28	蒋昂波, 王维维. ReLU激活函数优化研究[J]. 传感器与微系统, 2018, 37(2): 50-52.
	Jiang Angbo, Wang Weiwei. Research on Optimization of ReLU Activation Function[J]. Transducer and Microsystem Technologies, 2018, 37(2): 50-52.
29	史文策, 许江宁, 林恩凡. 陀螺仪的发展与展望[J]. 导航定位学报, 2021, 9(3): 8-12.
	Shi Wence, Xu Jiangning, Lin Enfan. Development and Prospect of Gyroscope[J]. Journal of Navigation and Positioning, 2021, 9(3): 8-12.
30	Pisharady P K, Vadakkepat P, Loh A P. Attention Based Detection and Recognition of Hand Postures Against Complex Backgrounds[J]. International Journal of Computer Vision, 2013, 101(3): 403-419.
31	Memo A, Minto L, Zanuttigh P. Exploiting Silhouette Descriptors and Synthetic Data for Hand Gesture Recognition[C]//Smart Tools and Apps for Graphics-Eurographics Italian Chapter Conference. The Eurographics Association, 2015.
32	Memo A, Zanuttigh P. Head-mounted Gesture Controlled Interface for Human-computer Interaction[J]. Multimedia Tools and Applications, 2018, 77(1): 27-53.
33	O'Brien H L, Cairns P, Hall M. A Practical Approach to Measuring User Engagement with the Refined User Engagement Scale(UES) and New UES Short Form[J]. International Journal of Human-Computer Studies, 2018, 112: 28-39.

卷积类型	Filter Shape	Stride	Output Size	卷积类型	Filter Shape	Stride	Output Size
MS-Conv	1×1、3×3、5×5	1	224×224×48	Dw Conv 5	3×3×256	1	28×28×256
Dw Conv 1	3×3×48	2	112×112×48	Pw Conv 5	1×1×256×256	1	28×28×256
Pw Conv 1	1×1×48×64	1	112×112×64	Dw Conv 6	3×3×256	2	14×14×256
Dw Conv 2	3×3×64	2	56×56×64	Pw Conv 6	1×1×256×512	1	14×14×512
Pw Conv 2	1×1×64×128	1	56×56×128	Dw Conv 7	3×3×512	2	7×7×512
Dw Conv 3	3×3×128	1	56×56×128	Pw Conv 7	1×1×512×1024	1	7×7×1024
Pw Conv 3	1×1×128×128	1	56×56×128	Dw Conv 8	3×3×1024	1	7×7×1024
Dw Conv 4	3×3×128	2	28×28×128	Pw Conv 8	1×1×1024×1024	1	7×7×1024
Pw Conv 4	1×1×128×256	1	28×28×256

组别	模型	准确率	召回率
1	MobileNet	95.68	94.03
2	MobileNet+ ELU	96.30	96.54
3	MobileNet+MS-Conv	97.13	97.82
4	MobileNet+MS-Conv+ELU	98.26	98.69

模型	A_NUS-II	A_C-S
本文	92.55	88.41
MobileNetV1	87.23	85.37
MobileNetV2	88.46	85.30
SqueezeNet	88.27	86.03
EfficientNet	90.63	87.34
AlexNet	89.31	86.29

模型	准确率/%	计算量/百万	参数量/百万
AlexNet	95.21	714	61.10
GoogleNet	90.26	1 504	6.62
ResNet50	94.32	4 111	25.56
MobileNetV1	95.68	575	4.21
MobileNetV2	95.10	314	3.50
MobileNetV3	93.47	312	3.52
ShuffleNet	92.39	148	2.28
SqueezeNet	95.06	823	1.25
EfficientNet	96.37	399	5.30
本文	98.26	580	4.23

模型	单次迭代速度	识别时间
MobileNetV1	652	0.132
本文	561	0.115

基于改进MobileNet网络的AR辅助手语字母识别方法

AR-assisted Sign Language Letter Recognition Method Based on Improved MobileNet Network

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 20

参考文献 33

相关文章 1

编辑推荐

Metrics

本文评价

字母	精确度	召回率	F1-score	字母	精确度	召回率	F1-score
A	0.99	0.99	1.00	N	0.95	0.97	0.96
B	1.00	1.00	1.00	O	1.00	1.00	0.99
C	1.00	0.99	0.99	P	0.96	0.98	0.98
D	1.00	0.98	0.99	Q	0.98	0.99	0.98
E	0.97	0.97	0.98	R	0.98	1.00	0.99
F	1.00	1.00	1.00	S	1.00	1.00	1.00
G	0.97	0.99	0.99	T	0.98	0.98	0.98
H	0.96	0.95	0.95	U	1.00	0.99	1.00
I	0.97	0.99	0.98	V	1.00	1.00	1.00
K	0.96	0.94	0.96	W	1.00	1.00	1.00
L	1.00	1.00	1.00	X	0.97	0.98	0.98
M	0.94	0.92	0.96	Y	1.00	1.00	1.00