AR-assisted Sign Language Letter Recognition Method Based on Improved MobileNet Network

doi:10.16182/j.issn1004731x.joss.22-0216

Abstract

Abstract:

An AR-assisted sign language letter recognition algorithm MS-MobileNet is proposed for the problems of sign language gestures needing to be standardized and low recognition rate. A multi-scale convolution module is designed to extract the low-level features and enhance the feature extraction ability. ELU activation function is used to retain the negative feature information, which combined with a lightweight MobileNet model for the web to improve the recognition accuracy and real-time performance for mobile AR applications. Test results show that compared with the original model, the recognition accuracy of MS-MobileNet on the datasets ASL-M, NUS-II and Creative Senz3D is improved by 2.58%, 5.32% and 3.04%, respectively. Based on improved network, a WebAR-assisted sign language collaborative interaction system is designed. After the evaluation test, the average user participation rate reached 8.2 points, and the single recognition time is less than 0.115 s. User's needs for immersive real-time sign language letter interaction is better met.

Key words: sign language letter recognition, MobileNet, multi-scale convolution, WebAR, collaborative interaction

CLC Number:

TP391

Chunhong Liu, Song Wang, Fupan Wang, Wensheng Tang, Yunqiang Pei, Dongsheng Tian, Yadong Wu. AR-assisted Sign Language Letter Recognition Method Based on Improved MobileNet Network[J]. Journal of System Simulation, 2023, 35(6): 1308-1321.

Figures/Tables 20

Fig. 1

Fig. 2

Fig. 3

Table 1

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Fig. 11

Table 2

Table 3

Table 4

Table 5

Table 6

Fig. 12

Fig. 13

Fig. 14

References 33

1	郝子煜, 阿里甫·库尔班, 李晓红, 等. 基于CapsNet的中国手指语识别[J]. 计算机应用研究, 2019, 36(10): 3157-3159.
	Hao Ziyu, Kuerban Alifu, Li Xiaohong, et al. Chinese Finger Language Recognition Using CapsNet[J]. Application Research of Computers, 2019, 36(10): 3157-3159.
2	杨淑莹, 赵敏, 郭杨杨, 等. 基于改进的EfficientDet的手语识别算法[J]. 微电子学与计算机, 2022, 39(2): 84-91.
	Yang Shuying, Zhao Min, Guo Yangyang, et al. Sign Language Recognition Algorithm Based on Improved EfficientDet[J]. Microelectronics & Computer, 2022, 39(2): 84-91.
3	Köpüklü O, Gunduz A, Kose N, et al. Real-time Hand Gesture Detection and Classification Using Convolutional Neural Networks[C]//2019 14th IEEE International Conference on Automatic Face & Gesture Recognition(FG 2019). Piscataway, NJ, USA: IEEE, 2019: 1-8.
4	Amaro N M, Piñero Pérez P Y, Montesinos D F P, et al. Tendencies in Augmented Reality[M]//Piñero Pérez P Y, Bello Pérez R E, Kacprzyk J. Artificial Intelligence in Project Management and Making Decisions. Cham: Springer International Publishing, 2022: 345-362.
5	梁欢, 陈一民, 李德旭, 等. 面向移动增强现实的手势交互方法[J]. 微型电脑应用, 2018, 34(5): 9-13.
	Liang Huan, Chen Yimin, Li Dexu, et al. The Method of Hand Gesture Interaction Based on Depth Learning and Hidden Markov Model[J]. Microcomputer Applications, 2018, 34(5): 9-13.
6	裴云强, 吴亚东, 王赋攀, 等. 基于改进L-K光流的WebAR信息可视分析方法[J]. 图学学报, 2020, 41(6): 962-969.
	Pei Yunqiang, Wu Yadong, Wang Fupan, et al. IV LKWA: An Information Visual Analysis Tool with Advanced L-K Optical Flow Based WebAR[J]. Journal of Graphics, 2020, 41(6): 962-969.
7	Qiao Xiuquan, Ren Pei, Guoshun Nan, et al. Mobile Web Augmented Reality in 5G and Beyond: Challenges, Opportunities, and Future Directions[J]. China Communications, 2019, 16(9): 141-154.
8	Lian K Y, Chiu C C, Hong Yongjie, et al. Wearable Armband for Real Time Hand Gesture Recognition[C]//2017 IEEE International Conference on Systems, Man, and Cybernetics(SMC). Piscataway, NJ, USA: IEEE, 2017: 2992-2995.
9	蔡兴泉, 郭天航, 臧坤, 等. 基于无线数据手套的手势识别方法研究[J]. 系统仿真学报, 2014, 26(1): 72-75, 84.
	Cai Xingquan, Guo Tianhang, Zang Kun, et al. Gesture Recognition Method Based on Wireless Data Glove[J]. Journal of System Simulation, 2014, 26(1): 72-75, 84.
10	Tarvekar M P. Hand Gesture Recognition System for Touch-less Car Interface Using Multiclass Support Vector Machine[C]//2018 Second International Conference on Intelligent Computing and Control Systems(ICICCS). Piscataway, NJ, USA: IEEE, 2018: 1929-1932.
11	Barbhuiya A A, Karsh R K, Dutta S. AlexNet-CNN Based Feature Extraction and Classification of Multiclass ASL Hand Gestures[C]//Proceeding of Fifth International Conference on Microelectronics, Computing and Communication Systems. Singapore: Springer Singapore, 2021: 77-89.
12	Hamid M F A, Zaman F H K. Hand Gesture Recognition Using Movidius Neural Compute Stick[C]//2019 IEEE 9th International Conference on System Engineering and Technology(ICSET). Piscataway, NJ, USA: IEEE, 2019: 510-514.
13	Jain A, Sethi A, Vishwakarma D K, et al. Ensembled Neural Network for Static Hand Gesture Recognition[C]//2021 12th International Conference on Computing Communication and Networking Technologies(ICCCNT). Piscataway, NJ, USA: IEEE, 2021: 1-5.
14	Howard A G, Zhu Menglong, Chen Bo, et al. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications[EB/OL]. (2017-04-17)[2022-02-26]. .
15	Liu Peiyu, Guo Jian, Lu Jiemin, et al. An sEMG Gesture Recognition Algorithm Based on Improved MobileNet[C]//2021 6th International Conference on Computational Intelligence and Applications(ICCIA). Piscataway, NJ, USA: IEEE, 2021: 169-173.
16	He Wangpeng, Liao Nannan, Liu Wei, et al. An Ultra-high Speed Gesture Recognition Algorithm Based on MobileNetV2[C]//Neural Computing for Advanced Applications. Singapore: Springer Singapore, 2021: 590-602.
17	Qian Siying, Ning Chenran, Hu Yuepeng. MobileNetV3 for Image Classification[C]//2021 IEEE 2nd International Conference on Big Data, Artificial Intelligence and Internet of Things Engineering(ICBAIE). Piscataway, NJ, USA: IEEE, 2021: 490-497.
18	Gupta N. Sign Language Recognition Using Diverse Deep Learning Models[C]//International Conference on Artificial Intelligence and Sustainable Engineering. Singapore: Springer Singapore, 2022: 463-475.
19	Zhang Kang, Lan Shengchang, Zhang Guiyuan. On the Effect of Training Convolution Neural Network for Millimeter-wave Radar-based Hand Gesture Recognition[J]. Sensors, 2021, 21(1): 259.
20	Qiang Baohua, Zhai Yijie, Zhou Mingliang, et al. SqueezeNet and Fusion Network-based Accurate Fast Fully Convolutional Network for Hand Detection and Gesture Recognition[J]. IEEE Access, 2021, 9: 77661-77674.
21	Gu Lingyun, Zhang Lin, Wang Zhaokui. Hierarchical Attention-based Astronaut Gesture Recognition: A Dataset and CNN Model[J]. IEEE Access, 2020, 8: 68787-68798.
22	Antoshchuk S, Kovalenko M, Sieck J. Gesture Recognition-based Human-computer Interaction Interface for Multimedia Applications[M]//Jat D S, Sieck J, Muyingi H N N, et al. Digitisation of Culture: Namibian and International Perspectives. Singapore: Springer Singapore, 2018: 269-286.
23	Xiao Mengting, Feng Zhiquan, Yang Xiaohui, et al. Multimodal Interaction Design and Application in Augmented Reality for Chemical Experiment[J]. Virtual Reality & Intelligent Hardware, 2020, 2(4): 291-304.
24	Yusnita L, Hadisukmana N, Wahyu R B, et al. Implementation of Real-time Static Hand Gesture Recognition Using Artificial Neural Network[C]//2017 4th International Conference on Computer Applications and Information Processing Technology(CAIPT). Piscataway, NJ, USA: IEEE, 2017: 1-6.
25	Otsu N. A Threshold Selection Method from Gray-level Histograms[J]. IEEE Transactions on Systems, Man, and Cybernetics, 1979, 9(1): 62-66.
26	刘文达. 基于单目摄像头的静态手势识别方法研究[D]. 武汉: 湖北工业大学, 2020.
	Liu Wenda. Research of Static Gesture Recognition Methods Based on Monocular Camera[D]. Wuhan: Hubei University of Technology, 2020.
27	景雨, 祁瑞华, 刘建鑫, 等. 基于改进多尺度深度卷积网络的手势识别算法[J]. 计算机科学, 2020, 47(6): 180-183.
	Jing Yu, Qi Ruihua, Liu Jianxin, et al. Gesture Recognition Algorithm Based on Improved Multiscale Deep Convolutional Neural Network[J]. Computer Science, 2020, 47(6): 180-183.
28	蒋昂波, 王维维. ReLU激活函数优化研究[J]. 传感器与微系统, 2018, 37(2): 50-52.
	Jiang Angbo, Wang Weiwei. Research on Optimization of ReLU Activation Function[J]. Transducer and Microsystem Technologies, 2018, 37(2): 50-52.
29	史文策, 许江宁, 林恩凡. 陀螺仪的发展与展望[J]. 导航定位学报, 2021, 9(3): 8-12.
	Shi Wence, Xu Jiangning, Lin Enfan. Development and Prospect of Gyroscope[J]. Journal of Navigation and Positioning, 2021, 9(3): 8-12.
30	Pisharady P K, Vadakkepat P, Loh A P. Attention Based Detection and Recognition of Hand Postures Against Complex Backgrounds[J]. International Journal of Computer Vision, 2013, 101(3): 403-419.
31	Memo A, Minto L, Zanuttigh P. Exploiting Silhouette Descriptors and Synthetic Data for Hand Gesture Recognition[C]//Smart Tools and Apps for Graphics-Eurographics Italian Chapter Conference. The Eurographics Association, 2015.
32	Memo A, Zanuttigh P. Head-mounted Gesture Controlled Interface for Human-computer Interaction[J]. Multimedia Tools and Applications, 2018, 77(1): 27-53.
33	O'Brien H L, Cairns P, Hall M. A Practical Approach to Measuring User Engagement with the Refined User Engagement Scale(UES) and New UES Short Form[J]. International Journal of Human-Computer Studies, 2018, 112: 28-39.

卷积类型	Filter Shape	Stride	Output Size	卷积类型	Filter Shape	Stride	Output Size
MS-Conv	1×1、3×3、5×5	1	224×224×48	Dw Conv 5	3×3×256	1	28×28×256
Dw Conv 1	3×3×48	2	112×112×48	Pw Conv 5	1×1×256×256	1	28×28×256
Pw Conv 1	1×1×48×64	1	112×112×64	Dw Conv 6	3×3×256	2	14×14×256
Dw Conv 2	3×3×64	2	56×56×64	Pw Conv 6	1×1×256×512	1	14×14×512
Pw Conv 2	1×1×64×128	1	56×56×128	Dw Conv 7	3×3×512	2	7×7×512
Dw Conv 3	3×3×128	1	56×56×128	Pw Conv 7	1×1×512×1024	1	7×7×1024
Pw Conv 3	1×1×128×128	1	56×56×128	Dw Conv 8	3×3×1024	1	7×7×1024
Dw Conv 4	3×3×128	2	28×28×128	Pw Conv 8	1×1×1024×1024	1	7×7×1024
Pw Conv 4	1×1×128×256	1	28×28×256

组别	模型	准确率	召回率
1	MobileNet	95.68	94.03
2	MobileNet+ ELU	96.30	96.54
3	MobileNet+MS-Conv	97.13	97.82
4	MobileNet+MS-Conv+ELU	98.26	98.69

模型	A_NUS-II	A_C-S
本文	92.55	88.41
MobileNetV1	87.23	85.37
MobileNetV2	88.46	85.30
SqueezeNet	88.27	86.03
EfficientNet	90.63	87.34
AlexNet	89.31	86.29

模型	准确率/%	计算量/百万	参数量/百万
AlexNet	95.21	714	61.10
GoogleNet	90.26	1 504	6.62
ResNet50	94.32	4 111	25.56
MobileNetV1	95.68	575	4.21
MobileNetV2	95.10	314	3.50
MobileNetV3	93.47	312	3.52
ShuffleNet	92.39	148	2.28
SqueezeNet	95.06	823	1.25
EfficientNet	96.37	399	5.30
本文	98.26	580	4.23

模型	单次迭代速度	识别时间
MobileNetV1	652	0.132
本文	561	0.115