A Precise Attention Tracking System Based on Computer Vision

doi:10.16182/j.issn1004731x.joss.23-FZ0829

Abstract

Abstract:

A precise attention tracking system based on machine vision is designed to address the difficulty in studying students' attention allocation. The system includes an image capture device and an accurate attention tracking algorithm. The image capture device can capture the clearer ocular images. The pupil center localization algorithm replaces VGG16 with lightweight MobileNetv3 and uses two-stage feature fusion and center keypoint prediction techniques to improve the speed and accuracy. The algorithm achieves a speed of up to 36 frames/s and 97.42% accuracy. The gaze tracking algorithm compensates for the head movements to achieve the meticulous gaze tracking. An interactive software for assessing cognitive abilities in school-age children is developed. The software calculates the eye movement indicators by using the collected gaze coordinates and evaluates the cognitive abilities based on psychological theory, and provides a reference for the psychology and education research.

Key words: pupil positioning, improved single shot multibox detector(SSD) algorithm, image gradient tilt algorithm, oriented FAST and rotated brief（Eye-ORB) algorithm, reading cognition, attention tracking

CLC Number:

TP391.7

Liu Jiyuan, Qi Hanwen, Liu Zhicheng, Fei Minrui, Zhang Kun. A Precise Attention Tracking System Based on Computer Vision[J]. Journal of System Simulation, 2023, 35(10): 2087-2100.

Figures/Tables 19

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Table 1

Table 2

Table 3

Table 4

Table 5

Table 6

Fig. 10

Table 7

Fig. 11

Fig. 12

References 29

1	Dong Wentao, Yang Lin, Gravina R, et al. ANFIS Fusion Algorithm for Eye Movement Recognition via Soft Multi-functional Electronic Skin[J]. Information Fusion, 2021, 71: 99-108.
2	Wang Xiaoming, Xiao Yineng, Deng Fangming, et al. Eye-movement-controlled Wheelchair Based on Flexible Hydrogel Biosensor and WT-SVM[J]. Biosensors, 2021, 11(6): 198.
3	Tesfamikael H H, Fray A, Mengsteab I, et al. Simulation of Eye Tracking Control Based Electric Wheelchair Construction by Image Segmentation Algorithm[J]. Journal of Innovative Image Processing, 2021, 3(1): 21-35.
4	Zhu Ji, Yang Hua, Liu Nian, et al. Online Multi-object Tracking With Dual Matching Attention Networks[C]//Computer Vision - ECCV 2018. Cham: Springer International Publishing, 2018: 379-396.
5	靳冰凌, 张震, 张子耀. 基于视觉的驾驶员疲劳特征提取方法[J]. 计算机技术与发展, 2018, 28(11): 193-197.
	Jin Bingling, Zhang Zhen, Zhang Ziyao. A Driving Fatigue Feature Extraction Method Based on Vision[J]. Computer Technology and Development, 2018, 28(11): 193-197.
6	陈加徐, 陆永华, 赵采仪, 等. 一种优化的自然光源下眼动视觉测量方法[J]. 应用科技, 2021, 48(4): 61-67.
	Chen Jiaxu, Lu Yonghua, Zhao Caiyi, et al. An Optimized Visual Measurement Method for Eye Movement Under Natural Light Source[J]. Applied Science and Technology, 2021, 48(4): 61-67.
7	宋琴琴, 杨国平. 基于视频图像的人眼动态追踪[J]. 计算机与数字工程, 2021, 49(7): 1457-1459, 1474.
	Song Qinqin, Yang Guoping. Human Eye Dynamic Tracking Based on Video Images[J]. Computer & Digital Engineering, 2021, 49(7): 1457-1459, 1474.
8	Wang Xiaoming, Zhao Xinbo, Zhang Yanning. Deep-learning-based Reading Eye-movement Analysis for Aiding Biometric Recognition[J]. Neurocomputing, 2021, 444: 390-398.
9	Brand J, Diamond S G, Thomas N, et al. Evaluating the Data Quality of the Gazepoint GP3 Low-cost Eye Tracker When Used Independently by Study Participants[J]. Behavior Research Methods, 2021, 53(4): 1502-1514.
10	Liu Wei, Anguelov D, Erhan D, et al. SSD: Single Shot MultiBox Detector[C]//Computer Vision-ECCV 2016. Cham: Springer International Publishing, 2016: 21-37.
11	Qassim H, Verma A, Feinzimer D. Compressed Residual-VGG16 CNN Model for Big Data Places Image Recognition[C]//2018 IEEE 8th Annual Computing and Communication Workshop and Conference (CCWC). Piscataway, NJ, USA: IEEE, 2018: 169-175.
12	Howard A, Sandler M, Chen Bo, et al. Searching for MobileNetV3[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway, NJ, USA: IEEE, 2019: 1314-1324.
13	Ma Dan, Lai Huicheng. Remote Sensing Image Matching Based Improved ORB in NSCT Domain[J]. Journal of the Indian Society of Remote Sensing, 2019, 47(5): 801-807.
14	Zhu Juntao, Gong Chaofei, Zhao Miaoxing, et al. Image Mosaic Algorithm Based on PCA-ORB Feature Matching[J]. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2020, XLII-3/W10: 83-89.
15	王素琴, 施文豪, 李兆歆, 等. 无人机航拍视频中的车辆检测方法[J]. 系统仿真学报, 2018, 30(7): 2776-2786.
	Wang Suqin, Shi Wenhao, Li Zhaoxin, et al. Vehicle Detection Method in UAV Aerial Video[J]. Journal of System Simulation, 2018, 30(7): 2776-2786.
16	柳有权, 裴雪, 李婉, 等. 基于邻近目标置信度评估的视觉目标跟踪与定位[J]. 系统仿真学报, 2020, 32(7): 1294-1300.
	Liu Youquan, Pei Xue, Li Wan, et al. Visual Tracking and Localization Based on Confidence Evaluation of Adjacent Targets[J]. Journal of System Simulation, 2020, 32(7): 1294-1300.
17	Li Xianguang, Ren Chao, Zhang Tengxu, et al. Unmanned Aerial Vehicle Image Matching Based on Improved RANSAC Algorithm and Surf Algorithm[J]. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2020, XLII-3/W10: 67-70.
18	He Qingfeng, Shahabi H, Shirzadi A, et al. Landslide Spatial Modelling Using Novel Bivariate Statistical Based Naïve Bayes, RBF Classifier, and RBF Network Machine Learning Algorithms[J]. Science of the Total Environment, 2019, 663: 1-15.
19	Villanueva A, Ponz V, Sesma-Sanchez L, et al. Hybrid Method Based on Topography for Robust Detection of Iris Center and Eye Corners[J]. ACM Transactions on Multimedia Computing, Communications, and Applications, 2013, 9(4): 25.
20	Martinikorena I, Cabeza R, Villanueva A, et al. Introducing I2head Database[C]//Proceedings of the 7th Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction. New York, NY, USA: Association for Computing Machinery, 2018: 1.
21	Jesorsky O, Kirchberg K J, Frischholz R W. Robust Face Detection Using the Hausdorff Distance[C]//Audio-and Video-based Biometric Person Authentication. Berlin, Heidelberg: Springer Berlin Heidelberg, 2001: 90-95.
22	Cai Haibin, Liu Bangli, Ju Zhaojie, et al. Accurate Eye Center Localization via Hierarchical Adaptive Convolution[C]//Proceedings of the 29th British Machine Vision Conference. Durham, UK: BMVA Press, 2018: 11504341.
23	Xia Yifan, Yu Hui, Wang Feiyue. Accurate and Robust Eye Center Localization via Fully Convolutional Networks[J]. IEEE/CAA Journal of Automatica Sinica, 2019, 6(5): 1127-1138.
24	Choi J H, Il Lee K, Kim Y C, et al. Accurate Eye Pupil Localization Using Heterogeneous CNN Models[C]//2019 IEEE International Conference on Image Processing (ICIP). Piscataway, NJ, USA: IEEE, 2019: 2179-2183.
25	Lee K I, Jeon J H, Song B C. Deep Learning-based Pupil Center Detection for Fast and Accurate Eye Tracking System[C]//Computer Vision-ECCV 2020. Cham: Springer International Publishing, 2020: 36-52.
26	Xiao Feng, Huang Kejie, Qiu Yue, et al. Accurate Iris Center Localization Method Using Facial Landmark, Snakuscule, Circle Fitting and Binary Connected Component[J]. Multimedia Tools and Applications, 2018, 77(19): 25333-25353.
27	胡艳红, 魏江, 梅少辉. 基于瞳孔角膜反射技术的视线估计方法[J]. 计算机工程与应用, 2018, 54(14): 7-10, 18.
	Hu Yanhong, Wei Jiang, Mei Shaohui. Gaze Estimation Method Based on Pupils and Corneal Reflection Technique[J]. Computer Engineering and Applications, 2018, 54(14): 7-10, 18.
28	李擎, 胡京尧, 迟健男, 等. 视线追踪中一种新的由粗及精的瞳孔定位方法[J]. 工程科学学报, 2019, 41(11): 1484-1492.
	Li Qing, Hu Jingyao, Chi Jiannan, et al. A New Pupil Localization Method From Rough to Precise in Gaze Tracking[J]. Chinese Journal of Engineering, 2019, 41(11): 1484-1492.
29	Akturk A O, Sahin I. Literature Review on Metacognition and Its Measurement[J]. Procedia-social and Behavioral Sciences, 2011, 15: 3731-3736.

算法	e ≤ 0.05	e ≤ 0.10	e ≤ 0.25
文献[22]	99.50	NA	NA
文献[23]	99.10	100	100
文献[24]	99.60	99.84	100
本文方法	99.77	99.97	100

精准度误差	BioID			I2Head
精准度误差	文献[22]	文献[25]	本文算法	文献[26]	文献[25]	本文算法
e≤0.05	89.48	96.71	97.63	91.24	99.00	99.42
e≤0.10	98.22	98.95	99.30	96.38	99.72	100
e≤0.25	NA	100	100	97.42	100	100

模型	MobileNet v3	二级特征融合模块	预测模块	AP/%	AP50/%	检测速度/(帧/s)
原始SSD模型				68.4	84.1	32
改进模型1	√			71.3	86.7	59
改进模型2	√	√		75.1	89.2	48
本文模型	√	√	√	83.4	97.2	43

图像采集装置	检测速度/(帧/s)	检测精度/%
自主研发采集装置	36	97.42
普通摄像头	25	92.68

组数	SURF	ORB	Eye-ORB
平均	61.6	133.8	169.4
第1组	53	134	200
第2组	62	136	180
第3组	61	128	150
第4组	65	140	146
第5组	67	131	171