一种基于机器视觉的精准注意力追踪系统

doi:10.16182/j.issn1004731x.joss.23-FZ0829

摘要/Abstract

摘要：

针对学生注意力分配困难和对学习影响等问题，提出一种基于机器视觉的精准注意力追踪系统。该系统包括图像采集装置和精准的注意力追踪算法。图像采集装置可以获得更清晰的眼部区域图像。瞳孔中心定位算法用轻量级的MobileNet v3替换VGG16(visual geometry group network)，采用两级特征融合和中心关键点预测技术，提高了检测速度和准确率。该算法检测速度可达36 帧/s，准确率为97.42%。视线追踪算法旨在解决头部偏移的影响，实现对视线的精确追踪。研发了一款面向学龄儿童的阅读认知评价交互软件。该软件利用采集到的视线坐标计算相关眼动指标，再通过心理学理论分析建模来评估学龄儿童的思维认知能力，为心理学和教育学相关领域研究提供了参考和借鉴。

关键词: 瞳孔定位, 改进型SSD(single shot multibox detector)算法, Eye-ORB(oriented FAST and rotated brief)算法, 阅读认知, 注意力追踪

Abstract:

A precise attention tracking system based on machine vision is designed to address the difficulty in studying students' attention allocation. The system includes an image capture device and an accurate attention tracking algorithm. The image capture device can capture the clearer ocular images. The pupil center localization algorithm replaces VGG16 with lightweight MobileNetv3 and uses two-stage feature fusion and center keypoint prediction techniques to improve the speed and accuracy. The algorithm achieves a speed of up to 36 frames/s and 97.42% accuracy. The gaze tracking algorithm compensates for the head movements to achieve the meticulous gaze tracking. An interactive software for assessing cognitive abilities in school-age children is developed. The software calculates the eye movement indicators by using the collected gaze coordinates and evaluates the cognitive abilities based on psychological theory, and provides a reference for the psychology and education research.

Key words: pupil positioning, improved single shot multibox detector(SSD) algorithm, image gradient tilt algorithm, oriented FAST and rotated brief（Eye-ORB) algorithm, reading cognition, attention tracking

中图分类号:

TP391.7

刘纪元,祁瀚文,刘志诚等 . 一种基于机器视觉的精准注意力追踪系统[J]. 系统仿真学报, 2023, 35(10): 2087-2100.

Liu Jiyuan,Qi Hanwen,Liu Zhicheng,et al . A Precise Attention Tracking System Based on Computer Vision[J]. Journal of System Simulation, 2023, 35(10): 2087-2100.

图/表 19

图1

图2

图3

图4

图5

图6

图7

图8

图9

表1

表2

表3

表4

表5

表6

图10

表7

图11

图12

参考文献 29

1	Dong Wentao, Yang Lin, Gravina R, et al. ANFIS Fusion Algorithm for Eye Movement Recognition via Soft Multi-functional Electronic Skin[J]. Information Fusion, 2021, 71: 99-108.
2	Wang Xiaoming, Xiao Yineng, Deng Fangming, et al. Eye-movement-controlled Wheelchair Based on Flexible Hydrogel Biosensor and WT-SVM[J]. Biosensors, 2021, 11(6): 198.
3	Tesfamikael H H, Fray A, Mengsteab I, et al. Simulation of Eye Tracking Control Based Electric Wheelchair Construction by Image Segmentation Algorithm[J]. Journal of Innovative Image Processing, 2021, 3(1): 21-35.
4	Zhu Ji, Yang Hua, Liu Nian, et al. Online Multi-object Tracking With Dual Matching Attention Networks[C]//Computer Vision - ECCV 2018. Cham: Springer International Publishing, 2018: 379-396.
5	靳冰凌, 张震, 张子耀. 基于视觉的驾驶员疲劳特征提取方法[J]. 计算机技术与发展, 2018, 28(11): 193-197.
	Jin Bingling, Zhang Zhen, Zhang Ziyao. A Driving Fatigue Feature Extraction Method Based on Vision[J]. Computer Technology and Development, 2018, 28(11): 193-197.
6	陈加徐, 陆永华, 赵采仪, 等. 一种优化的自然光源下眼动视觉测量方法[J]. 应用科技, 2021, 48(4): 61-67.
	Chen Jiaxu, Lu Yonghua, Zhao Caiyi, et al. An Optimized Visual Measurement Method for Eye Movement Under Natural Light Source[J]. Applied Science and Technology, 2021, 48(4): 61-67.
7	宋琴琴, 杨国平. 基于视频图像的人眼动态追踪[J]. 计算机与数字工程, 2021, 49(7): 1457-1459, 1474.
	Song Qinqin, Yang Guoping. Human Eye Dynamic Tracking Based on Video Images[J]. Computer & Digital Engineering, 2021, 49(7): 1457-1459, 1474.
8	Wang Xiaoming, Zhao Xinbo, Zhang Yanning. Deep-learning-based Reading Eye-movement Analysis for Aiding Biometric Recognition[J]. Neurocomputing, 2021, 444: 390-398.
9	Brand J, Diamond S G, Thomas N, et al. Evaluating the Data Quality of the Gazepoint GP3 Low-cost Eye Tracker When Used Independently by Study Participants[J]. Behavior Research Methods, 2021, 53(4): 1502-1514.
10	Liu Wei, Anguelov D, Erhan D, et al. SSD: Single Shot MultiBox Detector[C]//Computer Vision-ECCV 2016. Cham: Springer International Publishing, 2016: 21-37.
11	Qassim H, Verma A, Feinzimer D. Compressed Residual-VGG16 CNN Model for Big Data Places Image Recognition[C]//2018 IEEE 8th Annual Computing and Communication Workshop and Conference (CCWC). Piscataway, NJ, USA: IEEE, 2018: 169-175.
12	Howard A, Sandler M, Chen Bo, et al. Searching for MobileNetV3[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway, NJ, USA: IEEE, 2019: 1314-1324.
13	Ma Dan, Lai Huicheng. Remote Sensing Image Matching Based Improved ORB in NSCT Domain[J]. Journal of the Indian Society of Remote Sensing, 2019, 47(5): 801-807.
14	Zhu Juntao, Gong Chaofei, Zhao Miaoxing, et al. Image Mosaic Algorithm Based on PCA-ORB Feature Matching[J]. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2020, XLII-3/W10: 83-89.
15	王素琴, 施文豪, 李兆歆, 等. 无人机航拍视频中的车辆检测方法[J]. 系统仿真学报, 2018, 30(7): 2776-2786.
	Wang Suqin, Shi Wenhao, Li Zhaoxin, et al. Vehicle Detection Method in UAV Aerial Video[J]. Journal of System Simulation, 2018, 30(7): 2776-2786.
16	柳有权, 裴雪, 李婉, 等. 基于邻近目标置信度评估的视觉目标跟踪与定位[J]. 系统仿真学报, 2020, 32(7): 1294-1300.
	Liu Youquan, Pei Xue, Li Wan, et al. Visual Tracking and Localization Based on Confidence Evaluation of Adjacent Targets[J]. Journal of System Simulation, 2020, 32(7): 1294-1300.
17	Li Xianguang, Ren Chao, Zhang Tengxu, et al. Unmanned Aerial Vehicle Image Matching Based on Improved RANSAC Algorithm and Surf Algorithm[J]. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2020, XLII-3/W10: 67-70.
18	He Qingfeng, Shahabi H, Shirzadi A, et al. Landslide Spatial Modelling Using Novel Bivariate Statistical Based Naïve Bayes, RBF Classifier, and RBF Network Machine Learning Algorithms[J]. Science of the Total Environment, 2019, 663: 1-15.
19	Villanueva A, Ponz V, Sesma-Sanchez L, et al. Hybrid Method Based on Topography for Robust Detection of Iris Center and Eye Corners[J]. ACM Transactions on Multimedia Computing, Communications, and Applications, 2013, 9(4): 25.
20	Martinikorena I, Cabeza R, Villanueva A, et al. Introducing I2head Database[C]//Proceedings of the 7th Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction. New York, NY, USA: Association for Computing Machinery, 2018: 1.
21	Jesorsky O, Kirchberg K J, Frischholz R W. Robust Face Detection Using the Hausdorff Distance[C]//Audio-and Video-based Biometric Person Authentication. Berlin, Heidelberg: Springer Berlin Heidelberg, 2001: 90-95.
22	Cai Haibin, Liu Bangli, Ju Zhaojie, et al. Accurate Eye Center Localization via Hierarchical Adaptive Convolution[C]//Proceedings of the 29th British Machine Vision Conference. Durham, UK: BMVA Press, 2018: 11504341.
23	Xia Yifan, Yu Hui, Wang Feiyue. Accurate and Robust Eye Center Localization via Fully Convolutional Networks[J]. IEEE/CAA Journal of Automatica Sinica, 2019, 6(5): 1127-1138.
24	Choi J H, Il Lee K, Kim Y C, et al. Accurate Eye Pupil Localization Using Heterogeneous CNN Models[C]//2019 IEEE International Conference on Image Processing (ICIP). Piscataway, NJ, USA: IEEE, 2019: 2179-2183.
25	Lee K I, Jeon J H, Song B C. Deep Learning-based Pupil Center Detection for Fast and Accurate Eye Tracking System[C]//Computer Vision-ECCV 2020. Cham: Springer International Publishing, 2020: 36-52.
26	Xiao Feng, Huang Kejie, Qiu Yue, et al. Accurate Iris Center Localization Method Using Facial Landmark, Snakuscule, Circle Fitting and Binary Connected Component[J]. Multimedia Tools and Applications, 2018, 77(19): 25333-25353.
27	胡艳红, 魏江, 梅少辉. 基于瞳孔角膜反射技术的视线估计方法[J]. 计算机工程与应用, 2018, 54(14): 7-10, 18.
	Hu Yanhong, Wei Jiang, Mei Shaohui. Gaze Estimation Method Based on Pupils and Corneal Reflection Technique[J]. Computer Engineering and Applications, 2018, 54(14): 7-10, 18.
28	李擎, 胡京尧, 迟健男, 等. 视线追踪中一种新的由粗及精的瞳孔定位方法[J]. 工程科学学报, 2019, 41(11): 1484-1492.
	Li Qing, Hu Jingyao, Chi Jiannan, et al. A New Pupil Localization Method From Rough to Precise in Gaze Tracking[J]. Chinese Journal of Engineering, 2019, 41(11): 1484-1492.
29	Akturk A O, Sahin I. Literature Review on Metacognition and Its Measurement[J]. Procedia-social and Behavioral Sciences, 2011, 15: 3731-3736.

算法	e ≤ 0.05	e ≤ 0.10	e ≤ 0.25
文献[22]	99.50	NA	NA
文献[23]	99.10	100	100
文献[24]	99.60	99.84	100
本文方法	99.77	99.97	100

精准度误差	BioID			I2Head
精准度误差	文献[22]	文献[25]	本文算法	文献[26]	文献[25]	本文算法
e≤0.05	89.48	96.71	97.63	91.24	99.00	99.42
e≤0.10	98.22	98.95	99.30	96.38	99.72	100
e≤0.25	NA	100	100	97.42	100	100

模型	MobileNet v3	二级特征融合模块	预测模块	AP/%	AP50/%	检测速度/(帧/s)
原始SSD模型				68.4	84.1	32
改进模型1	√			71.3	86.7	59
改进模型2	√	√		75.1	89.2	48
本文模型	√	√	√	83.4	97.2	43

图像采集装置	检测速度/(帧/s)	检测精度/%
自主研发采集装置	36	97.42
普通摄像头	25	92.68

组数	SURF	ORB	Eye-ORB
平均	61.6	133.8	169.4
第1组	53	134	200
第2组	62	136	180
第3组	61	128	150
第4组	65	140	146
第5组	67	131	171