基于多模态脑机接口的虚拟现实康复训练系统

doi:10.16182/j.issn1004731x.joss.25-0836

摘要/Abstract

摘要：

人口老龄化导致认知与运动功能障碍的康复需求日益凸显。针对传统康复趣味性差、现有虚拟现实(VR)康复缺乏客观生理评估等问题，整合VR交互、近红外脑功能成像与动作捕捉技术，研发基于多模态脑机接口的VR康复训练系统。通过构建沉浸式认知运动结合的训练环境，引导用户完成上肢任务。通过招募受试者并同步采集脑网络数据与Kinect上肢运动参数进行多模态评估。结果显示，VR训练可优化前额叶网络效率，老年组在任务中呈现出更强的神经连接代偿机制，年轻组则表现出更高的运动效率。所提方法实现了康复的客观量化评估，为VR康复提供新颖多模态评估范式，对康复系统的适老化设计具有重要的指导意义。

关键词: 虚拟现实, 脑机接口, 近红外脑功能成像, 上肢康复, 多模态评估, 认知训练

Abstract:

The aging population has led to an increasing demand for rehabilitation for cognitive and motor functions. In response to the lack of interest in traditional rehabilitation and the absence of objective physiological assessment in existing virtual reality (VR) rehabilitation systems, a VR rehabilitation training system based on multimodal brain computer interface is developed by integrating VR interaction, near-infrared brain functional imaging, and motion capture technology. An immersive cognitive-motor integrated training environment was constructed to guide users in completing upper limb tasks. By recruiting subjects and synchronously collecting brain network data and Kinect upper limb motion parameters, multimodal assessment was performed. The results show that VR training can enhance the efficiency of the prefrontal network. Furthermore, during the tasks, the elderly group exhibited a stronger neural connections compensation mechanism, whereas the young group demonstrated higher motor efficiency. The proposed method enables objective quantitative assessment of rehabilitation, providing a novel multimodal assessment paradigm for VR rehabilitation and offering significant guidance for the age-friendly design of rehabilitation systems.

Key words: virtual reality(VR), brain-computer interface, functional near-infrared spectroscopy, upper-limb rehabilitation, multimodal assessment, cognitive training

中图分类号:

TP391.9

曲静,方凯宁,朱山童等 . 基于多模态脑机接口的虚拟现实康复训练系统[J]. 系统仿真学报, 2026, 38(1): 125-135.

Qu Jing,Fang Kaining,Zhu Shantong,et al . Virtual Reality Rehabilitation Training System Based on Multimodal Brain-computer Interface[J]. Journal of System Simulation, 2026, 38(1): 125-135.

图/表 8

图1

图2

图3

图4

表1

图5

图6

图7

参考文献 34

[1]	Garmany A, Yamada S, Terzic A. Longevity Leap: Mind the Healthspan Gap[J]. NPJ Regenerative Medicine, 2021, 6(1): 57.
[2]	Guillemot J R, Zhang Xue, Warner M E. Population Aging and Decline will Happen Sooner than We Think[J]. Social Sciences, 2024, 13(4): 190.
[3]	Hartshorne J K, Germine L T. When Does Cognitive Functioning Peak? The Asynchronous Rise and Fall of Different Cognitive Abilities Across the Life Span[J]. Psychological Science, 2015, 26(4): 433-443.
[4]	Yuan Manqiong, Long Xianxian, Zhang Zeyun, et al. Longitudinal Trajectory Effects of Different MCI Subtypes on General Cognitive and Daily Functions in a Population-based Cohort of Older Adults[J]. Journal of Psychiatric Research, 2024, 171: 296-305.
[5]	Jaeggi S M, Buschkuehl M, Jonides J, et al. Short- and Long-term Benefits of Cognitive Training[J]. Proceedings of the National Academy of Sciences of the United States of America, 2011, 108(25): 10081-10086.
[6]	Mirabito G, Verhaeghen P. The Effects of Mindfulness Interventions on Older Adults' Cognition: A Meta-analysis[J]. The Journals of Gerontology: Series B, 2023, 78(3): 394-408.
[7]	Yan Mingli, Zhao Yawei, Meng Qiuyan, et al. Effects of Virtual Reality Combined Cognitive and Physical Interventions on Cognitive Function in Older Adults with Mild Cognitive Impairment: A Systematic Review and Meta-analysis[J]. Ageing Research Reviews, 2022, 81: 101708.
[8]	樊振梅, 王春钚, 谢婷, 等. 肢体协调辅助装置结合VR训练对急性脑梗死颅内血管介入治疗患者步态平衡和表面肌电图的影响[J]. 中国卒中杂志, 2023, 18(2): 194-200.
	Fan Zhenmei, Wang Chunbu, Xie Ting, et al. Effects of Limb Coordination Device Combined with VR Training on Gait Balance and Surface Electromyography in Patients with Acute Cerebral Infarction After Intracranial Endovascular Therapy[J]. Chinese Journal of Stroke, 2023, 18(2): 194-200.
[9]	林宜杰, 张利, 徐琳峰, 等. 基于眼动技术的VR认知训练在卒中后右脑半球损伤注意障碍患者中的康复应用[J]. 中国康复, 2024, 39(2): 72-75.
	Lin Yijie, Zhang Li, Xu Linfeng, et al. Application of VR Cognitive Training Based on Eye Movement Technology in Rehabilitation of Patients with Right Hemi-sphere Injury After Stroke[J]. Chinese Journal of Rehabilitation, 2024, 39(2): 72-75.
[10]	韦卓男, 樊响, 余可妍, 等. 结构MRI联合基于VR眼动追踪技术的计算机化认知评估在阿尔茨海默病早期诊断中的应用价值[J]. 磁共振成像, 2024, 15(6): 49-53.
	Wei Zhuonan, Fan Xiang, Yu Keyan, et al. Application Value of Structural MRI Combined with Computerized Cognitive Assessment Based on VR Eye-tracking Technology in Early Diagnosis of Alzheimer's Disease[J]. Chinese Journal of Magnetic Resonance Imaging, 2024, 15(6): 49-53.
[11]	Mrakic-Sposta S, Di Santo S G, Franchini F, et al. Effects of Combined Physical and Cognitive Virtual Reality-based Training on Cognitive Impairment and Oxidative Stress in MCI Patients: A Pilot Study[J]. Frontiers in Aging Neuroscience, 2018, 10: 282.
[12]	Chatterjee K, Buchanan A, Cottrell K, et al. Immersive Virtual Reality for the Cognitive Rehabilitation of Stroke Survivors[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022, 30: 719-728.
[13]	Zhang Chengjie, Yu Suiran, Ji Jiancheng. CFI: A VR Motor Rehabilitation Serious Game Design Framework Integrating Rehabilitation Function and Game Design Principles with an Upper Limb Case[J]. Journal of NeuroEngineering and Rehabilitation, 2024, 21(1): 113.
[14]	Bueno-Vesga J A, Xu Xinhao, He Hao. The Effects of Cognitive Load on Engagement in a Virtual Reality Learning Environment[C]//2021 IEEE Virtual Reality and 3D User Interfaces (VR). Piscataway: IEEE, 2021: 645-652.
[15]	Kim H K, Park J, Choi Y, et al. Virtual Reality Sickness Questionnaire (VRSQ): Motion Sickness Measurement Index in a Virtual Reality Environment[J]. Applied Ergonomics, 2018, 69: 66-73.
[16]	王杨龙, 霍聪聪, 邵广健, 等. 基于功能性近红外光谱技术探讨神经反馈运动训练对脑卒中患者脑功能网络的影响[J]. 康复学报, 2025, 35(3): 250-259.
	Wang Yanglong, Huo Congcong, Shao Guangjian, et al. Exploring the Effects of Motor Training with Neurofeedback on Brain Functional Network in Stroke Patients Based on Functional Near-infrared Spectroscopy[J]. Rehabilitation Medicine, 2025, 35(3): 250-259.
[17]	Zhang Yanjie, Li Fan, Chang Danni. VR Rehabilitation System Evaluator: A fNIRS-based and LLM-enabled Evaluation Paradigm for Mild Cognitive Impairment[J]. Advanced Engineering Informatics, 2024, 62, Part B: 102734.
[18]	Bae S, Park H S. Development of Immersive Virtual Reality-based Hand Rehabilitation System Using a Gesture-controlled Rhythm Game with Vibrotactile Feedback: An fNIRS Pilot Study[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2023, 31: 3732-3743.
[19]	Kuan Cha, Wang Jinying, Li Yan, et al. A Novel Upper-limb Tracking System in a Virtual Environment for Stroke Rehabilitation[J]. Journal of NeuroEngineering and Rehabilitation, 2021, 18(1): 166.
[20]	Yang Kaizhi, Peng Liang, Tong Lina, et al. An Assessment Method for Upper Limb Rehabilitation Training Using Kinect[C]//2018 IEEE 8th Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER). Piscataway: IEEE, 2018: 949-953.
[21]	Qu Jing, Bu Lingguo, Zhao Lei, et al. A Training and Assessment System for Human-computer Interaction Combining fNIRS and Eye-tracking data[J]. Advanced Engineering Informatics, 2024, 62: 102765.
[22]	Hou Xin, Zhang Zong, Zhao Chen, et al. NIRS-KIT: A MATLAB Toolbox for Both Resting-state and Task fNIRS Data Analysis[J]. Neurophotonics, 2021, 8(1): 010802.
[23]	Qu Jing, Bu Lingguo, Chen Zhongxin, et al. ArmVR: Innovative Design Combining Virtual Reality Technology and Mechanical Equipment in Stroke Rehabilitation Therapy[J]. IEEE Transactions on Visualization and Computer Graphics, 2025, 31(5): 2288-2298.
[24]	Luke R, Larson E, Shader M J, et al. Analysis Methods for Measuring Passive Auditory fNIRS Responses Generated by a Block-design Paradigm[J]. Neurophotonics, 2021, 8(2): 025008.
[25]	Niso G, Bruña R, Pereda E, et al. Hermes: Towards an Integrated Toolbox to Characterize Functional and Effective Brain Connectivity[J]. Neuroinformatics, 2013, 11(4): 405-434.
[26]	Zhao Lei, Sun Haoran, Yang Fei, et al. A Multimodal Data Driven Rehabilitation Strategy Auxiliary Feedback Method: A Case Study[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022, 30: 1181-1190.
[27]	Nasreddine Z S, Phillips N A, Bédirian V, et al. The Montreal Cognitive Assessment, MoCA: A Brief Screening Tool for Mild Cognitive Impairment[J]. Journal of the American Geriatrics Society, 2005, 53(4): 695-699.
[28]	Liao Yingyi, Tseng H Y, Lin Yijia, et al. Using Virtual Reality-based Training to Improve Cognitive Function, Instrumental Activities of Daily Living and Neural Efficiency in Older Adults with Mild Cognitive Impairment[J]. European Journal of Physical and Rehabilitation Medicine, 2020, 56(1): 47-57.
[29]	Davis S W, Dennis N A, Daselaar S M, et al. Que PASA? The Posterior-anterior Shift in Aging[J]. Cerebral Cortex, 2008, 18(5): 1201-1209.
[30]	Bunzeck N, Steiger T K, Krämer U M, et al. Trajectories and Contributing Factors of Neural Compensation in Healthy and Pathological Aging[J]. Neuroscience & Biobehavioral Reviews, 2024, 156: 105489.
[31]	Tian Mi, Cai Yuchao, Zhang Jie. The Impact of Virtual Reality-based Products on Mild Cognitive Impairment Senior Subjects: An Experimental Study Using Multiple Sources of Data[J]. Applied Sciences, 2023, 13(4): 2372.
[32]	Su Chuanjun, Chiang C Y, Huang Jingyan. Kinect-enabled Home-based Rehabilitation System Using Dynamic Time Warping and Fuzzy Logic[J]. Applied Soft Computing, 2014, 22: 652-666.
[33]	Yücel M A, Anderson J E, Rogers D J, et al. Quantifying the Impact of Hair and Skin Characteristics on fNIRS Signal Quality for Enhanced Inclusivity[J/OL]. Nature Human Behaviour. (2025-09-02) [2025-09-10]. .
[34]	Roy S, Wu Jingyi, Cao Jiaming, et al. Exploring the Impact and Influence of Melanin on Frequency-domain Near-infrared Spectroscopy Measurements[J]. Journal of Biomedical Optics, 2024, 29(S3): S33310.

参数	老年组	年轻组	p值
女性比例/%	60	60
年龄(岁)	61.333 ± 3.994	22.467 ± 1.356	<0.001*
身高/cm	161.133 ± 8.927	167.867± 7.019	0.029*
体重/kg	64.800 ± 12.612	57.733 ± 9.106	0.089
MoCA量表分数	20.733 ± 6.692	28.600 ± 0.828	<0.001*