脑机接口（Brain-Computer Interface，BCI）技术是一种允许大脑和外部设备直接进行信息交流的新型人机交互方式。然而，目前单模态BCI系统分类准确率和信息传输率较低，难以满足实际应用需求。视线追踪（Gaze Tracking，GT）技术是一种基于视觉信号的人机交互方式，能够准确地捕捉用户的注视点。因此，本研究将视线追踪技术与脑电信号中的稳态视觉诱发电位（Steady-State Visual Evoked Potential，SSVEP）相结合，实现混合模态的交互系统，相较于单模态BCI系统具有更好的系统性能和人机交互体验。本文主要工作内容如下：

（1）针对典型相关分析（Canonical Correlation Analysis，CCA）算法未考虑个体差异性，以及在识别SSVEP信号过程中忽略了谐波分量信息等问题，本文提出了一种基于优化个体模板的多子带SSVEP信号识别算法（Optimize Individual Template-Filter Bank Canonical Correlation Analysis，OIT-FBCCA）。该算法利用联合空间滤波器对离线数据集中的共同特征进行优化，以构建优化的个体模板，并通过滤波器组将脑电信号分解为多个子带分量，分别与优化的个体模板进行相关分析，再加权平方和得到特征值，充分利用了谐波分量信息。在留一交叉验证实验中，结果显示本文所提算法在2秒时间窗下平均分类准确率为89.87±1.96%，相较于CCA、IT-CCA和FBCCA分别提升了12.1%、7.81%和6.1%，表明OIT-FBCCA算法具有更高的SSVEP信号识别性能。

（2）针对视线追踪技术存在头部自由度受限、鲁棒性较差等问题，本文提出了一种基于CS-MobileNetV3的多维度特征联合视线追踪模型。将注意力机制ECA模块和MobileNetV3中的SE模块融合，并在该网络最后的全连接层前嵌入CBAM注意力机制，得到改进后的CS-MobileNetV3模型，同时在全连接层中联合瞳孔坐标位置，使模型学习到多维度特征信息，提高模型的泛化能力。实验结果显示，本文提出的视线追踪模型在自建数据集上测试准确率达到94.57%，相较于ShuffleNetV2、VGG16、ResNet50、EfficientNet和MobileNetV3模型分别提升了6.2%、5.13%、7.43%、5.01%、7.84%，表明本文提出的视线追踪模型具有更强的鲁棒性和视线追踪性能。

（3）为了验证本文所提混合模态方法的有效性，搭建了一个基于GT-BCI的混合模态交互系统。该系统以护理床为应用平台，使用摄像头采集图像，同时利用Emotiv Epoc+脑电设备采集脑电数据，在服务器端分别对视线方向和脑电信号进行识别，通过D-S证据理论在决策层融合两种模态的识别结果，并将融合后的结果转化为对应的控制指令，通过蓝牙通信发送至护理床完成相应动作。5名被试者参与了交互实验，结果显示混合模态的平均准确率达到了92.86±2.52%，相较于单模态BCI和GT方式分别提高了10.72%和5.72%，表明混合模态具有更高的系统性能。

本研究将GT与BIC相结合，提升了单模态BCI系统的泛化性能，对于设计混合脑机接口的人机交互系统提供了新的思路。

Brain-Computer Interface (BCI) technology is a new type of human-computer interaction that allows direct information exchange between the brain and external devices. However, the current unimodal BCI system has low classification accuracy and information transfer rate, which is difficult to meet the practical application requirements. Gaze Tracking (GT) technology is a human-computer interaction method based on visual signals, which can accurately capture the user's gaze point. Therefore, this study combines the Gaze Tracking technology with Steady-State Visual Evoked Potential (SSVEP) in EEG signals to realize a mixed-modality interaction system, which has better system performance and human-computer interaction experience compared with the unimodal BCI system. The main work of this paper is as follows:

(1) Aiming at the problems that the Canonical Correlation Analysis (CCA) algorithm does not consider the individual variability and ignores the harmonic component information in the process of identifying SSVEP signals, this paper proposes an Optimize Individual Template-Filter Bank Canonical Correlation Analysis (OIT-FBCCA) algorithm to recognize SSVEP signals. The algorithm uses a joint spatial filter to optimize the common features in the offline dataset in order to construct the optimized individual template, and decomposes the EEG signals into multiple subband components through the filter bank, which are correlated with the optimized individual template respectively, and then weighted sum of squares to obtain the eigenvalues, which makes full use of the harmonic component information. In the leave-one-out cross-validation experiment, the results show that the proposed algorithm in this paper has an average classification accuracy of 89.87±1.96% under a 2-second time window, which is 12.1%, 7.81%, and 6.1% higher compared to CCA, IT-CCA, and FBCCA, respectively, and suggests that the OIT-FBCCA algorithm has a higher performance of recognizing SSVEP signals.

(2) Aiming at the problems of restricted head degrees of freedom and poor robustness of line-of-sight tracking technology, this paper proposes a multi-dimensional feature joint line-of-sight tracking model based on CS-MobileNetV3. The attention mechanism ECA module and the SE module in MobileNetV3 are fused, and the CBAM attention mechanism is embedded in front of the last fully-connected layer of this network to obtain the improved CS-MobileNetV3 model, while the joint pupil coordinate position in the fully-connected layer enables the model to learn the multidimensional feature information and improves the model's generalization ability. The experimental results show that the accuracy of the vision tracking model proposed in this paper reaches 94.57% when tested on the self-built dataset, which is improved by 6.2%, 5.13%, 7.43%, 5.01%, 7.84%, compared with the ShuffleNetV2, VGG16, ResNet50, EfficientNet, and MobileNetV3 models, respectively. It shows that the line-of-sight tracking model proposed in this paper has stronger robustness and line-of-sight tracking performance.

(3) In order to verify the effectiveness of the mixed-modal approach proposed in this paper, a GT-BCI-based mixed-modal interaction system was built. The system takes the nursing bed as the application platform, uses the camera to collect images, and at the same time uses the Emotiv Epoc+ EEG device to collect EEG data, recognizes the direction of vision and EEG signals at the server side, fuses the recognition results of the two modalities at the decision-making level through the theory of D-S evidence, and translates the fused results into the corresponding control commands, which are sent to the nursing bed to complete the corresponding actions through Bluetooth communication. 5 subjects participated in the interaction experiments. Five subjects participated in the interaction experiment, and the results showed that the average accuracy of the hybrid modality reached 92.86±2.52%, which was 10.72% and 5.72% higher compared to the unimodal BCI and GT approaches, respectively, indicating that the hybrid modality has higher system performance.

This study combines GT with BIC to improve the generalization performance of single-modal BCI systems, which provides new ideas for designing human-computer interaction systems with hybrid brain-computer interfaces.

[1] 姚伟伟, 王昭, 林惠等. 电动护理床升降机构的模糊PID自动化控制方法[J]. 机械制造与自动化, 2022, 51(03): 236-239.

[2] 周平, 陈思韵, 李文兮等. 促进康复动机的策略在卒中康复中的应用进展[J]. 中国康复医学杂志, 2024, 39(02): 280-287.

[3] Jetter H C, Reiterer H, Geyer F. Blended Interaction: understanding natural human-computer interaction in post-WIMP interactive spaces[J]. Personal and Ubiquitous Computing, 2014, 18: 1139-1158.

[4] Cruz A, Pires G, Lopes A, et al. A self-paced BCI with a collaborative controller for highly reliable wheelchair driving: Experimental tests with physically disabled individuals[J]. IEEE Transactions on Human-Machine Systems, 2021, 51(2): 109-119.

[5] 张汉明, 马金刚, 张宁宁等. 深度学习在癫痫检测中的应用进展[J]. 计算机工程与应用, 2023, 59(10): 35-47.

[6] 刘邈, 林超, 韩锦等. 基于P300特征的脑-机接口解码算法研究综述[J]. 信号处理, 2023, 39(08): 1367-1385.

[7] Sutter E E. The brain response interface: communication through visually-induced electrical brain responses[J]. Journal of Microcomputer Applications, 1992, 15(1): 31-45.

[8] 王金甲, 杨成杰. P300脑机接口控制智能家居系统研究[J]. 生物医学工程学杂志, 2014, 31(04): 762-766.

[9] Bi L, Lian J, Jie K, et al. A speed and direction-based cursor control system with P300 and SSVEP[J]. Biomedical Signal Processing and Control, 2014, 14: 126-133.

[10] Diez P F, Müller S M T, Mut V A, et al. Commanding a robotic wheelchair with a high-frequency steady-state visual evoked potential based brain-computer interface[J]. Medical engineering & physics, 2013, 35(8): 1155-1164.

[11] Cheng M, Gao X, Gao S, et al. Design and implementation of a brain-computer interface with high transfer rates[J]. IEEE transactions on biomedical engineering, 2002, 49(10): 1181-1186.

[12] 肖征东. 基于大鼠皮层锋电位及LFP信号的急性疼痛解码算法研究[D]. 浙江大学, 2019.

[13] 贺辉, 黄君浩. 基于眼动跟踪的人机交互应用[J]. 山东大学学报(工学版), 2021, 51(02): 1-8.

[14] Morimoto C H, Koons D, Amir A, et al. Pupil detection and tracking using multiple light sources[J]. Image and vision computing, 2000, 18(4): 331-335.

[15] Sugano Y, Matsushita Y, Sato Y, et al. An incremental learning method for unconstrained gaze estimation[C]//Computer Vision-ECCV 2008: 10th European Conference on Computer Vision, Marseille, France, October 12-18, 2008, Proceedings, Part III 10. Springer Berlin Heidelberg, 2008: 656-667.

[16] Kumar D, Sharma A. Electrooculogram-based virtual reality game control using blink detection and gaze calibration[C]//2016 International Conference on Advances in Computing, Communications and Informatics (ICACCI). IEEE, 2016: 2358-2362.

[17] Cheng Y, Lu F, Zhang X. Appearance-based gaze estimation via evaluation-guided asymmetric regression[C]//Proceedings of the European conference on computer vision (ECCV), 2018: 100-115.

[18] 柳忠起, 袁修干, 樊瑜波, 刘伟康, 卫勇. 模拟飞机着陆飞行中专家和新手眼动行为的对比[J]. 航天医学与医学工程, 2009, 05: 358-361.

[19] 胡大正. 自然光下基于单目摄像头的视线跟踪算法研究[D]. 华南理工大学, 2016.

[20] Wang X, Guo L, Yuan Z, et al. Towards eye-tracking-based technology on sight interpretation performance improvement[J]. Journal of Mechanics in Medicine and Biology, 2022, 22(9): 182-194.

[21] Pfurtscheller G, Allison B Z, Bauernfeind G, et al. The hybrid BCI[J]. Frontiers in neuroscience, 2010, 4: 1283.

[22] Lin K, Cinetto A, Wang Y, et al. An online hybrid BCI system based on SSVEP and EMG[J]. Journal of neural engineering, 2016, 13(2): 026020.

[23] Lim J H, Lee J H, Hwang H J, et al. Development of a hybrid mental spelling system combining SSVEP-based brain-computer interface and webcam-based eye tracking[J]. Biomedical Signal Processing and Control, 2015, 21: 99-104.

[24] Zhang Y, Xie J, Xu G, et al. Asynchronous eye-tracking-actuated switch for steady-state visual evoked potential based brain-computer interface applications[C]//2019 9th International IEEE/EMBS Conference on Neural Engineering (NER). IEEE, 2019: 360-364.

[25] Li Y, Pan J, Wang F, et al. A hybrid BCI system combining P300 and SSVEP and its application to wheelchair control[J]. IEEE Transactions on Biomedical Engineering, 2013, 60(11): 3156-3166.

[26] 谢平, 陈迎亚, 郝艳彪等. 基于脑肌电融合的混合脑机接口研究[J]. 中国生物医学工程学报, 2016, 35(01): 20-30.

[27] Shaikh M S. Fundamental engineering for brain-computer interfacing (bci): initiative for neuron-command operating devices[J]. Comput Biol Bioinfor, 2017, 5(4): 50-56.

[28] 刘彦俊. 面向运动想象脑电信号的时空频特征识别研究[D]. 广州大学, 2022.

[29] Wang K, Xu M, Wang Y, et al. Enhance decoding of pre-movement EEG patterns for brain-computer interfaces[J]. Journal of neural engineering, 2020, 17(1): 016033.

[30] 王迪, 陶庆, 张小栋等. 采用改进共空间模式算法的四类表情辅助脑电信号识别方法[J]. 西安交通大学学报, 2022, 56(12): 136-143.

[31] Huang J, Yang P, Wan B, et al. KDLPCCA-Based Projection for Feature Extraction in SSVEP-Based Brain-Computer Interfaces[J]. Journal of Shanghai Jiaotong University (Science), 2022: 1-8.

[32] 李为龙, 张睿芝, 高国雅等. 稳态视觉诱发电位信号的预处理滤波设计及实验评测[J]. 生物医学工程研究, 2023, 42(04): 319-328.

[33] 王丽娟, 聂坤, 高玉元等. 中国帕金森病重复经颅磁刺激治疗指南[J]. 中国神经精神疾病杂志, 2021, 47(10): 577-585.

[34] 吴自新, 李钊, 刘涛. 一种干涉仪测向解模糊的方法[J]. 电子技术与软件工程, 2021, (04): 95-97.

[35] Sanchez A, Vazquez C, Marker C, et al. Attentional disengagement predicts stress recovery in depression: an eye-tracking study[J]. Journal of abnormal psychology, 2013, 122(2): 303.

[36] 余璀璨, 李慧斌. 基于深度学习的人脸识别方法综述[J]. 工程数学学报, 2021, 38(04): 451-469.

[37] Klaib A F, Alsrehin N O, Melhem W Y, et al. Eye tracking algorithms, techniques, tools, and applications with an emphasis on machine learning and Internet of Things technologies[J]. Expert Systems with Applications, 2021, 166: 114037.

[38] Park S, Spurr A, Hilliges O. Deep pictorial gaze estimation[C]//Proceedings of the European conference on computer vision (ECCV), 2018: 721-738.

[39] 朱铱镤, 杜秀娟. 水下环境监测系统综述与展望[J]. 计算机工程与应用, 2023, 59(10): 65-74.

[40] 沈晓燕, 王雪梅, 王燕. 基于样本熵和模式识别的脑电信号识别算法研究[J]. 计算机工程与科学, 2020, 42(08): 1482-1488.

[41] Wu Y, Yang R, Chen W, et al. Research on Unsupervised Classification Algorithm Based on SSVEP[J]. Applied Sciences, 2022, 12(16): 8274.

[42] Meng X, Tian H, Chen X, et al. Numerical simulation of combustion surface regression based on Butterworth filter in hybrid rocket motor[J]. Acta Astronautica, 2023, 202: 400-410.

[43] 楚瑞博, 王剑, 张迁等. 基于小波收缩的改进阈值脑电信号去噪方法研究[J]. 现代电子技术, 2023, 46(11): 76-80.

[44] Houssein E H. Particle swarm optimization-enhanced twin support vector regression for wind speed forecasting[J]. Journal of Intelligent Systems, 2019, 28(5): 905-914.

[45] Zhang Y, Xu P, Cheng K, et al. Multivariate synchronization index for frequency recognition of SSVEP-based brain-computer interface[J]. Journal of neuroscience methods, 2014, 221: 32-40.

[46] 杨满, 钟子平, 韩锦等. 稳态视觉诱发电位解码算法研究综述[J]. 生物医学工程学杂志, 2022, 39(02): 416-425.

[47] Bin G, Gao X, Wang Y, et al. A high-speed BCI based on code modulation VEP[J]. Journal of neural engineering, 2011, 8(2): 025015.

[48] Chen X, Wang Y, Gao S, et al. Filter bank canonical correlation analysis for implementing a high-speed SSVEP-based brain-computer interface[J]. Journal of neural engineering, 2015, 12(4): 046008.

[49] 张国治, 陈康, 方荣行等. 基于DGA与鲸鱼算法优化Logit Boost-决策树的变压器故障诊断方法[J]. 电力系统保护与控制, 2023, 51(07): 63-72.

[50] 邵庆祝, 谢民, 汪伟等. 基于深度学习的电流互感器隐性故障诊断方法[J]. 自动化技术与应用, 2024, 43(03): 82-86.

[51] 董文豪, 张怀. 基于迁移学习的岩屑岩性识别[J]. 中国科学院大学学报, 2023, 40(06): 743-750.

[52] 潘秋景, 吴洪涛, 张子龙等. 基于多域物理信息神经网络的复合地层隧道掘进地表沉降预测[J]. 岩土力学, 2024, 45(02): 539-551.

[53] Howard A, Sandler M, Chu G, et al. Searching for mobilenetv3[C]//Proceedings of the IEEE/CVF international conference on computer vision, 2019: 1314-1324.

[54] 熊凯飞, 樊绍胜, 吴静. 基于改进YOLOv4的雾天变电站电力设备识别方法[J]. 无线电工程, 2022, 52(08): 1504-1512.

[55] 陈跃鹏, 任博博, 靳佳澍等. 基于并联卷积神经网络的运动模糊去除模型[J]. 华中科技大学学报(自然科学版), 2023, 51(09): 140-145.

[56] Hu J, Shen L, Sun G. Squeeze-and-excitation networks[C]//Proceedings of the IEEE conference on computer vision and pattern recognition, 2018: 7132-7141.

[57] Wang Q, Wu B, Zhu P, et al. ECA-Net: Efficient channel attention for deep convolutional neural networks[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020: 11534-11542.

[58] 郭乾宇, 武一, 刘华宾等. 基于损失自注意力机制的立体匹配算法研究[J]. 计算机应用研究, 2022, 39(07): 2236-2240.

[59] Ma N, Zhang X, Zheng H T, et al. Shufflenet v2: Practical guidelines for efficient cnn architecture design[C]//Proceedings of the European conference on computer vision (ECCV), 2018: 116-131.

[60] Simonyan K, Zisserman A. Very Deep Convolutional Networks for Large-Scale Image Recognition[J]. CoRR, 2014, abs/1409. 1556.

[61] He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE conference on computer vision and pattern recognition, 2016: 770-778.

[62] Tan M, Le Q. Efficientnet: Rethinking model scaling for convolutional neural networks[C]//International conference on machine learning. PMLR, 2019: 6105-6114.

[63] 张世醒, 韩德强, 范晓婧. 利用证据理论的多分类支持向量数据描述算法[J]. 西安交通大学学报, 2023, 57(02): 151-160.

[64] 李少年, 李毅, 魏列江等. 基于改进卡尔曼数据融合算法的温室物联网采集系统研究[J]. 传感技术学报, 2022, 35(04): 558-564.

[65] Jiang D, Wang Z. Research on mechanical equipment fault diagnosis method based on deep learning and information fusion[J]. Sensors, 2023, 23(15): 6999.