伴随着煤矿开采的现代化和智能化，矿下工作人员在进行井下作业时的安全问题已成为整个煤矿开采行业的最为关注的问题。为了降低由于矿工精神状态不佳而带来的矿下安全隐患，本课题从矿工的情绪状态和疲劳状态两种精神状态维度出发研究了多源信息融合下矿工精神状态识别和量化分析的问题。课题研究内容具体包括如下：

(1) 针对矿工的情绪状态识别问题，从人脸情绪信息和脑电情绪信息两部分实现情绪状态的判断。人脸情绪模态信息上，构建了适应矿工面部情绪识别的数据集并搭建了多尺度特征提取网络模型对数据集进行训练学习。脑电模态信息上，将脑电数据转换为脑电地形图并通过脑地形图识别网络模型对脑电地形图进行识别分析。在人脸信息和脑电信息的识别结果上，提出了自适应权值寻优算法实现脑电信息和人脸信息的决策层加权信息融合从而得出最终的情绪状态识别结果。实验结果表明，经过多模态信息融合后对中性、开心、生气等情绪状态识别精度为85.4%、88.6%、84.1%，分别较脑电模态和人脸模态下的识别精度提高了17.7%、6.6%、5.4%和18.6%、3.9%、19.9%。

(2) 针对矿工的疲劳状态识别问题，构建了一种能够对疲劳状态信息进行表征且包含了潜在导联信息特征的脑功率特征矩阵。脑功率特征矩阵利用不同波段脑电信号的功率谱特征构建了新的特征指标，并按照电极位置映射成特征矩阵。将脑功率特征矩阵在不同的分类识别模型中进行训练学习，得出CapsNet胶囊网络对疲劳状态识别精度为93.54%，较另外两种卷积识别模型识别精度分别提高了8.98%、5.68%。

(3) 针对精神状态的量化评估问题，提出了基于多源信息融合的矿工精神状态评估算法。利用评估算法将主客观下的精神状态信息进行量化融合得到最终的精神状态评估值。通过与设定的精神状态评估阈值比较得出矿工当前精神状态是否适合井下工作。12名受试者中仅有1名的评估结果有所偏差，表明了精神状态评估方法具有一定可行性。

本课题提出的多源信息融合的矿工精神状态评估方法，能够有效的融合矿工面部信息和脑电信息并对受试矿工的精神状态做出较为准确和直观的状态评估，基本上能够完成矿工的精神状态判别的任务。对矿工的井下工作和煤矿安全生产有一定的参考价值。

With the modernization and intelligence of coal mining, the safety of miners has become the most concerned issue in the entire coal mining industry. In order to reduce the hidden safety hazards in the mine caused by the poor mental state of the miners, this paper studies the problem of the identification and quantitative analysis of the miners' mental state under the fusion of multi-source information from the two mental state dimensions of the miners' emotional state and the fatigue state. The research contents of the subject specifically include:

(1) For the recognition of miners' emotional state, the judgment of emotional state is realized from two parts of facial emotional information and EEG emotional information. In terms of facial emotion modal information, a dataset suitable for facial emotion recognition of miners is constructed.  Also, a multi-scale feature extraction network model is built to train and learn the dataset. On the EEG modal information, the EEG data is converted into an EEG topographic map and the EEG topographic map is identified and analyzed through the brain topographic map recognition network model. On the recognition results of face information and EEG information, an adaptive weight optimization algorithm is proposed to realize the weighted information fusion of EEG information and face information at the decision level to obtain the final emotional state recognition result. The experimental results show that after multi-modal information fusion, the recognition accuracy of neutral, happy, angry and other emotional states is 85.4%, 88.6%, and 84.1%, which are higher 17.7%, 6.6%, 5.4% and 18.6%, 3.9%, 19.9% than the recognition accuracy of EEG mode and face mode respectively.

(2) Aiming at the fatigue state identification problem of miners, a brain power feature matrix that can represent the fatigue state information and contains the potential lead information features is constructed. The brain power feature matrix uses the power spectrum features of EEG signals in different bands to construct a new feature index. It maps into a feature matrix according to the electrode position. The brain power feature matrix is trained and learned in different classification and recognition models, and it is concluded that the recognition accuracy of CapsNet capsule network for fatigue state is 93.54%, which is 8.98% and 5.68% higher than the other two convolution recognition models respectively.

(3) Aiming at the quantitative assessment of mental state, a miners' mental state assessment algorithm based on multi-source information fusion is proposed. The evaluation algorithm is used to quantify and integrate the subjective and objective mental state information to obtain the final evaluation value of mental state. By comparing with the set mental state evaluation threshold, it is concluded whether the current mental state of miners is suitable for underground work. Only 1 of the 12 subjects had deviations in the assessment results, indicating that the mental state assessment method has certain feasibility.

The multi-source information fusion method for evaluating the mental state of miners proposed in this paper can effectively integrate the facial information and EEG information of the miners and make a more accurate and intuitive state evaluation of the tested miners' mental state. The task of mental state discrimination. It has a certain reference value for the underground work of miners and the safe production of coal mines.

[1] 郑翠,张玉峰,龙陆军.新时期我国主体能源消费结构调整优化研究[J].中国煤炭地质,2021,33(S1):49-51.

[2] 程莉,李争艳.基于Kaya-LMDI的中国煤炭消费需求影响因素分析[J].煤炭经济研究,2021,41(06):4-9.

[3] 聂聃, 王晓韡, 段若男, 等. 基于脑电的情绪识别研究综述[J]. 中国生物医学工程学报, 2012, 31(4): 595-606.

[4] 王冠男. 基于多特征的驾驶员疲劳驾驶识别[D]. 山西:太原科技大学, 2021.

[5] Nejati V, Khorrami A S, Fonoudi M. Neuromodulation of facial emotion recognition in health and disease: A systematic review[J]. Neurophysiologie Clinique, 2022.

[6] Alarcao S M, Fonseca M J. Emotions recognition using EEG signals: A survey[J]. IEEE Transactions on Affective Computing, 2017, 10(3): 374-393.

[7] 潘家辉, 何志鹏, 李自娜, 等. 多模态情绪识别研究综述[J]. 智能系统学报, 2020, 15(4): 633-645.

[8] Wang X, Yu C, Gu Y, et al. Multi‐Task and Attention Collaborative Network for Facial Emotion Recognition[J]. IEEJ Transactions on Electrical and Electronic Engineering, 2021, 16(4): 568-576. 8.

[9] 季欣欣, 邵洁, 钱勇生. 基于注意力机制和混合网络的小群体情绪识别[J]. 计算机工程与设计, 2020, 41(6): 1683.

[10] Li D, Liu J, Yang Z, et al. Speech emotion recognition using recurrent neural networks with directional self-attention[J]. Expert Systems with Applications, 2021, 173: 114683.

[11] Do L N, Yang H J, Nguyen H D, et al. Deep neural network-based fusion model for emotion recognition using visual data[J]. The Journal of Supercomputing, 2021, 77(10): 10773-10790.

[12] 蔡梓良, 郭苗苗, 杨新生,陈昕彤,徐桂芝.基于最大分类器差异域对抗方法的跨被试脑电情绪识别研究[J].生物医学工程学杂志,2021,38(03):455-462.

[13] Nie D, Wang X W, Shi L C, et al. EEG-based emotion recognition during watching movies[C]//2011 5th International IEEE/EMBS Conference on Neural Engineering. IEEE, 2011: 667-670.

[14] 王成龙, 韦巍, 李天永. 基于IMF能量矩的脑电情绪特征提取研究[J]. 现代电子技术, 2018, 41(20):4.

[15] Alsolamy M, Fattouh A. Emotion estimation from EEG signals during listening to Quran using PSD features[C]//2016 7th International Conference on Computer Science and Information Technology (CSIT). IEEE, 2016: 1-5.

[16] Duan R N, Zhu J Y, Lu B L. Differential entropy feature for EEG-based emotion classification[C]//2013 6th International IEEE/EMBS Conference on Neural Engineering (NER). IEEE, 2013: 81-84.

[17] 罗映雪, 贾博, 裘旭益, 等. 基于 Gamma 深度信念网络的飞行员脑疲劳状态识别[J]. 电子学报, 2020, 48(6): 1062.

[18] Knapik M, Cyganek B. Driver’s fatigue recognition based on yawn detection in thermal images[J]. Neurocomputing, 2019, 338: 274-292.

[19] 闫河, 杨晓龙, 张杨, 等. 基于 ASM 的驾驶员面部疲劳状态识别方法[J]. 计算机工程与设计, 2018, 39(10): 3240-3245.

[20] Yin Z, Zhang J. Task-generic mental fatigue recognition based on neurophysiological signals and dynamical deep extreme learning machine[J]. Neurocomputing, 2018, 283: 266-281.

[21] 郭孜政, 牛琳博, 吴志敏, 等. 基于 EEG 的驾驶疲劳识别算法及其有效性验证[J]. 北京工业大学学报, 2017, 43(6): 929-934.

[22] Wei L, Feng T, Zhao P, et al. Driver sleepiness detection algorithm based on relevance vector machine[J]. The Baltic Journal of Road and Bridge Engineering, 2021, 16(1): 118-139.

[23] 刘红梅. 通过静息态功能磁共振成像探讨解郁丸对WKY大鼠海马及前额叶皮层的脑功能活动的影响[D]. 山西: 山西医科大学,2020.

[24] 丁尚文, 钱志余, 李韪幍, 等. 低频光诱发脑电θ波段近似熵特征研究[J].激光与光电子学进展,2012,49(03):125-129.

[25] Alarcao S M, Fonseca M J. Emotions recognition using EEG signals: A survey[J]. IEEE Transactions on Affective Computing, 2017, 10(3): 374-393.

[26] 吕宝粮,张亚倩,郑伟龙.情感脑机接口研究综述[J].智能科学与技术学报,2021,3(01):36-48.

[27] Bajaj V, Rai K, Kumar A, et al. Rhythm-based features for classification of focal and non-focal EEG signals[J]. IET Signal Processing, 2017, 11(6): 743-748.

[28] 蔡桂元,史宇,吴文.自发性知觉经络反应的脑电频谱特征研究[J].中华神经医学杂志,2020,19(06):582-585.

[29] 王翔坤. 基于情绪诱发的头皮脑电信号研究[D]. 杭州:杭州电子科技大学,2021.

[30] 贾磊, 郑峥, 王沛, 等. 情绪诱发视盲任务中的情绪注意控制机制: 来自脑电 ERP 和时频分析的证据[J]. 心理科学, 2021 (4): 770.

[31] Cosottini M, Pesaresi I, Maritato P, et al. EEG topography-specific BOLD changes: a continuous EEG-fMRI study in a patient with focal epilepsy[J]. Magnetic Resonance Imaging, 2010, 28(3): 388-393.

[32] 黄飞, 尤启房, 杨晋吉. ASM 的手骨提取方法研究[J]. 计算机工程与应用, 2016, 3.

[33] Placidi G, Cinque L, Polsinelli M. A fast and scalable framework for automated artifact recognition from EEG signals represented in scalp topographies of Independent Components[J]. Computers in Biology and Medicine, 2021, 132: 104347.

[34] 唐成龙,谌颃,唐海春,吴泽锋.大数据背景下数据预处理方法研究运用[J].信息记录材料,2021,22(09):199-200.

[35] 刘志成, 王殿伟, 刘颖, 等. 基于二维伽马函数的光照不均匀图像自适应校正算法[J]. 北京理工大学学报, 2016, 36(2): 191-196.

[36] 黄飞, 尤启房, 杨晋吉. ASM 的手骨提取方法研究[J]. 计算机工程与应用, 2016, 3.

[37] Cheng M M, Mitra N J, Huang X, et al. Global contrast based salient region detection[J]. IEEE transactions on pattern analysis and machine intelligence, 2014, 37(3): 569-582.

[38] 李东民, 李静, 梁大川, 等. 基于多尺度先验深度特征的多目标显著性检测方法[J]. 自动化学报, 2019, 45(11): 2058-2070.

[39] 任银芝. 脑卒中的脑电信号特征提取研究[D]. 杭州: 杭州电子科技大学,2016.

[40] 张克军, 韩娜, 陈欣然, 等. 基于脑电图的新型情绪识别特征提取方法[J]. 华南师范大学学报 (自然科学版), 2019, 51(5): 6-11.

[41] Topic A, Russo M. Emotion recognition based on EEG feature maps through deep learning network[J]. Engineering Science and Technology, an International Journal, 2021, 24(6): 1442-1454.

[42] 熊珞琳, 毛帅, 唐漾, 等. 基于强化学习的综合能源系统管理综述[J]. 自动化学报, 2021, 47(10): 2321-2340.

[43] 赵旭, 黄光球, 江晋, 等. 基于深度强化学习的资源受限条件下的 DIDS 任务调度优化方法[J]. 控制与决策, 2021.

[44] Khaleghi B, Khamis A, Karray F O, et al. Multisensor data fusion: A review of the state-of-the-art[J]. Information fusion, 2013, 14(1): 28-447.

[45] 王海玉, 王映龙, 闵建亮, 等. 基于独立成分分析降噪与集成分类器的疲劳脑电分析[J]. 科学技术与工程, 2018, 32.

[46] Min J, Xiong C, Zhang Y, et al. Driver fatigue detection based on prefrontal EEG using multi-entropy measures and hybrid model[J]. Biomedical Signal Processing and Control, 2021, 69: 102857.

[47] 李昕, 李红红, 李长吾. 基于复杂度熵特征融合的高压力人群情感状态评估[J]. 中国生物医学工程学报, 2013, 32(3): 313-320.

[48] Martínez-Pérez L, Viso E, Touriño R, et al. Clinical evaluation of meibomian gland dysfunction in patients with keratoconus[J]. Contact Lens and Anterior Eye, 2021: 101495.

[49] 肖毅, 陈善广, 韩东旭, 等. 脑电信号多重粗粒化复杂度分析方法研究[J]. 电子科技大学学报, 2013, 42(3): 469-474.

[50] 周天一. 个体化脑电功率谱和脑网络分析方法的研究与应用[D]. 河北:燕山大学,2020.

[51] 陈芷若. 脑电功率值数据图, 脑电功率谱 (频谱) 曲线图和脑电功率谱 (频谱) 直方图和压缩谱阵图[J]. 现代电生理学杂志, 2013, 20(2): 118-124.

[52] 张洁,庞丽萍,完颜笑如,陈浩,王鑫,梁晋.基于脑电功率谱密度的作业人员脑力负荷评估方法[J].航空学报,2020,41(10):118-125.

[53] Li H, Wang D, Chen J, et al. Pre-service fatigue screening for construction workers through wearable EEG-based signal spectral analysis[J]. Automation in Construction, 2019, 106: 102851.

[54] 吕元杰. 脑波矿用头盔多源传感研究[D]. 西安:西安科技大学, 2017.

[55] Bismuth J, Vialatte F, Lefaucheur J P. Relieving peripheral neuropathic pain by increasing the power-ratio of low-β over high-β activities in the central cortical region with EEG-based neurofeedback: Study protocol for a controlled pilot trial (SMRPain study)[J]. Neurophysiologie Clinique, 2020, 50(1): 5-20.

[56] 王利, 艾玲梅, 王四万, 等. 驾驶疲劳脑电信号节律的特征分析[J]. 生物医学工程学杂志, 2012, 29(4): 629-633.

[57] 霍雪芹. 基于脑电和前额眼电融合的疲劳驾驶检测研究[D].上海:上海交通大学, 2017.

[58] Chen X, Lin J, Jin H, et al. The psychoacoustics annoyance research based on EEG rhythms for passengers in high-speed railway[J]. Applied Acoustics, 2021, 171: 107575.

[59] 储银雪. 基于脑电功率图谱的飞行员疲劳状态识别[D]. 上海:上海交通大学, 2019.

[60] Zhu J, Shen Z, Ni T. Multi-Frequent Band Collaborative EEG Emotion Classification Method Based on Optimal Projection and Shared Dictionary Learning[J]. Frontiers in Aging Neuroscience, 2022, 14.

[61] 罗丹,马军生.关于视觉特征与CapsNet的图像大数据分类研究[J].计算机仿真,2022,39(01):181-185.

[62] 王晓康. 基于脑机接口的脑波控制变量算法及轨道小车控制系统[D]. 西安:西安科技大学, 2016.