煤炭产业中，矿井下安全作业一直是矿山企业安全管理的核心内容。然而，由于人员异常行为所导致的矿山事故时有发生。近年来，随着人工智能技术不断发展，利用计算机视觉技术来识别矿井下人员异常行为是减少矿山事故的关键手段之一。由于矿井下生产环境存在低光照导致人员特征不明显，现有方法尚不能完全解决人员异常行为识别。基于此，本文从多模态融合的角度出发对矿井下人员异常行为开展研究。

具体内容如下：

1. 针对矿井下存在低光照问题，提出基于低光照增强的红外和可见光图像融合模型。设计低光照增强网络和纹理 –对比度增强融合网络，生成具有高对比度和纹理信息的融合图像；并设计分解损失函数和融合增强损失函数，减少融合后图像出现颜色失真等现象。所提出的模型在 LLVIP 公开数据集上进行性能评估和对比，在多项评估指标中均验证了模型优秀的融合性能，并且将所融合后的图像在不同检测网络中进行性能对比，解决了低光照存在的特征不明显所导致的错检、漏检等问题，且识别精度得 到显著提升。

2. 针对人员行为识别中单一模态特征表征不足的问题，利用多模态相互补充的优势，提出基于时空动态骨骼的 YOLOE 人员行为识别模型。首先利用 Openpose 提取人体骨骼模态数据，然后通过时空图卷积网络获取时空骨架序列图，实现 RGB 和时空骨骼模态融合，最后通过 YOLOE 检测网络实现人员行为识别。在各个模型设计的基础上，提出基于红外和可见光图像融合的井下人员异常行为识别模型识别矿井下人员异常行为。所提模型在 UCF101、HMDB51 等公开数据集上进行消融实验和对比实验，验证了多模态融合行为识别的有效性。最后在自建双模态异常行为数据集验证基于红外和可见光图像融合的井下人员异常行为识别模型的性能，实验结果表明，与单模态数据相比，本文提出的多模态融合模型在识别准确度上具有显著的性能优势。

基于红外和可见光图像融合的井下人员异常行为识别方法在公开数据集和自建双模态异常行为数据集分别取得良好的识别效果，为矿井下人员安全生产提供理论基础 和实验数据。

In the coal industry, underground safety operations have always been the core content of safety management for mining enterprises. However, mining accidents often occur due to abnormal human behavior. In recent years, with the continuous development of artificial intelligence technology, using computer vision technology to identify abnormal behavior of underground personnel is one of the key means to reduce mining accidents. Due to the low lighting conditions in the underground production environment, personnel characteristics are not clear,

and existing methods cannot fully solve the problem of identifying abnormal behavior of personnel. Based on this, this article conducts research on abnormal behavior of underground personnel from the perspective of multi-modal fusion. The specific content is as follows:

1. A fusion model for infrared and visible light images based on low light enhancement is proposed to address the issue of low lighting in underground mines. Design low light enhancement networks and texture contrast enhancement fusion networks to generate fused images with high contrast and texture information; And design decomposition loss function and fusion enhancement loss function to reduce color distortion and other phenomena in the fused image.

The proposed model was evaluated and compared on the LLVIP public dataset, and its excellent fusion performance was verified in multiple evaluation indicators. The fused images were compared in different detection networks to solve the problems of false positives and missed detections caused by unclear features in low lighting conditions, and the recognition accuracy was significantly improved.

2. In order to solve the problem of insufficient representation of single modal features in human behavior recognition, a YOLOE human behavior recognition model based on spatiotemporal dynamic skeleton is proposed by using the advantages of multi-modal complementation. Firstly, Openpose is used to extract human skeletal modal data. Then, a spatiotemporal graph convolutional network is used to obtain the spatiotemporal skeletal sequence diagram, achieving the fusion of RGB and spatiotemporal skeletal modalities. Finally, the YOLOE detection network is used to achieve human behavior recognition. On the basis of various model designs, a recognition model for abnormal behavior of underground personnel based on infrared andvisible light image fusion is proposed to identify abnormal behavior of underground personnel. The proposed model was subjected to ablation and comparative experiments on publicly available datasets such as UCF101 and HMDB51, verifying the effectiveness of multi-modal fusion behavior recognition. Finally, the performance of the underground personnel abnormal behavior recognition model based on infrared and visible light image fusion was validated on a self built dual-mode abnormal behavior dataset. The experimental results showed that the proposed multimodal fusion model had significant performance advantages in recognition accuracy compared to single-mode data.

The recognition method for abnormal behavior of underground personnel based on infrared and visible light image fusion has achieved good recognition results on public datasets and self built bimodal abnormal behavior datasets, providing theoretical basis and experimental data for the safety production of underground personnel.