煤矿辅助运输是煤矿生产系统的重要环节，运输状况直接关系到煤矿企业的安全和发展。在以胶轮车司机为主体的煤矿辅助运输系统中，胶轮车司机的风险行为一定程度上增大煤矿辅助运输事故发生的可能性，因此胶轮车司机风险行为的感知是安全监管的重中之重。本文基于深度学习的图像检测和人声检测技术，构建多模态胶轮车司机风险行为感知模型，以实现对胶轮车司机风险行为的感知，辅助煤矿安全管理人员决策，从而更好地预防和减少煤矿人因事故的发生，提升煤矿企业综合安全管理水平。本文主要研究内容及工作如下：

（1）通过对现有驾驶员行为检测技术的分析，明确深度学习的图像检测算法对胶轮车司机风险行为感知的合理性。运用文本挖掘对270份驾驶行为相关文献进行挖掘，结合胶轮车车辆配置与辅助运输环境特点，明确胶轮车司机常见风险行为类型；针对实际应用场景中光线不足导致图像清晰度较差等问题，首先对自建亮度数据集的图像进行亮度检测，采用Zero-DCE算法对亮度较差的数据进行图像增强；然后采用轻量级网络MobileNetV3_Large构建胶轮车司机风险行为感知模型，完成其在胶轮车司机风险行为自建数据集与State Farm的合并数据集的训练。测试结果表明，胶轮车司机风险行为感知的准确率达81.86%，相似度较高的动作感知准确率较低。

（2）通过比对多种语音处理技术的适用场景与特征，选取计算速度突出、独立性强的MFCC作为语音特征提取工具；分析胶轮车司机所处环境的语音特性和噪声特性，选取部分AISHELL-2和Noise-92作为混合音频数据集，借助Python语言和Keras框架，对混合音频数据集进行数量调整、长度切割、特征提取和VAD判决等操作，完成17320条音频数据的人声检测仿真实验。实验结果表明：人声检测准确率达93.83%，单条音频运行时间为8.97ms。

（3）针对相似行为在图像呈现上难以准确区分的问题，在图像相似度匹配、音频数据预处理的基础上，构建了基于深度学习的多模态胶轮车司机风险行为感知模型，通过人声检测将驾驶行为分为含人声情况的驾驶行为（IV）和无人声情况下的驾驶行为（EV）两类，再利用图像检测算法实现胶轮车司机驾驶行为的精确分类。仿真实验结果表明：多模态胶轮车司机风险行为感知模型准确度达94.57%，相较于单一模态图像检测和人声检测分别提高12.71%、0.74%。此外，综合分析矿井安全管理实际需求，初步设计了一套胶轮车司机风险行为感知与预警系统，该系统主要由图像处理模块、人声检测模块、风险行为感知与预警模块以及驾驶数据分析模块四部分构成，并依托PyQt5和Qt Designer软件完成各个窗体及界面的设计。

Coal mine auxiliary transportation is an essential link in the coal mine production system, and the transportation status is directly related to the safety and development of coal mining enterprises. In the coal mine auxiliary transportation system with rubber-tired vehicle drivers as the main body, the risk behavior of rubber-tired vehicle drivers increases the likelihood of coal mine auxiliary transportation accidents to a certain extent. Therefore, the perception of risk behavior of rubber-tired vehicle drivers is of utmost importance in safety supervision. This article is based on deep learning image detection and human voice detection technology to construct a multimodal rubber-tired vehicle drivers risk behavior perception model to achieve the perception of rubber-tired vehicle drivers' risk behavior, assist coal mine safety management personnel in decision-making, and better prevent and reduce the occurrence of human accidents in coal mines, and improve the comprehensive safety management level of coal mining enterprises. The main research content and work of this article are as follows:

(1) By analyzing existing driver behavior detection technologies, clarify the rationality of deep learning image detection algorithms in perceiving the risk behavior of rubber-tired vehicle drivers. Using text mining to mine 270 literature related to driving behavior, combined with the characteristics of rubber-tired vehicle configuration and auxiliary transportation environment, identify common risk behavior types of rubber-tired vehicle drivers; To address the issue of poor image clarity caused by insufficient lighting in practical application scenarios, firstly, brightness detection is performed on the images from the self-built brightness dataset, and the Zero-DCE algorithm is used to enhance the images of the data with poor brightness; Then, a lightweight network MobileNetV3_Large was used to construct a risk behavior perception model for rubber-tired vehicle drivers, and its training was completed on the self-built dataset of rubber-tired vehicle drivers' risk behavior and the merged dataset of State Farm. The test results show that the accuracy of risk behavior perception by rubber-tired vehicle drivers is 81.86%, while the accuracy of action perception with high similarity is lower.

(2) By comparing the applicable scenarios and features of various speech processing 

technologies, MFCC, with outstanding computational speed and strong independence, is selected as the speech feature extraction tool. Analyze the speech and noise characteristics of the environment where rubber-tired vehicle drivers are located, select some AISHELL-2 and Noise-92 as mixed audio datasets, and use Python language and Keras framework to perform operations such as quantity adjustment, length cutting, feature extraction, and VAD judgment on the mixed audio datasets. Complete human voice detection simulation experiments on 17320 audio data. The experimental results show that the accuracy of human voice detection is 93.83%, and the running time of single audio is 8.97ms.

(3) To address the difficulty in accurately distinguishing similar behaviors in image presentation, a multimodal rubber-tired vehicle drivers risk behavior perception model based on deep learning was constructed based on image similarity matching and audio data preprocessing. Through human voice detection, driving behavior was divided into two categories: including voice information (IV) and excluding voice information (EV), utilizing image detection algorithms to accurately classify the driving behavior of rubber-tired vehicle drivers. The simulation results show that the accuracy of the multimodal perception model of rubber-tired vehicle drivers' risk behavior is 94.57%, which is 12.71% and 0.74% higher than that of the single modal image detection and voice detection respectively. In addition, based on a comprehensive analysis of the actual needs of mine safety management, a preliminary design of a risk behavior perception and warning system for rubber-tired vehicle drivers has been carried out. The system is mainly composed of four parts: image processing module, voice detection module, risk behavior perception and warning module, and driving data analysis module. The design of various windows and interfaces is completed using PyQt5 and Qt Designer software.