煤矿综采工作面异常状况的视觉识别技术是近年来国内研究的热点。煤矿井下环境的煤尘和水雾使视频图像尘雾现象严重，导致采集到的图片质量差，难以满足井下异常状态监测的需求，因此，综采工作面的图像去雾和视觉异常检测方法研究具有重要意义和工程价值。本文结合图像去雾、目标检测技术、目标跟踪技术、动作识别技术，建立视觉异常检测模型，研究综采工作面异常状况的视觉检测方法，并将图像去雾与视觉异常检测模型移植到嵌入式平台上，实现液压支架护帮板、拖缆、大块煤、人员的异常检测。
  针对综采工作面视频图像尘雾现象严重，采集到的图片质量差，影响视觉异常检测准确率的问题，论文采用暗通道先验算法，并对高亮区域透射率进行优化解决煤矿图像含有高亮区域去雾失真的问题。针对图像去雾实时性差的问题，提出将改进暗通道先验算法中计算占比较大的最小值滤波、均值滤波、直方图统计进行并行加速处理，对列滤波进行转置优化，在GPU与嵌入式平台上实验表明可以满足实时去雾需求且去雾效果良好。
  基于深度学习的目标检测参数量和计算量大，存在计算资源和内存消耗大，难以在嵌入式平台上部署的问题，论文对YOLOv5s模型进行轻量化结合注意力机制改进，减少模型参数，提高推理速度；将YOLOv5s中原有的特征金字塔网络改进为AF-FPN，提高煤矿综采工作面多尺度目标的检测性能；用 -CIoU改进CIoU损失函数，提高煤矿综采目标的检测精度。借助公开数据集和自制的综采工作面异常数据集，验证改进算法，实验对比表明算法的有效性。
  针对液压支架护帮板、人员和拖缆的异常检测，基于改进YOLOv5s模型实现液压支架护帮板、人员和拖缆目标检测，获得标签与定位信息，采用标签组合分类实现液压支架护帮板的异常状态识别报警；通过判断人员定位坐标是否在危险区域，实现人员入侵的识别报警；通过判断拖缆的定位坐标是否在安全区域，实现拖缆脱轨槽异常的识别报警。借助综采工作面异常数据集，测试验证了上述方法的有效性。
  针对大块煤的滞留、堵塞的异常行为识别，提出一种基于改进YOLOv5s模型的DeepSORT的大块煤多目标跟踪算法，对连续追踪的大块煤目标连续50帧的最大距离进行计算，设置距离阈值判断实现大块煤行为异常识别，在异常数据集上实验表明，可以准确识别大块煤的滞留和堵塞状态。
  针对井下人员的不规范动作，构建Person_Action2021行为异常识别数据集，通过OpenPose提取人体骨架，构造成骨架时空图的形式，然后在构造的时空图上来进行时空图卷积来完成动作识别，实现人员行为异常识别。实验表明，该方法可以准确识别异常行为。
  搭建图像去雾和视觉异常检测嵌入式实验平台，对采集的煤矿井下综采工作面视频图像去雾预处理，并对液压支架护帮板异常检测、人员入侵检测、拖缆脱槽检测、大块煤行为异常识别、人员行为异常识别和复合异常检测算法进行实验验证，结果表明本文提出的图像去雾与视觉异常检测，初步可以满足实时性要求，准确率高，可以实现自动巡检需求，对煤矿综采工作面自动化监测和决策具有一定的借鉴意义。
 

  The visual recognition technology of abnormal conditions in fully mechanized coal mining face is a hot research topic in China in recent years. Coal dust and water mist in the coal mine underground environment make the video image fog phenomenon serious, resulting in poor quality of collected images, which is difficult to meet the needs of underground abnormal state monitoring. Therefore, the research on image defogging and visual anomaly detection method of fully mechanized mining face is of great significance and engineering value. In this paper, combined with image defogging, target detection technology, target tracking technology, and action recognition technology, the visual anomaly detection model was established, and the visual detection method of the abnormal condition of the fully mechanized mining face was studied. The image defogging and visual anomaly detection model was transplanted to the embedded platform to realize the abnormal detection of hydraulic support panel, towing cable, large coal, and personnel.
  The dust fog phenomenon of video in fully mechanized mining faces is serious, and the quality of collected images is poor, which affects the accuracy of visual anomaly detection. In this paper, based on the dark channel prior algorithm, the transmittance of the bright region was optimized to solve the problem of defogging distortion in coal mine images containing bright regions. An improved dark channel prior algorithm was proposed to solve the problem of the poor real-time performance of image defogging by parallel acceleration processing of minimum filtering, mean filtering and histogram statistics, and transposition optimization of column filtering. Experiments on GPU and embedded platform showed that it can meet the needs of real-time defogging and has good defogging effect.
  Target detection based on deep learning has a large number of parameters and computations, and there is a large consumption of computing resources and memory, which is difficult to deploy on embedded platforms. The YOLOv5s model was optimized by lightweight combined with an attention mechanism to reduce model parameters and improve reasoning speed. AF-FPN was used to optimize the original feature pyramid network of YOLOv5s to improve the detection performance of multi-scale targets in a fully mechanized coal mining face.                      -CIoU was used to optimize CIoU loss function to improve the detection accuracy of fully mechanized coal mining targets. The improved algorithm was verified by using the public data set and the self-made abnormal data set of a fully mechanized working face. The experimental comparison showed the effectiveness of the algorithm.
  Abnormal detection of hydraulic support face guard, personnel, and towing cables is a problem. The improved YOLOv5s were used to detect the target of the hydraulic support face guard, personnel, and towing cable to obtain the label and positioning information. The abnormal recognition alarm of the hydraulic support face guard was realized by label combination classification. Judge whether the personnel positioning coordinates in the dangerous area to achieve abnormal identification of personnel intrusion alarm. Determine whether the positioning coordinates of the towed cable can realize the abnormal identification and alarm of the towed cable from the track in the safe area. The effectiveness of the above method was verified by the test of the abnormal data set of a fully mechanized working face.
Identification of abnormal behavior of coal retention and blockage is a problem. This paper presented a multi-target tracking algorithm for bulk coal based on DeepSORT with improved YOLOv5s. Within 50 consecutive frames, the maximum distance of continuously tracked large coal was calculated. The distance threshold was set to realize the abnormal behavior recognition of large coal. Experiments on abnormal data sets showed that the retention and blockage state of bulk coal can be accurately identified.
Abnormal detection of irregular movement of underground personnel is a problem. This paper constructed Person_Action2021 behavioral anomaly recognition dataset. The human skeleton was extracted by OpenPose, and the skeleton space-time map was constructed. The skeleton space-time map was sent to the space-time map convolution to complete the action recognition and realize the abnormal behavior recognition. Experiments showed that this method can accurately identify abnormal behavior.
This paper built an embedded experimental platform for image dehazing and visual anomaly detection. The collected video of fully mechanized coal mining face in underground coal mines was preprocessed by image defogging, and the experimental verification was carried out on the abnormal detection of hydraulic support side guard plate, personnel intrusion detection, cable departure track detection, abnormal recognition of large coal behavior, abnormal recognition of personnel behavior and composite anomaly detection algorithm. The results showed that the proposed image defogging and visual anomaly detection can preliminarily meet the real-time requirements with high accuracy. It can realize the demand for automatic inspection and has certain reference significance for automatic monitoring and decision-making of fully mechanized coal mining faces.
 

[1] 边文越, 陈挺, 陈晓怡, 等. 世界主要发达国家能源政策研究与启示[J]. 中国科学院院刊, 2019, 34(04): 488-496.

[2] 景娇燕. 铁路煤炭运输低碳环保战略及措施分析[J]. 中国石油和化工标准与质量, 2020, 40(12): 133-134.

[3] 王国法, 刘峰, 孟祥军, 等. 煤矿智能化(初级阶段)研究与实践[J]. 煤炭科学技术, 2019, 47(08): 1-36.

[4] 张士海. 矿用巡检机器人在煤矿中的应用[J]. 煤矿机电, 2016(05): 76-79.

[5] 周静龙. 巡检机器人在煤矿的应用分析[J]. 煤, 2020, 29(05): 53+85.

[6] 徐桂涛. 煤矿视频监控系统中人员目标跟踪算法的研究与实现[D]. 武汉: 中国矿业大学, 2019.

[7] 张静. 基于视频的煤矿井下人形目标检测[D]. 西安: 西安科技大学, 2019.

[8] 王媛媛. 煤矿井下无线人员定位视频监控系统分析与设计[D]. 南京: 东南大学, 2019.

[9] 何宁, 王金宝, 鲍泓. 单幅图像去雾方法研究综述[J]. 北京联合大学学报(自然科学版), 2015, 29(03): 24-31.

[10] 蔡超峰. 局部直方图均衡化算法研究及其应用[D]. 郑州: 郑州大学, 2005.

[11] Land E H. The retinex theory of color vision[J]. Scientific american, 1977, 237(6): 108-129.

[12] Russo F. An image enhancement technique combining sharpening and noise reduction[J]. IEEE Transactions on Instrumentation and Measurement, 2002, 51(4): 824-828.

[13] Narasimhan S G, Nayar S K. Vision and the atmosphere[J]. International journal of computer vision, 2002, 48(3): 233-254.

[14] Narasimhan S G, Nayar S K. Contrast restoration of weather degraded images[J]. IEEE transactions on pattern analysis and machine intelligence, 2003, 25(6): 713-724.

[15] Schechner Y Y, Narasimhan S G, Nayar S K. Instant dehazing of images using polarization[C]//Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001. IEEE, 2001, 1: I-I.

[16] Tan R T. Visibility in bad weather from a single image[C]//2008 IEEE conference on computer vision and pattern recognition. IEEE, 2008: 1-8.

[17] Fattal R. Single image dehazing[J]. ACM transactions on graphics (TOG), 2008, 27(3): 1-9.

[18] He K, Sun J, Tang X. Single image haze removal using dark channel prior[J]. IEEE transactions on pattern analysis and machine intelligence, 2010, 33(12): 2341-2353.

[19] 蒋建国, 侯天峰, 齐美彬. 改进的基于暗原色先验的图像去雾算法[J]. 电路与系统学报, 2011, 16(02): 7-12.

[20] 王启明. 矿井高清图像去雾算法研究[D]. 阜新: 辽宁工程技术大学, 2018.

[21] 陈长华, 刘煜. 基于两次双边滤波的煤矿监控图像去雾研究[J]. 图学学报, 2017, 38(03): 418-424.

[22] Wu D, Zhang S. Research on image enhancement algorithm of coal mine dust[C]//2018 International Conference on Sensor Networks and Signal Processing (SNSP). IEEE, 2018: 261-265.

[23] 田子建, 王满利, 吴君, 等. 基于双域分解的矿井下图像增强算法[J]. 光子学报, 2019, 48(05): 107-119.

[24] 陆斌, 严利民, 陈志恒. 硬件优化的高清视频实时去雾算法[J]. 计算机应用研究, 2020, 37(12): 3807-3810.

[25] Lv X, Chen W, Shen I. Real-time dehazing for image and video[C]//2010 18th Pacific Conference on Computer Graphics and Applications. IEEE, 2010: 62-69.

[26] 陈志恒. 图像与视频去雾算法的研究及硬件实现[D]. 上海: 上海大学, 2020.

[27] 农健, 陈伟业. 视频监控系统异常目标检测与定位综述[J]. 电子技术与软件工程, 2022(02): 141-144.

[28] Horn B K P, Schunck B G. Determining optical flow[J]. Artificial intelligence, 1981, 17(1-3): 185-203.

[29] McIvor A M. Background subtraction techniques[J]. Proc. of Image and Vision Computing, 2000, 4: 3099-3104.

[30] Bradski G R. Computer vision face tracking for use in a perceptual user interface[J]. 1998.

[31] Reddy V, Sanderson C, Lovell B C. Improved anomaly detection in crowded scenes via cell-based analysis of foreground speed, size and texture[C]//CVPR 2011 WORKSHOPS. IEEE, 2011: 55-61.

[32] Tung F, Zelek J S, Clausi D A. Goal-based trajectory analysis for unusual behaviour detection in intelligent surveillance[J]. Image and Vision Computing, 2011, 29(4): 230-240.

[33] Benezeth Y, Jodoin P M, Saligrama V, et al. Abnormal events detection based on spatio-temporal co-occurences[C]//2009 IEEE conference on computer vision and pattern recognition. IEEE, 2009: 2458-2465.

[34] 范伟. 基于机器视觉的汽车车身零部件表面划痕检测的研究[D]. 南京: 南京理工大学, 2016.

[35] 李颀, 王康, 强华, 马琳. 基于颜色和纹理特征的异常玉米种穗分类识别方法[J]. 江苏农业学报, 2020, 36(01): 24-31.

[36] Sun J, Li C, Wu X J, et al. An effective method of weld defect detection and classification based on machine vision[J]. IEEE Transactions on Industrial Informatics, 2019, 15(12): 6322-6333.

[37] 张霞, 贺正然. 基于灰度与关键帧光流检测的视频异常判断[J]. 电子器件, 2019, 42(03): 718-721.

[38] 李清瑶, 邹皓, 赵群, 等. 基于帧间差分自适应法的车辆抛洒物检测[J]. 长春理工大学学报(自然科学版), 2018, 41(04): 108-113.

[39] Girshick R, Donahue J, Darrell T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2014: 580-587.

[40] Girshick R. Fast r-cnn[C]//Proceedings of the IEEE international conference on computer vision. 2015: 1440-1448.`

[41] Ren S, He K, Girshick R, et al. Faster r-cnn: Towards real-time object detection with region proposal networks[J]. Advances in neural information processing systems, 2015, 28.

[42] Redmon J, Farhadi A. Yolov3: An incremental improvement[J]. arXiv preprint arXiv:1804.02767, 2018.

[43] Bochkovskiy A, Wang C Y, Liao H Y M. Yolov4: Optimal speed and accuracy of object detection[J]. arXiv preprint arXiv:2004.10934, 2020.

[44] Ultralytics. "YOLOv5," https://github.com/ultralytics/yolov5.

[45] Jeong J, Park H, Kwak N. Enhancement of SSD by concatenating feature maps for object detection[J]. arXiv preprint arXiv:1705.09587, 2017.

[46] Xie T, Li X, Zhang X, et al. Detection of Atlantic salmon bone residues using machine vision technology[J]. Food Control, 2021, 123: 107787.

[47] 陈农田, 满永政, 宁威峰, 等. 基于深度学习的飞行员驾驶异常行为监测方法[J]. 安全与环境学报, 2022, 22(01): 249-255.

[48] 周得永, 高龙琴. 基于YOLOv3的太阳能电池板缺陷检测[J]. 南方农机, 2022, 53(04): 142-144.

[49] 韩佳彤, 张宏娜, 李召波, 等. 基于YOLOV5网络模型的市政道路检测识别[J]. 内蒙古大学学报(自然科学版), 2021, 52(05): 514-519.

[50] 张全, 张伟, 杨先凤, 等. 融合YOLOv5-ResNet级联网络的烟火检测方法[J/OL]. 安全与环境学报: 1-10[2022-03-15]. https://kns.cnki.net/kcms/detail/11.4537.X.20211218.0924.001.html.

[51] 李一鸣, 王潇. 基于YOLOv5s模型的轧钢表面缺陷检测[J]. 制造业自动化, 2021, 43(11): 117-119.

[52] 单鹏飞, 孙浩强, 来兴平, 等. 基于改进Faster R-CNN的综放煤矸混合放出状态识别方法[J/OL]. 煤炭学报: 1-13[2022-04-10]. https://kns.cnki.net/kcms/detail/11.2190.TD.20220222.1703.005.html.

[53] 郝帅, 张旭, 马旭, 等. 基于CBAM-YOLOv5的煤矿输送带异物检测[J/OL]. 煤炭学报: 1-11[2022-04-10]. https://kns.cnki.net/kcms/detail/11.2190.td.20220217.1713.004.html.

[54] 叶鸥, 窦晓熠, 付燕等. 融合轻量级网络和双重注意力机制的煤块检测方法[J]. 工矿自动化, 2021, 47(12): 75-80.

[55] Bolme D S, Beveridge J R, Draper B A, et al. Visual object tracking using adaptive correlation filters[C]//2010 IEEE computer society conference on computer vision and pattern recognition. IEEE, 2010: 2544-2550.

[56] Henriques J F, Caseiro R, Martins P, et al. Exploiting the circulant structure of tracking-by-detection with kernels[C]//European conference on computer vision. Springer, Berlin, Heidelberg, 2012: 702-715.

[57] Henriques J F, Caseiro R, Martins P, et al. High-speed tracking with kernelized correlation filters[J]. IEEE transactions on pattern analysis and machine intelligence, 2014, 37(3): 583-596.

[58] Ma C, Yang X, Zhang C, et al. Long-term correlation tracking[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2015: 5388-5396.

[59] Bewley A, Ge Z, Ott L, et al. Simple online and realtime tracking[C]//2016 IEEE international conference on image processing (ICIP). IEEE, 2016: 3464-3468.

[60] Wojke N, Bewley A, Paulus D. Simple online and realtime tracking with a deep association metric[C]//2017 IEEE international conference on image processing (ICIP). IEEE, 2017: 3645-3649.

[61] 高新闻, 沈卓, 许国耀, 等. 基于多目标跟踪的交通异常事件检测[J]. 计算机应用研究, 2021, 38(06): 1879-1883.

[62] 丁冰, 杨祖莨, 丁洁, 等. 基于改进YOLOv3的高速公路隧道内停车检测方法[J]. 计算机工程与应用, 2021, 57(23): 234-239.

[63] 李熙莹, 陆强, 张晓春, 等. 基于人车交互行为模型的上下客行为识别[J]. 中国公路学报, 2021, 34(07): 152-163.

[64] 田联房, 吴啟超, 杜启亮, 等. 基于人体骨架序列的手扶电梯乘客异常行为识别[J]. 华南理工大学学报(自然科学版), 2019, 47(04): 10-19.

[65] 李玉吉, 曹旭辉. 基于红外遥感信息的煤矿车辆异常行为检测研究[J]. 能源与环保, 2021, 43(12): 71-76.

[66] 周巧瑜, 曹扬, 詹瑾瑜, 等. 基于Yolo和GOTURN的景区游客翻越行为识别[J]. 计算机技术与发展, 2022, 32(01): 134-140.

[67] 卢锦亮, 吴广潮, 冯夫健, 等. 基于联合轨迹特征的徘徊行为识别方法[J]. 南京大学学报(自然科学), 2021, 57(05): 724-734.

[68] Ke Q, Bennamoun M, An S, et al. A new representation of skeleton sequences for 3d action recognition[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 3288-3297.

[69] Kim T S, Reiter A. Interpretable 3d human action analysis with temporal convolutional networks[C]//2017 IEEE conference on computer vision and pattern recognition workshops (CVPRW). IEEE, 2017: 1623-1631.

[70] Du Y, Wang W, Wang L. Hierarchical recurrent neural network for skeleton based action recognition[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2015: 1110-1118.

[71] Shahroudy A, Liu J, Ng T T, et al. Ntu rgb+ d: A large scale dataset for 3d human activity analysis[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 1010-1019.

[72] Kipf T N, Welling M. Semi-supervised classification with graph convolutional networks[J]. arXiv preprint arXiv:1609.02907, 2016.

[73] Yan S, Xiong Y, Lin D. Spatial temporal graph convolutional networks for skeleton-based action recognition[C]//Thirty-second AAAI conference on artificial intelligence. 2018.

[74] 于乃功, 柏德国. 基于姿态估计的实时跌倒检测算法[J]. 控制与决策, 2020, 35(11): 2761-2766.

[75] Hu Y, Lu M, Lu X. Feature refinement for image-based driver action recognition via multi-scale attention convolutional neural network[J]. Signal Processing: Image Communication, 2020, 81: 115697.

[76] 黎晓昀, 贾杰. 三维卷积和视频帧采样算法下斗殴检测技术[J]. 计算机工程与应用, 2021, 57(10): 241-245.

[77] 王泽杰, 沈超敏, 赵春, 等. 融合人体姿态估计和目标检测的学生课堂行为识别[J]. 华东师范大学学报(自然科学版), 2022(02): 55-66.

[78] 李学民. 矿用巡检机器人关键技术分析[J]. 煤矿机械, 2018, 39(05): 71-73.

[79] 严莉, 王玮, 刘荫, 等. 基于暗原色先验的图像去雾改进算法[J]. 中国高新科技, 2020(02): 45-47.

[80] McCartney E J. Optics of the atmosphere: scattering by molecules and particles[J]. New York, 1976.

[81] Zhao D, Xu L, Yan Y, et al. Multi-scale optimal fusion model for single image dehazing[J]. Signal Processing: Image Communication, 2019, 74: 253-265.

[82] Rezatofighi H, Tsoi N, Gwak J Y, et al. Generalized intersection over union: A metric and a loss for bounding box regression[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 658-666.

[83] Howard A, Sandler M, Chu G, et al. Searching for mobilenetv3[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019: 1314-1324.

[84] Hu J, Shen L, Sun G. Squeeze-and-excitation networks[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 7132-7141.

[85] Liu Y, Shao Z, Teng Y, et al. NAM: Normalization-based Attention Module[J]. arXiv preprint arXiv:2111.12419, 2021.

[86] Woo S, Park J, Lee J Y, et al. Cbam: Convolutional block attention module[C]//Proceedings of the European conference on computer vision (ECCV). 2018: 3-19.

[87] Wang J, Chen Y, Gao M, et al. Improved YOLOv5 network for real-time multi-scale traffic sign detection[J]. arXiv preprint arXiv:2112.08782, 2021.

[88] Zheng Z, Wang P, Liu W, et al. Distance-IoU loss: Faster and better learning for bounding box regression[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34(07): 12993-13000.

[89] He J, Erfani S, Ma X, et al. $\alpha $-IoU: A Family of Power Intersection over Union Losses for Bounding Box Regression[J]. Advances in Neural Information Processing Systems, 2021, 34.

[90] Cao Z, Simon T, Wei S E, et al. Realtime multi-person 2d pose estimation using part affinity fields[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 7291-7299.

[91] Wang J, Liu Z, Wu Y, et al. Mining actionlet ensemble for action recognition with depth cameras[C]//2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2012: 1290-1297.