带式输送机作为煤炭运输系统的重要组成部分，在煤矿的生产运输中扮演着重要角色。然而在煤炭运输过程中，大块物料砸落、输送带偏移都有可能引起输送带事故。与此同时，由于输送带自身设计和安装的问题，滚筒、托辊等高速旋转的位置都可能是伤人的挤夹点，因此有人员在周围活动时也极易发生不安全事故。为了降低设备运行过程中存在的安全隐患，本文研究并设计了基于机器视觉的煤矿带式输送机运行区域安全监测系统，并对系统中的关键算法：人员行为识别、大块物料监测以及输送带偏移进行了深入研究和必要的改进，并将最后的检测算法通过多组实验进行验证。本文的主要研究工作如下：

对于带式输送机运行区域周边矿工的不安全行为识别，为解决现有的矿区人员不安全行为识别时存在的环境干扰大，人员行为类型复杂难以识别的问题，提出一种基于改进时空图卷积网络（ST-GCN）的矿工不安全行为识别网络（NP-AGCN）。首先，利用多帧人体关键点构建的骨架时空图进行行为识别，减少煤矿复杂环境对行为识别的干扰;其次，针对原图结构无法学习非自然连接节点之间的关联关系，导致攀爬输送带、打架等行为识别率较低的问题，对图结构进行重构，改变原划分策略，提高模型对多关节交互行为的识别能力。最后，为了改善图卷积网络因感受野小而难以学习全局信息的问题，在图卷积中引入多头自注意机制，提高模型对不安全行为的识别能力。通过这种改进后的图卷积网络能够很好的对输送机周边矿工的各种不安全行为进行识别，从而降低输送机运行区域周边矿工受到伤害的可能。

对于输送带偏移及大块物料现象的检测，针对检测目标的不同分别提出两种不同的检测方式。两种方式都使用的是基于DETR的深度学习网络，在网络的末尾处通过替换检测头和分割头的方式来实现不同下游任务的分配。对大块物料的检测，由于检测的目标中经常会存在超大尺寸目标和遮挡目标，使用能够对这两个问题很好解决的Deformable-DETR目标检测网络并在网络后部加入大块物料判别算法。对输送带偏移的问题，采用实例分割的方式先将整个皮带的表面与背景分割出来，然后计算每张图像分割出的皮带之间的交并比来判定是否跑偏。最后，为了提高网络检测效率且降低在工业应用上模型维护难度，提出利用多任务深度学习的方式将两个不同的检测子任务共同进行训练。经过实验验证，针对不同问题分别提出的两种检测方法对于特定问题都能取得准确的检测结果，搭建的多任务深度学习网络也能在降低参数量的同时提升每个子任务的检测准确率。

最后，为了能将所研究的带式输送机运行区域安全监测算法应用到现实工作环境中，设计和实现带式输送机运行区域安全监测系统。根据实际工况需求、系统功能要求以及相关技术制定出系统设计方案，完成系统的软硬件选型、数据库的选择和搭建，并对系统通讯方式的选择和前后段通信接口进行说明，实现基于机器视觉的带式输送机运行区域安全监测系统。

As an important part of coal transportation system, belt conveyor plays an important role in coal mine production and transportation. However, in the process of coal transportation, large materials falling and conveyor belt deviation may cause conveyor belt accidents. At the same time, due to the design and installation of the conveyor belt itself, the roller, idler and other high-speed rotation positions may be hurt pinch points, so there are people around the activity is also prone to unsafe accidents. With the rapid development of machine vision technology, in order to make up for this defect, this paper studies and designs a set of coal mine belt conveyor transportation safety monitoring system based on machine vision, and the key algorithms in the system: Personnel behavior identification, bulk material monitoring and conveyor belt migration are deeply studied and necessary improvements are made, and the final detection algorithm is verified by several sets of experiments. The main research work of this paper is as follows: 

For the detection of unsafe behaviors of surrounding miners, in order to solve the existing problems of large environmental interference and difficult to identify the types of unsafe behaviors in mining areas, a mine unsafe behavior identification network (NP-AGCN) based on the improved Spatio-temporal graph convolution network (ST-GCN) is proposed. Firstly, multi-frame skeleton space-time map constructed by key points of human body is used for behavior recognition to reduce the interference of complex environment of coal mine. Secondly, in view of the problem that the original graph structure could not learn the association relationship between the unnatural connection nodes, which resulted in the low recognition rate of behaviors such as climbing belts and fighting, the graph structure was reconstructed and the original partitioning strategy was changed to improve the recognition ability of the model for multi-joint interaction. Finally, in order to improve the problem that the graph convolution network is difficult to learn global information due to the small sensitivity field, multiple self-attention mechanism is introduced into the graph convolution to improve the recognition ability of the model for unsafe behaviors. In order to verify the detection ability of the model in identifying unsafe behavior of personnel, tests were carried out on the public data set NTU-RGB+D and the self-built data set of unsafe behavior of miners. In the above data set, the recognition accuracy of the proposed model is 94.7% and 94.1% respectively, which is 6.4% and 7.4% higher than that of the original model, which verifies that the proposed model has good recognition accuracy. The improved graph convolutional network can well identify various unsafe behaviors of miners around the conveyor to avoid the occurrence of danger. 

For the detection of belt deviation and bulk material phenomenon, two different detection methods are proposed according to the different detection targets. Both methods use DETR-based deep learning networks, and at the end of the network, different downstream tasks are assigned by replacing detection headers and splitting headers. For the detection of bulk materials, as there are often oversized targets and shielded targets in the detection targets, Deformable-DETR target detection network which can solve these two problems well is used and the bulk material discrimination algorithm is added to the back of the network. For the problem of belt deviation, the surface of the whole belt and the background are segmented by the way of example segmentation, and then the intersection ratio between each image segmentation belt is calculated to determine whether the deviation is. Finally, in order to improve the efficiency of network detection and reduce the difficulty of model maintenance in industrial applications, a multi-task deep learning method is proposed to train two different detection subtasks together. Through experiments, the detection accuracy of bulk material detection network is 95.3%, and the detection accuracy of bulk material is up to 99%. The conveyor belt migration detection network has achieved 98.9% segmentation accuracy, and the identification accuracy of the migration determination algorithm is higher than other determination methods. The total number of parameters of the finally constructed multi-task learning model decreased by 14.3%, and the detection accuracy of different tasks increased by 1.8% and 0.1% respectively. 

Finally, in order to apply the transport safety monitoring model of the belt conveyor to the real working environment, the belt conveyor transport safety monitoring system is designed and implemented. According to the actual working condition requirements, system functional requirements and phase technology, the system design scheme is formulated, the system hardware and software selection, database selection and construction are completed, and the system communication mode selection and the front and rear section communication interface are described, and the belt conveyor transportation safety monitoring system based on machine vision is realized.

[1] 谢和平,吴立新,郑德志.2025年中国能源消费及煤炭需求预测[J].煤炭学报,2019,44(07):1949-1960.

[2] 刘海滨,李春贺.智慧矿山职业健康安全监管信息系统研究[J].煤炭科学技术,2019,47(03):87-92.

[3] 李爽,李丁炜,犹梦洁.煤矿安全态势感知预测系统设计及关键技术[J].煤矿安全,2020,51(05):244-248.DOI:10.13347/j.cnki.mkaq.2020.05.051.

[4] He D , Liu X , Zhong B . Sustainable belt conveyor operation by active speed control[J]. Measurement, 2020, 154:107458.

[5] 韩建辉,钱锋.关于煤炭采样方式及问题的探讨[J].山东化工,2019,48(22):117-118+122.DOI:10.19319/j.cnki.issn.1008-021x.2019.22.042.

[6] Yang Y , Hou C , Qiao T , et al. Longitudinal tear early-warning method for conveyor belt based on infrared vision[J]. Measurement, 2019, 147:106817.

[7] 张雷,张跃,李明雪,史新国,等.基于CSI的井下人员行为识别方法[J].物联网学报,2020,4(04):26-31.

[8] 冯志星.矿用带式输送机常见故障及故障诊断系统的设计研究[J].机械管理开发,2019,34(11):114-115.DOI:10.16525/j.cnki.cn14-1134/th.2019.11.047.

[9] 王文清,田柏林,冯海明,等.基于激光测距矿用带式输送机多参数检测方法研究[J].煤炭科学技术,2020,48(08):131-138.DOI:10.13199/j.cnki.cst.2020.08.016.

[10]徐欢,李振璧,姜媛媛,等.基于OpenCV的输送带跑偏自动检测算法研究[J].工矿自动化,2014,40(09):48-52.DOI:10.13272/j.issn.1671-251x.2014.09.012.

[11]Zimroz R, Król R. Failure analysis of belt conveyor systems for condition monitoring purposes[J]. Mining Science, 2009, 128(36): 255.

[12]Zhao L. Typical failure analysis and processing of belt conveyor[J]. Procedia Engineering, 2011, 26: 942-946.

[13]Y. Pang, G. Lodewijks, A novel embedded conductive detection system for intel-ligent conveyor belt monitoring, in: IEEE Int. Conf. Service Operations and Logistics, and Informatics, 2006, pp. 803–808.

[14]T. Nicolay, A. Treib, A. Blum, RF identification in the use of belt rip detection,Proc. IEEE Sens. (2004) 333–336.

[15]Liu, Y., Miao, C., Li, X., Xu, G., 2021. Research on deviation detection of belt conveyor based on inspection robot and deep learning. Complexity 2021, 1–15.

[16]Yang, X., Tian, M., Li, L., Lei, Z., Song, J., and Zhang, L., Research on Belt Failure Detection Technology for Belt Conveyor, Coal Mine Machinery, 2019, vol. 40, no. 2, pp. 133–136.

[17]Wang, J.; Liu, Q.; Dai, M. Belt vision localization algorithm based on machine vision and belt conveyor deviation detection. In Proceedings of the 2019 34rd Youth Academic Annual Conference of Chinese Association of Automation (YAC), Jinzhou, China, 6–8 June 2019; pp. 269–273.

[18]Wang, T., Dong, Z. & Liu, J. Research of Mine Conveyor Belt Deviation Detection System Based on Machine Vision. J Min Sci 57, 703–712 (2021). https://doi.org/10.1134/S1062739121040190

[19]Mei X, Miao C, Yang Y, et al. Rapid inspection technique for conveyor belt deviation[J]. Journal of Mechanical Engineering Research and Developments, 2016, 39(3): 653-662.

[20]Liu Y , Miao C , Li X , et al. Research on Deviation Detection of Belt Conveyor Based on Inspection Robot and Deep Learning[J]. Complexity, 2021.

[21]Su H , Yang C , Ferrigno G , et al. Improved Human–Robot Collaborative Control of Redundant Robot for Teleoperated Minimally Invasive Surgery[J]. IEEE Robotics & Automation Letters, 2019, 4(2):1447-1453.

[22]Hang S A , Yh B , Hrk A , et al. Improved recurrent neural network-based manipulator control with remote center of motion constraints: Experimental results[J]. Neural Networks, 2020, 131:291-299.

[23]Zhang M, Jiang K, Cao Y, et al. A deep learning-based method for deviation status detection in intelligent conveyor belt system[J]. Journal of Cleaner Production, 2022, 363: 132575.

[24]Zeng, C., Zheng, J., Li, J., 2019. Real-time conveyor belt deviation detection algorithm based on multi-scale feature fusion network. Algorithms 12, 205.

[25]Yang Y , Miao C , Li X , et al. On-line conveyor belts inspection based on machine vision[J]. Optik - International Journal for Light and Electron Optics, 2014, 125(19):5803-5807.

[26]Simonyan K, Zisserman A. Two-stream convolutional networks for action recognition in videos[J]. Advances in neural information processing systems, 2014, 27.

[27]Feichtenhofer C , Pinz A , Zisserman A . Convolutional Two-Stream Network Fusion for Video Action Recognition[J]. IEEE, 2016.

[28]Wang L, Xiong Y, Wang Z, et al. Temporal segment networks: Towards good practices for deep action recognition[C]//European conference on computer vision. Springer, Cham, 2016: 20-36.

[29]Wang L , Ge L , Li R , et al. Three-stream CNNs for action recognition[J]. Pattern Recognition Letters, 2017, 92(jun.1):33-40.

[30]Shahroudy A , Liu J , Ng T T , et al. NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analw'w'w'wysis[C]// IEEE Computer Society. IEEE Computer Society, 2016:1010-1019.

[31]Cao Z, Simon T, Wei S E, et al. Realtime multi-person 2d pose estimation using part affinity fields[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 7291-7299.

[32]Shotton, J.; Sharp, T.; Kipman, A.;Fitzgibbon, A.; Finocchio, M.; Blake, A.; Cook, M.; and Moore, R. 2011. Real-time human pose recognition in parts from single depth images. In CVPR.

[33]Li B, Dai Y, Cheng X, et al. Skeleton based action recognition using translation-scale invariant image mapping and multi-scale deep CNN[C]//2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW). IEEE, 2017: 601-604.

[34]Le T M, Inoue N, Shinoda K. A fine-to-coarse convolutional neural network for 3D human action recognition[J]. arXiv preprint arXiv:1805.11790, 2018.

[35]Ke Q, Bennamoun M, An S, et al. A new representation of skeleton sequences for 3d action recognition[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 3288-3297.

[36]Shahroudy A , Liu J , Ng T T , et al. NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis[C]// IEEE Computer Society. IEEE Computer Society, 2016:1010-1019.

[37]Zheng W, Li L, Zhang Z, et al. Relational network for skeleton-based action recognition[C]//2019 IEEE International conference on multimedia and expo (ICME). IEEE, 2019: 826-831.

[38]Yan S, Xiong Y, Lin D. Spatial temporal graph convolutional networks for skeleton-based action recognition[C]//Proceedings of the AAAI conference on artificial intelligence. 2018, 32(1).

[39]SHI Lei, ZHANG Yifan, CHENG Jia, et al. Two-stream adaptive graph convolutional networks for skeleton-based action recognition[C]. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019: 12026–12035.

[40]Alsawadi M S, Rio M. Skeleton Split Strategies for Spatial Temporal Graph Convolution Networks[J]. Computers, Materials & Continua, 2022, 71(3): 4643-4658.

[41]Hongye, Yang, Yuzhang, et al. PGCN-TCA: Pseudo Graph Convolutional Network With Temporal and Channel-Wise Attention for Skeleton-Based Action Recognition[J]. IEEE Access, 2020:10040-10047.

[42]Wu C , Wu X J , Kittler J . Spatial Residual Layer and Dense Connection Block Enhanced Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition[C]// 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). IEEE, 2019.

[43]Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Advances in neural information processing systems. 2017: 5998-6008.

[44]Parmar N, Vaswani A, Uszkoreit J, et al. Image transformer[C]//International conference on machine learning. PMLR, 2018: 4055-4064.

[45]Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[J]. arXiv preprint arXiv:2010.11929, 2020.

[46]Liu Z,Lin Y,Cao Y,et al.Swin transformer: Hierarchical vision transformer using shifted windows[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021:10012-10022.

[47]Esser P, Rombach R, Ommer B. Taming transformers for high-resolution image synthesis[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 12873-12883.

[48]Mazzia V, Angarano S, Salvetti F, et al. Action Transformer: A self-attention model for short-time pose-based human action recognition[J]. Pattern Recognition, 2022, 124: 108487.

[49]Carion N, Massa F, Synnaeve G, et al. End-to-end object detection with transformers[C]//Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16. Springer International Publishing, 2020: 213-229.

[50]Zhu X, Su W, Lu L, et al. Deformable detr: Deformable transformers for end-to-end object detection[J]. arXiv preprint arXiv:2010.04159, 2020.

[51]Zhao X , Zhou P , Xu K , et al. An Improved Character Recognition Framework for Containers Based on DETR Algorithm[J]. Sensors, 2021, 21(13):4612.

[52]Dong B, Zeng F, Wang T, et al. Solq: Segmenting objects by learning queries[J]. Advances in Neural Information Processing Systems, 2021, 34: 21898-21909.

[53]Caruana R. Multitask learning[M]. Springer US, 1998.

[54]Collobert, R. and Weston, J. (2008). A unified architecture for natural language processing. Proceedings of the 25th international conference on Machine learning ICML ’08, 20(1):160–167.

[55]Deng, L., Hinton, G. E., and Kingsbury, B. (2013). New types of deep neural network learning for speech recognition and related applications: An overview. 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 8599–8603.

[56]Duong, L., Cohn, T., Bird, S., and Cook, P. (2015). Low Resource Dependency Parsing: Cross-lingual Parameter Sharing in a Neural Network Parser. Proceedings ofthe 53rd Annual Meeting ofthe Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Short Papers), pages 845–850.

[57]Ramsundar B, Kearnes S, Riley P, et al. Massively multitask networks for drug discovery[J]. arXiv preprint arXiv:1502.02072, 2015.

[58]Zhang, C. H. and Huang, J. (2008). The sparsity and bias of the lasso selection in high-dimensional linear regression. Annals of Statistics, 36(4):1567–1594.

[59]Argyriou A, Evgeniou T, Pontil M. Multi-task feature learning, in ‘Advances in Neural Information Processing Systems 19’[J]. 2007.

[60]Yuan, M. and Lin, Y. (2006). Model selection and estimation in regression with grouped variables. Journal ofthe Royal Statistical Society: Series B (Statistical Methodology),68(1):49–67.

[61]Amir R. Zamir, Alexander Sax, William B. Shen,Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (MIT License Copyright (c) 2017 Stanford Vision and Learning Group), 2018.

[62]Trevor Standley, Amir R. Zamir, Dawn Chen,Leonidas Guibas, Jitendra Malik, and Silvio Savarese.Which tasks should be learned together in multi-task learning? arXiv: 1905.07553, cs.CV, 2019.

[63]Li C F, Xu Y D, Ding X H, et al. MultiR-Net: a novel joint learning network for COVID-19 segmentation and classification[J]. Computers in Biology and Medicine, 2022, 144: 105340.

[64]Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu,Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu,Chao Xu, and Wen Gao. Pre-trained image processing transformer. arXiv: 2012.00364, cs.CV, 2021.

[65]Hongje Seong, Junhyuk Hyun, and Euntai Kim. Video multitask transformer network. In 2019 IEEE/CVF In-ternational Conference on Computer Vision Workshop(ICCVW), pages 1553–1561, 2019.

[66]Eslam Mohamed and Ahmed El-Sallab. Spatio-temporal multi-task learning transformer for joint moving object detection and segmentation. arXiv:2106.11401, cs.CV, 2021.

[67]Yang L, Wang X, Zhu J, et al. Influencing factors, formation mechanism, and pre-control methods of coal miners′ unsafe behavior: A systematic literature review[J]. Frontiers in public health, 2022, 10.

[68]Yang L, Birhane G E, Zhu J, et al. Mining employees safety and the application of information technology in coal mining[J]. Frontiers in public health, 2021, 9: 709987.

[69]张利全. 带式输送机跑偏原因及其应对措施[J]. 科技情报开发与经济, 2008, 18(28):2.

[70]Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[J]. arXiv preprint arXiv:1409.1556, 2014.

[71]Kuhn H W. The Hungarian method for the assignment problem[J]. Naval research logistics quarterly, 1955, 2(1‐2): 83-97.

[72]Zhou J, Cui G, Hu S, et al. Graph neural networks: A review of methods and applications[J]. AI open, 2020, 1: 57-81.

[73]Hu J, Shen L, Sun G. Squeeze-and-excitation networks[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 7132-7141.

[74]Zhang D, Fang B, Yang W, et al. Robust inverse perspective mapping based on vanishing point[C]//Proceedings 2014 IEEE International Conference on Security, Pattern Analysis, and Cybernetics (SPAC). IEEE, 2014: 458-463.

[75]Zhang H, Wu C, Zhang Z, et al. Resnest: Split-attention networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 2736-2746.

[76]He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778.

[77]Zheng Z, Wang P, Liu W, et al. Distance-IoU loss: Faster and better learning for bounding box regression[C]//Proceedings of the AAAI conference on artificial intelligence. 2020, 34(07): 12993-13000.

[78]Liu W, Anguelov D, Erhan D, et al. Ssd: Single shot multibox detector[C]//Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14. Springer International Publishing, 2016: 21-37.

[79]Ren S, He K, Girshick R, et al. Faster r-cnn: Towards real-time object detection with region proposal networks[J]. Advances in neural information processing systems, 2015, 28.

[80]He K, Gkioxari G, Dollár P, et al. Mask r-cnn[C]//Proceedings of the IEEE international conference on computer vision. 2017: 2961-2969.

[81]Cai Z, Vasconcelos N. Cascade R-CNN: high quality object detection and instance segmentation[J]. IEEE transactions on pattern analysis and machine intelligence, 2019, 43(5): 1483-1498.

[82]Bolya D, Zhou C, Xiao F, et al. Yolact: Real-time instance segmentation[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 9157-9166.

[83]Wang X, Zhang R, Kong T, et al. Solov2: Dynamic and fast instance segmentation[J]. Advances in Neural information processing systems, 2020, 33: 17721-17732.