煤炭是我国重要的工业生产原材料和基础能源，随着人工成本逐渐提高、劳动力资源减少，实现煤炭开采运输的自动化与智能化成为煤炭产业主流发展趋势。煤炭异物分拣机器人是煤炭运输自动化的研究热点之一，而基于机器视觉的煤炭异物检测是异物分拣的核心技术。目前，煤炭异物检测主要存在数据集质量差、复杂特征目标检测精度低、传统检测方法定位效果差、网络模型冗余度高等问题。为解决以上问题，本文从煤炭异物图像数据扩增、复杂特征煤炭异物检测模型搭建、煤炭异物抓取位姿实时提取三个角度展开深入研究，以期提高煤炭异物检测算法的鲁棒性和适应性，为分拣机器人提供核心技术支撑。主要工作内容如下：

针对煤炭异物数据集样本量少、样本不平衡导致的煤炭异物检测模型特征提取困难、易过拟合的问题，研究基于改进StyleGAN的煤炭异物图像生成方法。通过引入双重自注意力模块与深度可分离卷积，使StyleGAN在生成高质量异物图像的同时减少总体参数量，缩短训练周期。实验结果表明，改进方法对生成异物图像的质量与多样性提升效果可观，利用生成图像进行数据扩增后，异物检测精度得到显著提高。

针对煤炭异物形状多变、尺度不均、相互遮挡的复杂特征所导致的低检测精度问题，研究基于改进BlendMask的煤炭异物检测方法。首先，在BlendMask模型的骨干网络中引入二代可变形卷积以增强多形变异物的特征提取能力；其次，利用双向加权特征金字塔网络改进特征融合路径，提高多尺度异物检测精度；最后，在金字塔网络后串联轻量混合注意力模块，加强模型对遮挡异物可见部位的关注程度。实验结果表明，改进方法切实提高了对复杂特征异物的检测精度，有效减少了误检与漏检现象的发生。 

针对传统煤炭异物检测方法难以提供有效抓取位姿信息及卷积神经网络的结构、参数存在大量冗余导致检测模型推理实时性差的问题，研究基于图形学算法与TensorRT加速部署的煤炭异物抓取位姿实时提取方法。首先，结合凸包边界旋转法与图像几何矩提取异物抓取位姿，为异物分拣提供可靠抓取信息；其次，利用TensorRT优化模型推理机制，实现模型快速前向推理。实验结果表明，经过位姿提取与TensorRT优化后，异物抓取的质心误差、角度误差及宽度误差得到了有效缩减，且达到了检测实时性需求。

基于煤炭异物分拣机器人平台应用需求，设计煤炭异物视觉检测系统，实现对带式输送机上煤炭异物的图像采集、实时检测、信息传输、持久化存储与结果可视化功能。在分拣机器人平台上进行静态检测实验、动态检测实验及系统实时性实验，实验结果表明，本文方法对静态复杂条件及1m/s带速内动态条件下的煤炭异物均能达到较好的检测效果，且能满足系统实时性需求，验证了所提算法的有效性和系统软件的可靠性。

Coal is an important raw material and basic energy for industrial production in China. With the gradual increase of labor cost and decrease of labor resources, the automation and intelligence of coal mining and transportation become the mainstream development trend of coal industry. Coal foreign object sorting robot is one of the research hotspots of coal transportation automation, and vision-based coal foreign object real-time accurate detection is the core technology of foreign object sorting. At present, coal foreign object detection mainly suffers from poor quality of dataset, low detection accuracy under complex features, poor localization effect of traditional detection methods, and high complexity of network models. In order to solve above problems, this paper conducts research from three perspectives, including foreign object image data augmentation, complex feature foreign object detection model construction, and real-time extraction of foreign object grasping poses, hoping to improve the robustness and adaptability of coal foreign object detection algorithm and provide core technology support for sorting robots. The main works are as follows.

 Aims the problems of difficult feature extraction and easy overfitting of the foreign objects detection model due to the small sample size and sample imbalance of coal foreign objects, a high-quality coal foreign object image generation method based on improved StyleGAN is investigated. By combining the dual self-attention mechanism and depth-separable convolution, the method enables the StyleGAN to generate high-quality foreign object images while reducing the total number of parameters and shortening the training period. Experimental results show that the quality and diversity of the generated foreign object images by the improved model are improved considerably, and the accuracy of the foreign object segmentation model is significantly improved after data augmentation using the generated images.

Aims the complex features of coal foreign objects with variable shapes, uneven scales and mutual occlusion leading to low detection accuracy, a coal foreign object detection method based on improved BlendMask is investigated. Firstly, DCN v2 is introduced in the backbone network of BlendMask model to enhance the feature extraction capability of polymorphic foreign objects; secondly, the feature fusion path is improved using bi-directional weighted feature pyramid network to improve the multi-scale foreign object detection accuracy; finally, a lightweight hybrid attention module after BiFPN to enhance the model's attention to the visible parts of the occluded foreign objects. The experimental results show that the improved method can effectively improve the detection accuracy of complex foreign objects, reduce the occurrence of false and missed detection.

Aims the problems that traditional coal foreign object detection methods are difficult to provide effective grasping pose information and CNN with large number of redundant parameters leading to poor inference of detection model in real-time, a real-time extraction method of foreign object grasping pose based on graphical method and TensorRT accelerated deployment is researched. Firstly, the convex hull boundary rotation method is combined with image geometric moments to extract foreign object grasping poses and provide reliable grasping information for foreign object sorting. Secondly, the model inference mechanism is optimized by TensorRT accelerated inference framework to achieve fast forward inference. The experimental results show that after the pose extraction and TensorRT optimization, the center-of-mass error, angle error and hand claw opening and closing error of foreign object grasping are effectively reduced, and the detection real-time requirement is achieved.

Based on the sorting requirements of the coal foreign object sorting robot, a coal foreign object vision detection system is designed to achieve image acquisition, real-time detection, information transmission, persistent storage and result visualization. The static detection experiments, dynamic detection experiments and system real-time experiments are conducted on the foreign matter sorting robot platform. The experimental results show that the method can achieve good detection effect for coal foreign objects under static complex conditions and dynamic conditions within 1m/s belt speed, and can meet the system real-time requirements, which verifies the effectiveness of the proposed algorithm and the reliability of the system software.

[1]王国法，任世华，庞义辉，等. 煤炭工业“十三五”发展成效与“双碳”目标实施路径[J]. 煤炭科学技术，2021.

[2]陈浮，王思遥，于昊辰，等. 碳中和目标下煤炭变革的技术路径[J]. 煤炭学报，2022, 47(04): 1452-1461.

[3]刘峰，郭林峰，赵路正. 双碳背景下煤炭安全区间与绿色低碳技术路径[J]. 煤炭学报，2022, 47(01): 1-15.

[4]刘峰，曹文君，张建明，等. 我国煤炭工业科技创新进展及“十四五”发展方向[J]. 煤炭学报，2021, 46(01): 1-15.

[5]Kiseleva T V, Mikhailov V G, Karasev V A. Management of local economic and ecological system of coal processing company[C]// IOP Conference Series: Earth and Environmental Science. IOP Publishing, 2016, 45(1): 012013.

[6]何永彬，母海龙，田娜, 等. 补连塔选煤厂杂物管控的措施[J]. 陕西煤炭，2019, 38(S1): 146-148.

[7]杨晨光，冯岸岸，朱金波，等. 智能分选中煤矸X射线识别技术的研究[J]. 安徽化工，2020, 46(03): 25-29+33.

[8]李曼，段雍，曹现刚，等. 煤矸分选机器人图像识别方法和系统[J]. 煤炭学报，2020, 45(10): 3636-3644.

[9]张琦，张荣梅，陈彬. 基于深度学习的图像识别技术研究综述[J]. 河北省科学院学报，2019, 36(03), 28-36.

[10]Su L, Cao X, Ma H, et al. Research on Coal Gangue Identification by Using Convolutional Neural Network[C]// 2018 2nd IEEE Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC). IEEE, 2018.

[11]曹亦俊，刘敏，邢耀文，等. 煤矿井下选煤技术现状和展望[J]. 采矿与安全工程学，2020, 37(01), 192-201.

[12]曹现刚，吴旭东，王鹏，等. 面向煤矸分拣机器人的多机械臂协同策略[J]. 煤炭学报，2019, 44(02), 763-774.

[13]曹现刚，郝朋英，王鹏，等. 多因素光照条件下高质量煤矸图像获取方法研究[J]. 煤炭科学技术，2023, 51(01): 455-463.

[14]曹现刚，刘思颖，王鹏，等. 面向煤矸分拣机器人的煤矸识别定位系统研究[J]. 煤炭科学技术, 2022, 50(01): 237-246.

[15]宋晓茹. 基于ARM和CPLD的煤矸石在线自动分选系统研究[D]. 西安：西安科技大学，2006.

[16]邬冠华，熊鸿建. 中国射线检测技术现状及研究进展[J]. 仪器仪表学报，2016, 37(08): 1683-1695.

[17]孔力，李红，徐恕宏，等. 双能γ射线透射法煤矸石在线识别与分选系统[J]. 华中理工大学学报，1997(10): 108-109.

[18]WATT J S, STERFFNER E J. Dual energy gamma-ray transmission techniques applied to on-line analysis in the coal and mineral industries[J]. International Journal of Applied Radiation & Isotopes, 1985, 36(11): 867-877.

[19]陈国杰，赵维义，朱 星. 基于单片机双能γ射线透射煤矸石在线识别仪[J]. 核电子学与探测技术，2004, 24(02): 140-142.

[20]余长军. 基于X射线的选煤厂块煤和块矸分选系统的研究[D]. 安徽：安徽理工大学，2017.

[21]刘富强，钱建生，王新红，等. 基于图像处理与识别技术的煤矿矸石自动分选[J]. 煤炭学报，2000, 25(5): 537-537．

[22]张万枝，王增才. 基于视觉技术的煤岩特征分析与识别[J]. 煤炭技术，2014, 33(10): 272-274.

[23]于国防，邹士威，秦聪. 图像灰度信息在煤矸石自动分选中的应用研究[J]. 工矿自动化，2012, 38(2): 36-39.

[24]王家臣，李良晖，杨胜利. 不同照度下煤矸图像灰度及纹理特征提取的实验研究[J]. 煤炭学报，2018, 43(11): 3051-3061.

[25]LE Ba Tuan，肖冬，毛亚纯，等. 可见、近红外光谱和深度学习CNN-ELM算法的煤炭分类[J]. 光谱学与光谱分析，2018, 38( 07) : 2107-2112.

[26]WANG Y, WANG Y, DANG L. Video detection of foreign objects on the surface of belt conveyor underground coal mine based on improved SSD[J]. Journal of Ambient Intelligence and Humanized Computing, 2020: 1-10.

[27]吴守鹏. 基于机器视觉的运煤皮带异物识别方法研究[D]. 江苏：中国矿业大学，2019.

[28]王卫东，张康辉，吕子奇. 基于深度学习的煤中异物机器视觉检测[J]. 矿业科学学报，2021, 6(1): 115-123.

[29]胡景皓. 基于深度学习的带式输送机非煤异物视频检测系统[D]. 山西：太原理工大学，2021.

[30]任志玲，朱彦存. 改进CenterNet算法的煤矿皮带运输异物识别研究[J]. 控制工程，2021.

[31]Zhong Z, Zheng L, Kang G, et al. Random Erasing Data Augmentation[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2017, 34(7).

[32]Liang D, Yang F, Zhang T, et al. Understanding Mixup Training Methods[J]. IEEE Access, 2018, PP: 1-1.

[33]胡璟皓，高妍，张红娟，等. 基于深度学习的带式输送机非煤异物识别方法[J]. 工矿自动化，2021, 47(06): 57-62+90.

[34]郝帅，张旭，马旭，等. 基于CBAM-YOLOv5的煤矿输送带异物检测[J]. 煤炭学报，2022, 47(11): 4147-4156.

[35]李曼，杨茂林，刘长岳，等. 基于图像的煤矸分选中图像照度调节方法[J]. 煤炭学报，2021, 46(S2): 1149-1158.

[36]Goodfellow I J, Pouget-Abadie J, Mirza M, et al. Generative Adversarial Networks[J]. Advances in Neural Information Processing Systems, 2014, 3: 2672-2680.

[37]Arjovsky M, Chintala S, Bottou L. Wasserstein generative adversarial networks[C]// International Conference on Machine Learning, Sydney, Australia, 2017: 214-223.

[38]Gulrajani I, Ahmed F, Arjovsky M, et al. Improved training of Wasserstein GANs[C]// Advances in Neural Information Processing Systems. 2017: 5767-5777.

[39]Radford A, Metz L, Chintala S. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[J]. Computer ence, 2015.

[40]Karras T, Aila T, Laine S, et al. Progressive Growing of GANs for Improved Quality, Stability, and Variation[C]// Proceedings of the International Conference on Learning Representations (ICLR), 2018.

[41]Andrew B, Jeff D H, Kare S Y. Large scale GAN training for high fidelity natural image synthesis[C]// Proceedings of the International Conference on Learning Representations (ICLR), 2019.

[42]Shi Z, Sang M, Huang Y. Defect Detection of MEMS Based on Data Augmentation, WGAN-DIV-DC, and a YOLOv5 Model[J]. Sensors, 2022, 22(23), 9400.

[43]邓源，施一萍，刘婕，等. 结合双通道WGAN-GP的多角度人脸表情识别算法研究[J]. 激光与光电子学进展，2022, 59(18): 137-147.

[44]王星，高峰，陈吉，等. 基于GAN网络的煤岩图像样本生成方法[J]. 煤炭学报，2021, 46(09): 3066-3078.

[45]Wang L, Wang X, Li B. A data expansion strategy for improving coal-gangue detection[J]. International Journal of Coal Preparation and Utilization, 2022:1-19.

[46]Karras Tero, Laine Samuli, Aila Timo. A Style-Based Generator Architecture for Generative Adversarial Networks[C]// IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2019: 4401-4410.

[47]Ma D, Liu J, Fang H, et al. A Multi-defect detection system for sewer pipelines based on StyleGAN-SDM and fusion CNN[J]. Construction and Building Materials, 2021,312: 125385.

[48]Hussin S, Yildirim R. StyleGAN-LSRO Method for Person Re-Identification[J]. IEEE Access, 2021, PP(99):1-1.

[49]Li M, Zhou G, Chen A, et al. FWDGAN-based data augmentation for tomato leaf disease identification[J]. Computers and Electronics in Agriculture, 2022, 194: 106779.

[50]Zhang H, Goodfellow I, Metaxas D, et al. Self-attention generative adversarial networks[C]// International conference on machine learning. PMLR, 2019: 7354-7363.

[51]Yang Y, Sun L, Mao X. Data Augmentation Based on Generative Adversarial Network with Mixed Attention Mechanism[J]. Electronics, 2022, 11(11), 1718.

[52]Gu W, Bai S, Kong L. A review on 2D instance segmentation based on deep neural networks[J]. Image and Vision Computing, 2022: 104401.

[53]张继凯，赵君，张然，等. 深度学习的图像实例分割方法综述[J]. 小型微型计算机系统，2021, 1(01): 161-171.

[54]Fathi A, Wojna Z, Rathod V, et al. Semantic instance segmentation via deep metric learning[J]. arXiv: 1703.10277, 2017.

[55]Shu L, Jia J, Fidler S, et al. SGN: Sequential Grouping Networks for Instance Segmentation[C]// 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, 2017.

[56]Kong S, Fowlkes C. Recurrent pixel embedding for instance grouping[C]// Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 9018-9028.

[57]Hariharan B, Arbeláez P, Girshick R, et al. Simultaneous detection and segmentation[C]// European conference on computer vision. Springer, Cham, 2014: 297-312.

[58]Pinheiro P O, Collobert R, Dollár P. Learning to segment object candidates[J]. Advances in neural information processing systems, 2015, 28.

[59]He K, Gkioxari G, Dollár P, et al. MaskR-CNN[C]// Proceedings of the IEEE International Conference on Computer Vision, 2017: 2961-2969.

[60]Huang Z, Huang L, Gong Y, et al. Mask scoring R-CNN[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019: 6409-6418.

[61]Li Y, Qi H, Dai J, et al. Fully convolutional instance-aware semantic segmentation[C]// Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 2359-2367.

[62]Bolya D, Zhou C, Xiao F, et al．YOLACT: real-time instance segmentation[C]// Proceedings of the IEEE International Conference on Computer Vision, 2019: 9157-9166.

[63]Bolya D, Zhou C, Xiao F, et al. Yolact++: Better real-time instance segmentation[J]. IEEE transactions on pattern analysis and machine intelligence, 2020.

[64]Hao Chen, Kunyang Sun, Zhi Tian, et al．BlendMask: Top-Down Meets Bottom-Up for Instance Segmentation[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020: 8573-8581.

[65]Lee Y, Park J. Centermask: Real-time anchor-free instance segmentation[C]// Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 13906-13915.

[66]Wang X, Kong T, Shen C, et al. Solo: Segmenting objects by locations[C]// European Conference on Computer Vision. Springer, Cham, 2020: 649-665.

[67]Wang X, Zhang R, Kong T, et al. Solov2: Dynamic and fast instance segmentation[J]. Advances in Neural information processing systems, 2020, 33: 17721-17732.

[68]Mahler J, Pokorny F T, Hou B, et al. Dex-Net 1.0: A cloud-based network of 3D objects for robust grasp planning using a Multi-Armed Bandit model with correlated rewards[C]// 2016 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2016.

[69]Mahler J, Liang J, Niyaz S, et al. Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics[J]. 2017.

[70]Mahler J, Matl M, Liu X, et al. Dex-Net 3.0: Computing Robust Vacuum Suction Grasp Targets in Point Clouds Using a New Analytic Model and Deep Learning[C]// 2018:1-8.

[71]Lenz I, Lee H, Saxena A. Deep Learning for Detecting Robotic Grasps[J]. The International Journal of Robotics Research, 2013, 34(4-5).

[72]Chu F J, Xu R, Patricio V. Real-World Multiobject, Multigrasp Detection[J]. IEEE Robotics and Automation Letters, 2018, 3:3355-3362.

[73]Weng Y, Sun Y, Jiang D, et al. Enhancement of real-time grasp detection by cascaded deep convolutional neural networks[J]. Concurrency and Computation: Practice and Experience, 2020.

[74]Liu D, Tao X, Yuan L, et al. Robotic objects detection and grasping in clutter based on cascaded deep convolutional neural network[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 71: 1-10.

[75]孙先涛，程伟，陈文杰，等. 基于深度学习的视觉检测及抓取方法[J/OL]. 北京航空航天大学学报：1-13[2023-03-01].

[76]楚红雨，冷齐齐，张晓强，等. 融入注意力机制的多模特征机械臂抓取位姿检测[J/OL]. 控制与决策：1-9[2023-03-03].

[77]Fu J, Liu J, Tian H, et al. Dual attention network for scene segmentation[C]// Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 3146-3154.

[78]Howard A G, Zhu M, Chen B, et al. Mobilenets:Efficient convolutional neural networks for mobile vision applications[J]. arXiv: 1704.04861, 2017.

[79]Heusel M, Ramsauer H, Unterthiner T, et al. Gans trained by a two time-scale update rule converge to a local nash equilibrium[J]. Advances in neural information processing systems, 2017, 30.

[80]Salimans T, Goodfellow I, Zaremba W, et al. Improved techniques for training gans[J]. Advances in neural information processing systems, 2016, 29.

[81]Lin T Y, Maire M, Belongie S, et al. Microsoft coco: Common objects in context[C]// European conference on computer vision. Springer, Cham, 2014: 740-755.

[82]Tian Z, Shen C, Chen H, et al. Fcos: Fully convolutional one-stage object detection[C]// Proceedings of the IEEE/CVF international conference on computer vision. 2019: 9627-9636.

[83]He K , Zhang X , Ren S , et al. Deep Residual Learning for Image Recognition[J]. IEEE, 2016.

[84]Lin T Y, Dollár P, Girshick R, et al. Feature pyramid networks for object detection[C]// Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 2117-2125.

[85]ZHU X, HU H, LIN S, et al. Deformable convnets v2: More deformable, better results[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 9308-9316.

[86]TAN M, PANG R, LE Q V. EfficientDet: Scalable and Efficient Object Detection[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020.

[87]方鹏，郝宏运，李腾飞，等. 基于注意力机制和可变形卷积的鸡只图像实例分割提取[J]. 农业机械学报，2021, 52(4): 257-265.

[88]LIU S, QI L, QIN H, et al. Path Aggregation Network for Instance Segmentation[J]. IEEE, 2018.

[89]张胜虎，马惠敏. 遮挡对于目标检测的影响分析[J]. 图学学报，2020, 41(06): 891-896.

[90]WANG X L, XIAO T T, JIANG Y N, et al. Repulsion loss: detecting pedestrians in a crowd[C]// Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 7774-7783.

[91]邹梓吟，盖绍彦，达飞鹏，等. 基于注意力机制的遮挡行人检测算法[J]. 光学学报，2021, 41(15): 157-165.

[92]Liu J, Zhao Y, Jia W, et al. DLNet: Accurate segmentation of green fruit in obscured environments[J]. Journal of King Saud University-Computer and Information Sciences, 2022, 34(9): 7259-7270.

[93]Woo S, Park J, Lee J Y, et al. Cbam: Convolutional block attention module[C]// Proceedings of the European conference on computer vision (ECCV), Cham: Springer, 2018: 3-19.

[94]Li G, Fang Q, Zha L, et al. HAM: Hybrid attention module in deep convolutional neural networks for image classification[J]. Pattern Recognition, 2022, 129: 108785.

[95]Wang Q, Wu B, Zhu P, et al. ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020.

[96]Jie H, Li S, Gang S. Squeeze-and-Excitation Networks[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2018

[97]程鹏飞，闫浩文，韩振辉. 一个求解多边形最小面积外接矩形的算法[J]. 工程图学学报，2008(01): 122-126.

[98]邓仕超，黄寅. 二值图像膨胀腐蚀的快速算法[J]. 计算机工程与应用, 2017, 53(05): 207-211.

[99]周立君，刘宇，白璐，等. 使用TensorRT进行深度学习推理[J]. 应用光学，2020, 41(02): 337-341.