在我国，煤炭不仅是主体能源，也是工业生产的关键原料。随着国家对“双碳”目标的推进，煤炭产业正迈向以绿色开采和低碳利用为核心的高质量发展阶段。在此过程中，自动化与智能化技术在煤炭开采及运输中日益成为主流趋势。特别在提升煤炭运输自动化水平方面，煤炭异物分拣机器人的研究成为一个焦点领域，其中，基于机器视觉的检测技术是该领域的核心技术之一。目前，煤炭异物检测主要存在异物数据集质量差、复杂特征下目标检测精度低、抓取特征提取结果误差大等问题。为解决以上问题，本文从异物图像数据扩增、高精度异物检测模型搭建、异物抓取特征提取三个角度展开深入研究，以期提高煤炭异物检测算法的鲁棒性和适应性，为分拣机器人提供核心技术支撑。主要工作内容如下：

针对煤炭异物检测中RGB-D样本量少及样本不平衡导致的模型特征提取困难和过拟合问题，研究基于组合数据的RGB-D图像数据集扩增方法。通过构建高质量的单类别RGB-D异物库和背景数据库、并引入随机分布点来合成带有标注文件的煤炭异物RGB-D图像，从而大幅度提高了数据的多样性。实验结果表明，利用本文方法进行数据扩增后的数据集训练异物检测模型，能有效提升检测模型的性能。

针对煤流中夹杂的异物对比度低、相互遮挡、异物图像缺乏目标空间与边缘等信息导致异物检测识别率低、定位误差大的问题，研究基于双金字塔网络的RGB-D煤炭异物检测方法。首先，通过引入 Depth 图像构建 RGB 图像与 Depth 图像的双特征金字塔网络，丰富异物特征的空间与边缘信息，提高检测精度；其次，提出特征层模态融合模块CAFM，以协同优化并融合 RGB 特征与 Depth 特征，增强网络对特征图中被遮挡异物可见部分的关注度，提高被遮挡异物检测精度；最后，使用双阶段检测头结构完成对煤炭异物的分类、回归与分割。实验结果表明，该方法平均分割精度为82.2%，平均检测时间为110.5ms，符合异物检测实时性与准确性要求。

针对传统位姿提取方法缺乏空间信息导致难以提供有效抓取位姿信息，研究基于点云的异物三维抓取位姿检测方法。首先，根据RGB、Depth、分割掩膜图像生成异物三维点云，通过点云包围盒算法与机械手几何尺寸确定异物的空间抓取位姿。针对缺乏抓取优先级序列导致的抓取位姿信息失效的问题，研究基于混合逻辑分析的异物抓取序列提取方法，通过读取异物深度图来确定高低排序，通过碰撞算法来确定异物遮挡情况来综合对煤炭异物的抓取优先级排序，获取煤炭异物的抓取序列。通过异物的空间抓取位姿与抓取序列获取合理的抓取特征，为异物检测模型的工业化部署奠定基础。

针对煤炭异物分拣机器人平台工业应用的需求，开发煤炭异物的视觉检测系统，实现对带式输送机上煤炭异物的图像采集、实时检测、信息传输、图像存储与结果可视化等功能。并在异物分拣机器人平台上进行静态及动态检测实验，实验结果表明，检测系统准确率与抓取特征结果取得良好的效果，系统响应平均时间为210ms，满足中低带速下检测时间要求，证明了所研究方法的可行性。

In China, coal is not only the main energy source but also vital for industrial production. As the country progresses towards its "dual carbon" objectives, the coal sector is evolving towards high-quality development characterized by green mining and low-carbon utilization. In this transition, automation and intelligent technology are becoming increasingly prevalent in coal extraction and transportation. Enhancing coal transport automation, research on sorting robots employing machine vision detection is a key focus. Challenges include dataset quality, precision in complex feature detection, and grasping result errors. This paper deeply investigates data augmentation, high-precision model construction, and pose sequence extraction to enhance coal foreign object detection robustness and adaptability, supporting robotic sorting technologically.The main contributions of this work are as follows:

Addressing the challenges of model feature extraction difficulty and overfitting caused by the scarcity and imbalance of RGB-D samples in coal foreign object detection, this study explores a method for augmenting RGB-D image datasets based on combined data. By constructing a high-quality single-category RGB-D foreign object library and background database, and introducing randomly distributed points to synthesize annotated coal foreign object RGB-D images, the diversity of the data is significantly enhanced. Experimental results demonstrate that training a foreign object detection model with the dataset augmented by the proposed method can effectively improve the performance of the detection model.

In response to the issues of low contrast, mutual occlusion, and lack of spatial and edge information in foreign object images within the coal stream, which lead to a low recognition rate and large positioning errors in foreign object detection, this research investigates an RGB-D coal foreign object detection method based on a dual pyramid network. Initially, by incorporating Depth images, a dual-feature pyramid network for RGB and Depth images is constructed to enrich the spatial and edge information of foreign object features, thereby improving detection accuracy. Subsequently, a feature layer modality fusion module (CAFM) is proposed to synergistically optimize and fuse RGB features with Depth features, enhancing the network's focus on the visible parts of occluded foreign objects and improving the detection precision of occluded objects. Finally, a two-stage detection head structure is employed to accomplish the classification, regression, and segmentation of coal foreign objects. Experimental results indicate that this method achieves an average segmentation precision of 82.2% with an average detection time of 110.5ms, meeting the requirements for real-time and accurate foreign object detection.

Addressing the issue where traditional pose extraction methods lack spatial information, making it difficult to provide effective grasping pose information, this study investigates a method for detecting three-dimensional grasping poses of foreign objects based on point clouds. Initially, a three-dimensional point cloud of the foreign object is generated from RGB, Depth, and segmentation mask images. The spatial grasping pose of the foreign object is determined by a point cloud bounding box algorithm in conjunction with the geometric dimensions of the robotic manipulator. To address the problem of invalidated grasping pose information due to the lack of a grasping priority sequence, a method for extracting the grasping sequence of foreign objects based on hybrid logical analysis is investigated. The depth map of the foreign object is read to determine the order of height, and a collision algorithm is used to determine the occlusion of the foreign object, thereby comprehensively sorting the grasping priority of coal foreign objects and obtaining the grasping sequence. By obtaining the spatial grasping pose and grasping sequence of the foreign objects, reasonable grasping features are acquired, laying the foundation for the industrial deployment of the foreign object detection model.

Based on the application requirements of the coal foreign object sorting robot platform, a coal foreign object visual detection system is designed, enabling functions such as image capture, real-time detection, information transmission, image storage, and result visualization of coal foreign objects on belt conveyors. Static and dynamic detection experiments are conducted on the foreign object sorting robot platform. The static detection experiment results show that the detection system's accuracy and grasping pose outcomes are effective, with an average system response time of 210ms, meeting the detection time requirements at medium and low belt speeds. The dynamic detection experiment results indicate that with a belt speed of less than 1m/s, the system can ensure a detection accuracy rate of over 80%, demonstrating the feasibility of the researched method.

[1]刘峰,郭林峰,赵路正.双碳背景下煤炭安全区间与绿色低碳技术路径[J].煤炭学报,2022,47(1):1-15.

[2]陈浮,王思遥,于昊辰,等.碳中和目标下煤炭变革的技术路径[J].煤炭学报,2022,47(04):1452-1461.

[3]刘峰,曹文君,张建明,等.我国煤炭工业科技创新进展及“十四五”发展方向[J].煤炭学报,2021,46(1) 1-15.

[4]卫小芳,王建国,丁云杰.煤炭清洁高效转化技术进展及发展趋势[J].中国科学院院刊, 2019, 34(04): 409-416.

[5]王国法,赵国瑞,任怀伟.智慧煤矿与智能化开采关键核心技术分析[J].煤炭学报, 2019, 44(01): 34-41.

[6]王国法,庞义辉,任怀伟,等.智慧矿山系统工程及关键技术研究与实践[J].煤炭学报,2024,49(01):181-202.

[7]曹现刚,刘思颖,王鹏,等.面向煤矸分拣机器人的煤矸识别定位系统研究[J].煤炭科学技术, 2022, 50(01): 237-246.

[8]Li M, Duan Y, He X, et al. Image positioning and identification method and system for coal and gangue sorting robot[J]. International Journal of Coal Preparation and Utilization, 2022, 42(6): 1759-1777.

[9]杨晨光,冯岸岸,朱金波,等.智能分选中煤矸X射线识别技术的研究[J].安徽化工,2020,46(03):25-29+33.

[10]李曼,段雍,曹现刚,等.煤矸分选机器人图像识别方法和系统[J].煤炭学报,2020, 45(10):3636-3644.

[11]赵跃民,张亚东,周恩会,等.清洁高效干法选煤研究进展与展望[J].中国矿业大学学报,2022,51(03):607-616.

[12]葛世荣,郝尚清,张世洪，等.我国智能化采煤技术现状及待突破关键技术[J].煤炭科学技术,2020,48(07):28-46.

[13]曹现刚,吴旭东,王鹏,等. 面向煤矸分拣机器人的多机械臂协同策略[J]. 煤炭学报, 2019,44(02), 763-774.

[14]曹现刚,郝朋英,王鹏,等.多因素光照条件下高质量煤矸图像获取方法研究[J].煤炭科学技术,2023,51(01):455-463.

[15]Cao X, Wei H, Wang P, et al. High quality coal foreign object image generation method based on StyleGAN-DSAD[J]. Sensors, 2022, 23(1): 374.

[16]宋晓茹.基于ARM和CPLD的煤矸石在线自动分选系统研究[D].西安:西安科技大学,2007.

[17]曹现刚,李莹,王鹏,等.煤矸石识别方法研究现状与展望[J].工矿自动化,2020,46(01):38-43.

[18]邬冠华,熊鸿建.中国射线检测技术现状及研究进展[J].仪器仪表学报,2016,37(08): 1683-1695

[19]孔力,李红,徐恕宏,等.双能γ射线透射法煤矸石在线识别与分选系统[J].华中理工大学学报,1997,(10):108-109+113.

[20]Watt J S, Steffner E J. Dual energy gamma-ray transmission techniques applied to on-line analysis in the coal and mineral industries[J]. The International Journal of Applied Radiation and Isotopes, 1985, 36(11): 867-877.

[21]袁华昕.基于X射线图像的煤矸石智能分选控制系统研究[D].沈阳:东北大学,2016.

[22]何晓明.基于X射线的煤与矸石自动识别方法研究[D].沈阳:东北大学,2013.

[23]余长军.基于X射线的选煤厂块煤和块矸分选系统的研究[D].淮南:安徽理工大学,2017.

[24]刘富强,钱建生,王新红,等.基于图像处理与识别技术的煤矿矸石自动分选[J].煤炭学报, 2000, 25(5):537-537．

[25]于国防,邹士威,秦聪.图像灰度信息在煤矸石自动分选中的应用研究[J].工矿自动化, 2012, 38(2):36-39.

[26]王家臣,李良晖,杨胜利.不同照度下煤矸图像灰度及纹理特征提取的实验研究[J].煤炭学报, 2018, 43(11):3051-3061.

[27]Wang Y, Wang Y, Dang L. Video detection of foreign objects on the surface of belt conveyor underground coal mine based on improved SSD[J]. Journal of Ambient Intelligence and Humanized Computing, 2023: 1-10.

[28]郝帅,张旭,马旭,等.基于CBAM-YOLOv5的煤矿输送带异物检测[J].煤炭学报,2022,47(11):4147-4156.

[29]Zhang K, Wang W, Lv Z, et al. Computer vision detection of foreign objects in coal processing using attention CNN[J]. Engineering Applications of Artificial Intelligence, 2021, 102: 104242.

[30]任志玲,朱彦存.改进CenterNet算法的煤矿皮带运输异物识别研究[J].控制工程,2023,30(04):703-711.

[31]程德强,徐进洋,寇旗旗,等.融合残差信息轻量级网络的运煤皮带异物分类[J].煤炭学报,2022,47(3):1361-1369.

[32]Zhong Z, Zheng L, Kang G, et al. Random erasing data augmentation[C]//Proceedings of the AAAI conference on artificial intelligence. 2020, 34(07): 13001-13008.

[33]Liang D, Yang F, Zhang T, et al. Understanding mixup training methods[J]. IEEE access, 2018, 6: 58774-58783.

[34]Rok B,Lusa L.SMOTEF or High-Dimensional Class-Imbalanced Data[J].BMC Bioinformatics, 2013, 14(1): 106-121.

[35]Inoue H.Data Augmentation By Pairing Samples For Images Classification[J].Arxiv Preprint Arxiv:1801.02929, 2018.

[36]Zhang H, Cisse M, Dauphin Y N, et al. Mixup: Beyond Empirical Risk Minimization[J]. Arxiv Preprint Arxiv:1710.09412, 2017.

[37]Cubuk E D, Zoph B, Mane D, et al. Autoaugment: Learning augmentation policies from data[J]. arXiv preprint arXiv:1805.09501, 2018.

[38]Kusner M J, Paige B, Hernández-Lobato J M. Grammar variational autoencoder[C]//International conference on machine learning. PMLR, 2017: 1945-1954.

[39]Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative Adversarial Nets[J]. Advances In Neural Information Processing Systems, 2014:2672–2680.

[40]胡璟皓,高妍,张红娟,等.基于深度学习的带式输送机非煤异物识别方法[J].工矿自动化,2021,47(06):57-62+90.

[41]李曼,杨茂林,刘长岳,等.基于图像的煤矸分选中图像照度调节方法[J].煤炭学报,2021,46(S2):1149-1158.

[42]王星,高峰,陈吉,等.基于GAN网络的煤岩图像样本生成方法[J].煤炭学报,2021,46(09):3066-3078.

[43]Wang L,Wang X,Li B. A Data Expansion Strategy For Improving Coal-Gangue Detection[J]. International Journal Of Coal Preparation And Utilization, 2022:1-19.

[44]Ma J, Ma Y, Li C.Infrared And Visible Image Fusion Methods And Applications: A Survey[J].Information Fusion, 2019:153-178.

[45]He Y, Chiu W C, Keuper M, et al. Std2p: Rgbd semantic segmentation using spatio-temporal data-driven pooling[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 4837-4846.

[46]Qu L, He S, Zhang J, et al. RGBD salient object detection via deep fusion[J]. IEEE transactions on image processing, 2017, 26(5): 2274-2285.

[47]高云,廖慧敏,黎煊,等.基于双金字塔网络的RGB-D群猪图像分割方法[J].农业机械学报,2020,51(07):36-43.

[48]Zhou H, Qi L, Wan Z, et al. RGB-D co-attention network for semantic segmentation[C]//Proceedings of the Asian conference on computer vision. 2020.

[49]汪丹丹,张旭东,范之国,等.基于RGB-D的反向融合实例分割算法[J].图学学报,2021,42(05):767-774.

[50]Gupta S, Girshick R, Arbeláez P, et al. Learning rich features from RGB-D images for object detection and segmentation[C]//Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VII 13. Springer International Publishing, 2014: 345-360.

[51]Cheng Y, Cai R, Li Z, et al. Locality-sensitive deconvolution networks with gated fusion for RGB-D indoor semantic segmentation[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 3029-3037.

[52]Lenz I, Lee H, Saxena A. Deep learning for detecting robotic grasps[J]. The International Journal of Robotics Research, 2015, 34(4-5): 705-724.

[53]Nguyen V D. Constructing stable force-closure grasps[C]//Proceedings of 1986 ACM Fall joint computer conference. 1986: 129-137.

[54]Faverjon B, Ponce J. On computing two-finger force-closure grasps of curved 2d objects[C]//Proceedings. 1991 IEEE International Conference on Robotics and Automation. IEEE, 1991: 424-429.

[55]Dizioğlu B, Lakshiminarayana K. Mechanics of form closure[J]. Acta mechanica, 1984, 52(1): 107-118.

[56]Miller A T, Knoop S, Christensen H I, et al. Automatic grasp planning using shape primitives[C]//2003 IEEE International Conference on Robotics and Automation (Cat. No. 03CH37422). IEEE, 2003, 2: 1824-1829.

[57]Mason M T, Salisbury Jr J K. Robot hands and the mechanics of manipulation[J]. 1985. .1134

[58]Bicchi A, Kumar V. Robotic grasping and contact: A review[C]//Proceedings 2000 ICRA. Millennium conference. IEEE international conference on robotics and automation. Symposia proceedings (Cat. No. 00CH37065). IEEE, 2000, 1: 348-353.

[59]Wang D, Liu C, Chang F, et al. High-performance pixel-level grasp detection based on adaptive grasping and grasp-aware network[J]. IEEE transactions on industrial electronics, 2021, 69(11): 11611-11621.

[60]Laili Y, Chen Z, Ren L, et al. Custom grasping: A region-based robotic grasping detection method in industrial cyber-physical systems[J]. IEEE Transactions on Automation Science and Engineering, 2022, 20(1): 88-100.

[61]Yu J, Zhou M, Gong D, et al. 6-DOF grasping pose detection method incorporating instance segmentation[C]//2023 IEEE International Conference on Real-time Computing and Robotics (RCAR). IEEE, 2023: 959-964.

[62]张千.多评价因素大视场下并联机器人堆叠串类水果抓取位姿检测研究[D].镇江:江苏大学,2021.

[63]郭慧,沈霞,王勇.智能获取装箱管状工件抓取位置的研究[J].图学学报,2015,36(3):452-456.

[64]Jabalameli A, Behal A. From single 2D depth image to gripper 6D pose estimation: A fast and robust algorithm for grabbing objects in cluttered scenes[J]. Robotics, 2019, 8(3): 63.

[65]颜培清,何炳蔚,雷阿唐,等.基于深度信息的多目标抓取规划方法研究[J].电子测量与仪器学报, 2016, 30(9):1342-1350.

[66]Keisuke M,Kawasaki S,Hiroshi M.Object grasping control method and apparatus[P]. United States:US2011/00741.71A1, 2011.03.31.

[67]Kong C,Wang S,Wang Y,et al. Application of AHP-FCA modeling in visual guided manipulator[C]//2017 2nd International Conference on Robotics and Automation Engineering (ICRAE). IEEE, 2017: 121-125.

[68]Valero E,Antonio AdÁN.Grasp Registration And Learning In Virtual Reality Environments[C]// WSEAS International Conference On Computational Intelligence,Man-Machine Systems And Cybernetics,2009,40-47.

[69]Zhang H, Lan X, Zhou X, et al. Visual manipulation relationship network for autonomous robotics[C]//2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids). IEEE, 2018: 118-125.

[70]曹现刚,刘思颖,王鹏,等.面向煤矸分拣机器人的煤矸识别定位系统研究[J].煤炭科学技术,2022,50(01):237-246.

[71]曹现刚,费佳浩,王鹏,等.基于多机械臂协同的煤矸分拣方法研究[J]. 煤炭科学技术, 2019, 47(04): 7-12

[72]Suzuki S. Topological structural analysis of digitized binary images by border following[J]. Computer vision, graphics, and image processing, 1985, 30(1): 32-46.

[73]朱云博,冯广斌,孙华刚,等.基于数学形态学的振动信号降噪和解调方法研究[J].机械科学与技术,2012,31(08):1261-1264.

[74]Ren S, He K, Girshick R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks[J]. IEEE transactions on pattern analysis and machine intelligence, 2016, 39(6): 1137-1149.

[75]He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778.

[76]Ma J, Ma Y, Li C. Infrared and visible image fusion methods and applications: A survey[J]. Information fusion, 2019, 45: 153-178.

[77]Ramachandram D, Taylor G W. Deep multimodal learning: A survey on recent advances and trends[J]. IEEE signal processing magazine, 2017, 34(6): 96-108.

[78]Hou Q, Zhou D, Feng J. Coordinate attention for efficient mobile network design[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 13713-13722.

[79]Wang X, Zhang R, Kong T, et al. Solov2: Dynamic and fast instance segmentation[J]. Advances in Neural information processing systems, 2020, 33: 17721-17732.

[80]Chen H, Sun K, Tian Z, et al. Blendmask: Top-down meets bottom-up for instance segmentation[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 8573-8581.

[81]He K, Gkioxari G, Dollár P, et al. Mask r-cnn[C]//Proceedings of the IEEE international conference on computer vision. 2017: 2961-2969.

[82]Huang Z, Huang L, Gong Y, et al. Mask scoring r-cnn[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 6409-6418.

[83]Ke L, Danelljan M, Li X, et al. Mask transfiner for high-quality instance segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 4412-4421.

[84]李钰龙,梁新武.融合注意力机制和多任务学习的机器人抓取检测算法[J].哈尔滨工业大学学报,2023,55(12):9-17.

[85]Jiang Y, Moseson S, Saxena A. Efficient grasping from rgbd images: Learning using a new rectangle representation[C]//2011 IEEE International conference on robotics and automation. IEEE, 2011: 3304-3311.

[86]Lenz I, Lee H, Saxena A. Deep learning for detecting robotic grasps[J]. The International Journal of Robotics Research, 2015, 34(4-5): 705-724.

[87]于瑞云,赵金龙,余龙,等.结合轴对齐包围盒和空间划分的碰撞检测算法[J].中国图象图形学报,2018,23(12):1925-1937.

[88]王伟,马峻,刘伟.基于OBB包围盒的碰撞检测研究与应用[J].计算机仿真,2009. 26(09):180-183+312.

[89]杨帆.基于B+树存储的AABB包围盒碰撞检测算法[J].计算机科学,2021,48(S1):331-333,348.