目前国内在煤矿智能化发展方面“采掘失衡”问题严重，掘进工作面智能化程度较低，巷道掘进施工中仍需要人工在本地对设备进行控制。煤矿井下环境恶劣，粉尘浓度较大，人工操作方式易造成超挖、欠挖且存在很多安全隐患。因此，实现掘进设备的智能化控制决策对推动智慧矿山建设具有重要意义。

论文针对掘进设备远程控制中存在的设备决策能力低，掘进效率不高，安全隐患大等问题，提出了一种数字孪生驱动的掘进设备控制决策方法，结合数字孪生技术、虚拟现实技术、深度强化学习技术，将掘进工作面及设备的虚拟空间与物理空间进行有机融合，使设备虚拟样机具备自主决策能力，在虚拟空间中完成碰撞检测、局部避障与路径规划，并将决策指令发送至物理空间，实现掘进设备的自主决策与远程控制。

针对掘进设备运行中存在的避让落煤、干涉设备、禁行区域等局部避障问题，提出了非结构化环境下的掘进设备局部避障策略。利用激光雷达将设备行进过程中的障碍物重建在虚拟环境中，通过Ray-Col碰撞检测方法对设备进行碰撞检测，根据碰撞检测结果执行避障行为，并在Unity3D中对该策略进行仿真验证，为掘进设备的路径规划决策奠定基础。

针对掘进设备在巷道中路径规划难度大的问题，提出了基于虚拟智能体的掘进设备全局路径规划方法。结合深度强化学习技术，以Markov决策过程为理论基础，在Critic-Actor学习框架下对传统PPO算法进行改进，通过奖惩机制建立基于Muti-PPO算法的掘进设备虚拟智能体，设计智能体的动作空间与状态空间，实现掘进设备自主规划决策。将其与PPO、SAC算法进行仿真对比，结果表明Muti-PPO算法的鲁棒性在三种工况下均达到最优。

针对掘进设备人机交互与远程控制效率低的问题，提出了“数据驱动、双向映射、碰撞检测、自主决策、人机协作”的远程控制策略。构建掘进设备控制决策数字孪生体模型，通过“虚拟空间样机建立、物理空间状态感知、虚实数据交互”将井下空间映射至数字化虚拟空间中。基于Unity3D开发人机交互平台，通过虚拟样机远程控制物理样机，同时利用物理样机传感器数据驱动虚拟样机同步变化，以此循环实现以设备自主决策为主，人工远程干预为辅的掘进设备控制决策。

最后搭建系统实验平台，分别对系统通讯性能、虚实同步运动性能、碰撞检测与局部避障功能、全局路径决策规划功能进行测试与验证。实验结果表明，系统通讯性能良好，能够将障碍物在虚拟空间中进行重建，并实现设备的碰撞检测与局部避障，在此基础上，智能体能够进行路径规划，并自主对物理空间下发控制决策指令。在系统运行过程中，虚实同步运动性能良好，误差精度满足井下施工要求。该研究为井下掘进设备智能化控制提供了新的思路。

At present, the problem of 'unbalanced development of coal mining and tunneling technology' in the intelligent development of coal mining in China is serious. The intelligent degree of tunneling face is low, and manual control of equipment is still required in roadway tunneling construction. Coal mine underground environment is bad, dust concentration is large, so manual operation is easy to cause overbreak, underbreak and there are many security risks. Therefore, the intelligent control decision of tunneling equipment is of great significance to promote the construction of ' smart mine '.

Aiming at the problems of low decision-making ability, low tunneling efficiency and large security risks in remote control of tunneling equipment, this paper studies a control decision method of tunneling equipment driven by digital twin. Combined with digital twin technology, virtual reality technology and deep reinforcement learning technology, the virtual space and physical space of excavation face and equipment are organically integrated, so that the virtual prototype of equipment has the ability of independent decision-making. It can complete collision detection, local obstacle avoidance and path planning in virtual space, and send decision instructions to physical space to realize autonomous planning and remote control of tunneling equipment.

In view of the local obstacle avoidance problems existing in tunneling roadway, such as avoiding coal falling, interference equipment, forbidden area and so on, the local obstacle avoidance strategy of tunneling equipment in unstructured environment is proposed. The laser radar is used to reconstruct the obstacles in the process of equipment running in the virtual environment. The Ray-Col collision detection method is used to detect the collision of the equipment, and the obstacle avoidance action is carried out according to the collision detection results. The strategy is simulated and verified in the Unity3D, which lays the foundation for the path planning decision of the tunneling equipment.

Aiming at the difficulty of path planning of tunneling equipment in roadway, a global path planning method of tunneling equipment based on virtual agent is proposed. Combined with deep reinforcement learning technology and based on Markov decision-making process, the traditional PPO algorithm is improved under the framework of Critic-Actor learning. The virtual agent of tunneling equipment based on Muti-PPO algorithm is established by reward and punishment mechanism, and the action space and state space of the agent are designed to realize the autonomous planning and decision of tunneling equipment. Simulation comparison with PPO and SAC algorithm，the simulation results show that the robustness of Muti-PPO algorithm is optimal under three working conditions.

Aiming at the low efficiency of human-computer interaction and remote control of tunneling equipment, a remote control strategy of ‘ data-driven, bidirectional mapping, collision detection, autonomous decision-making, and human-computer cooperation ’ is proposed. the digital twin model of equipment control decision is constructed, the underground space is mapped to the digital virtual space through "establishment of virtual space prototype, perception of physical space state and interaction of virtual and real data". Human-computer interaction platform is developed based on Unity3D, The remote control of the physical prototype is completed by controlling the virtual prototype. The sensor data of the physical prototype is used to drive the synchronous change of the virtual prototype, this cycle realizes the remote control based on equipment independent decision-making and supplemented by manual intervention.

Finally, a system experimental platform is built to test and verify the performance of system communication, virtual-real synchronous motion, collision detection and local obstacle avoidance , global path decision planning. The experimental results show that the communication performance of the system is good, the obstacles can be reconstructed in the virtual space, and the collision detection and local obstacle avoidance of the equipment can be realized. On this basis, the tunneling equipment agent can independently plan the path and realize the autonomous decision control of the physical space. In the process of system operation, the virtual and real synchronous motion performance is good, and the error accuracy meets the requirements of underground construction. This study provides a new idea for intelligent control of underground tunneling equipment.

[1] 国家能源集团.《国家能源集团煤矿智能化建设指南（试行）》[Z]. 2021-02-09.

[2] 王虹, 王步康, 张小峰, 等. 煤矿智能快掘关键技术与工程实践[J]. 煤炭学报, 2021, 46(7): 2068-2083.

[3] 张旭辉, 杨文娟, 薛旭升, 等. 煤矿远程智能掘进面临的挑战与研究进展[J]. 煤炭学报, 2022, 47(01): 579-597.

[4] 丁恩杰, 俞啸, 夏冰, 等. 矿山信息化发展及以数字孪生为核心的智慧矿山关键技术[J]. 煤炭学报, 2022, 47(01): 564-578.

[5] 李浩, 王昊琪, 刘根, 等. 工业数字孪生系统的概念、系统结构与运行模式[J]. 计算机集成制造系统, 2021, 27(12): 3373-3390.

[6] 陶飞, 张辰源, 张贺, 等. 未来装备探索: 数字孪生装备[J]. 计算机集成制造系统, 2022, 28(01): 1-16.

[7] Wang YC, Tao F, Zhang M, et al. Digital twin enhanced fault prediction for the autoclave with insufficient data[J]. Journal of Manufacturing Systems, 2021, 60(2021): 350-359.

[8] 陶飞, 张贺, 戚庆林, 等. 数字孪生模型构建理论及应用[J]. 计算机集成制造系统, 2021, 27(01): 1-15.

[9] 葛世荣, 张帆, 王世博, 等. 数字孪生智采工作面技术架构研究[J]. 煤炭学报, 2020, 45(06): 1925-1936.

[10] 王岩, 张旭辉, 曹现刚, 等. 掘进工作面数字孪生体构建与平行智能控制方法研究[J/OL]. 煤炭学报: 1-12[2022-03-17]. https://kns.cnki.net/kcms/detail/11.2190.TD.20220308.1050.0 02.html

[11] Wang F Y, Zheng N N, Cao D P, et al. Parallel driving in CPSS: A unified approach for transport automation and vehicle intelligence[J]. IEEE/CAA Journal of Automatica Sinica, 2017, 4(4): 577-587.

[12] 张旭辉, 张超, 王妙云, 等. 数字孪生驱动的悬臂式掘进机虚拟操控技术[J]. 计算机集成制造系统, 2021, 27(06): 1617-1628.

[13] 王妙云, 张旭辉, 马宏伟, 等. 远程控制综采设备碰撞检测与预警方法[J]. 煤炭科学技术, 2021, 49(09): 110-116.

[14] 张旭辉, 魏倩楠, 王妙云, 等. 悬臂式掘进机远程虚拟操控系统研究[J]. 煤炭科学技术, 2020, 48(11): 44-51.

[15] 张旭辉, 董润霖, 马宏伟, 等. 基于虚拟现实的煤矿救援机器人远程控制技术[J]. 煤炭科学技术, 2017, 45(05): 52-57.

[16] 刘大同, 郭凯, 王本宽, 等. 数字孪生技术综述与展望[J]. 仪器仪表学报, 2018, 39(11): 1-10.

[17] 戴晟, 赵罡, 于勇, 等. 数字化产品定义发展趋势: 从样机到孪生[J]. 计算机辅助设计与图形学报, 2018, 30(08): 1554-1562.

[18] 杜莹莹, 罗映, 彭义兵, 等. 基于数字孪生的工业机器人三维可视化监控[J/OL]. 计算机集成制造系统: 1-15[2022-03-17]. http://kns.cnki.net/kcms/detail/11.5946.TP.20220110.1032.002.html

[19] Lu Q C, Parliksd A K, Woodall P, et al. Developing a digital twin at building and city levels: case study of West Cambridge campus[J]. Journal of Management in Enginneering, 2020, 36(3): 0502004.

[20] Zhou M, Yan J, Feng D. Digital twin framework and its application to power grid online analysis[J]. CSEE Journal of Power and Energy Systems, 2019, 5(3): 391-398.

[21] Liu Q, Zhang H, Leng J W, et al. Digital twin-driven rapid individualized designing of automated flow-shop manufacturing system[J]. International Journal of Production Research, 2019, 57(12): 3903-3919.

[22] Bao J, Guo D S, Li J, et al. The modelling and operations for the digital twin in the context of manufacturing[J]. Enterprise Information Systems, 2019, 13(4): 534-556.

[23] 陶飞, 马昕, 胡天亮, 等. 数字孪生标准体系[J]. 计算机集成制造系统, 2019, 25(10): 2405-2418.

[24] 宁振波.《智能制造的本质》[J]. 自动化博览, 2021, 38(12): 7.

[25] Liu S, Wang L, Wang X V, et al. A framework of data-driven dynamic optimisation for smart production logistics[C]//IFIP International Conference on Advances in Production Management Systems. Springer, Cham, 2020: 213-221.

[26] Gao Y, Chang D, Chen C H, et al. Design of digital twin applications in automated storage yard scheduling[J]. Advanced Engineering Informatics, 2022, 51: 101477.

[27] Zakrajsek A J, Mall S. The development and use of a digital twin model for tire touchdown health monitoring[C]//58th AIAA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference. 2017: 0863.

[28] 李娟莉, 沈宏达, 谢嘉成, 等. 基于数字孪生的综采工作面工业虚拟服务系统[J]. 计算机集成制造系统, 2021, 27(02): 445-455.

[29] 陶飞, 刘蔚然, 张 萌, 等. 数字孪生五维模型及十大领域应用[J]. 计算机集成制造系统, 2019, 25(1): 1-18.

[30] 陈杨阳, 霍振龙, 刘智伟, 等. 我国煤矿运输机器人发展趋势及关键技术[J]. 煤炭科学技术, 2020, 48(07): 233-242.

[31] 张旭辉, 王妙云, 张雨萌, 等. 数据驱动下的工业设备虚拟仿真与远程操控技术研究[J]. 重型机械, 2018(05): 14-17.

[32] Chen C, Pan Y, Li D, et al. A virtual-physical collision detection interface for AR-based interactive teaching of robot[J]. Robotics and Computer-Integrated Manufacturing, 2020, 64(2): 101948.

[33] 杜春晖. 基于多技术融合的煤矿井下采掘运输设备防碰撞系统[J]. 煤炭学报, 2020, 45(S2): 1060-1068.

[34] 黄智, 魏鹏轩, 万从保, 等. 整体叶盘磨抛加工碰撞检测方法[J]. 计算机集成制造系统, 2020, 26(12): 3350-3358.

[35] 成居宝, 杜娟, 刘丽琴, 等. 基于数控机床特性的碰撞检测算法研究[J]. 组合机床与自动化加工技术, 2020(08): 101-105+110.

[36] Chen Y, Luo G, Mei Y, et al. UAV path planning using artificial potential field method updated by optimal control theory[J]. International Journal of Systems Science,2016,47(6): 1407-1420.

[37] Krauss R. Grids[J]. October, 1979, 9: 51-64.

[38] 田劼, 银晓琦, 文艺成. 基于混合IWO—PSO算法的掘进机截割轨迹规划方法[J]. 工矿自动化, 2021, 47 (12): 55-61.

[39] 王学武, 汤彬, 顾幸生. 焊接机器人避障策略研究[J]. 机械工程学报, 2019, 55(17): 77-84.

[40] 杜广泽, 张旭东, 邹渊, 等. 非结构道路场景下轮式无人车辆避障算法[J]. 兵工学报, 2020, 41(10): 2096-2105.

[41] Aggarwal S, Kumar N. Path planning techniques for unmanned aerial vehicles: A review, solutions, and challenges[J]. Computer Communications, 2020, 149: 270-299.

[42] Miyombo M E, Liu Y, Ayodeji A. A state-aware adaptive pathfinder for dynamic minimum dose path planning during an emergency in a complex radioactive environment[J]. Progress in Nuclear Energy, 2022, 146: 104154.

[43] Wang M, Cao J. A review of collision detection for deformable objects[J]. Computer Animation and Virtual Worlds, 2021, 32(5): e1987.

[44] 闫皎洁, 张锲石, 胡希平. 基于强化学习的路径规划技术综述[J]. 计算机工程, 2021, 47(10): 16-25.

[45] Duchoň F, Babinec A, Kajan M, et al. Path planning with modified a star algorithm for a mobile robot[J]. Procedia Engineering, 2014, 96: 59-69.

[46] Kadry S, Alferov G, Fedorov V. D-Star Algorithm Modification[J]. International Journal of Online & Biomedical Engineering, 2020, 16(8).

[47] Lee B C, An S, Kim S K. Devleopment of Racing Game using NevMesh Agent[C]//Proce-edings of the Korean Society of Computer Information Conference. Korean Society of Co-mputer Information, 2019: 73-74.

[48] Ivanov S, D'yakonov A. Modern deep reinforcement learning algorithms[J]. arXiv preprint arXiv:1906.10025, 2019.

[49] Rana K, Zaveri M. A-star algorithm for energy efficient routing in wireless sensor network[J]. Trends in Network and Communications, 2011: 232-241.

[50] 刘梦杰, 朱希安, 王占刚. 基于双向A*算法的矿井水灾逃生路径应用研究[J]. 煤炭工程, 2019(9): 42-47

[51] 鲍久圣, 张牧野, 葛世荣, 等. 基于改进A*和人工势场算法的无轨胶轮车井下无人驾驶路径规划[J/OL]. 煤炭学报: 1-14[2022-03-06]. https://kns.cnki.net/kcms/detail/11.2190.TD.20220228.1312.001.html

[52] Patel D D, Lalwani D I. Effect of algorithm parameters in development of spiral tool path for machining of 2.5 D star-shaped pockets[J]. International Journal of Computer Aided Engineering and Technology, 2019, 11(6): 727-746.

[53] 朱蟋蟀, 孙兵, 朱大奇. 基于改进D*算法的AUV三维动态路径规划[J]. 控制工程, 2021, 28(04): 736-743.

[54] 黄鲁, 周非同. 基于路径优化D*Lite算法的移动机器人路径规划[J]. 控制与决策, 2020, 35(04): 877-884.

[55] Wang B. End user oriented BIM enabled multi-functional virtual environment supporting building emergency planning and evacuation[D]. Cardiff University, 2014.

[56] 董润霖. 煤矿探测机器人环境重建与虚拟操控技术研究[D]. 西安: 西安科技大学, 2018.

[57] 王妙云. 煤矿井下四旋翼无人机虚拟远程操控关键技术研究[D]. 西安: 西安科技大学, 2020.

[58] Mnih V, Kavukcuoglu K, Silver D, et al. Playing atari with deep reinforcement learning[J]. arXiv preprint arXiv: 1312.5602, 2013.

[59] Kaelbling L P, Littman M L, Moore A W. Reinforcement learning: A survey[J]. Journal of artificial intelligence research, 1996, 4: 237-285.

[60] Lis, Xu X, Zuo L. Dynamic path planning of a mobile robot with improved Q-learning algorithm[C]. 2015 IEEE international conference on information and automation, IEEE, 2015: 409-414.

[61] Bae H, Kim G, Kim J, et al. Multi-robot path planning method using reinforcement learning[J]. Applied Sciences, 2019, 9(15): 3057.

[62] 王军, 杨云霄, 李莉. 基于改进深度强化学习的移动机器人路径规划[J]. 电子测量技术, 2021, 44(22): 19-24.

[63] 成怡, 郝密密. 改进深度强化学习的室内移动机器人路径规划[J]. 计算机工程与应用, 2021, 57(21): 256-262.

[64] 胡炼, 林潮兴, 杨伟伟, 等. URG-04LX 2维激光扫描测距传感器的应用试验研究[J]. 电子设计工程, 2014, 22(14): 1-3.

[65] 姜武华, 辛鑫, 陈无畏, 等. 基于信息融合的自动泊车系统多工况车位识别和决策规划[J]. 机械工程学报, 2021, 57(06): 131-141.

[66] 赵毓, 管公顺, 郭继峰, 等. 基于多智能体强化学习的空间机械臂轨迹规划[J]. 航空学报, 2021, 42(01): 266-276.

[67] 孙世光, 兰旭光, 张翰博, 等. 基于模型的机器人强化学习研究综述[J]. 模式识别与人工智能, 2022, 35(01): 1-16.

[68] 王学宁. 策略梯度增强学习的理论、算法及应用研究[D]. 长沙: 国防科学技术大学, 2006.

[69] Schulman J, Wolski F, Dhariwal P, et al. Proximal policy optimization algorithms[J]. arXiv: 1707.06347, 2017

[70] Heess N,Sriram S,Lemmon J,et al. Emergence of locomotion behaviours in rich environments[J]. arXiv: 1707.06347, 2017.

[71] 李跃, 邵振洲, 赵振东, 等. 面向轨迹规划的深度强化学习奖励函数设计[J]. 计算机工程与应用, 2020, 56(02): 226-232.

[72] 黄东晋, 蒋晨凤, 韩凯丽. 基于深度强化学习的三维路径规划算法[J]. 计算机工程与应用, 2020, 56(15): 30-36.