随着大数据、物联网技术在工业领域中的普及，煤矿生产朝着信息化、智能化方向发展，对煤矿设备数据进行采集、存储、处理、分析能够及时、准确地反映设备运行状态，是实现煤矿设备健康状态评估、寿命预测和故障诊断的重要基础。煤矿机电设备日趋大型化、复杂化，设备数据呈现爆炸式增长，而且数据多源异构性也愈发明显，造成数据无法高效共享和使用，传统煤矿设备数据管理平台已无法实现对海量数据的存储处理，更无法满足建设智慧煤矿的需求。因此，围绕煤矿设备海量多源异构数据，深入研究设备元数据统一描述、优化设备大数据存储查询方法、构建设备大数据管理平台对智慧煤矿的建设具有重要意义。

首先对煤矿设备数据的多源异构、高时效、高动态响应、高传输以及海量特性进行分析，结合煤矿设备数据管理平台需求，提出了基于Hadoop的煤矿设备数据管理平台整体架构，对数据采集层、数据描述层、数据存储层和数据处理层四个模块进行详细设计，最终实现数据平台的可视化操作与用户间的交互。

为了实现煤矿设备大数据的高效利用和共享，首先对煤矿设备数据集进行划分，对其中的设备元数据采用RDF资源描述框架进行标准化语义描述，并结合设备数据资源实例，验证描述方法的可行性和通用性，同时构建元数据本体模型并完成本体实例映射，从而提升煤矿设备数据间的互操作性。

针对传统的存储方法无法满足海量设备RDF元数据的存储需求，提出了一种基于Hbase的RDF元数据分布式存储模型，利用Hbase的列存储、易扩展等特性完成RDF元数据的存储与加载，并对两种存储结构进行优化，从而实现RDF数据的快速查询。

通过HDFS分布式文件系统完成煤矿设备文件数据的读写，但对大量小文件数据进行存储处理的过程中会消耗过多的集群资源并带来存取效率低下等问题。通过对小文件数据存储处理方法的深入研究，提出了一种小文件合并存储优化方法，提升了小文件存取的效率，同时设计了一种小文件合并处理优化方法以提高数据处理的效率。对小文件映射数据和煤矿设备元数据构建二级索引来实现非主键查询，从而提高了集群的数据查询性能。

最后以Hadoop分布式存储系统为架构，完成煤矿设备数据管理平台的搭建，分别在集群上完成小文件数据存储处理性能测试实验和二级索引查询性能测试实验，通过对实验结果的研究分析来证明所提出的数据存储查询优化方法的可行性与高效性。

With the wide application of big data and Internet of Things technology in industrial field, coal mine production is developing in the direction of informatization and intelligence. Collecting, storing, processing and analyzing data of coal mine equipment can reflect the running state of equipment timely and accurately, which is an important basis for realizing health state assessment, life prediction and fault diagnosis of coal mine equipment.Coal mine electromechanical equipment is becoming larger and more complicated,device data is growing explosively.Moreover, the multi-source heterogeneity of data is becoming more and more obvious,which make the data can’t be shared and used efficiently.The traditional data management platform of coal mine equipment has been unable to store and process massive data,nor can it meet the needs of building smart coal mines.Therefore, focusing on the massive multi-source heterogeneous data of coal mine equipment,researching the unified description of equipment metadata in depth, optimizing the big data of equipment storage method,building the equipment data management platform are of great importance in building smart coal mine.

Firstly, the characteristics of multi-source heterogeneous, high timeliness, high dynamic response, high transmission and massive about the coal mine equipment data are analyzed.Combined with the requirements of coal mine equipment data management platform,the overall architecture of coal mine equipment data management platform based on Hadoop is proposed.It is designed from four modules: data acquisition layer, data description layer, data storage layer and data processing layer.Finally, the visual operation of the data platform and interaction with users are realized.

In order to realize efficient sharing and utilization of coal mine equipment data.Firstly, the coal mine equipment data set is divided, and the equipment metadata is described by RDF resource description framework.Combined with the equipment data resource instance,verify the generality and feasibility of the description method. At the same time, build the metadata ontology model and complete the ontology instance mapping,thus to improve the interoperability between coal mine equipment data.

In the face of traditional storage methods can’t satisfy the storage requirements of  RDF metadata about the massive device.A distributed storage model of RDF metadata which based on HBase is proposed.Using the column storage, scalability and other characteristics of HBase to complete RDF metadata storage and loading.And two storage structures are optimized to realize fast query of RDF data.

Completing the reading and writing of device file data through HDFS.However, in the process of storing and dealing a large number of small file data will consume too many cluster resources and bring some problems such as low access efficiency. Through in-depth research on the storage and processing methods of small file data.A small file merge storage optimization method is proposed to improve the efficiency of small file access.Meanwhile, an optimization method of small file merging processing is designed to improve the efficiency of data processing.A secondary index is constructed for small file mapping data and coal mine equipment metadata,which realize non primary key query and improve the data query capability of the cluster.

Finally, the Hadoop distributed storage system is used as the framework to complete the construction of the coal mine equipment data management platform.Make the performance test experiment about the small file data storage processing and the secondary index query on the cluster.Verify the feasibility and efficiency of optimization methods about data storage and query,which through the research and analysis of the experimental results.

[1] 王国法.“十四五”煤矿智能化和煤炭高质量发展的思考[J].智能矿山,2021,2(01):1-6.

[2] 中华人民共和国国务院新闻办公室. 新时代的中国能源发展[N]. 人民日报,2020-12-22(010).

[3] 尹伟华.2021年能源行业形势分析与2022年展望[J].中国物价,2022(02):16-18.

[4] 谢和平,王金华,王国法,任怀伟,刘见中,葛世荣,周宏伟,吴刚,任世华.煤炭革命新理念与煤炭科技发展构想[J].煤炭学报,2018,43(05):1187-1197.

[5] 毛善君. 煤炭工业4.0[C]//.煤矿自动化与信息化——第25届全国煤矿自动化与信息化学术会议论文集.[出版者不详],2015:74-89.

[6] 崔亚仲,白明亮,李波.智能矿山大数据关键技术与发展研究[J].煤炭科学技术,2019,47(03):66-74.

[7] 曹现刚,罗璇,张鑫媛,张树楠,吴少杰.煤矿机电设备运行状态大数据管理平台设计[J].煤炭工程,2020,52(02):22-26.

[8] 王辉.基于大数据平台的煤矿机电设备数据综合管理系统[D].中国矿业大学,2021.

[9] 王国法,王虹,任怀伟,赵国瑞,庞义辉,杜毅博,张金虎,侯刚.智慧煤矿2025情景目标和发展路径[J].煤炭学报,2018,43(02):295-305.

[10] 杜毅博,赵国瑞,巩师鑫.智能化煤矿大数据平台架构及数据处理关键技术研究[J].煤炭科学技术,2020,48(07):177-185.

[11] 李婉梅.基于分布式查询的矿山设备语义互操作系统研究[D].中国矿业大学,2020.

[12] 龚振,范冰冰.数据集的语义关联发现方法研究[J].计算机应用与软件,2018,35(08):83- 86+185.

[13] 李善青,郑彦宁,赵辉,邢晓昭.大数据背景下科学元数据的重要问题研究[J].科技管理研究,2019,39(18):184-188.

[14] Fang H, Zhao B, Zhang X W, Yang X X. A United Framework for Large-Scale Resource Description Framework Stream Processing[J]. Journal of Computer Science and Technology,2019,34(4):13.

[15] Wylot M, Hauswirth M, Philippe Cudré-Mauroux, Sakr S. RDF Data Storage and Query Processing Schemes: A Survey[J]. ACM Computing Surveys, 2018, 51(4):1-36.

[16] Kalani F A. Semantic Web: XML and RDF Roles[J]. Oriental Journal of Computer Science and Technology, 2015, 8(3):247-254.

[17] 赵鑫,王然风,付翔.基于Hadoop生态圈的选煤数据中台设计[J].工矿自动化,2021,47 (12):121-127.

[18] 王凌晖,解云月,周美华.Hadoop分布式存储架构的性能分析[J].现代电子技术,2018,41(18):92-95.

[19] Zagan E, Danubianu M. HADOOP: A Comparative Study between Single-Node and Multi-Node Cluster[J]. International Journal of Advanced Computer Science and Applications, 2021, 12(2).

[20] 金国栋,卞昊穹,陈跃国,杜小勇.HDFS存储和优化技术研究综述[J].软件学报, 2020,31(01):137-161.

[21] Ghemawat S, Gobioff H, Leung S T. The Google file system[J]. Acm Sigops Operating Systems Review, 2003, 37(5):29-43.

[22] 程学旗,靳小龙,杨婧,徐君.大数据技术进展与发展趋势[J].科技导报,2016,34(14):49- 59.

[23] 尧炜,马又良.浅析Hadoop 1.0与2.0设计原理[J].邮电设计技术,2014(07):37-42.

[24] 王鹤鸣,郑良广,杨玉钊.基于大数据平台的能耗分析与管理系统[J].机车电传动,2019(04):107-111.

[25] 李婉玉. 基于ELK的商业银行大数据检索平台的设计与实现[D].西安电子科技大学,2019.

[26] Li Y, Eldawy A, Xue J, Knorozova N, Mokbel M F, Janardan R. Scalable computational geometry in MapReduce[J]. The VLDB Journal, 2019, 28(4):523-548.

[27] 张庆. 基于Hadoop的公交物联网海量采集数据的存储平台设计[D].北京工业大学,2016.

[28] 王辉. 基于大数据平台的煤矿机电设备数据综合管理系统[D].中国矿业大学,2021.

[29] 陈少龙. 基于Hadoop的煤矿设备数据规范化和清洗的研究[D].西安科技大学,2016.

[30] Berners-Lee T, Cailliau R, Groff J F. The World-Wide Web.[J]. Computer Networks and ISDN Systems,1992,25(4-5).

[31] Siegel M D, Madnick S E. A metadata approach to resolving semantic conflicts[C]//International Conference on Very Large Data Bases. 1991:133-145.

[32] Decker S, Melnik S. The Semantic Web: the roles of XML and RDF[J]. Internet Computing IEEE, 2000, 4(5):63-73.

[33] Candan K S, Liu H, Suvarna R. Resource description framework[J]. ACM SIGKDD Explorations Newsletter,2001,3(1).

[34] Berners-Lee T, Hendler J. Publishing on the semantic web.[J]. Nature, 2001, 410 (6832):1023-4.

[35] Kim Y H, Kim B G, Lim H C. The index organizations for RDF and RDF schema[C]//Advanced Communication Technology,2006.Icact 2006. the,International Conference.IEEE.2006:4 pp.-1874.

[36] 欧石燕.面向关联数据的语义数字图书馆资源描述与组织框架设计与实现[J].中国图书馆学报,2012,38(06):58-71.

[37] 杨阳. 一种异构数据描述与转换框架的研究与实现[D].西安电子科技大学,2014.

[38] 王浩然,徐建良,张巍. 一种XML元数据的自动语义标注方法[C]//.2008'中国信息技术与应用学术论坛论文集（二）.,2008:270-272.

[39] 李孝斌,尹超,尹胜.云制造环境下机床装备资源特性分析与语义描述方法[J].计算机集成制造系统,2014,20(09):2164-2171.

[40] 石湘,刘萍.基于知识元语义描述模型的领域知识抽取与表示研究——以信息检索领域为例[J].数据分析与知识发现,2021,5(04):123-133.

[41] Wang Y, Li Y Q, Li Y W, Shi Y L, Li W W. Research of Access Optimization of Small Files on Basis of B+Tree on Hadoop[J]. Advanced Computing and Communication Technologies 2018,562:197-204.

[42] Mackey G, Selrish S, Wang J.Improving metadata management for small files in HDFS[C]. 2009 IEEE International Conference on Cluster Computing and Workshops,2009:1-4.

[43] Chandrasekar S , Dakshinamurthy R , Seshakumar P G , et al. A novel indexing scheme for efficient handling of small files in Hadoop Distributed File System[C]// International Conference on Computer Communication & Informatics. IEEE, 2013.

[44] Gohil P, Panchal B, Dhobi J S . A novel approach to improve the performance of Hadoop in handling of small files[C]// IEEE International Conference on Electrical. IEEE, 2015.

[45] 曾杨. 基于Har的HDFS小文件存储改进[D].湖南大学,2017.

[46] 邹振宇,郑烇,王嵩,杨坚.基于HDFS的云存储系统小文件优化方案[J].计算机工程,2016,42(03):34-40+46.

[47] 王伟,徐文倩,赵少锋.云存储环境下基于HDFS的海量小文件装箱算法[J].通讯世界,2016(14):285-286.

[48] 李国,李汶晓,徐俊洁.航空货运中海量小文件的存储优化[J].计算机工程与设计,2018,39(05):1484-1489.

[49] 郑通,郭卫斌,范贵生.HDFS中海量小文件合并与预取优化方法的研究[J].计算机科学,2017,44(S2):516-519+541.

[50] 马振,哈力旦·阿布都热依木,李希彤.海量样本数据集中小文件的存取优化研究[J].计算机工程与应用,2018,54(22):80-84+98.

[51] 王玉福,刘贺扬,白亮.基于地质大数据工作面智能精准开采关键技术研究[J].煤炭技术,2022,41(03):176-180.

[52] 霍俊杰.露天煤矿大型设备远程监控平台及其关键技术研究[J].工矿自动化,2021,47 (S1):91-95.

[53] 曹现刚,段欣宇,张梦园,雷卓,李彦川.煤矿设备状态监测系统设计[J].工矿自动化,2021,47(05):101-105.

[54] 高有进,杨艺,常亚军,张幸福,李国威,连东辉,崔科飞,武学艺,魏宗杰.综采工作面智能化关键技术现状与展望[J].煤炭科学技术,2021,49(08):1-22.

[55] 丁远.煤矿安全监控系统接入工业互联网关键设备研究[J].煤矿安全,2021,52(12):138-141.

[56] 郝志峰,黄泽林,蔡瑞初,傅正佳,温雯,唐凯麟.基于YARN的分布式资源动态调度与协同分配系统[J].计算机工程,2021,47(02):226-232.

[57] 刘肃平,谭志平.基于大数据的辅机设备振动噪声监测分析平台[J].计算机工程与应用,2018,54(22):258-264.

[58] 丁奕齐. 面向领域知识图谱构建的知识抽取的研究和实现[D].北京邮电大学,2021.

[59] Berners-Lee T, Dimitroyannis D, Mallinckrodt A J, McKay S. World Wide Web[J]. Computers in Physics,1998,8(3).

[60] 袁毓林,曹宏.“语义网—本体知识—知识图谱”和语言研究[J].汉语学报,2021(01):8-19.

[61] 翟社平,高山,郭琳,李兆兆.一种基于HBase的语义数据存储模型[J].计算机应用与软件,2018,35(03):15-20.

[62] 甘瀛. RDF大图数据的分布式存储方案研究[D].天津大学,2018.

[63] 王媛媛,吕晓丹,胡琪,吴鸿川.基于HBase的RDF数据存储方案研究与设计[J].信息网络安全,2016(03):59-63.

[64] 朱道恒,秦学,刘君凤.一种基于HBase的RDF数据存储改进方法[J].软件,2019,40(12):13-17.

[65] 谢莉祥. 分布式文件系统元数据存取技术研究[D].重庆交通大学,2016.

[66] Dong B, Qiu J, Zheng Q, Zhong X, Li J, Li Y. A Novel Approach to Improving the Efficiency of Storing and Accessing Small Files on Hadoop: A Case Study by PowerPoint Files[C]// IEEE. IEEE, 2010:65-72.

[67] Zhang Y, Liu D. Improving the Efficiency of Storing for Small Files in HDFS[C]// Springer Berlin Heidelberg. Springer Berlin Heidelberg, 2013.

[68] 陈芳州. 基于HDFS的云环境数据存储优化技术研究[D].南京邮电大学,2019.

[69] 洪新利. 基于MapFile的HDFS小文件存取优化研究[D].南昌大学,2017.

[70] 赵亚楠,李朝奎,肖克炎,范建福.基于Hadoop的地质矿产大数据分布式存储方法[J].地质通报,2019,38(Z1):462-470.

[71] Xu H. Research on mass monitoring data Retrieval Technology based on HBase[J]. Journal of Physics Conference Series, 2021, 1871(1):012133.