本文针对回声状态网络最优储层结构设计的难题，提出强化学习对储层神经元进行自组织筛选和重构的方法，来解决冗余信息导致储层规模与具体任务不匹配的问题，并将其应用于矿井涌水量预警中。本文主要研究内容如下： (1) 针对回声状态网络储层中存在大量冗余神经元，会导致高维状态空间矩阵共线性且进一步影响网络预测性能的问题，提出了一种基于强化学习的储层神经元筛选优 化方法。该方法基于集成学习的思想构建多个随机初始化的储层，通过互信息评估储 层内每个神经元对整体网络的贡献，并结合强化学习的决策机制，筛选出对网络输出 有效的神经元，进而达到优化网络结构，提高预测性能的目的。在仿真实验中，采用 Mackey-Glass、Lorenz和PM2.5三组时间序列数据，以初始条件100×3为例，实验结果 表明，本文提出的优化算法将储层的神经元间接的控制在20%以内，同时取得了更好 的预测结果。相比其它优化方法，该储层筛选算法不仅获得了最小储层结构，还提升 了网络性能。 (2) 针对回声状态网络储层神经元存在高耦合现象，影响网络预测性能和稳定性的 问题，提出了一种基于强化学习的回声状态网络储层自组织重构方法。该方法结合强 化学习的决策机制，并经过Lyapunov证明其机制的稳定性。同时，利用回声状态特性 和奇异值分解原理，通过状态矩阵对储层神经元进行反映射重构，实现了对储层结构 的自动调整。该方法显著增强了储层内部动力学特性，并使网络能够更好地适应输入 数据的变化，提高了预测性能和稳定性。通过对Rossler和Laser时间序列数据的仿真验 证，结果表明，本文储层重构方法在保证预测能力和稳定性的前提下，直接有效地控 制了储层规模。 (3) 针对矿井涌水量数据复杂，难以预测的问题，利用了结构优化的回声状态网络 进行预测。通过对实际矿井涌水量的验证，该模型能够有效地预测矿井涌水量，并准 确判断预警情况，为及时采取相应措施提供帮助，还为矿山安全管理提供了更可靠的 工具和方法。 经上述分析，研究实验结果充分证明了本文提出的回声状态网络结构优化方法的 有效性。该方法不仅实现了对储层规模的控制，还提升了网络性能，为回声状态网络 在实际应用中的推广提供了可靠的方法和理论支持。

The challenging problem of optimal reservoir structure design in Echo State Networks (ESN) is addressed by proposing a method that utilizes reinforcement learning for the selforganization and reconstruction of reservoir neurons in this paper. The mismatch between reservoir size and specific tasks caused by redundant information, with applications in mine water inrush warning, is addressed in this study. The main research contributions are summarized as follows: Firstly, a reinforcement learning-based optimization method for reservoir neuron selection is proposed to mitigate the collinearity issue in the high-dimensional state space matrix caused by redundant neurons, which adversely affects network prediction performance. Multiple randomly initialized reservoirs are constructed using ensemble learning principles. The contribution of each neuron in the reservoir to the overall network is evaluated using mutual information, and reinforcement learning's decision mechanism is leveraged to select neurons that effectively contribute to the network output, thereby optimizing the network structure and improving prediction performance. In simulation experiments, data sets including Mackey-Glass, Lorenz, and PM2.5 are employed, with an initial condition of 100×3. Experimental results demonstrate that the proposed optimization algorithm indirectly controls the reservoir neurons within 20% and achieves superior prediction results. Compared to other optimization methods, the minimum reservoir structure is obtained, and network performance is significantly enhanced. Secondly, a reinforcement learning-based self-organizing reconstruction method for ESN reservoirs is proposed to address the high coupling issue among reservoir neurons, which affects network prediction performance and stability. The decision-making procedure from reinforcement learning is integrated, and its stability is mathematically verified through the principles of Lyapunov stability theory. Leveraging the echo state property and singular value decomposition principle, the reservoir structure is automatically adjusted by reflecting the reservoir neurons through the state matrix, significantly enhancing the internal dynamics of the reservoir and improving the network's adaptability to input data changes. The effectiveness of the reservoir reconstruction method is demonstrated through simulation validation using Rossler and Laser datasets, with reservoir size being directly controlled while ensuring prediction capability and stability. Finally, aiming at the complexity and difficulty of predicting mine water inflow, an optimized echo state network was introduced for prediction. The verification of actual mine water inflow data demonstrated that the model can effectively predict mine water inflow and accurately judge early warning situations. This provides assistance for timely taking corresponding measures and offers more reliable tools and methods for mine safety management. Based on the above analysis, the effectiveness of the proposed method for optimizing the echo state network structure is conclusively demonstrated by the experimental results of this paper. The control over reservoir scales is facilitated, and network performance is enhanced by this method, providing a reliable methodology and theoretical foundation for the practical application of echo state networks.

[1] Russell S J, Norvig P. Artificial intelligence: a modern approach[M]. Pearson: Prentice Hall, 2016.

[2] 马晨, 沈超, 蔺琛皓, 李前, 王骞, 李琦, 管晓宏. 针对自动驾驶智能模型的攻击与防御[J/OL]. 计算机学报: 2024-03-15.

[3] Li M, Zhang W, Hu B, Kang J M, Wang Y Q, Lu S F. Automatic assessment of

depression and anxiety through encoding pupil-wave from HCI in VR scenes[J]. ACM Transactions on Multimedia Computing, Communications and Applications, 2023, 20(2): 1-22.

[4] Gao C, Cheng S J. The deep learning model for physical intelligence education and its functional realization path[J]. Soft Computing, 2023, 27(15): 10827-10838.

[5] 刘敏, 张魁星, 李丽萍, 徐娟娟, 李翔, 魏本征. 基于残差注意力神经网络模型的癫痫脑电信号分类[J]. 北京生物医学工程, 2023, 42(3): 263-270.

[6] Bas E, Egrioglu E, Cansu T. Robust training of median dendritic artificial neural networks for time series forecasting[J]. Expert Systems with Applications, 2024,238:1-14.

[7] 王雨露, 李飞, 杨震, 黄山, 张罡, 詹曙. 基于深度前馈神经网络的多因子人体表面积计算模型[J]. 计算机工程与科学, 2023, 45(01): 119-126.

[8] Jiang J C, Wang H Z, Xie J, Guo X T, Guan Y, Yu Q B. Medical knowledge embedding based on recursive neural network for multi-disease diagnosis[J]. Elsevier,2020,103:1-15.

[9] 胡敏, 高永, 吴昊, 王晓华, 黄忠. 融合边缘检测和递归神经网络的视频表情识别[J]. 电子测量与仪器学报, 2020, 34(07): 103-111.

[10] Miao P, Shen Y J, Li Y J, Bao L. Finite-time recurrent neural networks for solving nonlinear optimization problems and their application[J]. Neurocomputing, 2016, 177: 120-129.

[11] 耿磊, 傅洪亮, 陶华伟, 卢远, 郭歆莹, 赵力. 基于动态卷积递归神经网络的语音情感识别[J/OL]. 计算机工程: 2023-03-29.

[12] Bengio Y, Simard P, Frasconi P. Learning long-term dependencies with gradient descent is difficult[J]. IEEE Trans on Neural Networks, 1994, 5(2), 157-166.

[13] 庄爱军. 基于WA-ESN的建筑起重机械故障检测[J]. 机械与电子,2021, 39(01): 67-75.

[14] Han S I, Lee J M. Fuzzy echo state neural networks and funnel dynamic surface control for prescribed performance of a nonlinear dynamic system[J]. IEEE Transactions on Industrial Electronics, 2013, 61(2): 1099-1112.

[15] Jaeger H. The “echo state” approach to analysing and training recurrent neural networks-with an erratum note[R]. Bonn, Germany: German National Research Center for Information Technology GMD Technical Report, 2001.

[16] 刘丽丽, 刘玉玺, 王河山. 偏置剪枝叠式自编码回声状态网络的时序预测[J]. 计算机工程与设计, 2024, 45(01): 212-219.

[17] Ding L, Bai Y L, Fan M H, Yu Q H, Zhu Y J, Chen X Y. Serial-parallel dynamic echo state network: A hybrid dynamic model based on a chaotic coyote optimization algorithm for wind speed prediction[J]. Expert Systems with Applications,2023,212:1-19.

[18] 李无言, 王志强, 蒋永年, 郭亚. 基于优化回声状态网络的溶解氧预测建模[J]. 控制工程, 2023, 30(03): 520-528.

[19] Liu C, Li Y L, Duan Z X, Chu Z S, Ma Z F. Echo state network-based robust tracking control for unknown constrained nonlinear systems by using integral reinforcement learning[J]. IEEE Access, 2024, 12: 15133-15144.

[20] Rodan A, Tino P. Minimum complexity echo state network[J]. IEEE Transactions on Neural Networks, 2010, 22(1): 131-144.

[21] Cui H , Liu X, Li L. The architecture of dynamic reservoir in the echo state network[J]. Chaos: An Interdisciplinary Journal of Nonlinear Science, 2012, 22(3): 1-10.

[22] Xue Y B, Yang L, Haykin, Simon. Decoupled echo state networks with lateral inhibition[J]. Neural Networks, 2007, 20(3): 365-376.

[23] Qiao J F, Li F J, Han H G, Li W J. Growing echo-state network with multiple subreservoirs[J]. IEEE Transactions on Neural Networks and Learning Systems, 2016, 28(2): 391-404.

[24] Wang H, Yan X. Improved simple deterministically constructed cycle reservoir network with sensitive iterative pruning algorithm[J]. Neurocomputing, 2014, 145: 353-362.

[25] 王磊, 乔俊飞, 杨翠丽, 朱心新. 基于灵敏度分析的模块化回声状态网络修剪算法[J]. 自动化学报, 2019, 45(06): 136-145．

[26] Yang D Y, Li T, Guo Z J, Li Q. Multi-scale convolutional echo state network with an effective pre-training strategy for solar irradiance forecasting[J]. IEEE Access, 2024, 12: 13442-13452.

[27] Ozturk M C, Xu D, Principe J C. Analysis and design of echo state networks[J]. Neural Computation, 2007, 19(1): 111-138.

[28] Li D Y, Liu F, Qiao J F, Li R. Structure optimization for echo state network based on contribution[J]. Tsinghua Science and Technology, 2019, 24(1): 97-105.

[29] Chen Q, Jin Y C, Song Y D. Fault-tolerant adaptive tracking control of Euler-Lagrange systems – An echo state network approach driven by reinforcement learning[J]. Neurocomputing, 2022, 484: 109-116.

[30] Jaeger H, Haas H. Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication[J]. Science, 2004, 304(5667): 78-80.

[31] Dutoit X, Schrauwen B, Van Campenhout J, Stroobandt D, Van Brussel H, Nuttin M. Pruning and regularization in reservoir computing[J]. eurocomputing, 2009, 72(7-9): 1534-1546.

[32] 韩敏, 任伟杰, 许美玲. 一种基于L1范数正则化的回声状态网络[J]. 自动化学报, 2014, 40(11): 2428-2435.

[33] Xu M L, Han M. Adaptive elastic echo state network for multivariate time series prediction[J]. IEEE Transactions on Cybernetics, 2016, 46(10): 2173-2183.

[34] Wang Z G, Zeng Y R, Wang S R, Wang L. Optimizing echo state network with backtracking search optimization algorithm for time series forecasting[J]. Engineering Applications of Artificial Intelligence, 2019, 81: 117-132.

[35] Liu J X, Sun T N, Luo Y L, Yang S, Cao Y, Zhai J. Echo state network optimization using binary grey wolf algorithm[J]. Neurocomputing. 2020, 385: 310-318.

[36] 李保健, 程春田, 武新宇, 王森. 日径流预报贝叶斯回声状态网络方法[J]. 中国科学:技术科学, 2014, 44(09): 1004-1012.

[37] Qiao J F, Wang L, Yang C L. Adaptive lasso echo state network based on modified Bayesian information criterion for nonlinear system modeling[J]. Neural Computing and Applications, 2019, 31(10): 6163-6177.

[38] Ahmed Z, Memon M Q, Memon A, Munshi P, Memon M J. Echo state network

optimization using hybrid-structure based gravitational search algorithm with square quadratic programming for time series prediction[J]. nternational Arab Journal of

Information. Technology, 2022, 19(3A): 530-535.

[39] Liu Z Y, Xu X H, Pan M Y, Loo C K, Li S X. Weighted error-output recurrent echo kernel state network for multi-step water level prediction[J]. Applied Soft Computing, 2023, 137: 110-131.

[40] Ferreira A A, Ludermir T B, De Aquino R R B. An approach to reservoir computing design and training[J]. Expert systems with applications, 2013, 40(10): 4172-4182.

[41] 许美玲, 王依雯. 基于改进差分进化和回声状态网络的时间序列预测研究[J]. 自动化学报, 2021, 47(07): 1589-1597.

[42] Chouikhi N, Ammar B, Rokbani N, Alimi A M. PSO-based analysis of Echo State Network parameters for time series forecasting[J]. Applied Soft Computing, 2017, 55: 211-225.

[43] Lun S X, Hu H F, Yao X S. The modified sufficient conditions for echo state property and parameter optimization of leaky integrator echo state network[J]. Applied Soft Computing, 2019, 77: 750-760.

[44] 张昭昭, 朱应钦, 乔俊飞, 余文. 一种基于行为空间的回声状态网络参数优化方法[J]. 信息与控制, 2021, 50(05): 556-565.

[45] 潘诗媛, 华志广, 王光伟, 赵冬冬, 窦满峰. 基于级联回声状态网络的氢燃料电池剩余使用寿命预测[J/OL]. 中国电机工程学报: 2024-03-05.

[46] Viehweg J, Worthmann K, Mäder P. Parameterizing echo state networks for multi-step time series prediction[J]. Neurocomputing, 2023, 522: 214-228.

[47] 吴忠强, 戚松岐, 尚梦瑶, 申丹丹. 基于优化回声状态网络的微电网等效建模[J]. 计量学报, 2021, 42(07): 923-929.

[48] 翟肖昂, 宋金玲, 康燕, 李院夫, 林琢. 基于回声状态网络的水质预测方法[J]. 河北科技师范学院学报, 2022, 36(02): 82-88.

[49] Yuen S C Y, Yao G, Johnson E. Augmented reality: An overview and five directions for AR in education[J]. Journal of Educational Technology Development and Exchange (JETDE), 2011, 4(1): 1-11.

[50] 郭艺, 王枫, 甘甫平, 闫柏琨. 基于移动平均模型和指数平滑模型的岩溶泉泉流量预测[J]. 河北地质大学学报, 2020, 43(04): 19-25.

[51] Brockwell P J. Continuous-time ARMA processes[J]. Handbook of statistics, 2001, 19: 249-276.

[52] 肖珊, 陈建勇, 林斌, 龙建勋. 自回归积分移动平均模型在长沙市蝇密度预测中的应用[J].中国媒介生物学及控制杂志, 2023, 34(06): 788-793.

[53] 马晶, 王梅, 谯小伟, 曹文珮, 李娟生.ARIMA季节性模型在预测兰州市丙肝发病人数中的应用[J]. 中国卫生统计, 2022, 39(01): 98-100+105.

[54] 杨雨桐, 张利, 杨玖. 基于指数平滑法的噪声污染预测及应用[J].资源节约与环保, 2024(03): 74-78.

[55] 王磊. 回声状态网络优化设计及应用研究[D]. 北京:北京工业大学, 2020.

[56] Yildiz I B, Jaeger H, Kiebel S J. Re-visiting the echo state property[J]. Neural Networks, 2012, 35: 1-9.

[57] Buehner M, Young P. A tighter bound for the echo state property [J]. IEEE Transactions on Neural Networks, 2006, 17(3): 820-824.

[58] Wainrib G, Galtier M N. A local echo state property through the largest Lyapunov exponent[J]. Neural Networks, 2016, 76:39-45.

[59] Yao X S, Wang Z S, Zhang H G. Prediction and identification of discrete-time dynamic nonlinear systems based on adaptive echo state network[J]. Neural Networks, 2019, 113: 11-19.

[60] Li X, Bi F R, Yang X, Bi X Y. An echo state network with improved topology for time series prediction[J]. IEEE Sensors Journal, 2022, 22(6): 5869-5878.

[61] Yang C L, Nie K Z, Qiao J F, Wang D L. Robust echo state network with sparse online learning[J]. Information Sciences, 2022, 594: 95-117.

[62] 郭伟, 姚欢, 张昭昭, 朱应钦. 基于强化学习的储层神经元筛选优化方法[J/OL]. 控制与决策: 2024-03-10.

[63] Kobialka H U, Kayani U. Echo state networks with sparse output connections[C]//Artificial Neural Networks – ICANN 2010: 20th International Conference, Thessaloniki, Greece, September 15-18, 2010, Proceedings, Part I 20. Springer Berlin Heidelberg, 2010: 356-361.

[64] Kraskov A, Stögbauer H, Grassberger P. Estimating mutual information[J]. Physical review E, 2004, 69(6): 138-151.

[65] Chen Y C. A tutorial on kernel density estimation and recent advances[J]. Biostatistics & Epidemiology, 2017, 1(1): 161-187.

[66] Liu H, Yu C M, Yu C Q, Chen C, Wu H P. A novel axle temperature forecasting method based on decomposition, reinforcement learning optimization and neural network[J]. Advanced Engineering Informatics, 2020, 44: 1-10.

[67] Gallicchio C, Micheli A, Pedrelli L. Design of deep echo state networks[J]. Neural Networks, 2018, 108: 33-47.

[68] Chen X F, Luo X, Jin L, Li S, Liu M. Growing echo state network with an inverse-free weight update strategy[J]. IEEE Transactions on Cybernetics, 2022, 53(2): 753-764.

[69] Li Y, Li F J. PSO-based growing echo state network[J]. Applied Soft Computing, 2019, 85: 347-355.

[70] 王磊, 苏中, 乔俊飞, 赵静. 基于增量式学习的正则化回声状态网络[J]. 控制与决策, 2022: 37(03): 661-668.

[71] 谭大国. BP神经网络在矿井涌水量预测中的应用[J]. 制造业自动化, 2015, 37(05): 66-68.

[72] 张宪峰, 魏久传, 张延飞, 吴霞, 李孝朋. 基于主成分分析与BP神经网络的矿井涌水量预测研究[J]. 煤炭技术, 2018, 37(06): 201-203.

[73] 吴煌, 杨智成, 李梦华. 基于长短期记忆神经网络的矿井涌水量预测[J]. 中国水运, 2023, 23(03): 25-27.

[74] 连会青, 李启兴, 王瑞, 夏向学, 张庆, 黄亚坤, 任正端, 康佳. 基于深度学习的LSTMGRU复合模型矿井涌水量预测方法研究[J/OL].煤矿安全, 2024-01-25.