波达方向（Direction of Arrival, DOA）估计是阵列信号处理领域的重要问题，在雷达、声纳、无线通信等军民领域都有着广泛的应用。近年来，嵌套阵列的提出解决了均匀线性阵列（Uniform Linear Array, ULA）性能和成本之间的矛盾，然而针对嵌套阵列信号处理的相关研究多集中在阵型改进和算法设计方面，对于阵列误差下的嵌套阵列DOA估计方法研究较少。另一方面，针对阵列误差下的DOA估计方法大多基于ULA提出，不适用于嵌套阵列。因此，本文基于嵌套阵列，研究阵元位置误差和互耦误差下的DOA估计方法，主要研究内容如下：

1．针对嵌套阵列DOA估计方法在阵元位置误差下性能失效的问题，提出一种基于阵列流型分离的原子范数最小化（AMS-ANM）算法。利用单辅助信源获得实际阵元位置，建立虚拟阵列接收信号模型，通过阵列流型分离技术提取阵列流型矩阵的范德蒙德结构，从而能够获得原子范数最小化的数学模型。实验结果表明，AMS-ANM算法可以实现阵元位置误差下的欠定DOA估计，在低信噪比、小快拍的情况下能够有效提高估计精度和空间分辨率，对不同阵元位置误差均具有鲁棒性。

2．针对互耦误差下现有嵌套阵列DOA估计方法的性能有待提高的问题，提出迭代加权低秩矩阵恢复（IW-LRMR）算法，对接收数据协方差矩阵进行扩展，将加权处理的方法引入到低秩矩阵恢复算法，增强信号的稀疏性，并利用互耦矩阵的结构特点，用维数较低的互耦系数表征互耦矩阵，通过联合迭代求解高维Toeplitz矩阵和互耦系数。实验结果表明，IW-LRMR算法能够有效地避免互耦效应的影响，DOA估计精度较高，而且对邻近信号也有效。

Direction of arrival (DOA) estimation is an important issue in the field of array signal processing, and has wide applications in radar, sonar, wireless communication and other military and civilian fields. In recent years, the nested array has been proposed to solve the contradiction between the performance and cost of the uniform linear array (ULA). However, the nested array signal processing mainly focuses on the improvement of the array structure and the algorithm design, and the DOA estimation for nested array with model error is less studied. On the other hand, most of DOA estimation methods about model errors are based on ULA, which can not be used for the nested array. Therefore, this paper conducts research on the DOA estimation method for the nested array with the array element location errors and mutual coupling. The main research contents are as follows:

1. Aiming at the problem that the performance of DOA estimation methods for nested array fails in array element location errors, the array manifold separated atomic norm minimization (AMS-ANM) algorithm is proposed. The actual array element location is obtained by using a single auxiliary source, and a receiving signal model of the virtual array is established. The vandermonde structure of the array manifold is extracted by the array manifold separation technology, so that the mathematical model of atomic norm minimization can be obtained. The experimental results show that AMS-ANM algorithm can achieve underdetermined DOA estimation in array element location errors, effectively improve the estimation accuracy and spatial resolution with low signal-to-noise ratio and few snapshots, and is robust to different array element location errors.

2. Aiming at the problem that the performance of the existing DOA estimation methods for the nested array needs to be improved with the mutual coupling, an iterative weighted low rank matrix reconstruction (IW-LRMR) algorithm is proposed. The received data covariance matrix is extended, and the weighted processing is introduced into the low rank matrix reconstruction algorithm to enhance signal sparsity. The mutual coupling coefficients with lower dimension to express the mutual coupling matrix due to the structure features of the mutual coupling matrix, and the high-dimensional Toeplitz matrix and mutual coupling coefficient are obtained by joint iteration. The experimental results show that the IW-LRMR algorithm can effectively avoid the influence of mutual coupling, has high DOA estimation accuracy, and is also effective for adjacent signals.