图像修复是图像处理领域中的重要研究方向之一，其目标是对信息传递过程受噪音影响或受污染的缺损图像进行合理填充，生成符合人主观视觉需求的修复结果。对修复结果的生成图像结构、纹理质量、清晰度有着极高要求。传统的图像修复算法基于图像的低级像素信息计算相似度来完成对破损图像的修复，不具有提取深层语义的信息，在面对大规模破损时修复质量不佳；现阶段基于深度学习的图像修复技术能够充分提取图像的深层语义信息，适用范围更广，但是在面对大面积破损时，修复结果仍然会出现伪影、结构模糊、修复边缘结构断层等问题。此外，实际应用中图像的缺损区域往往呈现随机不规则的形状，当前的图像修复算法研究在面对随机缺损时修复质量不佳。因此，本文为了解决当前图像修复技术出现的问题，设计了一个基于边缘生成的两阶段图像修复算法，本文的主要工作内容如下：

（1）基于堆叠生成网络设计了一个两阶段图像修复网络，通过分析人工修复技术的原理，将修复过程分为边缘预测和纹理填补两个阶段。第一阶段网络为边缘生成网络，使用 Canny 边缘提取算子对破损图像进行边缘提取，将提取到的破损边缘图作为输入，训练网络进行边缘预测，对破损图像进行结构修复。第二阶段网络为修复填补网络，使用第一阶段网络生成的边缘图作为结构引导和破损图像进行合并作为输入，最终输出修复结果。两阶段网络生成器设计为编码器、解码器和中间层结构，对图像进行特征提取融合，判别器使用谱归一化马尔可夫判别器提升网络训练稳定性。

（2）设计多尺度特征融合模块代替传统的残差网络作为两阶段网络中间层，使用符合混合扩张卷积理论扩张率的不同扩张卷积提升模块感受野，同时保留对邻近信息的提取，使用通道注意力机制对不同卷积结果进行权重分配，训练后的第二阶段修复填补网络在 Places365-Standard 场景数据集上相较于对比算法，峰值信噪比提升了 0.58 dB，结构相似度提升了 0.86%。

（3）使用门控卷积对代替编码器解码器中的普通卷积，为多尺度特征融合模块添加门控机制，实现网络训练时对有效和无效像素特殊的处理，提高网络对有效部分信息的利用，在部分像素缺失和大面积损坏情况下，能够更好地对局部特征进行精确描述。在修复填补网络中引入上下文注意力机制模块对特征图进行前后板块划分，将背景板块重塑为卷积核，通过和前景板块的卷积计算注意力分数，使网络在色彩填补过程中可以从遥远空间位置获取纹理特征信息。改进后的网络在 Places365-Standard 场景数据集上相较于对比算法，峰值信噪比提升了 1.44 dB，结构相似度提升了 2.26%。在人脸数据集 CelebA上进行人脸修复实验，相较于对比算法峰值信噪比提升了 2.12 dB，结构相似度提升了2.58%。

Image inpainting is one of the important research directions in the field of image processing, and its goal is to reasonably fill in the defective images affected or polluted by noise in the information transmission process, and generate restoration results that meet the subjective visual needs of people. There are extremely high requirements for the image structure, texture quality and clarity of the restoration results. At present, the image restoration technology based on deep learning can fully extract the deep semantic information of the image and has a wider range of applications, but in the face of large-scale damage, the restoration results still have problems such as artifacts, blurred structures, and repaired edge structure faults. In addition, the defective areas of images in practical applications often show random irregular shapes, and the current image restoration algorithm research has poor repair quality in the face of random defects. Therefore, in order to solve the problems of current image inpainting technology, a two-stage image inpainting algorithm based on edge generation is designed, and the main contects of this thesis is as follows:

(1) A two-stage image inpainting network was designed based on the stacked generation network StackGAN, and the restoration process was divided into two stages: edge prediction and texture filling by analyzing the principle of artificial restoration technology. In the first stage, the network is an edge generation network, in which the Canny edge extraction operator is used to extract the edge of the damaged image, and the extracted edge map is used as the input to train the network for edge prediction and structural repair of the damaged image. The second stage network is the repair and filling network, and the edge map generated by the first-stage network is used as the structure guide and the damaged image is merged as the input, and the repair result is finally output. The two-stage network generator is designed as an encoder, decoder and middle-layer structure, which performs feature extraction and fusion on the image, and the discriminator uses the spectral normalized Markov discriminator to improve the stability of network training.

(2) The multi-scale feature fusion module is designed to replace the traditional residual network as the middle layer of the two-stage network, and the receptive field of the module is improved by using different expansion convolutional expansion rates that conform to the expansion rate of the hybrid expansion convolution theory, while retaining the extraction of adjacent information, and the channel attention mechanism is used to distribute the weights of different convolution results, and the PSNR and SSIM of the second-stage repair filling network after training are increased by 0.58 dB and the SSIM is increased by 0.86% compared with the comparison algorithm on the Places365-Standard scene dataset.

(3) Gated convolution pairs are used to replace the ordinary convolutions in the encoder decoder, and a gating mechanism is added to the multi-scale feature fusion module, so as to realize the special processing of valid and invalid pixels during network training, improve the utilization of effective part information by the network, and better describe the local features accurately in the case of missing pixels and large-scale damage. In the repair filling network, the context attention mechanism module is introduced to divide the front and back plates of the feature map, the background plate is reshaped into a convolutional kernel, and the attention score is calculated by convolution with the foreground plate, so that the network can obtain texture feature information from distant spatial locations in the process of color filling. Compared with the comparison algorithm, the PSNR and SSIM of the improved network are increased by 1.44 dB and the SSIM is increased by 2.26% compared with the comparison algorithm on the Places365-Standard scene dataset. Compared with the comparison algorithm, the PSNR increased by 2.12 dB and SSIM is increased by 2.58%.

[1]彭进业,余喆,屈书毅,等.基于深度学习的图像修复方法研究综述[J].西北大学学报(自然科学版),2023,53(06):943-963.

[2]李月龙,高云,闫家良等.基于深度神经网络的图像缺损修复方法综述[J].计算机学报,2021,44(11):2295-2316.

[3]罗海银,郑钰辉.图像修复方法研究综述[J].计算机科学与探索,2022,16(10):2193-2218.

[4]Bertalmio M, Sapiro G, Caselles V, et al. Image inpainting[C]//Proceedings of the 27th annual conference on Computer graphics and interactive techniques. 2000: 417-424.

[5]SHEN J, CHAN T F. Mathematical models for local nontexture inpaintings[J]. SIAM Journal on Applied Mathe matics, 2002, 62(3): 1019-1043.

[6]EFROS A A, LEUNG T K.Texture synthesis by non-parametric sampling[C]//Proceedings of the seventh IEEE international conference on computer vision. Corfu,Greece, September, 1999, 2: 1033-1038.

[7]ASHIKHMIN M. Synthesizing natural textures[C]//Proceedings of the 2001 symposium on Interactive 3D graphics. New York, March, 2001: 217-226.

[8]CRIMINISI A, PÉREZ P, TOYAMA K. Region filling and object removal by exemplar-based image inpainting[J]. IEEE Transactions on image processing, 2004, 13(9):1200-1212.

[9]BARNES C, SHECHTMAN E, FINKELSTEIN A, et al. PatchMatch: A randomized correspondence algorithm for structural image editing[J]. ACM Trans. Graph., 2009,28(3): 24.

[10]李奇,李龙,王卫,等.基于改进Criminisi算法的破损纺织品文物图像修复[J].激光与光电子学进展,2023,60(16):173-182.

[11]HAYS J, EFROS A A. Scene completion using millions of photographs[J]. ACM Transactions on Graphics (ToG),2007, 26(3): 4-es.

[12]Pathak D, Krahenbuhl P, Donahue J, et al. Context encoders: Feature learning by inpainting[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 2536-2544.

[13]Yang C, Lu X, Lin Z, et al. High-resolution image inpainting using multi-scale neural patch synthesis[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 6721-6729.

[14]Iizuka S, Simo-Serra E, Ishikawa H. Globally and locally consistent image completion[J]. ACM Transactions on Graphics (ToG), 2017, 36(4): 1-14.

[15]Liao L, Hu R, Xiao J, et al. Edge-aware context encoder for image inpainting[C]//2018 IEEE International conference on acoustics, speech and signal processing (ICASSP). IEEE, 2018: 3156-3160.

[16]YAN Z, LI X, LI M, et al. Shift-net: Image inpainting via deep feature rearrangement[C]//Proceedings of the European conference on computer vision (ECCV). Munich, Germany, September 8-14, 2018: 1-17.

[17]Liu G, Reda F A, Shih K J, et al. Image inpainting for irregular holes using partial convolutions[C]//Proceedings of the European conference on computer vision (ECCV). 2018: 85-100.

[18]Wang N, Li J, Zhang L, et al. MUSICAL: Multi-Scale Image 25 Learning for Inpainting[C]//IJCAI. 2019: 3748-3754.

[19]RONNEBERGER O, FISCHER P, BROX T. U-net: Convolutional networks for biomedical image segmentation[C]//International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, Munich, Germany, October 5-9, 2015: 234-241.

[20]Hong X, Xiong P, Ji R, et al. Deep fusion network for image completion[C]//Proceedings of the 27th ACM international conference on multimedia. 2019: 2033-2042.

[21]YU Y, ZHANG L. High-fidelity image inpainting with GAN inversion[C]// European Conference on Computer Vision (ECCV) 2022 Lecture Notes in Computer Science, vol 13676. Springer ,242-258.

[22]GU J, SHEN Y, ZHOU B. Image processing using multi-code GAN prior[C]//Proceedings of the 2020 IEEE/CVF Confer ence on Computer Vision and Pattern Recognition, Seattle, Jun 14-19, 2020. Piscataway: IEEE, 2020: 3009-3018.

[23]POIRIER-GINTER Y, LALONDE J F. Robust unsupervised StyleGAN image restoration[C]//Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Rec ognition, Vancouver, Jun 18-22, 2023. Piscataway: IEEE, 2023: 22292-22301.

[24]Wang T, Gao J, et al. ISAR Image Inpainting Algorithm Based on DCGAN [C]//2021 International Symposium on Antennas and Propagation (ISAP), Taipei, Taiwan, 2021:1-2.

[25]Samba S, Aminata N, et al. A New Spectral Intrinsic Decomposition based Image Inpainting Method [C]//2022 International Conference on Electrical, Computer and Energy Technologies (ICECET), Prague, Czech Republic, 2022:1-5.

[26]Yu J, Lin Z, Yang J, et al. Generative image inpainting with contextual attention[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 5505-5514.

[27]Zhang H, Xu T, Li H, et al. StackGAN: text to photo-realistic image synthesis with stacked generative adversarial networks[C].//Proceedings of the IEEE International Conference on Computer Vision. Venice, Italy, 2017: 5907-5915

[28]Yu J, Lin Z, Yang J, et al. Free-form image inpainting with gated convolution[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 4471-4480.

[29]Song Y, Yang C, Lin Z, et al. Contextual-based image inpainting: Infer, match, and translate[C]//Proceedings of the European Conference on Computer Vision (ECCV). 2018: 3-19.

[30]Nazeri K, Ng E, Joseph T, et al. Edgeconnect: Generative image inpainting with adversarial edge learning [C]//2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), Seoul, Korea (South), 2019: 3265-3274.

[31]DENG X C, YU Y. Ancient mural inpainting via structure information guided two-branch model [J]. Heritage Science, 2023, 11(1): 1-17.

[32]Li J, He F, Zhang L, et al. Progressive reconstruction of visual structure for image inpainting[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 5962-5971.

[33]MA Y Q, LIU X L, BAI S H, et al. Coarse-to-fine image inpainting via region-wise convolutions and nonlocal correlation [C]//Proceedings of the 28th Intenational Joint Conference on Artificial Intelligence. Macao:IJCAI,2019: 3123-3129.

[34]Lahiri A, Jain A K, Agrawal S, et al. Prior guided gan based semantic inpainting[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 13696-13705.

[35]杨红菊,李丽琴,王鼎.联合语义分割与边缘重建的深度学习图像修复[J].中国图象图形学报,2022,27(12):3553-3565.

[36]章登勇,文凰,李峰,等.基于双分支网络的图像修复取证方法[J].网络与信息安全学报,2022,8(06):110-122.

[37]孙宇,李晗.基于结构-纹理分解的数字图像分层修复算法[J].计算机仿真,2023,40(06):220-224.

[38]Liu H, Jiang B, Xiao Y, et al. Coherent semantic attention for image inpainting[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019: 4170-4179.

[39]Sagong M, Shin Y, Kim S, et al. Pepsi: Fast image inpainting with parallel decoding network[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 11360-11368.

[40]Shin Y G, Sagong M C, Yeo Y J, et al. Pepsi++: Fast and lightweight network for image inpainting[J]. IEEE transactions on neural networks and learning systems, 2020, 32(1): 252-265.

[41]Zeng Y, Fu J, Chao H, et al. Aggregated contextual transformations for high-resolution image inpainting[J]. IEEE Transactions on Visualization and Computer Graphics, July 2023:3266-3280.

[42]Demir U, Unal G. Patch-based image inpainting with generative adversarial networks[J]. arXiv preprint arXiv:1803.07422, 2018.

[43]LI H A, WANG G Y, GAO K, et al. A gated convolution and self-attention-based pyramid image inpainting network[J]. Journal of Circuits, Systems and Computers, 2022, 31(12) : 2250208:1-18.

[44]Yuting L, Heng Liu, et al. Face Image Inpainting Based on Mask Gridding Autoencoder and Global-Local Refinement Fusion [C]//2023 16th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), Taizhou, China, 2023:1-6.

[45]武文杰,王红蕾.基于生成对抗网络的遮挡人脸图像修复的改进与实现[J].计算机应用与软件,2021,38(01):217-221+249.

[46]Zeng Y, Fu J, Chao H, et al. Learning pyramid-context encoder network for high-quality image inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 1486-1494.

[47]Liu H, Jiang B, Song Y, et al. Rethinking image inpainting via a mutual encoder-decoder with feature equalizations[C]//Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer International Publishing, 2020: 725-741.

[48]Li J, Wang N, Zhang L, et al. Recurrent feature reasoning for image inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 7760-7768.

[49]Zheng C, Cham T J, Cai J. Pluralistic image completion[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 1438-1447.

[50]Lahiri A, Jain A K, Agrawal S, et al. Prior guided gan based semantic inpainting[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 13696-13705.

[51]Zhao L, Mo Q, Lin S, et al. Uctgan: Diverse image inpainting based on unsupervised cross-space translation[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 5741-5750.

[52]LIU H, WAN Z, HUANG W, et al. PD-GAN: probabilistic diverse GAN forimage inpainting[C]//Proceeding of the 2021 IEEE Conference on Computer Vision and Pattern Recognition, Jun 19-25, 2021. Piscataway: IEEE, 2021: 9371-9381.

[53]Aref A, Mehran E. Multi-Modality Image Inpainting Using Generative Adversarial Networks[C]// 2023 Congress in Computer Science, Computer Engineering, & Applied Computing (CSCE), Las Vegas, NV, USA, 2023:1909-1916.

[54]Dongsik Y, Jeong-Gi K, et al. DIFAI: Diverse Facial Inpainting using StyleGAN Inversion [C]// 2022 IEEE International Conference on Image Processing (ICIP), Bordeaux, France, 2022:1141-1145.

[55]Phutke S S, Murala S. Diverse receptive field based adversarial concurrent encoder network for image inpainting[J]. IEEE Signal Processing Letters, 2021:1873-1877.

[56]Phutke S S, KULKARNI A, VIPPARTHI S K, et al. Blind image inpainting via omni-dimensional gated attention and wavelet queries[C]// 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops(CVPRW). Vancouver: IEEE, 2023:1251-1260.

[57]Chen L C, Papandreou G, Kokkinos I, et al. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs[J]. IEEE transactions on pattern analysis and machine intelligence, 2017, 40(4): 834-848.

[58]He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778

[59]PQNQU W, PENGFEI C, et al., Understanding Convolution for Semantic Segmentation [C]. IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA, 2018: 12-15.

[60]Hu J, Shen L, Albanie S, et al. Squeeze-and-Excitation Networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(8): 2011-2023.

[61]L. Yuan, C. Ruan, et al. Image Inpainting Based on Patch-GANs[J]. IEEE Access, vol. 7, 2019: 46411-46421.

[62]Miyato T, Kataoka T, Koyama M, et al. Spectral normalization for generative adversarial networks[J]. arXiv preprint arXiv:1802.05957, 2018.