Image restoration for irregular holes based on dual discrimination generation countermeasure network

基于双判别生成对抗网络的不规则孔洞图像修复

¹ School of Information Science and Engineering, Yunnan University, Kunming 650000, China
² Yunnan Communications Investment and Construction Group Co., Ltd, Kunming 650000, China

Received: 1 July 2020

Abstract

In order to solve the problem that the global and local generated countermeasure network cannot inpaint the random irregular large holes, and to improve the standard convolution generator, which demonstrates the defects of color difference and blur, a network architecture of inpainting irregular large holes in an image based on double discrimination generation countermeasure network is proposed. Firstly, the image generator is a U-net architecture defined by partial convolution. The normalized partial convolution only completes the end-to-end mask update for the effective pixels. The skip link in U-net propagates the context information of the image to the higher resolution, and optimizes the training results of the model with the weighted loss function of reconstruction loss, perception loss and wind grid loss. Subsequently, the adversary loss function, the dual discrimination network including the synthetic discriminator and the global discriminator are trained separately to judge the consistency between the generated image and the real image. Finally, the weighted loss functions are trained together with generating network and double discrimination network to further enhance the detail and overall consistency of the inpainted area and make the inpainted results more natural. The simulation experiment is carried out on the Place 365 standard database. The subjective and objective experimental results show that the results of the proposed method has reasonable overall and detail semantic consistency than those of the existing methods when they are used to repair random, irregular and large-area holes. The proposed method effectively overcomes the defects of blurry details, color distortion and artifacts.

摘要

为解决现有算法在修复随机、不规则大面积孔洞时出现色差和细节模糊的缺陷，提出了基于双判别生成对抗网络的不规则大孔洞图像修复网络架构。图像生成器是部分卷积定义的U-Net架构，归一化的部分卷积仅对有效像素完成端到端的掩码更新，U-Net中的跳过链接将图像的上下文信息向更高层分辨率传播，用重建损失、感知损失和风格损失的加权损失函数优化模型的训练结果；使用对抗性损失函数，单独训练包含合成判别器和全局判别器的双判别网络，判断生成图像与真实图像的一致性；加权所有损失函数，结合生成网络和双判别网络一起训练，进一步增强待修复区域的细节和整体一致性，使修复结果更自然。在Place365标准数据库上进行仿真实验，实验结果表明：提出方法在处理随机、不规则、大面积孔洞修复时，其结果的整体和细节语义一致性优于现有方法的结果，有效克服了细节模糊、颜色失真和出现伪影等缺陷。