Aggregated Contextual Transformations for High-Resolution Image Inpainting

By Yanhong Zeng, Jianlong Fu, Hongyang Chao, Baining Guo. 2021

324 citations recorded by the source corpus.

Image inpainting that completes large free-form missing regions in images is a promising yet challenging task. State-of-the-art approaches have achieved significant progress by taking advantage of generative adversarial networks (GAN). However, these approaches can suffer from generating distorted structures and blurry textures in high-resolution images (e.g., <inline-formula><tex-math notation="LaTeX">$512\times 512$</tex-math><alternatives><mml:math><mml:mrow><mml:mn>512</mml:mn><mml:mo>×</mml:mo><mml:mn>512</mml:mn></mml:mrow></mml:math><inline-graphic xlink:href="fu-ieq1-3156949.gif"/></alternatives></inline-formula>). The challenges mainly drive from (1) image content reasoning from distant contexts, and (2) fine-grained texture synthesis for a large missing region. To overcome these two challenges, we propose an enhanced GAN-based model, named <bold>A</bold>ggregated C<bold>O</bold>ntextual-<bold>T</bold>ransformation GAN (<bold>AOT-GAN</bold>), for high-resolution image inpainting. Specifically, to enhance context reasoning, we construct the generator of AOT-GAN by stacking multiple layers of a proposed AOT block. The AOT blocks aggregate contextual transformations from various receptive fields, allowing to capture both informative distant image contexts and rich patterns of interest for context reasoning. For improving texture synthesis, we enhance the discriminator of AOT-GAN by training it with a tailored mask-prediction task. Such a training objective forces the discriminator to distinguish the detailed appearances of real and synthesized patches, and in turn facilitates the generator to synthesize clear textures. Extensive comparisons on Places2, the most challenging benchmark with 1.8 million high-resolution images of 365 complex scenes, show that our model outperforms the state-of-the-art. A user study including more than <bold>30 subjects</bold> further validates the superiority of AOT-GAN. We further evaluate the proposed AOT-GAN in practical applications, e.g., logo removal, face editing, and object removal. Results show that our model achieves promising completions in the real world. We release codes and models in <uri>https://github.com/researchmm/AOT-GAN-for-Inpainting</uri>.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Improving Diffusion Models for Inverse Problems using Manifold Constraints (2022, methodology): “We also include comparisons against supervised learning based base-lines: LaMa [43], AOT-GAN [52], ICT [49], and DSI [35].”
RePaint: Inpainting using Denoising Diffusion Probabilistic Models (2022, methodology): “As seen in Figure 4 and 5, AOT [50] fails completely, while the others either produce blurry images or generate visible artifacts, or both.”
DiffIR: Efficient Diffusion Model for Image Restoration (2023, background): “Up to now, researchers have carried out CNN’s study with different perspectives and obtained more elaborate network architecture designs and learning schemes, such as residual block [29, 81, 6], GAN [21, 65, 48], attention [86, 66, 11, 72, 71, 68, 73], knowledge distillation [67], and others [26, 19”