Image Inpainting With Learnable Bidirectional Attention Maps

By Chaohao Xie, Shaohui Liu, Chao Li, Ming-Ming Cheng, Wangmeng Zuo, Xiao Liu, Shilei Wen, Errui Ding. 2019

265 citations recorded by the source corpus.

Most convolutional network (CNN)-based inpainting methods adopt standard convolution to indistinguishably treat valid pixels and holes, making them limited in handling irregular holes and more likely to generate inpainting results with color discrepancy and blurriness. Partial convolution has been suggested to address this issue, but it adopts handcrafted feature re-normalization, and only considers forward mask-updating. In this paper, we present a learnable attention map module for learning feature re-normalization and mask-updating in an end-to-end manner, which is effective in adapting to irregular holes and propagation of convolution layers. Furthermore, learnable reverse attention maps are introduced to allow the decoder of U-Net to concentrate on filling in irregular holes instead of reconstructing both holes and known regions, resulting in our learnable bidirectional attention maps. Qualitative and quantitative experiments show that our method performs favorably against state-of-the-arts in generating sharper, more coherent and visually plausible inpainting results. The source code and pre-trained models will be available at: https://github.com/Vious/LBAM_inpainting/.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, background): “…al., 2019; Jo & Park, 2019; Lahiri et al., 2017; Liu et al., 2019a; Nazeri et al., 2019; Ren et al., 2019; Sagong et al., 2019; Wang et al., 2018; Xie et al., 2019; Xiong et al., 2019; Yan et al., 2018; Yang et al., 2017; 2019a; Yu et al., 2018; Yu et al., 2019; Zeng et al., 2019; Lahiri et al.,…”
Image Fine-grained Inpainting (2020, unknown): “corrective effect on a few problematic results produced by the baseline model. And we give the quantitative results in Table IV, which validates the A CORRESPONDENCE SUBMITTED TO JOURNAL 8 Input LBAM [38] PICNet [12] DMFN (Ours) Fig. 10. Inpainted images with irregular masks on Paris StreetView. Inp”
DiffIR: Efficient Diffusion Model for Image Restoration (2023, background): “Up to now, researchers have carried out CNN’s study with different perspectives and obtained more elaborate network architecture designs and learning schemes, such as residual block [29, 81, 6], GAN [21, 65, 48], attention [86, 66, 11, 72, 71, 68, 73], knowledge distillation [67], and others [26, 19”
Image Inpainting via Conditional Texture and Structure Dual Generation (2021, background): “Moreover, updated versions of vanilla convolution are investigated [13, 27, 36], where operations are masked and normalized to be conditioned only on valid pixels, achieving promising performance for irregular corruptions.”
Flow-edge Guided Video Completion (2020, background): “Several improved architecture designs have been proposed to handle free-form holes [23,40,44] and leverage predicted structures (e.”
Learning Oracle Attention for High-Fidelity Face Completion (2020, methodology): “The second category of methods [31, 19, 27] typically involve training a deep CNN in an encoder-decoder structure to predict each pixel of the missing area.”
Contrastive Learning for Diverse Disentangled Foreground Generation (2022, background): “Recently, deep learning methods [8,18,33,35,48,50,52,53,55,59,28,57,61] dramatically boosted the quality, in terms of both visual quality and semantic coherence.”
Aggregated Contextual Transformations for High-Resolution Image Inpainting (2021, methodology): “An illustration of the Residual block (a) used in the state-of-the-art deep inpainting models [17], [42], [51] and the proposed AOT block (b).”
Image Inpainting by End-to-End Cascaded Refinement With Mask Awareness (2021, methodology): “Bidirectional Attentional Maps [49] (LBAM) is tested with the source code and the pertrained models provided by its authors.”
Image Inpainting with Learnable Feature Imputation (2020, methodology): “he reader to zoom-in on missing regions. For non handpicked qualitative examples, see Appendix 5. Qualitative Results Figure 4 shows a set of hand-picked examples, Figure 5 shows examples selected by [41], and Appendix 5 includes a large set of examples selected by Image Inpainting with Learnable Fe”