MAT: Mask-Aware Transformer for Large Hole Image Inpainting

By Wenbo Li, Zhe Lin, Kun Zhou, Lu Qi, Yi Wang, Jiaya Jia. Computer Vision and Pattern Recognition, 2022

503 citations recorded by the source corpus.

Recent studies have shown the importance of modeling long-range interactions in the inpainting problem. To achieve this goal, existing approaches exploit either standalone attention techniques or transformers, but usually under a low resolution in consideration of computational cost. In this paper, we present a novel transformer-based model for large hole inpainting, which unifies the merits of transformers and convolutions to efficiently process high-resolution images. We carefully design each component of our framework to guarantee the high fidelity and diversity of recovered images. Specifically, we customize an inpainting-oriented transformer block, where the attention module aggregates non-local information only from partial valid tokens, indicated by a dynamic mask. Extensive experiments demonstrate the state-of-the-art performance of the new model on multiple benchmark datasets. Code is released at https://github.com/fenglinglwb/MAT.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/f2736077d96cccdc8e95ac92fdc3dd3916885d29

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yulun Zhang
2. Yapeng Tian
3. Wenjun Zeng
4. Luc Van Gool
5. Zhibo Chen
6. Radu Timofte
7. Xin Jin
8. Lei Zhang

Citations in the corpus

• Inpaint Anything: Segment Anything Meets Image Inpainting (2023, background): “The success of deep learning has brought new opportunities [13, 10, 8, 4], and all these SOTA methods can be categorized from multiple perspectives, e.”
• DiffIR: Efficient Diffusion Model for Image Restoration (2023, methodology): “Presently, the transformer has been adopted in numerous vision tasks, such as image recognition [17, 60], segmentation [62, 69, 87, 49], object detection [5, 89], and image restoration [7, 38, 74, 36, 8].”