Learning to Incorporate Structure Knowledge for Image Inpainting
By Jie Yang, Zhiquan Qi, Yong Shi. 2020
146 citations recorded by the source corpus.
This paper develops a multi-task learning framework that attempts to incorporate the image structure knowledge to assist image inpainting, which is not well explored in previous works. The primary idea is to train a shared generator to simultaneously complete the corrupted image and corresponding structures — edge and gradient, thus implicitly encouraging the generator to exploit relevant structure knowledge while inpainting. In the meantime, we also introduce a structure embedding scheme to explicitly embed the learned structure features into the inpainting process, thus to provide possible preconditions for image completion. Specifically, a novel pyramid structure loss is proposed to supervise structure learning and embedding. Moreover, an attention mechanism is developed to further exploit the recurrent structures and patterns in the image to refine the generated structures and contents. Through multi-task learning, structure embedding besides with attention, our framework takes advantage of the structure knowledge and outperforms several state-of-the-art methods on benchmark datasets quantitatively and qualitatively.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Arsenii Ashukha
- Elizaveta Logacheva
- Victor S. Lempitsky
- Roman Suvorov
- Hongyu Yang
- Anton Mashikhin
- Kate Saenko
- Trevor Darrell
In Computer Vision and Pattern Recognition
- Arsenii Ashukha
- Elizaveta Logacheva
- Roman Suvorov
- Anton Mashikhin
- Hongyu Yang
- Xiefan Guo
- Di Huang
- Aleksei Silvestrov
Citations in the corpus
- Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, methodology): “Several works suggest two-stage methods that use completion of other structure types as an intermediate step: salient edges in [32], semantic segmentation maps in [44], foreground object contours in [48], gradient maps in [52], and edge-preserved smooth images in [36].”
- Image Inpainting via Conditional Texture and Structure Dual Generation (2021, background): “[32] introduce a multi-task framework to generate sharp edges by adding structural constraints.”