Image completion with structure propagation

By Jian Sun, Lu Yuan, Jiaya Jia, Harry Shum. 2005

666 citations recorded by the source corpus.

In this paper, we introduce a novel approach to image completion, which we call structure propagation. In our system, the user manually specifies important missing structure information by extending a few curves or line segments from the known to the unknown regions. Our approach synthesizes image patches along these user-specified curves in the unknown region using patches selected around the curves in the known region. Structure propagation is formulated as a global optimization problem by enforcing structure and consistency constraints. If only a single curve is specified, structure propagation is solved using Dynamic Programming. When multiple intersecting curves are specified, we adopt the Belief Propagation algorithm to find the optimal patches. After completing structure propagation, we fill in the remaining unknown regions using patch-based texture synthesis. We show that our approach works well on a number of examples that are challenging to state-of-the-art techniques.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Free-Form Image Inpainting With Gated Convolution (2018, methodology): “To improve image inpainting, different user guidance is explored including dots or lines [Ashikhmin 2001; Barnes et al. 2009; Drori et al. 2003; Sun et al. 2005], structures [Huang et al. 2014], transformation or distortion information [Huang et al. 2013; Pavi´c et al. 2006] and image exemplars…”
Image Inpainting via Generative Multi-column Convolutional Neural Networks (2018, methodology): “To preserve structure, patch priority computation speciﬁes the patch ﬁlling order [4, 7, 8, 21].”
Image Completion Using Global Optimization (2006, background): “[12] require the user to specify the curves on which the most salient missing structures reside, while Drori et al .”
PatchMatch: a randomized correspondence algorithm for structural image editing (2009, methodology): “Likewise, the best completion algorithms offer tools to guide the result by providing hints for the computer [Sun et al. 2005].”
Semantic Image Inpainting with Progressive Generative Networks (2018, background): “Although non-semantic image inpainting approaches have been studied for decades [2, 5– 7, 9, 17, 25, 27, 31], they cannot generate semantically meaningful content, and thus are not suitable to handle the semantic image inpainting task where large proportions of areas are corrupted in images.”
Statistics of Patch Offsets for Image Completion (2012, methodology): “Matching-based methods [6–10] explicitly match patches in the region to be ﬁlled with those in the known region and copy the content.”
Coherent Semantic Attention for Image Inpainting (2019, background): “On one hand, the architecture of rough network is the same as pix to pix [19].”
A Survey of Human-in-the-loop for Machine Learning (2021, methodology): “The image inpainting frameworks proposed by previous studies can be bifurcated as exemplarbased [122, 123, 124] approaches and deep learning-based [125, 126, 127] methods.”
Interactive image completion with perspective correction (2006, background): “[31], we do not set all gradients on the common boundary to zero but only at those pixels where the gradient in the source fragment is below a certain theshold.”
Wide-Context Semantic Image Extrapolation (2019, background): “It is also a natural choice to use inpainting methods [1, 5, 7, 20, 23, 25, 37] for extrapolating images.”
Get Out of my Picture! Internet-based Inpainting (2009, background): “Except in some simplified cases, these local approaches fail when the region is larger than a few pixels across, which has led to algorithms such as those of Criminisi et al. [11], Sun et al. [ 26 ], and Bertalmio et al. [8] being proposed, which try to preserve both large-scale salient structures a”
Context-Aware Semantic Inpainting (2017, background): “Many algorithms on recovering holes in images or videos have been proposed [8]–[14].”
Shift-Net: Image Inpainting via Deep Feature Rearrangement (2018, neutral): “In exemplar-based inpainting [2, 3, 5, 7, 9, 17, 18, 22, 23, 25, 32, 37, 39, 43, 44, 46], the completion is conducted from the exterior to the interior of the missing part by searching and copying best matching patches from the known region.”
Image Inpainting : Overview and Recent Advances (2014, background): “Structure can be identified in a supervised way, as in [57], where the user specifies curves corresponding to important missing structures (e.g., object boundaries) in the unknown region.”
Image Inpainting by End-to-End Cascaded Refinement With Mask Awareness (2021, background): “Traditional patch-based image inpainting approaches such as [1], [3], [5], [21]–[23] typically propagated appearance information from the remained regions or other source images into the missing regions through various manually defined similarity metrics between patches.”
Image Completion Using Efficient Belief Propagation Via Priority Scheduling and Dynamic Pruning (2007, background): “1) Completion Via Energy Refinement: One advantage of posing image completion as an optimization problem is that one can now refine completion simply by refining the energy function (i.e., adding more terms to it), e.g., to favor spatial coherence during image completion (i.e., fill the target…”
Fast, automatic and fine-grained tampered JPEG image detection via DCT coefficient analysis (2009, methodology): “2(c)) refers to a block in the tampered region or along the synthesis edge and an unchanged block is a block in the unchanged region.”