Video Inpainting of Complex Scenes

By Alasdair Newson, Andrés Almansa, Matthieu Fradet, Yann Gousseau, Patrick Pérez. 2014

264 citations recorded by the source corpus.

We propose an automatic video inpainting algorithm which relies on the optimisation of a global, patch-based functional. Our algorithm is able to deal with a variety of challenging situations which naturally arise in video inpainting, such as the correct reconstruction of dynamic textures, multiple moving objects and moving background. Furthermore, we achieve this in an order of magnitude less execution time with respect to the state-of-the-art. We are also able to achieve good quality results on high definition videos. Finally, we provide specific algorithmic details to make implementation of our algorithm as easy as possible. The resulting algorithm requires no segmentation or manual input other than the definition of the inpainting mask, and can deal with a wider variety of situations than is handled by previous work. 1. Introduction. Advanced image and video editing techniques are increasingly common in the image processing and computer vision world, and are also starting to be used in media entertainment. One common and difficult task closely linked to the world of video editing is image and video " inpainting ". Generally speaking, this is the task of replacing the content of an image or video with some other content which is visually pleasing. This subject has been extensively studied in the case of images, to such an extent that commercial image inpainting products destined for the general public are available, such as Photoshop's " Content Aware fill " [1]. However, while some impressive results have been obtained in the case of videos, the subject has been studied far less extensively than image inpainting. This relative lack of research can largely be attributed to high time complexity due to the added temporal dimension. Indeed, it has only very recently become possible to produce good quality inpainting results on high definition videos, and this only in a semi-automatic manner. Nevertheless, high-quality video inpainting has many important and useful applications such as film restoration, professional post-production in cinema and video editing for personal use. For this reason, we believe that an automatic, generic video inpainting algorithm would be extremely useful for both academic and professional communities.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Generative Image Inpainting with Contextual Attention (2018, methodology): “It has many applications in photo editing, image-based rendering and computational photography [3, 23, 28, 29, 34, 39].”
Coarse-to-Fine Image Inpainting via Region-wise Convolutions and Non-Local Correlation (2019, methodology): “It can be widely applied in many tasks [Barnes et al., 2009a; Newson et al., 2014; Park et al., 2017; Simakov et al., 2008], such as photo editing, image-based rendering, computational photography, etc. Till now, there have been many methods proposed for generating desirable ∗Corresponding…”
Deep Video Inpainting (2019, background): “[17] extend this by developing a spatiotemporal version of PatchMatch [2] to strengthen the temporal coherence and speed up the patch matching.”
Learning Joint Spatial-Temporal Transformations for Video Inpainting (2020, methodology): “temporal transformer. 2 Related Work To develop high-quality video inpainting technology, many eorts have been made on lling missing regions with spatially and temporally coherent contents in videos [2,13,18,24,29,33]. We discuss representative patch-based methods and deep generative models for vide”
Flow-edge Guided Video Completion (2020, methodology): “Up until a few years ago, most methods used patch-based synthesis techniques [14,26,39].”