Deep Video Inpainting

By Dahun Kim, Sanghyun Woo, Joon-Young Lee, In-So Kweon. 2019

235 citations recorded by the source corpus.

Video inpainting aims to fill spatio-temporal holes with plausible content in a video. Despite tremendous progress of deep neural networks for image inpainting, it is challenging to extend these methods to the video domain due to the additional time dimension. In this work, we propose a novel deep network architecture for fast video inpainting. Built upon an image-based encoder-decoder model, our framework is designed to collect and refine information from neighbor frames and synthesize still-unknown regions. At the same time, the output is enforced to be temporally consistent by a recurrent feedback and a temporal memory module. Compared with the state-of-the-art image inpainting algorithm, our method produces videos that are much more semantically correct and temporally smooth. In contrast to the prior video completion method which relies on time-consuming optimization, our method runs in near real-time while generating competitive video results. Finally, we applied our framework to video retargeting task, and obtain visually pleasing results.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Free-Form Image Inpainting With Gated Convolution (2018, background): “The latter approach [15, 49, 45, 46, 38, 37, 48, 26, 52, 33, 35, 19] can exploit semantics learned from large scale datasets to synthesize contents in nonstationary images in an end-to-end fashion.”
FlowFormer: A Transformer Architecture for Optical Flow (2022, background): “In many downstream video tasks, such as action recognition [46,37,61], video inpainting [28,51,13], video super-resolution [30,5,39], and frame interpolation [52,33,20], optical flow serves as a fundamental component providing dense correspondences as valuable clues for prediction.”
Artificial intelligence in the creative industries: a review (2020, methodology): “VINet [157] employs an encoder-decoder model using an RNN to estimate optical flow, processing multiple input frames concatenated with the previous inpainting results.”
Flow-edge Guided Video Completion (2020, methodology): “Our method substantially improves the performance over state-of-the-art algorithms [14,20,22,26,27,42] on the three metrics.”
Learning Joint Spatial-Temporal Transformations for Video Inpainting (2020, methodology): “Since the training set of DAVIS is limited (60 videos with at most 90 frames for each), we initialize model weights by a pre-trained model on YouTube-VOS following the settings used in [16,33].”
VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning (2021, background): “In the case of images, generative approaches often operate on pixel space [5, 10, 35, 61, 70], which can be extremely expensive for larger input size like videos and has hence limited the widespread adoption of these methods.”