Learning Joint Spatial-Temporal Transformations for Video Inpainting

By Yanhong Zeng, Jianlong Fu, Hongyang Chao. 2020

353 citations recorded by the source corpus.

High-quality video inpainting that completes missing regions in video frames is a promising yet challenging task. State-of-the-art approaches adopt attention models to complete a frame by searching missing contents from reference frames, and further complete whole videos frame by frame. However, these approaches can suffer from inconsistent attention results along spatial and temporal dimensions, which often leads to blurriness and temporal artifacts in videos. In this paper, we propose to learn a joint Spatial-Temporal Transformer Network (STTN) for video inpainting. Specifically, we simultaneously fill missing regions in all input frames by self-attention, and propose to optimize STTN by a spatial-temporal adversarial loss. To show the superiority of the proposed model, we conduct both quantitative and qualitative evaluations by using standard stationary masks and more realistic moving object masks. Demo videos are available at this https URL.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet (2021, background): “Recent works have explored to apply Transformers to various vision tasks: image classification [5, 12], object detection [3, 59, 56, 8, 34], segmentation [4, 40], image enhancement [4, 45], image generation [27], video processing [58, 51], and 3D point cloud processing [54].”
CvT: Introducing Convolutions to Vision Transformers (2021, background): “…a viable alternative to the convolutional neural networks (CNNs) in visual recognition tasks, such as classiﬁcation [11, 30], object detection [3, 45, 43, 8, 28], segmentation [33, 36], image enhancement [4, 40], image generation [24], video processing [42, 44] and 3D point cloud processing [12].”
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding (2021, methodology): “In addition to image classification, Transformers have also been applied to other compute vision tasks, including object detection [3, 54, 50, 10], segmentation [41, 44], image enhancement [4, 46], image generation [28, 6], video processing [48, 53], and vision-language tasks [26, 35, 7, 33, 21, 20,”
Incorporating Convolution Designs into Visual Transformers (2021, methodology): “Besides, recent work also apply Transformers to various vision tasks, including object detection [5, 52], segmentation [41], image enhancement [6, 44] and video processing [47, 51].”
All Tokens Matter: Token Labeling for Training Better Vision Transformers (2021, background): “Some works also extend transformers for video and 3D point cloud processing [50, 53, 60].”
Aggregated Contextual Transformations for High-Resolution Image Inpainting (2021, background): “Significant progress has been made by the emergence of deep feature learning [15] and adversarial training [16] for image inpainting [13], [14], [17], [18].”
TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classication (2021, methodology): “Recently, Transformer was also applied in many computer vision tasks, including object detection [16, 17], segmentation [18, 19], image enhancement [20, 21] and video processing [34].”
A Survey on Vision Transformer (2020, methodology): “proposes a spatial-temporal transformer network [144] for this task.”
Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling (2021, background): “It has intrigued various vision tasks [12, 20], such as object classification [6, 9], detection [4, 76], segmentation [59, 73], pose estimation [17, 27], image generation [37], video processing [69, 75], and multimodality learning [41].”
Stripformer: Strip Transformer for Fast Image Deblurring (2022, background): “Recently, transformers have achieved comparable or even better performance than CNN models in several vision applications such as image classification [12], object detection [13], semantic segmentation [24], inpainting [25], and super-resolution [26].”
Refiner: Refining Self-attention for Vision Transformers (2021, background): “…of transformers motivate researchers to explore its application on computer vision tasks, including image enhancement [9, 59], image classiﬁcation [19, 60, 66], object detection [6, 68, 13, 64], segmentation [54, 34, 64], video processing [62, 67] and 3D point cloud data processing [63, 22].”
TransGAN: Two Transformers Can Make One Strong GAN (2021, methodology): “Its popularity among computer vision tasks rises recently (Parmar et al., 2018; Yang et al., 2020; Zeng et al., 2020; Carion et al., 2020; Wu et al., 2020; Chen et al., 2020a).”
TransVG: End-to-End Visual Grounding with Transformers (2021, background): “Inspired by the great success of transformers in neural machine translation, a series of transformers [5, 6, 8, 14, 22, 51, 62, 67] applied to vision tasks have been proposed.”
FlowFormer: A Transformer Architecture for Optical Flow (2022, background): “Since then, transformer-based architectures has been introduced into many other vision tasks, such as detection [4], point cloud processing [15,59], image restoration [6,31], video inpainting[57,32], etc, and achieves state-of-the-art in most tasks.”
TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up (2021, methodology): “We are inspired by the recent success of transformer architectures in computer vision [32, 33, 34].”
PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer (2021, neutral): “Recently, some works introduce vision transformer for video understanding tasks such as action recognition [1, 3, 4, 15, 20, 38, 41], action detection [36, 57, 61, 72], video super-resolution [5], video inpainting [32, 70], and 3D animation [9].”