VRT: A Video Restoration Transformer

By Jingyun Liang, Jiezhang Cao, Yuchen Fan, Kai Zhang, Rakesh Ranjan, Yawei Li, Radu Timofte, Luc Van Gool. IEEE Transactions on Image Processing, 2022

422 citations recorded by the source corpus.

Video restoration aims to restore high-quality frames from low-quality frames. Different from single image restoration, video restoration generally requires to utilize temporal information from multiple adjacent but usually misaligned video frames. Existing deep methods generally tackle with this by exploiting a sliding window strategy or a recurrent architecture, which are restricted by frame-by-frame restoration. In this paper, we propose a Video Restoration Transformer (VRT) with parallel frame prediction ability. More specifically, VRT is composed of multiple scales, each of which consists of two kinds of modules: temporal reciprocal self attention (TRSA) and parallel warping. TRSA divides the video into small clips, on which reciprocal attention is applied for joint motion estimation, feature alignment and feature fusion, while self attention is used for feature extraction. To enable cross-clip interactions, the video sequence is shifted for every other layer. Besides, parallel warping is used to further fuse information from neighboring frames by parallel feature warping. Experimental results on five tasks, including video super-resolution, video deblurring, video denoising, video frame interpolation and space-time video super-resolution, demonstrate that VRT outperforms the state-of-the-art methods by large margins (up to 2.16dB) on fourteen benchmark datasets. The codes are available at https://github.com/JingyunLiang/VRT.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/52c7dc185708962d58367fd8c710e3f34e26d9dd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Chen Change Loy
3. Ming-Hsuan Yang
4. Chao Dong
5. Dacheng Tao
6. Lei Zhang
7. Xiaogang Wang
8. Chunhua Shen

In Media Technology

1. Xiangyu Chen
2. Jiantao Zhou
3. X. Zhang
4. Jian‐Yuan Sun
5. Xiaojie Chu
6. Chao Dong
7. Liangyu Chen
8. Radu Timofte

Citations in the corpus

• Activating More Pixels in Image Super-Resolution Transformer (2022, background): “Due to the impressive performance, Transformer has also been introduced for low-level vision tasks [5,54,37,29,3,62,28,26].”
• Simple Baselines for Image Restoration (2022, background): “GLU or its variants has verified their effectiveness in NLP[29,9,8], and there is a prosperous trend of them in computer vision[30,37,16,19].”