Spatio-Temporal Filter Adaptive Network for Video Deblurring

By Shangchen Zhou, Jiawei Zhang, Jin-shan Pan, Haozhe Xie, Wangmeng Zuo, Jimmy S. J. Ren. 2019

232 citations recorded by the source corpus.

Video deblurring is a challenging task due to the spatially variant blur caused by camera shake, object motions, and depth variations, etc. Existing methods usually estimate optical flow in the blurry video to align consecutive frames or approximate blur kernels. However, they tend to generate artifacts or cannot effectively remove blur when the estimated optical flow is not accurate. To overcome the limitation of separate optical flow estimation, we propose a Spatio-Temporal Filter Adaptive Network (STFAN) for the alignment and deblurring in a unified framework. The proposed STFAN takes both blurry and restored images of the previous frame as well as blurry image of the current frame as input, and dynamically generates the spatially adaptive filters for the alignment and deblurring. We then propose the new Filter Adaptive Convolutional (FAC) layer to align the deblurred features of the previous frame with the current frame and remove the spatially variant blur from the features of the current frame. Finally, we develop a reconstruction network which takes the fusion of two transformed features to restore the clear frames. Both quantitative and qualitative evaluation results on the benchmark datasets and real-world videos demonstrate that the proposed algorithm performs favorably against state-of-the-art methods in terms of accuracy, speed as well as model size.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment (2021, methodology): “The recurrent framework is a popular structure adopted in various video processing tasks such as super-resolution [8, 10–12, 14, 24], deblurring [22, 38], and frame interpolation [33].”
AIM 2020 Challenge on Real Image Super-Resolution: Methods and Results (2020, methodology): “Inspired by other image deblurring tasks [30,47,46], a pre-deblur module is used before the residual groups to pre-process blurry inputs and improve super-resolution accuracy.”
Artificial intelligence in the creative industries: a review (2020, background): “A filter adaptive convolution (FAC) has been proposed in the STFAN network [112], where the convolutional kernel is acquired from the feature values in a spatially varying manner.”
VRT: A Video Restoration Transformer (2022, methodology): “Existing video restoration methods can be mainly divided into two categories: sliding window-based methods [1], [2], [3], [4], [5], [6] and recurrent methods [7], [8], [9], [10], [11], [12], [13], [14], [15], [16].”