Adversarial Spatio-Temporal Learning for Video Deblurring

By Kaihao Zhang, Wenhan Luo, Yiran Zhong, Lin Ma, Wei David Liu, Hongdong Li. IEEE Transactions on Image Processing, 2018

341 citations recorded by the source corpus.

Camera shake or target movement often leads to undesired blur effects in videos captured by a hand-held camera. Despite significant efforts having been devoted to video-deblur research, two major challenges remain: 1) how to model the spatio-temporal characteristics across both the spatial domain (i.e., image plane) and the temporal domain (i.e., neighboring frames) and 2) how to restore sharp image details with respect to the conventionally adopted metric of pixel-wise errors. In this paper, to address the first challenge, we propose a deblurring network (DBLRNet) for spatial-temporal learning by applying a 3D convolution to both the spatial and temporal domains. Our DBLRNet is able to capture jointly spatial and temporal information encoded in neighboring frames, which directly contributes to the improved video deblur performance. To tackle the second challenge, we leverage the developed DBLRNet as a generator in the generative adversarial network (GAN) architecture and employ a content loss in addition to an adversarial loss for efficient adversarial training. The developed network, which we name as deblurring GAN, is tested on two standard benchmarks and achieves the state-of-the-art performance.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8c55acd5ec82df6dab3d13338244f7e7eef52e7c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Chen Change Loy
3. Radu Timofte
4. Ming-Hsuan Yang
5. Dacheng Tao
6. Lei Zhang
7. Chao Dong
8. Yu Qiao

Citations in the corpus

• VRT: A Video Restoration Transformer (2022, methodology): “Method DeepDeblur [66] SRN [89] DBN [82] DBLRNet [116] STFAN [123] STTN [37] SFE [104] EDVR [99] TSP [71] PVDNet [80] GSTA [84] ARVo [41] VRT (ours)”
• EDVR: Video Restoration With Enhanced Deformable Convolutional Networks (2019, background): “Several approaches [34, 50] directly fuse multiple frames without explicit temporal alignment, because the existence of blur increases the difficulty of motion estimation.”
• Deblurring by Realistic Blurring (2020, background): “Recently, deep learning methods have achieved a great success in the areas of object recognition [9, 44, 43, 19, 18] and image reconstruction including video deblurring [45], video dehazing [25], and other GAN-based generation tasks [29, 24, 39, 37].”
• Spatiotemporal Knowledge Distillation for Efficient Estimation of Aerial Video Saliency (2019, methodology): “To address the feature issue, deep saliency models [40]–[43] proposed to use feature representation learned from data by using Convolutional Neural Networks (CNNs) [44], [45].”