Fast Image Processing with Fully-Convolutional Networks

By Qifeng Chen, Jia Xu, Vladlen Koltun. 2017

356 citations recorded by the source corpus.

We present an approach to accelerating a wide variety of image processing operators. Our approach uses a fully-convolutional network that is trained on input-output pairs that demonstrate the operator’s action. After training, the original operator need not be run at all. The trained network operates at full resolution and runs in constant time. We investigate the effect of network architecture on approximation accuracy, runtime, and memory footprint, and identify a specific architecture that balances these considerations. We evaluate the presented approach on ten advanced image processing operators, including multiple variational models, multiscale tone and detail manipulation, photographic style transfer, nonlocal dehazing, and nonphoto- realistic stylization. All operators are approximated by the same model. Experiments demonstrate that the presented approach is significantly more accurate than prior approximation schemes. It increases approximation accuracy as measured by PSNR across the evaluated operators by 8.5 dB on the MIT-Adobe dataset (from 27.5 to 36 dB) and reduces DSSIM by a multiplicative factor of 3 com- pared to the most accurate prior approximation scheme, while being the fastest. We show that our models general- ize across datasets and across resolutions, and investigate a number of extensions of the presented approach.

Subfields: Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Getting to 99% Accuracy in Interactive Segmentation (2020, neutral): “[11] VGG16 hypercolumn, CAN [29] 1:1 none”
Speech Denoising with Deep Feature Losses (2018, background): “The adaptive normalization operator Γ in our network matches the one proposed in [31] and improves performance and training speed.”
Learning to Optimize: Training Deep Neural Networks for Interference Management (2017, background): “learns computationally consuming operations in image processing through convolutional networks [22].”
Low-Light Image Enhancement via a Deep Hybrid Network (2019, methodology): “[43] use a fully-convolutional network to model the operators.”
Single Image Reflection Separation with Perceptual Losses (2018, background): “Our network f is a fully convolutional network that has a similar network architecture to the context aggregation network [33, 4].”
Miss Detection vs. False Alarm: Adversarial Learning for Small Object Segmentation in Infrared Images (2019, methodology): “The weights of the generators are initialized using the identity initialization technique [5], while the weights of the discriminator is initialized by following the literature [16].”
Learned perceptual image enhancement (2017, methodology): “The enhancement CNN is a context aggregation network (CAN) [5, 36], which generates enhanced image c W ( x ) , in which weights W are trained by our perceptual loss.”
Deep Photo Enhancer: Unpaired Learning for Image Enhancement from Photographs with GANs (2018, methodology): “[6] approximated existing filters using a fully convolutional network.”
Fast End-to-End Trainable Guided Filter (2018, methodology): “Concretely, we develop a fully-convolutional neural network serving as F (I), which is composed of two convolution layers, between which are an adaptive normalization layer [10] and a leaky ReLU layer.”
Gated Context Aggregation Network for Image Dehazing and Deraining (2018, methodology): “…of our smoothed dilated resblock in alleviating the gridding artifacts, we compare it with the previous widely-used exponentially dilated resblock [7, 9, 25], where the dilation rates of adjacent res-blocks are increased exponentially (e.g., 2, 4, 8, 16, 32) are also very curious about whether…”
Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining (2018, methodology): “We implement it by extending Context Aggregation Net (CAN) [11,12] with Squeezeand-Excitation (SE) blocks [29], and name it as SE Context Aggregation Net (SCAN).”
Low-Light Image and Video Enhancement Using Deep Learning: A Survey (2021, background): “Moreover, although some general photo enhancement methods [45], [46], [47], [48], [49], [50], [51], [52], [53] can improve the brightness of images to some extent, we omit them in this survey as they are not designed to handle diverse low-light conditions.”
Interactive Image Segmentation with Latent Diversity (2018, methodology): “As described in Section 3, we used the context aggregation network (CAN) for dense prediction [44, 7].”
Attention Guided Low-Light Image Enhancement with a Large Scale Low-Light Simulation Dataset (2019, background): “Previous research [12] demonstrates the effectiveness of dilated convolution in image processing.”
Classification-Driven Dynamic Image Enhancement (2017, background): “Several recent works have shown that convolutional neural networks (CNN) [2, 3, 23, 27, 39, 40] can successfully WLS Filter [a]”
A Fully Convolutional Two-Stream Fusion Network for Interactive Image Segmentation (2018, result): “Note that, compared to the proposed FCTSFN, DEXTR achieves the reported performance with more training data (Pascal VOC 2012 + SBD [35]), with an online hard example mining (OHEM) [36] based training strategy and a more advanced base network (ResNet-101 [10]); similarly, LDN achieve its reported per”