Deep bilateral learning for real-time image enhancement

By Michael Gharbi, Jiawen Chen, Jonathan T. Barron, Samuel W. Hasinoff, Frédo Durand. ACM Transactions on Graphics, 2017

838 citations recorded by the source corpus.

Performance is a critical challenge in mobile image processing. Given a reference imaging pipeline, or even human-adjusted pairs of images, we seek to reproduce the enhancements and enable real-time evaluation. For this, we introduce a new neural network architecture inspired by bilateral grid processing and local affine color transforms. Using pairs of input/output images, we train a convolutional neural network to predict the coefficients of a locally-affine model in bilateral space. Our architecture learns to make local, global, and content-dependent decisions to approximate the desired image transformation. At runtime, the neural network consumes a low-resolution version of the input image, produces a set of affine transformations in bilateral space, upsamples those transformations in an edge-preserving fashion using a new slicing node, and then applies those upsampled transformations to the full-resolution image. Our algorithm processes high-resolution images on a smartphone in milliseconds, provides a real-time viewfinder at 1080p resolution, and matches the quality of state-of-the-art approximation techniques on a large class of image operators. Unlike previous work, our model is trained off-line from data and therefore does not require access to the original operator at runtime. This allows our model to learn complex, scene-dependent transformations for which no reference implementation is available, such as the photographic edits of a human retoucher.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/dd656e5f21003a33b10eb3574a093774a7081d46

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images (2021, background): “The resulting renderings can be retouched like any raw photograph: here we show (d, left) a dark all-in-focus exposure with a simple global tonemap and (d, right) a brighter, synthetically refocused exposure postprocessed by HDRNet [17].”
Deep Photo Enhancer: Unpaired Learning for Image Enhancement from Photographs with GANs (2018, background): “[7] proposed a fast approximation for existing filters.”
EnlightenGAN: Deep Light Enhancement Without Paired Supervision (2019, background): “HDR-Net [24] incorporated deep networks with the ideas of bilateral grid processing and local affine color transforms with pairwise supervision.”
Single image portrait relighting (2019, methodology): “…aims to decrease the dynamic range of an input image while preserving its local contrast, can also be thought of as a coarse approximation to relighting, and has been explored through classic algorithms [Farbman et al. 2008; Paris et al. 2011] and learning-based approaches [Gharbi et al. 2017].”
Guided Collaborative Training for Pixel-wise Semi-Supervised Learning (2020, methodology): “We regard HDRNet [13] as the task model.”
Learning Blind Video Temporal Consistency (2018, methodology): “Recent advances of deep convolutional neural networks (CNNs) have led to the development of many powerful image processing techniques including, image filtering [30,37], enhancement [10,24,38], style transfer [17,23,29], colorization [19,41], and general image-to-image translation tasks [21,27,43].”
Learning Guided Convolutional Network for Depth Completion (2019, background): “Gharbi et al. [41] generate afﬁne parameters by a deep network to perform color transforms for image enhancement.”
Learning a Deep Single Image Contrast Enhancer from Multi-Exposure Images (2018, methodology): “In [29], a CNN is trained to set the parameters of bilateral filters, which are then used to enhance an input image to a desired image edited by professional photographers.”
A Deep Learning System to Screen Novel Coronavirus Disease 2019 Pneumonia (2020, positive): “With the rapid development of computer technology, digital image processing technology has been widely applied in the medical field, including organ segmentation and image enhancement and repair, thereby providing support for subsequent medical diagnosis [13,14].”
Faceshop (2018, methodology): “Recently, numerous deep learning techniques have been proposed for global editing applications such as general image enhancing [Gharbi et al. 2017; Yan et al. 2016], grayscale to color transformation [Iizuka et al.”
Fast End-to-End Trainable Guided Filter (2018, methodology): “Our primary baseline is Deep Bilateral Learning (DBL) [18], which shares similar architecture to ours and achieves a good balance between quality and speed.”
Learning Image-Adaptive 3D Lookup Tables for High Performance Photo Enhancement in Real-Time (2020, methodology): “The ﬁrst category of methods learn a dense pixel-to-pixel mapping between the input and output image pairs [7], or predict pixel-wise transformations to enhance the input image [2], [6], [10].”
Portrait shadow manipulation (2020, methodology): “2012), the HDRnet dataset (Gharbi et al. 2017), and our own captures.”
Learning Enriched Features for Real Image Restoration and Enhancement (2020, methodology): “Input image HDRNet [35] DPE [18] DeepUPE [106] MIRNet (Ours) Ground-truth Fig.”