Data-driven hallucination of different times of day from a single outdoor photo

By Yichang Shih, Sylvain Paris, Frédo Durand, William T. Freeman. 2013

216 citations recorded by the source corpus.

We introduce "time hallucination": synthesizing a plausible image at a different time of day from an input image. This challenging task often requires dramatically altering the color appearance of the picture. In this paper, we introduce the first data-driven approach to automatically creating a plausible-looking photo that appears as though it were taken at a different time of day. The time of day is specified by a semantic time label, such as "night". Our approach relies on a database of time-lapse videos of various scenes. These videos provide rich information about the variations in color appearance of a scene throughout the day. Our method transfers the color appearance from videos with a similar scene as the input photo. We propose a locally affine model learned from the video for the transfer, allowing our model to synthesize new color data while retaining image details. We show that this model can hallucinate a wide range of different times of day. The model generates a large sparse linear system, which can be solved by off-the-shelf solvers. We validate our methods by synthesizing transforming photos of various outdoor scenes to four times of interest: daytime, the golden hour, the blue hour, and nighttime.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Image-to-Image Translation with Conditional Adversarial Networks (2016, background): “, [14, 23, 18, 8, 10, 50, 30, 36, 16, 55, 58]), despite the fact that the setting is always the same: predict pixels from pixels.”
Generative Visual Manipulation on the Natural Image Manifold (2016, methodology): “The data term relaxes the color constancy assumption by introducing a locally affine color transfer model A [32] while the spatial and color regularization terms encourage smoothness in both the motion and color change.”
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks (2017, background): “However, obtaining paired training data can be difﬁcult and expensive.”
Learning Temporal Transformations from Time-Lapse Videos (2016, methodology): “[8] hallucinates an input scene image at a different time of day by making use of a timelapse video dataset exhibiting lighting changes in an example-based color transfer technique.”
Universal Style Transfer via Feature Transforms (2017, methodology): “Existing style transfer methods are mostly example-based [13, 25, 24, 7, 21].”
Deep Photo Style Transfer (2017, methodology): “[15] generate visually satisfying results.”
Synthesizing Images of Humans in Unseen Poses (2018, background): “Examples include scene segmentation [18, 21], surface normal prediction [2], coloring [31], style transfer [12, 23], edge detection [29] and sketch inversion [5].”
A Style-Aware Content Loss for Real-time HD Style Transfer (2018, methodology): “Inspired by Image Analogies [18], approaches [11,30,39,40] are based on finding dense correspondence between content and style image and often require image pairs to depict similar content.”
Manipulating Attributes of Natural Scenes via Hallucination (2018, methodology): “Some data-driven approaches [Laffont et al. 2014; Shih et al. 2013] leverage the timelapse video datasets taken for same scene to capture scene variations that occur at different times.”
A Closed-form Solution to Photorealistic Image Stylization (2018, methodology): “Local methods [28, 27, 35, 12, 33] perform spatial color mapping through finding dense correspondences between the content and style photos based on either low-level or high-level features.”
MIXGAN: Learning Concepts from Different Domains for Mixture Generation (2018, background): “…network to learn a mapping from paired images, e.g. translating sketches to photos [Isola et al., 2017; Eigen and Fergus, 2015; Hertzmann et al., 2001; Johnson et al., 2016; Laffont et al., 2014; Long et al., 2015; Shih et al., 2013; Wang and Gupta, 2016; Xie and Tu, 2015; Zhang et al., 2016].”
Visual attribute transfer through deep image analogy (2017, methodology): “Compared with other local color transfer methods, e.g. shih et al.[Shih et al. 2013] and Luan et al.[Luan et al. 2017] in Figure 21(c), our method can produce visually comparable results to theirs.”
The Contextual Loss for Image Transformation with Non-Aligned Data (2018, background): “Finally, there are also approaches tailored to a specific domain and style, such as faces or time-of-day in city-scape images [29,30].”
Deep bilateral learning for real-time image enhancement (2017, methodology): “…features, with coefficients defined by the channels of the sliced feature map ¯ A : Interpolated affine transformations similar to this have been used successfully for matting [Levin et al. 2008], intrinsic image decomposition [Bousseau et al. 2009] and time of day transfer [Shih et al. 2013].”