View Synthesis by Appearance Flow

By Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, Alexei A. Efros. 2016

734 citations recorded by the source corpus.

We address the problem of novel view synthesis: given an input image, synthesizing new images of the same object or scene observed from arbitrary viewpoints. We approach this as a learning task but, critically, instead of learning to synthesize pixels from scratch, we learn to copy them from the input image. Our approach exploits the observation that the visual appearance of different views of the same instance is highly correlated, and such correlation could be explicitly learned by training a convolutional neural network (CNN) to predict appearance flows – 2-D coordinate vectors specifying which pixels in the input view could be used to reconstruct the target view. Furthermore, the proposed framework easily generalizes to multiple input views by learning how to optimally combine single-view predictions. We show that for both objects and scenes, our approach is able to synthesize novel views of higher perceptual quality than previous CNN-based techniques.

Subfields: Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Photographic Image Synthesis with Cascaded Refinement Networks (2017, background): “Researchers have also studied image inpainting [32], superresolution [3, 17, 19], novel view synthesis [10, 45, 51], and interactive image manipulation [52].”
Spatially Transformed Adversarial Examples (2018, background): “Zhou et al. (2016a) adopted the spatial transformers for synthesizing novel views of the same object and has shown that a geometric method can produce more realistic results compared to pure pixel-based methods.”
Learning Category-Specific Mesh Reconstruction from Image Collections (2018, methodology): “Instead, we take inspiration from [38] and formulate this task as that of predicting the appearance flow.”
3D Shape Induction from 2D Views of Multiple Objects (2016, background): “More recently, convolutional networks have been trained to generate views of 3D objects given their attributes and camera parameters [8], to generate 3D shape given a 2D view of the object [29], and to generate novel views of an object [36].”
Unsupervised Learning of 3D Structure from Images (2016, background): “More recent work [19, 5, 4, 37] addresses some of these limitations by learning parts of the encoding-decoding pipeline depicted in ﬁgure 2 in separate stages.”
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (2018, background): “Here, the bilinear sampling operator interpolates the pixel values bilinearly with respect to a location (x, y) and its four neighbouring pixels in the image, similar to [42].”
Tracking Emerges by Colorizing Videos (2018, background): “A related but diﬀerent line of work explores how to learn geometric properties or cycle consistencies with self-supervision, for example for motion capture or correspondence [20–24].”
Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks (2016, neutral): “Thus, in our work, rather than modeling a motion prior as a context-free distribution, we propose to model the conditional motion distribution of future frames given an input image by incorporating a series of low-and high-level image cues.”
Unsupervised Learning of Depth and Ego-Motion from Video (2017, methodology): “Recently, end-to-end learning has been applied to reconstruct novel views by transforming the input based on depth or ﬂow, e.g., DeepStereo [10], Deep3D [51] and Appearance Flows [54].”
Generative Image Inpainting with Contextual Attention (2018, methodology): “For appearance flow [42], we train on the same framework except that the contextual attention layer is replaced with a convolution layer to directly predict 2-D pixel offsets as attention.”
Learning to Decompose and Disentangle Representations for Video Prediction (2018, background): “Instead of directly generating the pixels, transformation-based models focus on predicting the difference/transformation between frames and lead to sharper results [5, 8, 21, 35, 39, 40, 44, 45].”
Gibson Env: Real-World Perception for Embodied Agents (2018, methodology): “via an encoderdecoder like architecture that directly renders pixels [30, 52, 88] or predicts a flow map for pixels [99].”
Scribbler: Controlling Deep Image Synthesis with Sketch and Color (2016, background): “In general, deep image synthesis can be conditioned on any input vector [46], such as attributes [50], 3d viewpoint parameters and object identity [8], image and desired viewpoint [55], or grayscale image [54, 17, 23] .”
Pluralistic Image Completion (2019, background): “More recently, several works [42, 38, 35] exploited spatial attention [16, 46] to get high-frequency details.”
Video Frame Interpolation via Adaptive Separable Convolution (2017, methodology): “Recently, Zhou et al . developed an method that employs a convolutional neural network to estimate appearance ﬂow and then uses this estimation to warp input pixels to create a novel view [64].”
An Unsupervised Learning Model for Deformable Medical Image Registration (2018, methodology): “The layer has since been used for dense spatial transformations as well [34, 46].”
DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization (2024, background): “Novel view synthesis aims to generate unseen views of the same object or scene from a set of given images [1, 59].”
Multi-content GAN for Few-Shot Font Style Transfer (2017, positive): “For example, given a single-view image, encoder-decoder architectures have been proposed to hallucinate novel views of faces [15, 31, 4], bodies [35], and other rigid objects [37, 25].”