Neural scene representation and rendering

By S. M. Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S. Morcos, Marta Garnelo, Avraham Ruderman, Andrei A. Rusu, Ivo Danihelka, Karol Gregor and 12 more. 2018

674 citations recorded by the source corpus.

A scene-internalizing computer program To train a computer to “recognize” elements of a scene supplied by its visual sensors, computer scientists typically use millions of images painstakingly labeled by humans. Eslami et al. developed an artificial vision system, dubbed the Generative Query Network (GQN), that has no need for such labeled data. Instead, the GQN first uses images taken from different viewpoints and creates an abstract description of the scene, learning its essentials. Next, on the basis of this representation, the network predicts what the scene would look like from a new, arbitrary viewpoint. Science, this issue p. 1204 A computer vision system predicts how a 3D scene looks from any viewpoint after just a few 2D views from other viewpoints. Scene representation—the process of converting visual sensory data into concise descriptions—is a requirement for intelligent behavior. Recent work has shown that neural networks excel at this task when provided with large, labeled datasets. However, removing the reliance on human labeling remains an important open problem. To this end, we introduce the Generative Query Network (GQN), a framework within which machines learn to represent scenes using only their own sensors. The GQN takes as input images of a scene taken from different viewpoints, constructs an internal representation, and uses this representation to predict the appearance of that scene from previously unobserved viewpoints. The GQN demonstrates representation learning without human labels or domain knowledge, paving the way toward machines that autonomously learn to understand the world around them.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Learning Representations by Maximizing Mutual Information Across Views (2019, methodology): “, 2018, Eslami et al., 2018] rely on explicit density estimation, our model uses the contrastive bound from CPC [van den Oord et al., 2018], which has been further analyzed by McAllester and Stratos [2018], and Poole et al. [2019]. Another line of prior work considers relating the information shared”
ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition (2021, background): “[9] SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse,”
Reconciling deep learning with symbolic artificial intelligence: representing objects and relations (2019, background): “For example, a generative query network learns to render a 3D scene from any viewpoint given images from a few sample viewpoints [30].”
Shaping Belief States with Generative Environment Models for RL (2019, background): “As previous work has shown, globally coherent environment maps can be learned by position conditioned models [15].”
On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset (2019, methodology): “Beyond the evaluation of representation learning algorithms, the proposed dataset can likewise be used for other tasks such as 3D reconstruction and scene rendering [9] or learning compositional visual concepts [17].”
MONet: Unsupervised Scene Decomposition and Representation (2019, methodology): “On the other hand, Generative Query Networks [Eslami et al., 2018] have demonstrated impressive modelling of rich 3D scenes, but do not explicitly factor representations into objects and are reliant on privileged view information as part of their training.”
Conditional Neural Processes (2018, background): “Generative Query Networks (GQN), for example, predict new viewpoints in 3D scenes given some context observations using a similar training regime to NPs (Eslami et al., 2018).”
Neural Processes (2018, background): “Finally, NPs and CNPs themselves can be seen as generalizations of recently published generative query networks (GQN) which apply a similar training procedure to predict new viewpoints in 3D scenes given some context observations (Eslami et al., 2018).”
Meta-Learning in Neural Networks: A Survey (2020, background): “[213] render scenes from novel views, and talking faces are generated from little data by learning the initialization of an adversarial model for quick adaptation [214].”
GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations (2019, methodology): “We conduct experiments on two canonical and publicly available datasets: the GQN dataset (Es-lami et al. [9]) and ShapeStacks (Groth et al. [10]).”
Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs (2019, background): “Other types of generative models already use convolutional latent distributions [Finn et al., 2015, Levine et al., 2015, Eslami et al., 2018], unlike the ﬂat vectors we consider here.”
Taming VAEs (2018, methodology): “It is inspired by the empirical observation that some high-capacity VAEs such as ConvDRAW[5, 16] may reach much lower reconstruction errors compared to a level that is perceptually distinguishable, at the expense of a weak compression rate (large KL).”
Dream to Control: Learning Behaviors by Latent Imagination (2019, background): “Several objectives for learning representations for control have been proposed (Watter et al., 2015; Jaderberg et al., 2016; Oord et al., 2018; Eslami et al., 2018).”
Pluralistic Image Completion (2019, methodology): “2 “CVAE”).”
Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations (2019, methodology): “SRNs could be explored in a probabilistic framework [2, 3], enabling sampling of feasible scenes given a set of observations.”
SynSin: End-to-End View Synthesis From a Single Image (2019, neutral): “[14] use no implicit 3D representation.”
Genie: Generative Interactive Environments (2024, background): “…2018; Oh et al., 2015), which enable next-frame prediction that is conditioned on action inputs (Bamford and Lucas, 2020; Chiappa et al., 2017; Eslami et al., 2018; Hafner et al., 2020, 2021; Kim et al., 2020, 2021; Micheli et al., 2023; Nunes et al., 2020; Pan et al., 2022; Robine et al.,…”
Adaptive Posterior Learning: few-shot learning with a surprise-based memory module (2019, positive): “…task and thus meta-learning algorithms have been successfully applied to a wide range of challenges like RL (Wang et al., 2016; Finn et al., 2017), program induction (Devlin et al., 2017) few-shot classiﬁcation (Koch et al., 2015; Vinyals et al., 2016) and scene understanding (Eslami et al., 2018).”