Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations

By Mehdi S. M. Sajjadi, H. Meyer, Etienne Pot, Urs M. Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lucic, Daniel Duckworth, Alexey Dosovitskiy and 3 more. 2021

260 citations recorded by the source corpus.

A classical problem in computer vision is to infer a 3D scene representation from few images that can be used to render novel views at interactive rates. Previous work focuses on reconstructing pre-defined 3D representations, e.g. textured meshes, or implicit representations, e.g. radiance fields, and often requires input images with precise camera poses and long processing times for each novel scene. In this work, we propose the Scene Representation Transformer (SRT), a method which processes posed or unposed RGB images of a new area, infers a “set-latent scene representation ”, and synthesises novel views, all in a single feed-forward pass. To calculate the scene representation, we propose a generalization of the Vision Transformer to sets of images, enabling global information integration, and hence 3D reasoning. An efficient decoder transformer parameterizes the light field by attending into the scene representation to render novel views. Learning is supervised end-to-end by minimizing a novel-view reconstruction error. We show that this method outperforms recent baselines in terms of PSNR and speed on synthetic datasets, including a new dataset created for the paper. Further, we demonstrate that SRT scales to support interactive visualization and semantic segmentation of real-world outdoor environments using Street View imagery.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models (2023, background): “Some follow-up works study coordinate-dependent latents [Chibane et al. 2020; Peng et al. 2020; Sajjadi et al. 2022] that combine traditional data structures such as regular grids with the neural field concept.”
Novel View Synthesis with Diffusion Models (2022, methodology): “In recent literature, geometry-free approaches (i.e., methods without explicit geometric inductive biases like those introduced by volume rendering) such as Light Field Networks (LFN) (Sitzmann et al., 2021) and Scene Representation Transformers (SRT) (Sajjadi et al., 2021) have achieved re-sults co”
CAT3D: Create Anything in 3D with Multi-View Diffusion Models (2024, methodology): “To condition on the camera pose, we use a camera ray representation (“raymap”) that is the same height and width as the latent representations [37, 68] and encodes the ray origin and direction at each spatial location.”
MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images (2024, methodology): “The performance of feed-forward NeRF models progressively improved with the use of feature matching information [3, 5], Trans-formers [10, 25, 30] and 3D volume representations [3, 44].”
Object Scene Representation Transformer (2022, methodology): “Unless stated otherwise, all investigations in this section are conducted on the MSN-Hard dataset with 5 input views.”
Generative Novel View Synthesis with 3D-Aware Diffusion Models (2023, methodology): “Scene Representation Trans-formers [23] utilize a transformer-based approach to merge information from multiple views, which is effective for NVS on both simple and complex scenes.”
ReconFusion: 3D Reconstruction with Diffusion Priors (2023, unknown): “These networks leverage large collections of posed multiview data across many scenes [8, 11, 19, 21, 44, 55, 57, 60, 67, 71].”
PixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction (2023, background): “While the above methods infer inter-pretable 3D representations in the form of signed distances or radiance fields, recent light field scene representations trade interpretability for faster rendering [10, 37, 43, 46, 47].”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, unknown): “Compared to NeRF models, Scene Rendering Trans-former (SRT) [125] (November 2021) took a different approach to volume rendering.”
LRM: Large Reconstruction Model for Single Image to 3D (2023, unknown): “…to model the image-to-3D data distribution (Choy et al., 2016; Yan et al., 2016; Dai et al., 2017; Xu et al., 2019; Wu et al., 2020; M¨uller et al., 2022; Sajjadi et al., 2022; Goel et al., 2023), where a compact latent code is trained to carry the texture, geometry, and pose details of the target.”