Object Scene Representation Transformer
By Mehdi S. M. Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Paveti'c, Mario Luvci'c, Leonidas J. Guibas, Klaus Greff, Thomas Kipf. 2022
120 citations recorded by the source corpus.
A compositional understanding of the world in terms of objects and their geometry in 3D space is considered a cornerstone of human cognition. Facilitating the learning of such a representation in neural networks holds promise for substantially improving labeled data efficiency. As a key step in this direction, we make progress on the problem of learning 3D-consistent decompositions of complex scenes into individual objects in an unsupervised fashion. We introduce Object Scene Representation Transformer (OSRT), a 3D-centric model in which individual object representations naturally emerge through novel view synthesis. OSRT scales to significantly more complex scenes with larger diversity of objects and backgrounds than existing methods. At the same time, it is multiple orders of magnitude faster at compositional rendering thanks to its light field parametrization and the novel Slot Mixer decoder. We believe this work will not only accelerate future architecture exploration and scaling efforts, but it will also serve as a useful tool for both object-centric as well as neural scene representation learning communities.
Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Aerospace Engineering
- Peter R. Florence
- Sergey Levine
- Karol Hausman
- Tianhe Yu
- Andy Zeng
- Yevgen Chebotar
- Fei Xia
- Marc Toussaint
In Computer Vision and Pattern Recognition
- Sergey Levine
- Luc Van Gool
- Trevor Darrell
- Li Fei-Fei
- Jitendra Malik
- Dacheng Tao
- Xiaogang Wang
- Andrew Zisserman
In Model Architectures
- Sergey Levine
- Pierre Sermanet
- Jonathan Tompson
- Corey Lynch
- Quoc V. Le
- Tianhe Yu
- Marc Toussaint
- Karol Hausman
Citations in the corpus
- PaLM-E: An Embodied Multimodal Language Model (2023, methodology): “Based on SRT (Sajjadi et al., 2022b), OSRT learns 3D-centric neural scene representations through a novel view synthesis task.”
- Transfer Visual Prompt Generator across LLMs (2023, unknown): “[54] NF-ResNet-50 [6] NF-ResNet-50 GPT-2-like † [42] No Flamingo [2] NFNet-F6 [6]+Resampler [21] Resampler Chinchilla [18] Xattn-Dense PaLM-E [16] ViT [15] / OSRT [49] All PaLM [11] No BLIP-2 [30] EVA-CLIP [52] + Q-Former [30] Q-Former OPT [60] / Flan-T5 [12] No Training Paradigm.”
- A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “As the low-level policy executes actions, PaLM-E can also replan based on changes in the environment.”