Object-Centric Learning with Slot Attention

By Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, Thomas Kipf. 2020

1,186 citations recorded by the source corpus.

Learning object-centric representations of complex scenes is a promising step towards enabling efficient abstract reasoning from low-level perceptual features. Yet, most deep learning approaches learn distributed representations that do not capture the compositional properties of natural scenes. In this paper, we present the Slot Attention module, an architectural component that interfaces with perceptual representations such as the output of a convolutional neural network and produces a set of task-dependent abstract representations which we call slots. These slots are exchangeable and can bind to any object in the input by specializing through a competitive procedure over multiple rounds of attention. We empirically demonstrate that Slot Attention can extract object-centric representations that enable generalization to unseen compositions when trained on unsupervised object discovery and supervised property prediction tasks.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, neutral): “…for object detection (Hu et al., 2018; Carion et al., 2020), video processing (Wang et al., 2018; Sun et al., 2019), image classiﬁcation (Wu et al., 2020), unsupervised object discovery (Locatello et al., 2020), or uniﬁed text-vision tasks (Chen et al., 2020c; Lu et al., 2019; Li et al., 2019).”
Training data-efficient image transformers & distillation through attention (2020, background): “, 2018), unsupervised object discovery (Locatello et al., 2020), and text-vision tasks (Chen et al.”
Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “Several other works use attention to process or refine latent spaces that interface with the input/output data using task- or domain-specific architectures [12, 45, 92].”
Perceiver: General Perception with Iterative Attention (2021, background): “…domain-agnostic analogue of models that stacks self-attention on top of convolutional feature maps to perform cheap but global processing on top or in conjunction with otherwise spatially localized convolutional feature maps (e.g. Carion et al. 2020; Locatello et al. 2020; Wang et al. 2021).”
Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation (2021, methodology): “This applies even to transformers with random “position” embeddings, which similarly to TSPN (Kosiorek et al., 2020) and Slot Attention (Locatello et al., 2020) use noise to make elements Next, we aim to evaluate our contributions in Section 3.”
PaLM-E: An Embodied Multimodal Language Model (2023, methodology): “, 2022a): rather than relying on external knowledge about objects, they are discovered in an unsupervised way through inductive biases in the architecture (Locatello et al., 2020).”
Scalable Adaptive Computation for Iterative Generation (2022, background): “, 2020), Slot Attention (Locatello et al., 2020), and BigBird (Zaheer et al.”
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, methodology): “Let us take Capsule Networks (Sabour et al., 2017) and the Slot Attention module (Locatello et al., 2020) in the computer vision field as examples: these modules try to create object-centric and part-centric representations inside the latent space of the network.”
Towards Causal Representation Learning (2021, background): “learning [10, 39, 83, 86, 87, 138, 155, 160, 262, 255] can also be considered a special case of disentangled factorization as discussed here.”
On the Binding Problem in Artificial Neural Networks (2020, methodology): “…neural networks (Joulin and Mikolov, 2015; Graves et al., 2016), in self-attention models (Vaswani et al., 2017; Dehghani et al., 2019; Locatello et al., 2020), in Recurrent Independent Mechanisms (RIMs; (Goyal et al., 2019)), albeit without having a common format, and in certain…”
How to Represent Part-Whole Hierarchies in a Neural Network (2021, methodology): “One way to avoid using a mixture for modeling the diﬀerent types of part is to have a set of identical, “universal” capsules, each of which contains enough knowledge to model any type of part [Locatello et al., 2020, Srivastava et al., 2019, Sun et al., 2020b].”
Multiscale Vision Transformers (2021, methodology): “Self-attention mechanisms has been used for image understanding [88, 120, 57, 7], unsupervised object recognition [80] as well as vision and language [83, 71].”
LambdaNetworks: Modeling Long-Range Interactions Without Attention (2021, background): “, 2020); object detection and object-centric tasks (Wang et al., 2018; Hu et al., 2018a; Carion et al., 2020; Locatello et al., 2020); video tasks (Sun et al.”
GroupViT: Semantic Segmentation Emerges from Text Supervision (2022, methodology): “The Grouping Block works similarly to a single iteration of the previously proposed Slot Attention mechanism [46].”
GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields (2020, background): “To this end, many works [9, 25, 39, 43, 44, 48, 54, 71, 74, 97, 98] investigate how disentangled representations can be learned from data without explicit supervision.”
Generative Adversarial Transformers (2021, methodology): “Works such as Capsule networks [62] leverage ideas from psychology about Gestalt principles [34, 64], perceptual grouping [6] or analysis-by-synthesis [4], and like us, introduce ways to piece together visual elements to discover compound entities or, in the cases of Set Transformers [50] and A(2)-N”