Set Transformer

By Juho Lee, Yoonho Lee, Jungtaek Kim, Adam R. Kosiorek, Seungjin Choi, Yee Whye Teh. 2018

267 citations recorded by the source corpus.

Many machine learning tasks such as multiple instance learning, 3D shape recognition, and few-shot image classification are defined on sets of instances. Since solutions to such problems do not depend on the order of elements of the set, models used to address them should be permutation invariant. We present an attention-based neural network module, the Set Transformer, specifically designed to model interactions among elements in the input set. The model consists of an encoder and a decoder, both of which rely on attention mechanisms. In an effort to reduce computational complexity, we introduce an attention scheme inspired by inducing point methods from sparse Gaussian process literature. It reduces the computation time of self-attention from quadratic to linear in the number of elements in the set. We show that our model is theoretically attractive and we evaluate it on a range of tasks, demonstrating the state-of-the-art performance compared to recent methods for set-structured data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, background): “[53, 58, 59]), with the exception of the Deep Set Prediction Network (DSPN) [31, 60], learn an ordered representation of the output set with learned per-element initialization, which prevents these approaches from generalizing to a different set cardinality at test time.”
Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “As discussed in [35], this means each layer scales quadratically in compute and memory, which currently makes it impossible to apply Transformers on high-dimensional data like images without some form of preprocessing.”
Scaling Vision Transformers (2021, methodology): “In particular, we evaluate global average pooling (GAP) and multihead attention pooling (MAP) [24] to aggregate representation from all patch tokens.”
Scalable Adaptive Computation for Iterative Generation (2022, background): “Recurrent Interface Networks are perhaps most similar to Set Transformers (Lee et al., 2018) and Perceivers (Jaegle et al., 2021b;a), which also leverage a set of latents for input-agnostic computation.”
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, methodology): “With the help of the Set Transformer [179] in combining information between layers, SCAE discovers constituents of the image and organizes them into a smaller set”
Stacked Capsule Autoencoders (2019, methodology): “We first encode all input points (which take the role of part capsules) with Set Transformer (Lee et al., 2019)—a permutation-invariant encoder h based on attention mechanisms—into K object capsules.”
Few-Shot and Continual Learning with Attentive Independent Mechanisms (2021, background): “These mechanisms compete and attend to the input representation in a top-down fashion using crossattention [30, 29].”
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding (2021, methodology): “The second is the memory-based mechanism, including Compressive Transformers [30] and Set Transformer [19].”
Simple Open-Vocabulary Object Detection with Vision Transformers (2022, methodology): “For the image representation, we use multihead attention pooling (MAP) [25,43] to aggregate token representation.”
Luna: Linear Unified Nested Attention (2021, methodology): “…et al., 2018), quantization (Shen et al., 2020; Fan et al., 2020), sparse attention (Parmar et al., 2018; Kitaev et al., 2020), side memory module (Lee et al., 2019; Gupta and Berant, 2020; Goyal et al., 2021), and low-rank or compressed context (Wang et al., 2019; Ainslie 3We observed that Luna…”
Unsupervised Attributed Multiplex Network Embedding (2019, methodology): “Q can be any pooling method that can handle permutation invariant input, such as set2set (Vinyals, Bengio, and Kudlur 2015) or Set Transformer (Lee et al. 2019).”
Set Prediction without Imposing Structure as Conditional Density Estimation (2020, methodology): “Some very recent works (Kosiorek et al., 2020; Carion et al., 2020; Locatello et al., 2020; Karl et al., 2020) respect the permutation symmetry in the model, by applying the Transformer (Vaswani et al., 2017; Lee et al., 2019)”
Better Set Representations For Relational Reasoning (2020, methodology): “ that naturally imposes a set-structured latent representation. 2.2. Set Encoders and Set Decoders Both set encoders and set decoders have been well-studied (Zaheer et al.,2017;Zhang et al.,2019;2020;Lee et al., 2018). Although they can be applied within our frameSet-Structured Latent Representation”
Right for the Right Concept: Revising Neuro-Symbolic Concepts by Interacting with their Explanations (2020, methodology): “(ii) To this end, we devise a novel NeuroSymbolic concept learner, combining Slot Attention [28] and Set Transformer [24] in an end-to-end differentiable fashion.”
3D Object Detection with Pointformer (2020, background): “Set Transformer [16] uses attention mechanisms to model interactions among elements in the input set.”
End-to-end Recurrent Multi-Object Tracking and Trajectory Prediction with Relational Reasoning (2019, background): “[11]), which is able to account for higher-order interactions between set elements when computing their representations, thereby allowing rich information exchange, and it can do so in a permutation-equivariant manner.”
Efficient Attentions for Long Document Summarization (2021, methodology): “…are tailored for an autoregressive decoder only (Dai et al., 2019); memory methods use a separate memory module to attend to full sequences (Lee et al., 2019), which share a similar theoretical foundation as global tokens; and kernel methods over attentions require training models from…”