Conditional Set Generation with Transformers

By Adam R. Kosiorek, Hyunjik Kim, Danilo Jimenez Rezende. 2020

44 citations recorded by the source corpus.

A set is an unordered collection of unique elements--and yet many machine learning models that generate sets impose an implicit or explicit ordering. Since model performance can depend on the choice of order, any particular ordering can lead to sub-optimal results. An alternative solution is to use a permutation-equivariant set generator, which does not specify an order-ing. An example of such a generator is the DeepSet Prediction Network (DSPN). We introduce the Transformer Set Prediction Network (TSPN), a flexible permutation-equivariant model for set prediction based on the transformer, that builds upon and outperforms DSPN in the quality of predicted set elements and in the accuracy of their predicted sizes. We test our model on MNIST-as-point-clouds (SET-MNIST) for point-cloud generation and on CLEVR for object detection.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Molecular Biology, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Yoshua Bengio
3. Jianfeng Gao
4. Quoc V. Le
5. Percy Liang
6. Dacheng Tao
7. Trevor Darrell
8. Ruslan Salakhutdinov

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Trevor Darrell
4. Andrea Vedaldi
5. Jitendra Malik
6. Andrew Zisserman
7. Ming-Hsuan Yang
8. Xiaogang Wang

In Molecular Biology

1. Max Welling
2. Victor Garcia Satorras
3. Emiel Hoogeboom
4. Thomas Kipf
5. Yan Zhang
6. Clément Vignac
7. Simon Lacoste-Julien
8. Mohit Bansal

In Generative Models

1. Emiel Hoogeboom
2. Thomas Kipf
3. Max Welling
4. Victor Garcia Satorras
5. Alexey Dosovitskiy
6. Simon Lacoste-Julien
7. Francesco Locatello
8. Stefano Ermon

Citations in the corpus

• Object-Centric Learning with Slot Attention (2020, methodology): “[61] Adam R Kosiorek, Hyunjik Kim, and Danilo J Rezende.”
• Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation (2021, background): “This applies even to transformers with random “position” embeddings, which similarly to TSPN (Kosiorek et al., 2020) and Slot Attention (Locatello et al., 2020) use noise to make elements Next, we aim to evaluate our contributions in Section 3.”
• Set Prediction without Imposing Structure as Conditional Density Estimation (2020, methodology): “Augmenting the loss function with set size RMSE alleviates some of the issues with set size, but leads to decreases in shape reconstruction performance (Kosiorek et al., 2020).”
• Equivariant Diffusion for Molecule Generation in 3D (2022, methodology): “Some examples are autoregressive methods such as (Liu et al., 2018; You et al., 2018; Liao et al., 2019), and one-shot approaches such as (Simonovsky & Komodakis, 2018; De Cao & Kipf, 2018; Bresson & Laurent, 2019; Kosiorek et al., 2020; Krawczuk et al., 2021).”
• Right for the Right Concept: Revising Neuro-Symbolic Concepts by Interacting with their Explanations (2020, methodology): “Deep Set Prediction Networks [58], Transformer Set Prediction Networks [20] or Mask-RCNN based models [11] are viable alternatives to the Slot Attention module as concept embedding module.”
• multiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoning (2021, background): “Machine Learning over Sets: Set-based ML models (Zaheer et al., 2017; Lee et al., 2018; Zhang et al., 2019a; Kosiorek et al., 2020) have a wide range of applications including generating multiple image captions (Vinyals et al., 2015), generating diverse translations (Cho et al., 2014; Bahdanau et…”