FSPool: Learning Set Representations with Featurewise Sort Pooling

By Yan Zhang, Jonathon S. Hare, Adam Prügel-Bennett. 2019

94 citations recorded by the source corpus.

Traditional set prediction models can struggle with simple datasets due to an issue we call the responsibility problem. We introduce a pooling method for sets of feature vectors based on sorting features across elements of the set. This can be used to construct a permutation-equivariant auto-encoder that avoids this responsibility problem. On a toy dataset of polygons and a set version of MNIST, we show that such an auto-encoder produces considerably better reconstructions and representations. Replacing the pooling function in existing set encoders with FSPool improves accuracy and convergence speed on a variety of datasets.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, background): “[34] Yan Zhang, Jonathon Hare, and Adam Prügel-Bennett.”
Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation (2021, methodology): “Zhang et al. (2019) ﬁnd that FSPool-based encoders (Zhang et al., 2020) achieved by far the best results among the encoders they have tried.”
Learning the Best Pooling Strategy for Visual Semantic Embedding (2020, methodology): “The baselines include: • Generalized Mean Pooling (GeM) [36], an adaptive pooling function with a single trainable parameter, and is popular in image search literature; • Feature-sorting Pooling (FSPool) [51], a learnable pooling that handles variable-sized inputs by interpolating a fixed-size learn”
Better Set Representations For Relational Reasoning (2020, methodology): “Speciﬁcally, H embed is implemented by ﬂattening the ﬁnal feature map and passing it through a linear layer, and H agg is a set encoder that processes each element individually with a 3-layer MLP, followed by FSPool [24].”
Set Prediction without Imposing Structure as Conditional Density Estimation (2020, methodology): “FSPool (Zhang et al., 2020b) is applied as the permutation invariant aggregator.”
Conditional Structure Generation through Graph Variational Generative Adversarial Nets (2019, methodology): “Finally, the idea of enabling permutation-invariance in an encoder-decoder system also applies to problems like set generation, where we notice the recent development of models that achieve it by backpropagating through an order-invariant set encoder [49], which can be further improved with more sop”
Conditional Set Generation with Transformers (2020, methodology): “However imposing such an ordering can lead to a pathology that is commonly referred to as the responsibility problem (Zhang et al., 2019; 2020); there exist points in the output set space where a small change in set space (as measured by a set loss) requires a large change in the generative model’s…”
multiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoning (2021, background): “…where a subset of the input text explain a prediction, (2) free-form or natural language explanations (Camburu et al., 2018; Ra-jani et al., 2019; Zhang et al., 2020; Kumar and Talukdar, 2020) that are not constrained to the input, and (3) structured explanations that range from…”
Deep Set Prediction Networks (2019, methodology): “[30] construct an auto-encoder that pools a set into a feature vector where information from the encoder is shared with their decoder.”