Deep Sets

By Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnab'as P'oczos, Ruslan Salakhutdinov, Alex Smola. 2017

3,061 citations recorded by the source corpus.

In this paper, we study the problem of designing objective functions for machine learning problems defined on finite \emph{sets}. In contrast to traditional objective functions defined for machine learning problems operating on finite dimensional vectors, the new objective functions we propose are operating on finite sets and are invariant to permutations. Such problems are widespread, ranging from estimation of population statistics \citep{poczos13aistats}, via anomaly detection in piezometer data of embankment dams \citep{Jung15Exploration}, to cosmology \citep{Ntampaka16Dynamical,Ravanbakhsh16ICML1}. Our main theorem characterizes the permutation invariant objective functions and provides a family of functions to which any permutation invariant objective function must belong. This family of functions has a special structure which enables us to design a deep network architecture that can operate on sets and which can be deployed on a variety of scenarios including both unsupervised and supervised learning tasks. We demonstrate the applicability of our method on population statistic estimation, point cloud classification, set expansion, and image tagging.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Deep Perm-Set Net: Learn to predict sets with unknown permutation and cardinality using deep neural networks (2018, background): “Another recent work proposed by Zaheer et al. (2017) is a deep learning framework which can deal with sets as input with different sizes and permutations.”
A simple neural network module for relational reasoning (2017, background): “unstructured inputs coming from CNNs and LSTMs. There has also been some recent work on reasoning about sets, although this work does not explicitly reason about the relations of elements within sets [47]. Grounding spatial relations Although grounding language in spatial percepts has a long-standin”
Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation (2021, background): “A DeepSets encoder (Zaheer et al., 2017) – which is based on sum pooling – has the same gradients for equal elements, which would make DSPN set-equivariant.”
Meta-Learning Probabilistic Inference for Prediction (2018, methodology): “We use permutation-invariant instance-pooling operations to process these sets similarly to Qi et al. (2017) and as formalized in Zaheer et al. (2017).”
Relational inductive biases, deep learning, and graph networks (2018, methodology): “Such an approach is the essence of the Deep Sets and related models (Zaheer et al., 2017; Edwards and Storkey, 2016; Pevn`y and Somol, 2017), which we explore further in Section 4.2.3.”
Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects (2018, methodology): “To condition DISC on propagated latent variables (Line 4 in Algorithm 3), we encode the latter by using a two-layer MLP similarly to Zaheer et al., 2017, l Training was done by optimising the FIVO bound, which is known to be tighter than the IWAE bound for sequential latent variable models…”
Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions (2018, background): “In light of other recent work (Zaheer et al., 2017) it can be seen as a permutation equivariant set function.”
Set Transformer (2018, methodology): “…shape recognition (Wu et al., 2015; Shi et al., 2015; Su et al., 2015; Charles et al., 2017), sequence ordering (Vinyals et al., 2016), and various set operations (Muandet et al., 2012; Oliva et al., 2013; Edwards & Storkey, 2017; Zaheer et al., 2017) can also be viewed as the set-input problems.”
How Powerful are Graph Neural Networks? (2018, background): “The proof extends the setting in (Zaheer et al., 2017) from sets to multisets.”
SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks (2020, neutral): “Linear DeepSet [46] Tensor Field [28] Set Transformer [16]”
Set2Graph: Learning Graphs From Sets (2020, methodology): “We used F 2 , ( F 1 , F 2 ) , and F 3 (resp.) for these learning tasks. φ is implemented using DeepSets (Zaheer et al., 2017) with 5 layers with output dimension d 1 ∈ { 5 , 10 , 80 , 512 } ; ψ is implemented with an MLP, m , with { 2 , 3 , 6 } layers with input dimension d 2 deﬁned by d 1 and β .…”
Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data (2019, background): “ing convolutions on 3D voxel grids [27, 43]. While it seems natural to extend the convolution operations from 2D to 3D, it is shown that performing convolutions on a point cloud is not a trivial task [30, 49]. The difﬁculty stems from the fact that a point cloud has no well-deﬁned order of points on”
Provably Powerful Graph Networks (2019, background): “We note that Proposition 1 is a generalization of lemma 6 in Zaheer et al. (2017) to the case of multisets of vectors.”
Gauge Equivariant Convolutional Networks and the Icosahedral CNN (2019, positive): “The work presented in this paper brings together two hereto-fore disparate lines of research: equivariant and geometrical deep learning.”