Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs

By Nicholas Watters, Loic Matthey, Christopher P. Burgess, Alexander Lerchner. 2019

188 citations recorded by the source corpus.

We present a simple neural rendering architecture that helps variational autoencoders (VAEs) learn disentangled representations. Instead of the deconvolutional network typically used in the decoder of VAEs, we tile (broadcast) the latent vector across space, concatenate fixed X- and Y-"coordinate" channels, and apply a fully convolutional network with 1x1 stride. This provides an architectural prior for dissociating positional from non-positional features in the latent distribution of VAEs, yet without providing any explicit supervision to this effect. We show that this architecture, which we term the Spatial Broadcast decoder, improves disentangling, reconstruction accuracy, and generalization to held-out regions in data space. It provides a particularly dramatic benefit when applied to datasets with small objects. We also emphasize a method for visualizing learned latent spaces that helped us diagnose our models and may prove useful for others aiming to assess data representations. Finally, we show the Spatial Broadcast Decoder is complementary to state-of-the-art (SOTA) disentangling techniques and when incorporated improves their performance.

Subfields: Generative Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

In Representation & Self-Supervised Learning

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, methodology): “[28] Nicholas Watters, Loic Matthey, Christopher P Burgess, and Alexander Lerchner.”
SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition (2020, background): “tect some foreground objects from the middle base that is above the water. One reason for this behavior is because we limit the capacity of the background module by using a spatial broadcast decoder (Watters et al., 2019) which is much weaker when compared to other decoders like sub-pixel convolutio”
On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset (2019, background): “To bypass this limitation, many recent state-of-the-art disentanglement models [48, 19, 5, 14, 6] have heavily relied on synthetic toy datasets, trying to solve a simplified version of the problem in the hope that the conclusions drawn might likewise be valid for real-world settings.”
MONet: Unsupervised Scene Decomposition and Representation (2019, methodology): “As discussed above, we utilised a weight modulating the latent KL in the loss [Higgins et al., 2017, Burgess et al., 2018] and a broadcast decoder [Watters et al., 2019] in MONet to encourage disentangling of the latent representation at the feature level.”
GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations (2019, methodology): “Subsequently, a variational auto-encoder ( VAE ) (Kingma and Welling [19], Rezende et al. [20]) with a spatial broadcast decoder (Watters et al. [21]) models individual components.”
Generative Neurosymbolic Machines (2020, background): “(3) we reduce the number of layers for the spatial broadcast decoder [49] from 4 to 3.”
Learning to Give Checkable Answers with Prover-Verifier Games (2021, unknown): “For the autoencoding auxiliary head, we used a Spatial Broadcast Decoder [35] with 3 convolutional layers with 32 channels, kernel size of 3 and LeakyReLU activations.”
Reconstructing continuous distributions of 3D protein structure from cryo-EM images (2019, background): “…networks in computer vision: Using spatial (i.e. pixel) coordinates as features to a convolutional decoder to improve generative modeling has been proposed many times, with recent work computing each image as a function of a ﬁxed coordinate lattice and latent variables (Watters et al. (2019)).”
Hierarchical Relational Inference (2020, background): “Typically, these approaches assume access to supervised state descriptions, and only few works also infer object representations from raw visual images (van Steenkiste et al. 2018; Veerapaneni et al. 2019; Watters et al. 2019).”
Object Scene Representation Transformer (2022, background): “Our novel Slot Mixer differs significantly from the standard choice in the literature where Spatial Broadcast (SB) decoders [31] (Fig.”
Conditional Object-Centric Learning from Video (2021, methodology): “We decode slot representations after application of the corrector using a slot-wise Spatial Broadcast Decoder (Watters et al., 2019) to produce per-slot RGB predictions of the optical flow (or reconstructed frame) and an alpha mask.”
On Translation Invariance in CNNs: Convolutional Layers Can Exploit Absolute Spatial Location (2020, background): “Explicitly adding absolute spatial location information helps for patch matching [67, 71], generative modeling [101], semantic segmentation [36, 100], instance segmentation [72].”