SCAN: Learning Abstract Hierarchical Compositional Visual Concepts

By Irina Higgins, Nicolas Sonnerat, Loic Matthey, Arka Pal, Christopher P. Burgess, Matthew Botvinick, Demis Hassabis, Alexander Lerchner. 2017

53 citations recorded by the source corpus.

The natural world is infinitely diverse, yet this diversity arises from a relatively small set of coherent properties and rules, such as the laws of physics or chemistry. We conjecture that biological intelligent systems are able to survive within their diverse environments by discovering the regularities that arise from these rules primarily through unsupervised experiences, and representing this knowledge as abstract concepts. Such representations possess useful properties of compositionality and hierarchical organisation, which allow intelligent agents to recombine a finite set of conceptual building blocks into an exponentially large set of useful new concepts. This paper describes SCAN (Symbol-Concept Association Network), a new framework for learning such concepts in the visual domain. We first use the previously published beta-VAE (Higgins et al., 2017a) architecture to learn a disentangled representation of the latent structure of the visual world, before training SCAN to extract abstract concepts grounded in such disentangled visual primitives through fast symbol association. Our approach requires very few pairings between symbols and images and makes no assumptions about the choice of symbol representations. Once trained, SCAN is capable of multimodal bi-directional inference, generating a diverse set of image samples from symbolic descriptions and vice versa. It also allows for traversal and manipulation of the implicit hierarchy of compositional visual concepts through symbolic instructions and learnt logical recombination operations. Such manipulations enable SCAN to invent and learn novel visual concepts through recombination of the few learnt concepts.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Molecular Biology, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Molecular Biology

In Representation & Self-Supervised Learning

Citations in the corpus

Environmental drivers of systematicity and generalization in a situated agent (2019, background): “…designed to support some form of symbolic computation (such as graphs (Battaglia et al., 2018), modular-components deﬁned by symbolic parses (Andreas et al., 2016; Bahdanau et al., 2018), explicit latent variables (Higgins et al., 2017) or other neuro-symbolic hybrid methods (Mao et al., 2019)).”
Understanding disentangling in β-VAE (2018, background): “The disentangled representations learnt by β -VAE have been shown to be important for learning a hierarchy of abstract visual concepts conducive of imagination [17] and for improving transfer performance of reinforcement learning policies, including simulation to reality transfer in robotics [16].”
Measuring the tendency of CNNs to Learn Surface Statistical Regularities (2017, methodology): “In [16] a variational setup is used to learn visual concepts and [32] aims to learn abstract relations be-tween objects in natural scene images.”
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision (2019, background): “Siddharth et al. (2017); Higgins et al. (2018) learned disentangled representations using deep generative models.”
Measuring abstract reasoning in neural networks (2018, background): “…also been substantial progress in both reasoning and abstract representation learning in neural nets (Botvinick et al., 2017; LeCun et al., 2015; Higgins et al., 2016; 2017), the extent to which these models exhibit anything like general abstract reasoning is the subject of much debate (Garnelo…”
Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs (2019, methodology): “A number of new variations of the VAE objective have been developed to alleviate this problem, though all of them introduce additional hyperparameters [Higgins et al., 2017a, Burgess et al., 2018, Kim and Mnih, 2017, Chen et al., 2018].”
Small Sample Learning in Big Data Era (2018, background): “Alternatively, Higgins et al. (2018) described a new framework of symbol-concept association network (SCAN), able to discover and learn an implicit hierarchy of abstract concepts from as few as five symbol-image pairs per concept.”
Unsupervised Discovery of Parts, Structure, and Dynamics (2019, background): “Higgins et al. (2018) learned an implicit hierarchy of abstract concepts from a few symbol-image pairs.”
Concept Learning with Energy-Based Models (2018, background): “A recent framework of [15] focuses on visual concepts such as color and shape, where concepts are deﬁned in terms of distributions over latent variables produced by a variational autoencoder.”
Neural Style Transfer: A Review (2017, unknown): “learning [118] and transfer learning [119].”
Locality and compositionality in zero-shot learning (2019, background): “…supervised label classiﬁer ( FC ) • Unsupervised / reconstruction based / generative : Variational auto-encoders( VAE , Kingma & Welling, 2013), β -VAE (Higgins et al., 2017a), Adversarial auto-encoders ( AAE , Makhzani et al., 2015), • Local self-supervision and variants : Augmented Multiscale…”
Beyond imitation: Zero-shot task transfer on robots by learning concepts as cognitive programs (2018, background): “In contrast to datasets that measure pixel accurate image reconstructions in simplistic settings (54), use of the tabletop world recognizes the schematic nature of concepts (18), and enables the evaluation of sensorimotor representations for their generalization to settings that are different from t”
Joint-VAE: Learning Disentangled Joint Continuous and Discrete Representations (2018, background): “Disentangled representations have also been applied to reinforcement learning (Higgins et al. (2017a)) and for learning visual concepts (Higgins et al. (2017b)).”