On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset

By Muhammad Waleed Gondal, Manuel Wüthrich, Ðorðe Miladinovic, Francesco Locatello, Martin Breidt, V. Volchkov, J. Akpo, Olivier Bachem, Bernhard Schölkopf, Stefan Bauer. 2019

170 citations recorded by the source corpus.

Learning meaningful and compact representations with disentangled semantic aspects is considered to be of key importance in representation learning. Since real-world data is notoriously costly to collect, many recent state-of-the-art disentanglement models have heavily relied on synthetic toy data-sets. In this paper, we propose a novel data-set which consists of over one million images of physical 3D objects with seven factors of variation, such as object color, shape, size and position. In order to be able to control all the factors of variation precisely, we built an experimental platform where the objects are being moved by a robotic arm. In addition, we provide two more datasets which consist of simulations of the experimental setup. These datasets provide for the first time the possibility to systematically investigate how well different disentanglement methods perform on real data in comparison to simulation, and how simulated data can be leveraged to build better representations of the real world. We provide a first experimental study of these questions and our results indicate that learned models transfer poorly, but that model and hyperparameter selection is an effective means of transferring information to the real world.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, methodology): “, 2004), Shapes3D (Kim and Mnih, 2018) and MPI3D (Gondal et al., 2019).”
On the Transfer of Disentangled Representations in Realistic Settings (2020, background): “Transfer of learned disentangled representations from simulation to the real world has been recently investigated by Gondal et al. (2019) on the MPI3D dataset, and previously by Higgins et al. (2017b) in the context of reinforcement learning.”
Weakly-Supervised Disentanglement Without Compromises (2020, methodology): “…noise; we assume that noise is explicitly modeled as a latent variable and its effect is manifested through g? as done by (Bengio et al., 2013; Locatello et al., 2019b; Higgins et al., 2018; 2017a; Suter et al., 2019; Reed et al., 2015; LeCun et al., 2004; Kim & Mnih, 2018; Gondal et al., 2019).”
On the Fairness of Disentangled Representations (2019, background): “Indeed, isolating each independent factor of variation into the independent components of a representation vector should make it both interpretable and simplify downstream prediction tasks [6, 7, 29, 35, 56, 58, 60, 74, 82, 87, 89, 90, 1, 28].”
Disentangling Factors of Variation Using Few Labels (2019, background): “The hope is that such representations will be interpretable, maximally compact, allow for counterfactual reasoning and be useful for a large variety of downstream tasks (Bengio et al., 2013; Peters et al., 2017; LeCun et al., 2015; Bengio et al., 2007; Schmidhuber, 1992; Lake et al., 2017; Goodfello”
Structural Autoencoders Improve Representations for Generation and Transfer (2020, methodology): “We train our proposed architectures as well as several baselines on three different datasets: 3D-Shapes [25], and two variants ("toy" and "real") of the MPI3D Disentanglement dataset [26], as well as Celeb-A [27].”
Disentangling Factors of Variations Using Few Labels (2020, background): “The hope is that such representations will be interpretable, maximally compact, allow for counterfactual reasoning and be useful for a large variety of downstream tasks (Bengio et al., 2013; Peters et al., 2017; LeCun et al., 2015; Bengio et al., 2007; Schmidhuber, 1992; Lake et al., 2017; Goodfello”
Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding (2020, methodology): “We include results on all DisLib datasets, dSprites [47], Cars3D [48], SmallNORB [49], Shapes3D [50], MPI3D [51], in Tables 15, 16, 17, 18, and 19, respectively.”
D’ya Like DAGs? A Survey on Structure Learning and Causal Discovery (2021, methodology): “[139]) user specified adds edges with probability p = 2e d2−d Linear, GP Add, GP Mix, Sigmoid Add and Sigmoid Mix mixed graph data CausalWorld [2] comprehensive robotics dataset MPI3D [71] visual disentanglement dataset Pendulum-light-shadow [257] image data Phase coupled oscillator [7] physical rel”
Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style (2021, background): “2 also on the MPI3D-real dataset [38] containing > 1 million real images with ground-truth annotations of 3D objects being moved by a robotic arm.”