DARLA: Improving Zero-Shot Transfer in Reinforcement Learning

By Irina Higgins, Arka Pal, Andrei A. Rusu, Loic Matthey, Christopher P. Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, Alexander Lerchner. 2017

447 citations recorded by the source corpus.

Domain adaptation is an important open problem in deep reinforcement learning (RL). In many scenarios of interest data is hard to obtain, so agents may learn a source policy in a setting where data is readily available, with the hope that it generalises well to the target domain. We propose a new multi-stage RL agent, DARLA (DisentAngled Representation Learning Agent), which learns to see before learning to act. DARLA's vision is based on learning a disentangled representation of the observed environment. Once DARLA can see, it is able to acquire source policies that are robust to many domain shifts - even with no access to the target domain. DARLA significantly outperforms conventional baselines in zero-shot domain adaptation scenarios, an effect that holds across a variety of RL environments (Jaco arm, DeepMind Lab) and base RL algorithms (DQN, A3C and EC).

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, methodology): “Maintaining the terminology, the disentanglement scores in (Higgins et al., 2017a; Kim and Mnih, 2018; Ridgeway and Mozer, 2018; Eastwood and Williams, 2018; Suter et al., 2019) focus on disentanglement in the sense of (Eastwood and Williams, 2018) and (Chen et al., 2018; Kumar et al., 2018) on…”
Recurrent World Models Facilitate Policy Evolution (2018, methodology): “Our approach may complement sim2real approaches outlined in previous work [4, 33].”
Reconciling deep learning with symbolic artificial intelligence: representing objects and relations (2019, background): “The resulting representations are not only advantageous from the point of view of interpretability, but are also beneﬁcial to downstream tasks such as reinforcement learning [34] and learning hierarchical concepts [35 ].”
On the Binding Problem in Artificial Neural Networks (2020, background): “…are disentangled can be approached by optimizing for some notion of (statistical) independence between features (e.g. Schmidhuber, 1992c; Chen et al., 2016; Higgins et al., 2017a), sparse feature updates across time (Whitney, 2016), or independent controllability of features (Thomas et al., 2017).”
On the Transfer of Disentangled Representations in Realistic Settings (2020, background): “Disentangled representations hold the promise of generalization to unseen scenarios (Higgins et al., 2017b), increased interpretability (Adel et al., 2018; Higgins et al., 2018) and faster learning on downstream tasks (van Steenkiste et al., 2019; Locatello et al., 2019a).”
Weakly-Supervised Disentanglement Without Compromises (2020, methodology): “…noise; we assume that noise is explicitly modeled as a latent variable and its effect is manifested through g? as done by (Bengio et al., 2013; Locatello et al., 2019b; Higgins et al., 2018; 2017a; Suter et al., 2019; Reed et al., 2015; LeCun et al., 2004; Kim & Mnih, 2018; Gondal et al., 2019).”
Shaping Belief States with Generative Environment Models for RL (2019, background): “A non-exhaustive list includes pixel control [53], contrastive predictive coding (CPC) [54], action-conditional CPC [20], frame reconstruction [35, 55], next-frame prediction [18, 56, 20, 57] and successor representations [58, 59, 60].”
On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset (2019, background): “angled representation learning is to learn variables Z ithat capture the independent mechanisms G i. sequence modeling, curiosity-based exploration or even domain adaptation in reinforcement learning [45, 25, 37, 16, 18, 29, 47]. While the research community is in general agreement on the importance”
On the Fairness of Disentangled Representations (2019, methodology): “onal assumptions on the effect of the factors of variation were also studied [39, 16, 47, 31, 91, 26, 19, 41, 93, 62, 53, 81, 8] and applied in the sequential data and reinforcement learning settings [88, 85, 57, 69, 37, 38]. Overall, the disentanglement literature is interested in isolating the eff”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Hence, learning to solve an up-stream task from a disentangled representation is expected to require fewer samples and be easier in general [66, 6, 28, 29, 57].”
Multi-Object Representation Learning with Iterative Variational Inference (2019, methodology): “Recent breakthroughs in unsupervised representation learning (Higgins et al., 2017a; Makhzani et al., 2015; Chen et al., 2016) tend to focus on data where a single object of interest is placed in front of some background (e.g. dSprites, 3D Chairs, CelebA).”
Variational Autoencoders Pursue PCA Directions (by Accident) (2018, background): “a representation is found it allows for interpretable latent code manipulation, which is desirable in a variety of applications; recently, for example, in reinforcement learning [39, 18, 11, 41, 34].”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, methodology): “While prior work (Steenbrugge et al., 2018; Laversanne-Finot et al., 2018; Nair et al., 2018; Higgins et al., 2017b; 2018) successfully applied disentanglement methods such as β-VAE on a variety of downstream tasks, it is not clear to us that these approaches and trained models performed well…”
Towards a Definition of Disentangled Representations (2018, background): “For example, subsequent tasks may safely ignore certain parts of such a representation to form abstractions [41, 53], or use novel recombinations of values within the subspaces for improved generalisation [40].”
Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologies (2018, background): “The representations learnt through such methods have been shown to help in transfer scenarios and with data efficiency for policy learning in the Reinforcement Learning (RL) context [22].”
Understanding disentangling in β-VAE (2018, background): “The disentangled representations learnt by β -VAE have been shown to be important for learning a hierarchy of abstract visual concepts conducive of imagination [17] and for improving transfer performance of reinforcement learning policies, including simulation to reality transfer in robotics [16].”
Visual Reinforcement Learning with Imagined Goals (2018, methodology): “These methods use the learned representation as a substitute for the state for the policy, but require additional information, such as access to the ground truth reward function based on the true state during training time [16, 14, 48, 11, 21, 17], expert trajectories [44], human demonstrations [42]”
Building Generalizable Agents with a Realistic and Rich 3D Environment (2018, background): “In DeepMind lab (Beattie et al., 2016; Higgins et al., 2017), diversity in the environment is introduced by pixel-level color or texture changes and maze layouts.”
Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task (2017, background): “al pipelined approaches which often suffer from propagation of errors between each stage of the pipeline. End-to-end approaches have had success both in the real world [1,2,3] and in simulated worlds [4,5,6,7]. Learning end-to-end controllers in simulation is an attractive alternative to using physi”