Learning Plannable Representations with Causal InfoGAN

By Thanard Kurutach, Aviv Tamar, Ge Yang, Stuart J. Russell, Pieter Abbeel. 2018

199 citations recorded by the source corpus.

In recent years, deep generative models have been shown to 'imagine' convincing high-dimensional observations such as images, audio, and even video, learning directly from raw data. In this work, we ask how to imagine goal-directed visual plans – a plausible sequence of observations that transition a dynamical system from its current configuration to a desired goal state, which can later be used as a reference trajectory for control. We focus on systems with high-dimensional observations, such as images, and propose an approach that naturally combines representation learning and planning. Our framework learns a generative model of sequential observations, where the generative process is induced by a transition in a low-dimensional planning model, and an additional noise. By maximizing the mutual information between the generated observations and the transition in the planning model, we obtain a low-dimensional representation that best explains the causal nature of the data. We structure the planning model to be compatible with efficient planning algorithms, and we propose several such models based on either discrete or continuous states. Finally, to generate a visual plan, we project the current and goal observations onto their respective states in the planning model, plan a trajectory, and then use the generative model to transform the trajectory to a sequence of observations. We demonstrate our method on imagining plausible visual plans of rope manipulation.

Subfields: Computer Vision and Pattern Recognition, Reinforcement Learning, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Generative Models

Citations in the corpus

Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “Traditionally, this problem has been solved through optimization [28, 29] or symbolic reasoning [24, 26], but more recently machine learning has been applied to aspects of the problem via learned representations, learned task-primitives, and more [30, 31, 32, 33, 34, 35, 36, 37, 38].”
COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration (2019, background): “Recent works have made progress in this regard (Finn et al., 2016; Ha and Schmidhuber, 2018; Kaiser et al., 2019; Kurutach et al., 2018; Sodhani et al., 2019; Zhang et al., 2018), but model-free alternatives are still hard to beat in terms of asymptotic performance.”
RoboNet: Large-Scale Multi-Robot Learning (2019, methodology): “This method is related to a range of recently proposed techniques that use predictive models for vision-based control [28, 29, 30, 31, 32].”
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (2018, background): “Prior work on action-conditioned video prediction has studied predicting synthetic video game images [16], [29], 3D point clouds [19], and real-world images [30], [31], [32], using both direct autoregressive frame prediction [31], [32], [33] and latent variable models [34], [35].”
Local Information Agent Modelling in Partially-Observable Environments (2021, methodology): “…state representations for improving the sample sample efﬁciency of RL algorithms using reconstruction-based [Corneil et al., 2018, Igl et al., 2018, Kurutach et al., 2018, Gregor et al., 2019, Gelada et al., 2019] or non-reconstruction-based methods [Laskin et al., 2020, Zhang et al., 2020], using…”
Time Reversal as Self-Supervision (2018, unknown): “manipulation [40] TRASS is applicable.”
Bayesian Relational Memory for Semantic Visual Navigation (2019, methodology): “From a reinforcement learning perspective, our work is related to the model-based approaches [14, 45, 61, 44, 32], in the sense that we model the environment layout via a re-”
Learning Semantic Embedding Spaces for Slicing Vegetables (2019, background): “Few works have looked at learning an embedding space for planning using raw pixel inputs [8], [9], [3], [10], [11].”
Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation (2019, background): “A number of recent works have explored reaching novel goals using only self-supervision (Finn & Levine, 2016; Eysenbach et al., 2019; Kurutach et al., 2018; Wang et al., 2019; Jayaraman et al., 2019; Nair et al., 2018a).”
Regression Planning Networks (2019, methodology): “For example, Causal InfoGAN [8] learns a latent binary representation that can be used jointly with graph-planning algorithm.”
Planning with Goal-Conditioned Policies (2019, background): “A number of papers have studied embedding high-dimensional observations into a low-dimensional latent space for planning [60, 16, 62, 22, 29].”
Learning Causal State Representations of Partially Observable Environments (2019, methodology): “We plan on extending the Distributed Experience Replay algorithm [26] with extra asynchronous workers dedicated respectively to the construction of continuous and discrete world models.”
Learning Markov State Abstractions for Deep Reinforcement Learning (2021, background): “, 2018), suitability for planning (Kurutach et al., 2018), information compression (Abel et al.”
ProgPrompt: Generating Situated Robot Task Plans using Large Language Models (2022, background): “Recent works have explored learning-based task & motion planning, using methods such as representation learning, hierarchical learning, language as planning space, learning compositional skills and more [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26].”
Self-supervised Learning of Image Embedding for Continuous Control (2019, methodology): “Other recent methods have proposed to learn an image embedding spaces where the planning problem could be easier [Kurutach et al., 2018, Zhang et al., 2018].”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “More complex hierarchical models, such as neural networks and decision trees, allow the robot to learn taskspecific features that provide better generalization [235, 334, 117, 174, 321, 110, 230, 353, 431].”
Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks (2021, background): “Alternatively, a number of recent works have studied long horizon visual planning [52, 53, 73, 74, 75, 50, 76, 77, 78], often by combining a structured search approach with learned components like generative models [73, 50, 76] or distance functions [53, 34].”
Causal Induction from Visual Observations for Goal Directed Tasks (2019, unknown): “Causal modeling has been explored in several contexts, including image classiﬁcation (Chalupka et al., 2014), generative models (Kocaoglu et al., 2017), robot planning (Kurutach et al., 2018), policy learning (Buesing et al., 2018), and transfer learning (Bengio et al., 2019).”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, methodology): “(16) By comparing (15) and (16), we can see that the generator of InfoGAN is similar to that of cGANs.”