Self-Supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigation

By Gregory Kahn, Adam Villaflor, Bosen Ding, Pieter Abbeel, Sergey Levine. 2017

315 citations recorded by the source corpus.

Enabling robots to autonomously navigate complex environments is essential for real-world deployment. Prior methods approach this problem by having the robot maintain an internal map of the world, and then use a localization and planning method to navigate through the internal map. However, these approaches often include a variety of assumptions, are computationally intensive, and do not learn from failures. In contrast, learning-based methods improve as the robot acts in the environment, but are difficult to deploy in the real-world due to their high sample complexity. To address the need to learn complex policies with few samples, we propose a generalized computation graph that subsumes value-based model-free methods and model-based methods, with specific instantiations interpolating between model-free and model-based. We then instantiate this graph to form a navigation model that learns from raw images and is sample efficient. Our simulated car experiments explore the design decisions of our navigation model, and show our approach outperforms single-step and $N$ -step double Q-learning. We also evaluate our approach on a real-world RC car and show it can learn to navigate through a complex indoor environment with a few hours of fully autonomous, self-supervised training. Videos of the experiments and code can be found at github.com/gkahn13/gcg

Subfields: Aerospace Engineering, Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

A General Framework for Uncertainty Estimation in Deep Learning (2019, background): “In order to account for the safety of the platform, however, previous works showed the importance of quantifying uncertainties [14], [13].”
Driving Policy Transfer via Modularity and Abstraction (2018, background): “nown for its high sample complexity and is conducted mainly in simulation [32,13,12]. Application of deep reinforcement learning to real vehicles has only been demonstrated in restricted environments [23]. Finally, some approaches have explored the space between traditional modular pipelines and end”
BADGR: An Autonomous Self-Supervised Learning-Based Navigation System (2020, methodology): “The most similar works to our BADGR system are GCG [8] and CAPs [9].”
Composable Action-Conditioned Predictors: Flexible Off-Policy Learning for Robot Navigation (2018, methodology): “Prediction has been an active area of research in computer vision, including prediction of future images [9, 10, 11], motion [12], and even physical events [13, 14, 15].”
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation (2018, methodology): “The image s and action a are inputs into our network, and the arg max in Equation (1) is evaluated by means of a stochastic optimization algorithm that can handle non-convex and multimodal optimization landscapes, similarly to [19] and [20].”
Lifelong Navigation (2020, background): “As for physical robot navigation, learning approaches typically either imitate a conventional planner [19] or learn from trial-and-error using reinforcement learning [3].”
Neural Autonomous Navigation with Riemannian Motion Policy (2019, background): “Another line of research adopts reinforcement learning (RL) [23], [24], [25] for self-supervised training.”
A CCELERATING O NLINE R EINFORCEMENT L EARNING VIA M ODEL -B ASED M ETA -L EARNING (2021, methodology): “Our model will be based off recent work, GCG and MuZero (Kahn et al., 2018; Schrittwieser et al., 2020) which learns a latent dynamics model that predicts a future sequence of states in latent space and a future sequence of rewards from these latents.”
Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learning (2018, background): “[20] propose a selfsupervised deep RL model for navigation.”
γ-Models: Generative Temporal Difference Learning for Infinite-Horizon Prediction (2020, background): “…(Nguyen & Widrow, 1990; Jordan & Rumelhart, 1992; Heess et al., 2015), and incorporating model-based planning without explicitly predicting future observations (Tamar et al., 2016; Silver et al., 2017; Oh et al., 2017; Kahn et al., 2018; Amos et al., 2018; Schrittwieser et al., 2019).”
RoboTHOR: An Open Simulation-to-Real Embodied AI Platform (2020, background): “For example, [79] address navigation towards a given input image, [10] address navigation towards a point in a scene and [30] learn a collision-free navigation policy.”
Visual Semantic Navigation using Scene Priors (2018, methodology): “Kahn et al. (2018) propose a method based on model-free and model-based RL to learn navigation policies using a few samples.”
Toward a Brain-Inspired System: Deep Recurrent Reinforcement Learning for a Simulated Self-Driving Agent (2019, methodology): “A recent work (Kahn et al., 2018) adopted double Q learning with recurrency and computation graphs to tackle a robotics navigation task.”
A survey of uncertainty in deep neural networks (2021, background): “[325], [28], [326], [327] demonstrated uncertainty-aware, real world application of a reinforcement learning algorithm for robotics, while [328], [329] proposed to leverage spatial information, on top of MC-dropout.”
Unifying Map and Landmark Based Representations for Visual Navigation (2017, background): “Most such approaches [6, 11, 15, 26, 28] study task-agnostic collision avoidance, and don’t build representations for spaces or plan paths.”
Deep Visual MPC-Policy Learning for Navigation (2019, methodology): “Learning-based methods including Deep Reinforcement Learning (DRL) [4, 27, 28] and Imitation Learning (IL) [29] have been used to obtain navigation policies.”
Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (2024, background): “Kang et al. (102) showed the values of 1) task-specific pre-training in simulation for learning generalizable visual representation and 2) the use of real-world data for learning accurate dynamics (79).”
A Lifelong Learning Approach to Mobile Robot Navigation (2021, background): “As for physical robot navigation, learning approaches typically either imitate an expert [19], [20] or learn from trial-and-error using reinforcement learning [3], [21].”