Composable Action-Conditioned Predictors: Flexible Off-Policy Learning for Robot Navigation

By Gregory Kahn, Adam Villaflor, Pieter Abbeel, Sergey Levine. 2018

22 citations recorded by the source corpus.

A general-purpose intelligent robot must be able to learn autonomously and be able to accomplish multiple tasks in order to be deployed in the real world. However, standard reinforcement learning approaches learn separate task-specific policies and assume the reward function for each task is known a priori. We propose a framework that learns event cues from off-policy data, and can flexibly combine these event cues at test time to accomplish different tasks. These event cue labels are not assumed to be known a priori, but are instead labeled using learned models, such as computer vision detectors, and then `backed up' in time using an action-conditioned predictive model. We show that a simulated robotic car and a real-world RC car can gather data and train fully autonomously without any human-provided labels beyond those needed to train the detectors, and then at test-time be able to accomplish a variety of different tasks. Videos of the experiments and code can be found at this https URL

Subfields: Artificial Intelligence, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “…in the conventional online RL setting (Tamar et al., 2016; Dosovitskiy and Koltun, 2016; Oh et al., 2017), and in the offline RL setting, such techniques have been applied effectively to learning navigational policies for mobile robots from previously collected data (Kahn et al., 2018, 2020).”
BADGR: An Autonomous Self-Supervised Learning-Based Navigation System (2020, methodology): “The learned predictive model, similarly to [4], [5], takes as input the current sensor observations and a sequence of future intended actions, and predicts the future navigational events.”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “This straightforward approach is known to fail in the offline RL setting, both in theory and practice, due to distribution shift and model-bias Ross & Bagnell (2012); Kidambi et al. (2020). In order to overcome these challenges, offline model-based algorithms like MOReL Kidambi et al. (2020) and MOP”
Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning (2021, methodology): “Model-based methods [18, 50, 13, 56, 19] attempt to learn the model from offline data, with minimal modification to the algorithm.”