Thinking While Moving: Deep Reinforcement Learning with Concurrent Control

By Ted Xiao, Eric Jang, Dmitry Kalashnikov, Sergey Levine, Julian Ibarz, Karol Hausman, Alexander Herzog. 2020

47 citations recorded by the source corpus.

We study reinforcement learning in settings where sampling an action from the policy must be done concurrently with the time evolution of the controlled system, such as when a robot must decide on the next action while still performing the previous action. Much like a person or an animal, the robot must think and move at the same time, deciding on its next action before the previous one has completed. In order to develop an algorithmic framework for such concurrent control problems, we start with a continuous-time formulation of the Bellman equations, and then discretize them in a way that is aware of system delays. We instantiate this new class of approximate dynamic programming methods via a simple architectural extension to existing value-based deep reinforcement learning algorithms. We evaluate our methods on simulated benchmark tasks and a large-scale robotic grasping task where the robot must "think while moving.''

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “The model is given how much of the previous action is left to execute [87].”
Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, methodology): “The Everyday Robots robot, a mobile manipulator with RGB observations, is placed in an office kitchen to interact with common objects using concurrent [99] continuous closed-loop control.”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, methodology): “Acting at the Same Time The MDP formulation assumes synchronous execution: the observed state remains unchanged until the action is applied.”
PI-QT-Opt: Predictive Information Improves Multi-Task Robotic Reinforcement Learning at Scale (2022, methodology): “QT-Opt is able to leverage large-scale, multi-task datasets in simulation and the real world [12, 13, 14, 15].”
Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models (2022, methodology): “A mobile manipulator robot with a parallel-jaw gripper, an over-the-shoulder RGB camera, and a 7 DoF arm is placed in an ofﬁce kitchen to interact with common objects using concurrent [46] continuous closed-loop control from pixels.”