Deep reinforcement learning with successor features for navigation across similar environments

By Jingwei Zhang, Jost Tobias Springenberg, Joschka Boedecker, Wolfram Burgard. 2016

314 citations recorded by the source corpus.

In this paper we consider the problem of robot navigation in simple maze-like environments where the robot has to rely on its onboard sensors to perform the navigation task. In particular, we are interested in solutions to this problem that do not require localization, mapping or planning. Additionally, we require that our solution can quickly adapt to new situations (e.g., changing navigation goals and environments). To meet these criteria we frame this problem as a sequence of related reinforcement learning tasks. We propose a successor-feature-based deep reinforcement learning algorithm that can learn to transfer knowledge from previously mastered navigation tasks to new problem instances. Our algorithm substantially decreases the required learning time after the first task instance has been solved, which makes it easily adaptable to changing environments. We validate our method in both simulated and real robot experiments with a Robotino and compare it to a set of baseline methods including classical planning-based navigation.

Subfields: Artificial Intelligence, Computer Networks and Communications, Computer Vision and Pattern Recognition, Reinforcement Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Representation & Self-Supervised Learning

Citations in the corpus

A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, positive): “…with only pixel inputs (Mnih et al., 2015, 2016), to learning policies capable of driving real robotic systems in navigation (Zhu et al., 2017b; Zhang et al., 2017a; Tai et al., 2017) and manipulation (Levine et al., 2016; Yu et al., 2018) tasks, the learning power of deep networks drives the…”
The AdobeIndoorNav Dataset: Towards Deep Reinforcement Learning based Real-world Indoor Robot Visual Navigation (2018, background): “…strong navigation performance in the indoor environments, the recently proposed deep reinforcement learning (DRL) based methods [1], [2], [3], [4] open up a new possibility that avoids the complexity of constructing these sophisticated SLAM pipelines and directly learns model-free…”
(CAD)$^2$RL: Real Single-Image Flight without a Single Real Image (2016, background): “Several recent works have extended deep RL methods to real-world robotics applications using either lowdimensional estimated state [10] or by collecting an exhaustive real-world dataset under gridworld-like assumptions [43].”
Successor Features for Transfer in Reinforcement Learning (2016, neutral): “[29] propose similar architectures to jointly learn ψ̃ π (s, a), φ̃(s, a, s′) and w̃.”
Cognitive Mapping and Planning for Visual Navigation (2017, background): “[8, 16, 46] show results in synthetic mazes, and only [10, 53, 62] show results with real images.”
SplitNet: Sim2Sim and Task2Task Transfer for Embodied Visual Navigation (2019, background): “For increased task generalization, others have proposed using the successor representation [38, 41] which decomposes the reward and Q-functions into a state-action feature φs,a, a successor feature ψs,a and a task reward vector w.”
Knowledge Transfer in Multi-Task Deep Reinforcement Learning for Continuous Control (2020, background): “Zhang et al. [25] proposed a DRL algorithm that can learn to transfer knowledge from previously-mastered navigation tasks to new problem instances through successor representation learning.”
One-Shot Reinforcement Learning for Robot Navigation with Interactive Replay (2017, background): “In robotics, agents have been trained to navigate using laser sensors in simulation [15] and simulated depth imagery [16]; structure-based sensors tend to exhibit less difference between simulation and reality (a narrower “reality gap”) than does visual sensing.”
Sharing Experience in Multitask Reinforcement Learning (2019, background): “Studies in [Borsa et al., 2016; Zhang et al., 2017] introduce the idea of integrating successor features induced from the neural encoder into multi-task learning framework, in which a linear dependency between representations across shared structure environments is assumed to enable knowledge to tra”
Distributed multi-robot collision avoidance via deep reinforcement learning for navigation in complex scenarios (2020, methodology): “DNNs have also been integrated with model predictive control to control robotic systems (Lenz et al., 2015; Zhang et al., 2016b).”
Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systems (2019, methodology): “directly from raw sensor inputs: target-driven navigation [5], successor feature RL for transferring navigation policies [6], and using auxiliary tasks to boost DRL training [7].”
Global Pose Estimation with an Attention-Based Recurrent Network (2018, methodology): “In simulated environments, [31] uses stacked LSTM in a goal-driven RL problem with auxilary tasks of depth prediction and loop-closure classification, while [48] added successor features to ease transfer from previously mastered navigation tasks to new ones.”
Cross-View Policy Learning for Street Navigation (2019, background): “A successorfeature-based deep RL algorithm that can learn to transfer knowledge from previously mastered navigation tasks to new problem instances was proposed in [46].”
Socially Compliant Navigation Through Raw Depth Inputs with Generative Adversarial Imitation Learning (2017, background): “from raw sensory inputs [5], [6], [7], [8].”
Attentive Multi-Task Deep Reinforcement Learning (2019, methodology): “ic weights, the best that can be done is to interpolate between con icting tasks. In contrast, our method allows con icting tasks to be learned in separate networks. One interesting line of research [15,31,3,2,16] capitalizes on transferring knowledge based on successor features, i.e., shared enviro”
Two Body Problem: Collaborative Visual Task Completion (2019, background): “Consequently the task is well suited for an RL formulation, a perspective which has become popular recently [62, 1, 16, 17, 33, 42, 86, 59, 5, 8, 90, 25, 36, 91, 37].”