Reinforced Imitation: Sample Efficient Deep Reinforcement Learning for Mapless Navigation by Leveraging Prior Demonstrations

By Mark Pfeiffer, Samarth Shukla, Matteo Turchetta, César Cadena, Andreas Krause, Roland Yves Siegwart, Juan I. Nieto. 2018

184 citations recorded by the source corpus.

This letter presents a case study of a learning-based approach for target-driven mapless navigation. The underlying navigation model is an end-to-end neural network, which is trained using a combination of expert demonstrations, imitation learning (IL) and reinforcement learning (RL). While RL and IL suffer from a large sample complexity and the distribution mismatch problem, respectively, we show that leveraging prior expert demonstrations for pretraining can reduce the training time to reach at least the same level of the performance compared to plain RL by a factor of 5. We present a thorough evaluation of different combinations of expert demonstrations, different RL algorithms, and reward functions, both in simulation and on a real robotic platform. Our results show that the final model outperforms both standalone approaches in the amount of successful navigation tasks. In addition, the RL reward function can be significantly simplified when using pretraining, e.g., by using a sparse reward only. The learned navigation policy is able to generalize to unseen and real-world environments.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

1. Bo Liu
2. Peter Stone
3. Shixiang Shane Gu
4. Scott Fujimoto
5. Kai Zhu
6. Tao Zhang
7. Xuesu Xiao
8. Sergey Levine

In Computer Vision and Pattern Recognition

1. Peter Stone
2. Bo Liu
3. Scott Fujimoto
4. Shixiang Shane Gu
5. Kai Zhu
6. Tao Zhang
7. Xuesu Xiao
8. Luc Van Gool

Citations in the corpus

• A Minimalist Approach to Offline Reinforcement Learning (2021, methodology): “, 2018], used for pre-training [Pfeiffer et al., 2018], modifying the replay buffer [Večerík et al.”
• Lifelong Navigation (2020, methodology): “Learning-based Navigation Data-driven machine learning techniques have also been widely applied to navigation problems [10, 11, 12, 13, 14, 15, 16, 17, 18].”
• Deep reinforcement learning based mobile robot navigation: A review (2021, methodology): “In addition, expert demonstration can provide an auxiliary experience for DRL, which can be used to address the sparse reward problem[38, 83].”
• Lyapunov-based Safe Policy Optimization for Continuous Control (2019, unknown): “Other use cases of CMDP constraints in robot navigation include collision avoidance (Pfeiffer et al., 2018) or limiting total battery usage throughout the task.”
• A Lifelong Learning Approach to Mobile Robot Navigation (2021, unknown): “b) Learning-based navigation: Data-driven machine learning techniques have also been widely applied to navigation problems [10]–[18].”