Learning Navigation Behaviors End-to-End With AutoRL

By Hao-Tien Lewis Chiang, Aleksandra Faust, Marek Fiser, Anthony Francis. 2018

261 citations recorded by the source corpus.

We learn end-to-end point-to-point and path-following navigation behaviors that avoid moving obstacles. These policies receive noisy lidar observations and output robot linear and angular velocities. The policies are trained in small, static environments with AutoRL, an evolutionary automation layer around reinforcement learning (RL) that searches for a deep RL reward and neural network architecture with large-scale hyper-parameter optimization. AutoRL first finds a reward that maximizes task completion and then finds a neural network architecture that maximizes the cumulative of the found reward. Empirical evaluations, both in simulation and on-robot, show that AutoRL policies do not suffer from the catastrophic forgetfulness that plagues many other deep reinforcement learning algorithms, generalize to new environments and moving obstacles, are robust to sensor, actuator, and localization noise, and can serve as robust building blocks for larger navigation tasks. Our path-following and point-to-point policies are, respectively, 23% and 26% more successful than comparison methods across new environments.

Subfields: Aerospace Engineering, Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

BADGR: An Autonomous Self-Supervised Learning-Based Navigation System (2020, background): “However, these methods have typically focused on pure geometric reasoning, require on-policy data, and often utilize simulation due to constraints such as sample efficiency [28, 8, 29, 30, 31, 32].”
Search on the Replay Buffer: Bridging Planning and Reinforcement Learning (2019, background): “Some prior works have also combined planning algorithms with RL (Chiang et al., 2019; Faust et al., 2018; Savinov et al., 2018a), ﬁnding that the combination yields agents adept at reaching distant goals.”
Adaptive Power System Emergency Control Using Deep Reinforcement Learning (2019, methodology): “Inspired by a recent work [40], we plan to automate this part in future efforts.”
Lyapunov-based Safe Policy Optimization for Continuous Control (2019, background): “…high-dimensional control problems, including video games (Mnih et al., 2015), board games (Silver et al., 2016), robot locomotion (Lillicrap et al., 2016), manipulation (Levine et al., 2016; Kalashnikov et al., 2018), navigation (Faust et al., 2018), and obstacle avoidance (Chiang et al., 2019).”
HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators (2019, background): “In robotics, this success has mostly been restricted to either navigation [7, 8, 9, 10] or manipulation with stationary arms [2, 11, 12, 13, 14].”
Language to Rewards for Robotic Skill Synthesis (2023, methodology): “While reward design can be automated, these techniques are sample-inefficient and still requires manual specification of an objective indicator function for each task [20].”
Combining Optimal Control and Learning for Visual Navigation in Novel Environments (2019, background): “Several works also use end-to-end learning for navigation using laser scans [12, 13, 14], for training and combining a local planner with a higher level roadmap for long range navigation [15, 16, 17, 18], or visual servoing [19].”
Deep reinforcement learning based mobile robot navigation: A review (2021, methodology): “(1) Expansion of network input Considering that the agent cannot uniquely distinguish its state based on current observations, the simplest solution is to add several previous observation frames as network inputs to improve its ability to distinguish among states [36, 51, 72, 75, 76] .”
Lifelong Navigation (2020, methodology): “Learning-based Navigation Data-driven machine learning techniques have also been widely applied to navigation problems [10, 11, 12, 13, 14, 15, 16, 17, 18].”
How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies? (2019, background): “While there are promising successes in applying RL algorithms directly on real systems ([7], [8], [9], [10], [11]), their performance on more complex systems remains bottle-necked by the relative data inefﬁciency of RL algorithms.”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, methodology): “One approach to reducing the burden of tuning hyperparameters is to use automated hyperparameter tuning methods (Chiang et al. 2019).”
Meta-learning curiosity algorithms (2020, background): “Finally, Chiang et al. (2019); Faust et al.”
Learning to Walk in the Real World with Minimal Human Effort (2020, background): “robotic applications, including autonomous driving [5, 41], navigation [15], and manipulation [33, 36, 2].”
Eureka: Human-Level Reward Design via Coding Large Language Models (2023, unknown): “Several prior works have studied automated reward search through evolutionary algorithms (Niekum et al., 2010; Chiang et al., 2019; Faust et al., 2019).”
Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (2024, methodology): “Instead of replacing the entire navigation stack with an RL policy, modular approaches replace specific components like the local planner (75) or the exploration algorithm (76) with RL, enabling better performance than classical baselines.”