DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames

By Erik Wijmans, Abhishek Kadian, Ari S. Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, Dhruv Batra. 2019

650 citations recorded by the source corpus.

We present Decentralized Distributed Proximal Policy Optimization (DD-PPO), a method for distributed reinforcement learning in resource-intensive simulated environments. DD-PPO is distributed (uses multiple machines), decentralized (lacks a centralized server), and synchronous (no computation is ever "stale"), making it conceptually simple and easy to implement. In our experiments on training virtual robots to navigate in Habitat-Sim, DD-PPO exhibits near-linear scaling -- achieving a speedup of 107x on 128 GPUs over a serial implementation. We leverage this scaling to train an agent for 2.5 Billion steps of experience (the equivalent of 80 years of human experience) -- over 6 months of GPU-time training in under 3 days of wall-clock time with 64 GPUs. This massive-scale training not only sets the state of art on Habitat Autonomous Navigation Challenge 2019, but essentially "solves" the task -- near-perfect autonomous navigation in an unseen environment without access to a map, directly from an RGB-D camera and a GPS+Compass sensor. Fortuitously, error vs computation exhibits a power-law-like distribution; thus, 90% of peak performance is obtained relatively early (at 100 million steps) and relatively cheaply (under 1 day with 8 GPUs). Finally, we show that the scene understanding and navigation policies learned can be transferred to other navigation tasks -- the analog of "ImageNet pre-training + task-specific fine-tuning" for embodied AI. Our model outperforms ImageNet pre-trained CNNs on these transfer tasks and can serve as a universal resource (all models and code are publicly available).

Subfields: Computer Vision and Pattern Recognition, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Habitat 2.0: Training Home Assistants to Rearrange their Habitat (2021, methodology): “The visual input is encoded using a CNN, concatenated with embeddings of proprioceptive-sensing and goal coordinates, and fed to a recurrent actor-critic network, trained with DD-PPO [12] for 100 Million steps of experience (see Appendix C.”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, methodology): “ments in the real world, Habitat Challenge 2020 has moved on to the more challenging task [87] of RGBD-based online localization without the GPS and compass.”
CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration (2022, methodology): “We train agents independently in Habitat [59] and RoboTHOR [17] simulation environments for 60M steps each, using DD-PPO [60, 67] and the AllenAct [66] framework.”
SQA3D: Situated Question Answering in 3D Scenes (2022, background): “Notable tasks including robotic navigation (Das et al., 2018; Anderson et al., 2018; Savva et al., 2019; Chen et al., 2019; Wijmans et al., 2019b; Qi et al., 2020; Deitke et al., 2022) and vision-based manipulation (Kolve et al., 2017; Puig et al., 2018; Xie et al., 2019; Shridhar et al., 2020a;b;…”
Occupancy Anticipation for Efficient Exploration and Navigation (2020, methodology): “While our approach generalizes well to this setting, DD-PPO [72] fails (0 SPL) due to its reliance on perfect odometry.”
Exploiting Scene-specific Features for Object Goal Navigation (2020, methodology): “In [29], a scalable Reinforcement Learning algorithm on multiple GPUs capable of solving the PointGoal Navigation task almost perfectly has been presented.”
Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning (2020, methodology): “Decentralized Distributed PPO (Wijmans et al., 2020) optimizes the distributed policy gradient setup for multi-GPU clusters and resource-intensive environments by parallelizing the learners and significantly reducing the network throughput required.”
Seeing the Un-Scene: Learning Amodal Semantic Maps for Room Navigation (2020, methodology): “ts a bathroom in the house but this region does not appear inside the crop as it does not fall inside the crop w.r.t. the agent’s current location. Inspired by the recent progress in point navigation [32,41], we reduce the room navigation problem to point navigation. We train a network f point to pr”
A Survey on Vision--Language--Action Models for Embodied AI (2024, neutral): “Real : SayCan setup LEO [159] ConvNeXt [160], PointNet++ [161] Vicuna-7B Concat MLP Yes Sim-Mani : CLIPort task subset; Sim-Navi : Habitat-web 3D-LLM [57] CLIP, 3D-CLR [162], ConceptFusion [163] Flamingo, BLIP-2”
Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI (2021, methodology): “We use a standard agent architecture for training on different datasets [30].”
Learning Object-conditioned Exploration using Distributed Soft Actor Critic (2020, background): “Researchers have studied a wide variety of navigation problems, such as metric navigation [9, 10, 11, 12], targetdriven semantic navigation [7, 13, 14], and exploration [15, 12, 16, 17, 18, 19].”
Topological Planning with Transformers for Vision-and-Language Navigation (2020, background): “Such approaches often require millions of frames of experience [11, 50, 52], and yet performance substantially degrades without ground truth odometry [11, 52].”
Curious Representation Learning for Embodied Intelligence (2021, methodology): “We posit that in PointNav, in our data-efficient experimental setting, vision is not crucial to obtain good performance, since the policy is given a compass, but note that in the large-scale RL setting [45] shows that vision is indeed helpful for navigation.”
Diffusion-based Generation, Optimization, and Planning in 3D Scenes (2023, background): “namic model [5,11,46,65,71,75], lacking a trajectory-level optimization for long-horizon planning.”