SplitNet: Sim2Sim and Task2Task Transfer for Embodied Visual Navigation

By Daniel Gordon, Abhishek Kadian, Devi Parikh, Judy Hoffman, Dhruv Batra. 2019

81 citations recorded by the source corpus.

We propose SplitNet, a method for decoupling visual perception and policy learning. By incorporating auxiliary tasks and selective learning of portions of the model, we explicitly decompose the learning objectives for visual navigation into perceiving the world and acting on that perception. We show improvements over baseline models on transferring between simulators, an encouraging step towards Sim2Real. Additionally, SplitNet generalizes better to unseen environments from the same simulator and transfers faster and more effectively to novel embodied navigation tasks. Further, given only a small sample from a target domain, SplitNet can match the performance of traditional end-to-end pipelines which receive the entire dataset

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey of Embodied AI: From Simulators to Research Tasks (2021, positive): “[98] provides a more modular approach.”
CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration (2022, result): “Most similar to our work is that of Gordon et al. [25] who decouple navigation into perception and acting, enabling faster transfer to new simulated environments.”
Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Performance? (2019, background): “More recently, Gordon et al. [23] showed that decoupling perception and policy learning modules can aid in generalization to unseen environments, as well as between different environment datasets.”
Auxiliary Tasks Speed Up Learning PointGoal Navigation (2020, unknown): “Closely related to our work is that of Gordon et al. [24] who show that auxiliary tasks can be leveraged to improve transfer to new tasks and new simulation environments ( i.e . synthetic to photorealistic).”
DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames (2019, methodology): “We examine transferring our agents to the following tasks (Gordon et al., 2019) – Flee The agent maximizes its geodesic distance from its starting location.”
How to Train PointGoal Navigation Agents on a (Sample and Compute) Budget (2020, unknown): “In this space, PointGoal Navigation has seen considerable interest recently with works proposing to leverage hierarchical models [24], pre-trained representations [25, 28], auxiliary-tasks [23], or distribution and very deep networks [16] to improve performance and/or sample efficiency.”
Audio-Visual Embodied Navigation (2019, methodology): “We assume an idealized GPS sensor, following prior work [61, 15, 38, 55].”
CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation (2022, background): “2, 3 [25] Daniel Gordon, Abhishek Kadian, Devi Parikh, Judy Hoffman, and Dhruv Batra.”
SoundSpaces: Audio-Visual Navigation in 3D Environments (2019, methodology): “Agents are allowed a time horizon of 500 actions for all tasks, similar to [59,47,14,37,52].”
Are We Making Real Progress in Simulated Environments? Measuring the Sim2Real Gap in Embodied Visual Navigation (2019, background): “3 [23] Daniel Gordon, Abhishek Kadian, Devi Parikh, Judy Hoffman, and Dhruv Batra.”