Audio-Visual Embodied Navigation

By Changan Chen, Unnat Jain, Carl Schissler, Sebastià Vicenc Amengual Garí, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip W. Robinson, Kristen Grauman. 2019

28 citations recorded by the source corpus.

Moving around in the world is naturally a multisensory experience, but today's embodied agents are deaf - restricted to solely their visual perception of the environment. We introduce audio-visual navigation for complex, acoustically and visually realistic 3D environments. By both seeing and hearing, the agent must learn to navigate to an audio-based target. We develop a multi-modal deep reinforcement learning pipeline to train navigation policies end-to-end from a stream of egocentric audio-visual observations, allowing the agent to (1) discover elements of the geometry of the physical space indicated by the reverberating audio and (2) detect and follow sound-emitting targets. We further introduce audio renderings based on geometrical acoustic simulations for a set of publicly available 3D assets and instrument AI-Habitat to support the new sensor, making it possible to insert arbitrary sound sources in an array of apartment, office, and hotel environments. Our results show that audio greatly benefits embodied visual navigation in 3D spaces.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Developmental Biology, Multimodal Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Developmental Biology

In Multimodal Learning

In Robotics & Control

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “The past few years have seen audio studied in computer vision tasks [245] for action classification [110,226], object categorization [125,234], source localization and tracking [14,197,212] and embodied navigation [33].”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, background): “The concept of real-to-sim revolves around capturing real-world information such as tactile perception [131], human-level motor control [132] and audio inputs [133] in addition to visual sensory inputs and integrating them for the development of more realistic embodied AI simulators that can effecti”
Look, Listen, and Act: Towards Audio-Visual Embodied Navigation (2019, background): “[19] also poses the task of audio-visual embodied navigation on the Habitat platform [20].”
A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Tasks (2020, background): “Recently, with visual and acoustic rendering, agents have been trained for audio-visual embod-ied navigation [19,31].”
Multi-modal Self-Supervision from Generalized Data Transformations (2020, methodology): “Futhermore, multi-modal learning has been used for several applications such as lip reading [4, 20, 21], audio-visual source separation and localization [3, 7, 28, 29, 39, 77, 87, 91, 113, 114], speech recognition [2, 73, 84], efficient inference [30, 56], egocentric action recognition [51] and navi”