Look, Listen, and Act: Towards Audio-Visual Embodied Navigation
By Chuang Gan, Yiwei Zhang, Jiajun Wu, Boqing Gong, Josh Tenenbaum. 2019
168 citations recorded by the source corpus.
A crucial ability of mobile intelligent agents is to integrate the evidence from multiple sensory inputs in an environment and to make a sequence of actions to reach their goals. In this paper, we attempt to approach the problem of Audio-Visual Embodied Navigation, the task of planning the shortest path from a random starting location in a scene to the sound source in an indoor environment, given only raw egocentric visual and audio sensory data. To accomplish this task, the agent is required to learn from various modalities, i.e., relating the audio signal to the visual environment. Here we describe an approach to audio-visual embodied navigation that takes advantage of both visual and audio pieces of evidence. Our solution is based on three key ideas: a visual perception mapper module that constructs its spatial memory of the environment, a sound perception module that infers the relative location of the sound source from the agent, and a dynamic path planner that plans a sequence of actions based on the audio-visual observations and the spatial memory of the environment to navigate toward the goal. Experimental results on a newly collected Visual-Audio-Room dataset using the simulated multi-modal environment demonstrate the effectiveness of our approach over several competitive baselines.
Subfields: Cognitive Neuroscience, Experimental and Cognitive Psychology, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Cognitive Neuroscience
In Experimental and Cognitive Psychology
In Signal Processing
Citations in the corpus
- A Survey of Embodied AI: From Simulators to Research Tasks (2021, background): “In this work [99], artificial agents pick multiple sensory observations such as vision and sound signal of the target objects and figure out the shortest trajectory to navigation from its starting location to the source of the sounds.”
- Occupancy Anticipation for Efficient Exploration and Navigation (2020, unknown): “However, our anticipation model is generic and can be easily integrated with most map-based embodied navigation models [20,11,18].”
- The rise and potential of large language model based agents: a survey (2025, unknown): “Previous studies demonstrate a basic framework that includes a dynamic path planner that uses visual and auditory observations along with spatial memories to plan a series of actions [265,280].”
- The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark Towards Physically Realistic Embodied AI (2021, unknown): “Popular tasks include point-goal navigation [31], audio-visual navigation [16, 9], vision-language navigation [1], and semantic-goal navigation [8].”
- SoundSpaces: Audio-Visual Navigation in 3D Environments (2019, background): “low-cost game audio in [29]), and it offers 5,000× more audio data and 15× more environments.”