Room-and-Object Aware Knowledge Reasoning for Remote Embodied Referring Expression

By Chen Gao, Jinyu Chen, Si Liu, Luting Wang, Qiong Zhang, Qi Wu. 2021

106 citations recorded by the source corpus.

The Remote Embodied Referring Expression (REVERIE) is a recently raised task that requires an agent to navigate to and localise a referred remote object according to a high-level language instruction. Different from related VLN tasks, the key to REVERIE is to conduct goal-oriented exploration instead of strict instruction-following, due to the lack of step-by-step navigation guidance. In this paper, we propose a novel Cross-modality Knowledge Reasoning (CKR) model to address the unique challenges of this task. The CKR, based on a transformer-architecture, learns to generate scene memory tokens and utilise these informative history clues for exploration. Particularly, a Room-and-Object Aware Attention (ROAA) mechanism is devised to explicitly perceive the room- and object-type information from both linguistic and visual observations. Moreover, through incorporating commonsense knowledge, we propose a Knowledge-enabled Entity Relationship Reasoning (KERR) module to learn the internal-external correlations among room- and object-entities for agent to make proper action at each viewpoint. Evaluation on REVERIE benchmark demonstrates the superiority of the CKR model, which significantly boosts SPL and REVERIE-success rate by 64.67% and 46.05%, respectively. Code is available at: https://github.com/alloldman/CKR.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jiafei Duan
2. Samson Yu
3. Cheston Tan
4. Yicong Hong
5. Gengze Zhou
6. Qi Wu
7. Zenan Li
8. Wentao Zhao

In Computer Vision and Pattern Recognition

1. Jiafei Duan
2. Samson Yu
3. Cheston Tan
4. Yicong Hong
5. Gengze Zhou
6. Junchi Yan
7. Qi Wu
8. Tangyao Li

Citations in the corpus

• A Survey of Embodied AI: From Simulators to Research Tasks (2021, background): “Simulators with both a high visual fidelity and realistic physics are highly sought after by the robotics communities as they provide the ideal test-bed for various robotic tasks such as navigation and interaction tasks [110], [111].”
• NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (2023, methodology): “When navigating to a viewpoint with unexpected observation, NavGPT can plan to explore the environment and use commonsense knowledge to assist decision-making, similar to VLN methods incorporate external knowledge(Li et al. 2022b; Gao et al. 2021; Li et al. 2023b).”
• NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classification (2023, background): “, text classification [46], vision navigation [12], physics simulation [30], etc.”