Vision-Dialog Navigation by Exploring Cross-Modal Memory
By Yi Zhu, Fengda Zhu, Zhaohuan Zhan, Bingqian Lin, Jianbin Jiao, Xiaojun Chang, Xiaodan Liang. 2020
54 citations recorded by the source corpus.
Vision-dialog navigation posed as a new holy-grail task in vision-language disciplinary targets at learning an agent endowed with the capability of constant conversation for help with natural language and navigating according to human responses. Besides the common challenges faced in visual language navigation, vision-dialog navigation also requires to handle well with the language intentions of a series of questions about the temporal context from dialogue history and co-reasoning both dialogs and visual scenes. In this paper, we propose the Cross-modal Memory Network (CMN) for remembering and understanding the rich information relevant to historical navigation actions. Our CMN consists of two memory modules, the language memory module (L-mem) and the visual memory module (V-mem). Specifically, L-mem learns latent relationships between the current language interaction and a dialog history by employing a multi-head attention mechanism. V-mem learns to associate the current visual views and the cross-modal memory about the previous navigation actions. The cross-modal memory is generated via a vision-to-language attention and a language-to-vision attention. Benefiting from the collaborative learning of the L-mem and the V-mem, our CMN is able to explore the memory about the decision making of historical navigation actions which is for the current step. Experiments on the CVDN dataset show that our CMN outperforms the previous state-of-the-art model by a significant margin on both seen and unseen environments.
Subfields: Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Xiaodan Liang
- Mohit Bansal
- Xiaojun Chang
- Luc Van Gool
- Hao Hao Tan
- Yi Zhu
- Jianfeng Gao
- Trevor Darrell
Citations in the corpus
- A Survey of Embodied AI: From Simulators to Research Tasks (2021, methodology): “The current work [101] in this area focuses on having a Cross-modal Memory Network (CMN) to help with remembering and understanding of the rich information related to the past navigation actions and make decisions for the current steps in navigation.”
- Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information (2021, background): “…of research in VLN utilizes soft attention over individual words for cross- modal grounding in both the natural language instruction and the visual scene (Wang et al., 2018, 2019; Tan et al., 2019; Landi et al., 2019; Xia et al., 2020; Wang et al., 2020b,a; Xiang et al., 2020; Zhu et al., 2020b).”
- Vision-Language Navigation With Self-Supervised Auxiliary Reasoning Tasks (2019, background): “Previous works in Vision-Language Navigation have focused on improving the ability of perceiving the vision and language inputs [13, 10, 42] and cross-modal matching [41, 47].”