Learning to Assemble Neural Module Tree Networks for Visual Grounding

By Daqing Liu, Hanwang Zhang, Zhengjun Zha, Feng Wu. IEEE International Conference on Computer Vision, 2018

319 citations recorded by the source corpus.

Visual grounding, a task to ground (i.e., localize) natural language in images, essentially requires composite visual reasoning. However, existing methods over-simplify the composite nature of language into a monolithic sentence embedding or a coarse composition of subject-predicate-object triplet. In this paper, we propose to ground natural language in an intuitive, explainable, and composite fashion as it should be. In particular, we develop a novel modular network called Neural Module Tree network (NMTree) that regularizes the visual grounding along the dependency parsing tree of the sentence, where each node is a neural module that calculates visual attention according to its linguistic feature, and the grounding score is accumulated in a bottom-up direction where as needed. NMTree disentangles the visual grounding from the composite reasoning, allowing the former to only focus on primitive and easy-to-generalize patterns. To reduce the impact of parsing errors, we train the modules and their assembly end-to-end by using the Gumbel-Softmax approximation and its straight-through gradient estimator, accounting for the discrete nature of module assembly. Overall, the proposed NMTree consistently outperforms the state-of-the-arts on several benchmarks. Qualitative results show explainable grounding score calculation in great detail.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/e504843bfd3ca81b859895439bbe19efc1f3dc61

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Deconfounded Image Captioning: A Causal Retrospect (2020, background): “Such a strategy is also widely applied in other vision-language tasks like VQA [50], [51], Grounding [52], [53], and Visual Dialogue [54], e.g., [54] exploits the scene graph containing both visual and semantic knowledge of the image to complete the dialogue.”
Multi-Task Collaborative Network for Joint Referring Expression Comprehension and Segmentation (2020, methodology): “To validate our approach, we conduct extensive experiments on three benchmark datasets, i.e. , RefCOCO, Re-fCOCO+ and RefCOCOg, and compare MCN to a set of state-of-the-arts (SOTAs) in both REC and RES [42, 38, 40, 16, 18, 37].”
Visual Grounding with Transformers (2021, methodology): “Early attempts [2, 3, 4, 5] view visual grounding as a special case of text-based image retrieval and frame it as a retrieval task on a set of candidate regions in a given image.”
Neurosymbolic Programming (2021, methodology): “To this end, it follows the lead of recent work on gradient-based architecture search for neural networks Liu et al., 2019b and defines a continuous relaxation of the space of programs.”
Towards Lightweight Transformer Via Group-Wise Transformation for Vision-and-Language Tasks (2022, methodology): “Compare to SOTA methods like NMTree [63], the performance gains of LW-Transformer are up to 6.1% (on RefCOCO+), greatly showing the generalization ability of LW-Transformer.”
FindIt: Generalized Localization with Natural Language Queries (2022, methodology): “One-stage [7,44,89,13,55] and two-stage [94,91,27,83,87,50] methods have been popular for these tasks.”
DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Grounding (2022, methodology): “The top-down methods (Hong et al. 2020; Hu et al. 2016; Liu, Wang, and Yang 2017; Liu et al. 2018; Nagaraja, Morariu, and Davis 2016; Mao et al. 2015) view the task as a ranking task with a set of boxes predicted by a pre-trained object detector, which limits their performance.”
Referring Transformer: A One-step Approach to Multi-task Visual Grounding (2021, background): “, [16, 31]) and train it to output the segmented masks [18, 29, 38, 53, 56].”
Modality Shifting Attention Network for Multi-Modal Video Question Answering (2020, background): “Sundry efforts that have made progress towards binding the two fields include [8, 5, 22, 32] in visual grounding, [31, 30, 36, 25] in image/video captioning, [10, 2, 35, 37] in video moment retrieval, and”
Auto-Encoding Scene Graphs for Image Captioning (2018, background): “Thus, the scene graph can provide a beneficial prior for other vision tasks like VQA [40, 43],image generation [13, 59], and visual grounding [28].”
TransVG: End-to-End Visual Grounding with Transformers (2021, methodology): “Among two-stage methods, modular attention network [59], various graphs [48, 52, 53], and multi-modal tree [28] are designed to better model the multi-modal relationships.”
ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language (2019, background): “Another line of research investigates how to improve visual grounding using more advanced methods such as incorporating syntax [33, 15], using graph attention networks [58, 62], speaker-listener models [35, 66], as well as weakly supervised methods [59, 68, 9] and zero-shot settings for unseen…”
More Grounded Image Captioning by Distilling Image-Text Matching Model (2020, background): “Many works [39, 32, 5, 17, 62, 30] belong to the first category which requires expensive ground truth annotations.”
Improving One-stage Visual Grounding by Recursive Sub-query Construction (2020, methodology): “NMTREE [24] parses the query with a dependency tree parser [2] and links each tree node with a visual region.”