Modeling Relationships in Referential Expressions with Compositional Modular Networks

By Ronghang Hu, Marcus Rohrbach, Jacob Andreas, Trevor Darrell, Kate Saenko. 2016

428 citations recorded by the source corpus.

People often refer to entities in an image in terms of their relationships with other entities. For example, the black cat sitting under the table refers to both a black cat entity and its relationship with another table entity. Understanding these relationships is essential for interpreting and grounding such natural language expressions. Most prior work focuses on either grounding entire referential expressions holistically to one region, or localizing relationships based on a fixed set of categories. In this paper we instead present a modular deep architecture capable of analyzing referential expressions into their component parts, identifying entities and relationships mentioned in the input expression and grounding them all in the scene. We call this approach Compositional Modular Networks (CMNs): a novel architecture that learns linguistic analysis and visual inference end-to-end. Our approach is built around two types of neural modules that inspect local regions and pairwise interactions between regions. We evaluate CMNs on multiple referential expression datasets, outperforming state-of-the-art approaches on all tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, background): “Vision and language : While substantial work exists on vision-language representation learning e.g., MAttNet (Yu et al., 2018b), CMN (Hu et al., 2017), VQA (Antol et al., 2015), CLEVR (Johnson et al., 2017), ViLBERT (Lu et al., 2019), they lack embodied or sequential decision making.”
From Recognition to Cognition: Visual Commonsense Reasoning (2018, background): “An orthogonal line of work has been on referring expressions: asking to what image region a natural language sentence refers to [60, 52, 65, 87, 88, 59, 36, 33].”
Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, background): “Furthermore, pairing different objects in a scene [13, 28, 60, 78] is also common.”
Learning to Reason: End-to-End Module Networks for Visual Question Answering (2017, methodology): “In contrast, our model obtains the textual input using soft attention over question words similar to [12].”
Inferring and Executing Programs for Visual Reasoning (2017, neutral): “Module networks are an example of reasoningaugmented models that use a syntactic parse of a question to determine the architecture of the network [1, 2, 17].”
MAttNet: Modular Attention Network for Referring Expression Comprehension (2018, methodology): “For example, expressions like “left dog” and “man in red” are hard to model using [9].”
Neural Baby Talk (2018, background): “Another related line of work is on resolving referring expressions [21] (or description-based object retrieval [36, 17, 18, 39] – given a description of an object in the image, identify which object is being referred to) or referring expression generation [21, 29, 31, 50] (given an object in the ima”
Speaker-Follower Models for Vision-and-Language Navigation (2018, background): “Existing work in visual grounding [38, 30, 25, 40, 35] has addressed the problem of passively perceiving a static image and mapping a referential expression to a bounding box [38, 30, 25] or a segmentation mask [24, 27, 55], exploring various techniques including proposal generation [10] and relatio”
Grounding Visual Explanations (2018, methodology): “Our aim here is to compare our phrase-critic model with the baseline visual explanation model [8] and the grounding model [10].”
Are we pretraining it right? Digging deeper into visio-linguistic pretraining (2020, methodology): “Visio-Linguistic Pretraining (V&L) Contrary to earlier works in vision and language [15, 19, 24, 32, 33, 56, 57] which designed specialized architectures for different tasks, many recently introduced models for V&L [2,12,25,26,30,31,47,48,59] provide a common architecture that can be pretrained…”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “Approaches such as compositional modular networks (Hu et al., 2017b) analyzed referential expressions by identifying entities and relationships mentioned in the input expression and grounding them all in the scene.”
Improving One-stage Visual Grounding by Recursive Sub-query Construction (2020, neutral): “[29,30,14,50,48,27,52,51,42,46,40,32].”
Bayesian Relational Memory for Semantic Visual Navigation (2019, background): “Similar ideas of using semantic knowledge to enhance reasoning have been applied to image classification [34, 56], segmentation [52, 64], situation recognition [33], visual question answering [57, 9, 53, 28, 26], image retrieval [29] and relation detection [62].”
Reward Learning from Narrated Demonstrations (2018, methodology): “Our model is a variation of the referential expression detector of [25].”
FindIt: Generalized Localization with Natural Language Queries (2022, methodology): “One-stage [7,44,89,13,55] and two-stage [94,91,27,83,87,50] methods have been popular for these tasks.”
More Grounded Image Captioning by Distilling Image-Text Matching Model (2020, background): “Many works [39, 32, 5, 17, 62, 30] belong to the first category which requires expensive ground truth annotations.”
Referring Transformer: A One-step Approach to Multi-task Visual Grounding (2021, methodology): “Models Visual Pretrain Multi- RefCOCO RefCOCO+ RefCOCOg Features Images task val testA testB val testA testB val-u test-u Two-stage: CMN [19] VGG16 None × - 71.”
A Real-Time Cross-Modality Correlation Filtering Method for Referring Expression Comprehension (2019, background): “Most of them [20, 9, 35, 32, 27, 28] mainly focus on exploring how to mine context information from the language and image or model the relationship between referents, for example, MAttNet [32] proposed a modular attention model to capture multi-modality context information.”