Modeling Context Between Objects for Referring Expression Understanding

By Varun K. Nagaraja, Vlad I. Morariu, Larry S. Davis. 2016

569 citations recorded by the source corpus.

Referring expressions usually describe an object using properties of the object and relationships of the object with other objects. We propose a technique that integrates context between objects to understand referring expressions. Our approach uses an LSTM to learn the probability of a referring expression, with input features from a region and a context region. The context regions are discovered using multiple-instance learning (MIL) since annotations for context objects are generally not available for training. We utilize max-margin based MIL objective functions for training the LSTM. Experiments on the Google RefExp and UNC RefExp datasets show that modeling context between objects provides better performance than modeling only object properties. We also qualitatively show that our technique can ground a referring expression to its referred region along with the supporting context region.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Learning from Simulated and Unsupervised Images through Adversarial Training (2016, neutral): “Large labeled training datasets are becoming increasingly important with the recent rise in high capacity deep neural networks [4, 20, 48, 48, 1, 24, 17].”
Generalized Decoding for Pixel, Image, and Language (2022, neutral): “Since the emergence of vision trans-formers, works like LAVT [81] enhance the cross-modal interactions from the very beginning, which led to SoTA on RefCOCO [86], RefCOCO+ [86] and G-Ref [52, 55].”
TVQA: Localized, Compositional Video Question Answering (2018, neutral): “, 2016) and referring expression comprehension (Hu et al., 2016; Yu et al., 2016; Nagaraja et al., 2016; Yu et al., 2017, 2018b).”
Speaker-Follower Models for Vision-and-Language Navigation (2018, neutral): “Existing work in visual grounding [38, 30, 25, 40, 35] has addressed the problem of passively perceiving a static image and mapping a referential expression to a bounding box [38, 30, 25] or a segmentation mask [24, 27, 55], exploring various techniques including proposal generation [10] and relatio”
Modeling Relationships in Referential Expressions with Compositional Modular Networks (2016, neutral): “The problem of grounding referential expressions can be naturally formulated as a retrieval problem over image regions [19, 10, 24, 7, 32, 20].”
Self-Monitoring Navigation Agent via Auxiliary Progress Estimation (2019, neutral): “…of vision tasks like image captioning (Hu et al., 2016; Rohrbach et al., 2016; Lu et al., 2018), visual question answering (Lu et al., 2016b; Agrawal et al., 2018), relationship detection (Lu et al., 2016a; Ma et al., 2018) and referral expression (Nagaraja et al., 2016; Gavrilyuk et al., 2018).”
Finding "It": Weakly-Supervised Reference-Aware Visual Grounding in Instructional Videos (2018, unknown): “Recent works also aim to ground expressions in phrases beyond object categories [15, 20, 32, 34, 39, 43, 49, 54].”
Recursive Visual Attention in Visual Dialog (2018, unknown): “Thanks to the rapid development of deep neural networks and the high quality of large-scale real-world datasets, researchers have achieved inspiring progress in a range of vision-language tasks, including image captioning [32, 8, 34, 3], referring expression grounding [22, 23, 35], and visual questi”
Temporally Grounding Natural Sentence in Video (2018, unknown): “More recent studies (Hu et al., 2017; Nagaraja et al., 2016) improve the performance with the aid of modeling relationships between objects.”
URVOS: Unified Referring Video Object Segmentation Network with a Large-Scale Benchmark (2020, unknown): “For the fair comparison with [10], we pre-trained our model on a large-scale referring image segmentation benchmark [20]; our method turns out to be better than [10] under the same pretraining environment.”
Unleashing Text-to-Image Diffusion Models for Visual Perception (2023, unknown): “We evaluate our VPD on the widely used RefCOCO [58], RefCOCO+ [58], and G-Ref [35].”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, unknown): “, 2016) VGG LSTM MMI SGD 7 (Nagaraja et al., 2016) VGG LSTM Neg.”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, background): “Vision-language results on nine tasks [1, 28, 59, 91, 124, 129, 130, 136, 209] and three evaluation-only benchmarks [106, 113, 120].”
LAVT: Language-Aware Vision Transformer for Referring Image Segmentation (2021, unknown): “Our LAVT achieves 72.73%, 62.14%, 61.24%, and 60.50% overall IoU on the validation sets of RefCOCO [64], RefCOCO+ [64], G-Ref (UMD partition) [44], and G-Ref (Google partition) [42], improving the state of the art for these datasets by absolute margins of 7.08%, 6.64%, 6.84%, and 8.57%,…”
Recurrent Multimodal Interaction for Referring Image Segmentation (2017, unknown): “Although the dataset has primarily been used for referring object detection [26, 37, 29], where the goal is to return a bounding box of the referred object, it is also suitable for referring image segmentation, since the original MS COCO annotation contains segmentation masks.”
Vision-Language Transformer and Query Generation for Referring Segmentation (2021, unknown): “The maximum length for the input language expression is set to 15 for RefCOCO and RefCOCO+ and 20 for G-Ref.”