Visual Referring Expression Recognition: What Do Systems Actually Learn?

By Volkan Cirik, Louis-philippe Morency, Taylor Berg-Kirkpatrick. 2018

67 citations recorded by the source corpus.

We present an empirical analysis of state-of-the-art systems for referring expression recognition – the task of identifying the object in an image referred to by a natural language expression – with the goal of gaining insight into how these systems reason about language and vision. Surprisingly, we find strong evidence that even sophisticated and linguistically-motivated models for this task may ignore linguistic structure, instead relying on shallow correlations introduced by unintended biases in the data selection and annotation process. For example, we show that a system trained and tested on the input image without the input referring expression can achieve a precision of 71.2% in top-2 predictions. Furthermore, a system that predicts only the object category given the input can achieve a precision of 84.2% in top-2 predictions. These surprisingly positive results for what should be deficient prediction scenarios suggest that careful analysis of what our models are learning – and further, how our data is constructed – is critical as we seek to make substantive progress on grounded language tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Connecting Vision and Language with Localized Narratives (2019, background): “In Out - - Image captioning [54, 58, 55] Out In - - Text-to-image Generation [45, 50, 59] In Out - Out Dense image captioning [23, 60, 27], Dense Relational Captioning [27] In Out - In Controllable and Grounded Captioning [11] In In - Out Phrase grounding [14] In In + Out - - Visual Question Answeri”
Words Aren’t Enough, Their Order Matters: On the Robustness of Grounding Visual Referring Expressions (2020, methodology): “Similar to (Cirik et al., 2018b), we distort the test split using two methods: (a) randomly shufﬂe words in a referring expression, and (b) delete all the words in the expression except for nouns and adjectives.”
Visual Spatial Reasoning (2022, background): “In particular, spatial reasoning has been found to be particularly challenging for current models, and much more challenging than capturing properties of individual entities (Kuhnle et al., 2018; Cirik et al., 2018; Akula et al., 2020), even for state-of-theart models such as CLIP (Radford et al.”
Challenges and Prospects in Vision and Language Research (2019, background): “Current benchmarks cannot reliably ascertain whether an algorithm has learned to represent objects and their attributes properly, and it is often easy to produce a correct response by “guessing” prominent objects in the scene (Cirik et al., 2018).”
Shifting the Baseline: Single Modality Performance on Visual Navigation & QA (2018, background): “This work extends observations made in both the Computer Vision (Goyal et al., 2018; Cirik et al., 2018) and Natural Language (Mudrakarta et al.”
Few-shot Object Grounding and Mapping for Natural Language Robot Instruction Following (2020, background): “Our language-conditioned segmentation problem is a variant of referring expression recognition [31, 32, 33, 34, 35].”
Vision-Language Models for Vision Tasks: A Survey (2023, background): “Several relevant surveys have been conducted which focus on VLMs on various vision-language tasks instead, such as visual question answering [57], natural language for visual reasoning [58] and visual referring expression [59], and phrase grounding [60].”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “A few approaches (Dai et al., 2017; Shetty et al., 2017) have leveraged adversarial training, while Vijayakumar et al. (2016) used diverse beam search to decode diverse image captions in English.”