Words Aren’t Enough, Their Order Matters: On the Robustness of Grounding Visual Referring Expressions

By Arjun Reddy Akula, Spandana Gella, Yaser Al-Onaizan, Song-Chun Zhu, Siva Reddy. 2020

62 citations recorded by the source corpus.

Visual referring expression recognition is a challenging task that requires natural language understanding in the context of an image. We critically examine RefCOCOg, a standard benchmark for this task, using a human study and show that 83.7% of test instances do not require reasoning on linguistic structure, i.e., words are enough to identify the target object, the word order doesn’t matter. To measure the true progress of existing models, we split the test set into two sets, one which requires reasoning on linguistic structure and the other which doesn’t. Additionally, we create an out-of-distribution dataset Ref-Adv by asking crowdworkers to perturb in-domain examples such that the target object changes. Using these datasets, we empirically show that existing methods fail to exploit linguistic structure and are 12% to 23% lower in performance than the established progress for this task. We also propose two methods, one based on contrastive learning and the other based on multi-task learning, to increase the robustness of ViLBERT, the current state-of-the-art model for this task. Our datasets are publicly available at https://github.com/aws/aws-refcocog-adv.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers (2021, background): “At the same time, datasets do not always capture what they are designed to measure; e.g. , Akula et al. (2020) show that models can do well on a referring expression task while ignoring the linguistic structure.”
An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models (2020, background): “A concurrent work (Akula et al., 2020) shows that MTL improves robustness on advesarial examples in visual grounding.”
Visual Spatial Reasoning (2022, background): “In particular, spatial reasoning has been found to be particularly challenging for current models, and much more challenging than capturing properties of individual entities (Kuhnle et al., 2018; Cirik et al., 2018; Akula et al., 2020), even for state-of-theart models such as CLIP (Radford et al.”
Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality (2022, background): “…[22], distracting models with visual similarity between images [7, 33], distracting models with textual similarity between many suitable captions [1,17], collecting more diverse image-caption pairs beyond the predominately English and North American/Western European datasets [50], probing for an…”