What Does BERT with Vision Look At?

By Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, Kai-Wei Chang. 2020

167 citations recorded by the source corpus.

Pre-trained visually grounded language models such as ViLBERT, LXMERT, and UNITER have achieved significant performance improvement on vision-and-language tasks but what they learn during pre-training remains unclear. In this work, we demonstrate that certain attention heads of a visually grounded language model actively ground elements of language to image regions. Specifically, some heads can map entities to image regions, performing the task known as entity grounding. Some heads can even detect the syntactic relations between non-entity words and image regions, tracking, for example, associations between verbs and regions corresponding to their arguments. We denote this ability as syntactic grounding. We verify grounding both quantitatively and qualitatively, using Flickr30K Entities as a testbed.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, background): “Recent VLP analyses [11, 33] show that VLP [51, 12, 32] learns the latent alignments between the semantically corresponded region-word or region-region pairs.”
Scaling Up Vision-Language Pretraining for Image Captioning (2021, background): “Along the journey of VLP, researchers have investigated different training strategies [11,31], robustness [25], compression [9, 10, 42], probing analysis [3, 27], and the extension to video-text modeling [21, 24, 38, 39, 50].”
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “In Table 2, we compare LLaMA-Adapter with existing popular VQA methods [65, 33, 34] and large language models [27, 4, 74] on ScienceQA datset.”
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (2022, neutral): “Inspired by the recent remarkable performance achieved for VQA [33, 32, 10, 9, 26, 7, 8], in this paper, we further extensively benchmark SCIENCEQA with a wide range of attention-based [1, 33, 21, 9] and Transformer-based [30, 26, 27, 7] methods.”
Unified Pre-training for Program Understanding and Generation (2021, methodology): “Later, BERT’s task-independent pre-training approach is rigorously studied (Devlin et al., 2019; Liu et al., 2019; Solaiman et al., 2019; Feng et al., 2020; Sun et al., 2019; Li et al., 2020).”
Kaleido-BERT: Vision-Language Pre-training on Fashion Domain (2021, background): “Another notable study is the recent ACL work [41] in which the authors creatively demonstrate that the attention head of the VL model can perform entity grounding and syntactic grounding.”
A Survey of Deep Learning for Mathematical Reasoning (2022, unknown): “The graph-based information can also be embedded when encoding the input mathematical sequences (Zhang et al., 2020b; Shen and Jin, 2020; Li et al., 2020b; Wu et al., 2021a).”
Playing Lottery Tickets with Vision and Language (2021, background): “Probing analysis [3, 40] further reveals the secrets that lead to the success of these pre-trained models.”
DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Grounding (2022, methodology): “DETR (Carion et al. 2020) introduces an end-to-end Transformer (Vaswani et al. 2017)-based solution for the ﬁrst time.”
Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training (2022, methodology): “Moreover, the effects of using different types of image encoding are also unclear, including regionbased (Li et al., 2020b; Zhang et al., 2021; Hu et al., 2021), grid-based (Wang et al., 2021), and patch-based (Kim et al., 2021; Li et al., 2021a).”
Spatial-Temporal Transformer for Dynamic Scene Graph Generation (2021, methodology): “To further bridge the vision and language domains, different Bert-like large-scale pretrained models are also developed, like Caption-Based Image Retrieval and Visual Commonsense Reasoning (VCR) [43, 32, 50].”
COFAR: Commonsense and Factual Reasoning in Image Search (2022, methodology): “In VL models , we use modern text-to-image retrieval methods, namely VSE++ (Faghri et al., 2018), and competitive vision-and-language transformers such as Visual-BERT (Li et al., 2020), ViLBERT (Lu et al., 2019), and VinVL (Zhang et al., 2021).”
Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models (2023, unknown): “[27] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.”