ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph

By Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, Haifeng Wang. 2020

419 citations recorded by the source corpus.

We propose a knowledge-enhanced approach, ERNIE-ViL, which incorporates structured knowledge obtained from scene graphs to learn joint representations of vision-language. ERNIE-ViL tries to build the detailed semantic connections (objects, attributes of objects and relationships between objects) across vision and language, which are essential to vision-language cross-modal tasks. Utilizing scene graphs of visual scenes, ERNIE-ViL constructs Scene Graph Prediction tasks, i.e., Object Prediction, Attribute Prediction and Relationship Prediction tasks in the pre-training phase. Specifically, these prediction tasks are implemented by predicting nodes of different types in the scene graph parsed from the sentence. Thus, ERNIE-ViL can learn the joint representations characterizing the alignments of the detailed semantics across vision and language. After pre-training on large scale image-text aligned datasets, we validate the effectiveness of ERNIE-ViL on 5 cross-modal downstream tasks. ERNIE-ViL achieves state-of-the-art performances on all these tasks and ranks the first place on the VCR leaderboard with an absolute improvement of 3.7%.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

A Multimodal Framework for the Detection of Hateful Memes (2020, background): “Recent developments in the ﬁeld, such as ERNIE-ViL [37], a multimodal model that incorporates structured knowledge from scene graphs [13] in its pretraining tasks, is another interesting avenue for future work.”
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation (2022, neutral): “Unlike previous works (An-339 derson et al., 2018; Chen et al., 2020; Li et al.,340 2020; Yu et al., 2021; Zhang et al., 2021; Kim341 et al., 2021b; Li et al., 2021a) that treat the VQA342 task as a discriminative problem, we let the model343 generate answers freely, which is more aligned344 with…”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “Ca pt io n/ O bj D et -b as ed ERNIE-ViL-Large [123] 79.”
Scaling Up Vision-Language Pretraining for Image Captioning (2021, methodology): “Since the birth of ViLBERT [30] and LXMERT [40], we have witnessed a boom of methods for vision-language pre-training [6,7,13,22,26, 37, 45, 46].”
FILIP: Fine-grained Interactive Language-Image Pre-Training (2021, methodology): “We compare our FILIP model against methods with complex attention layers including Unicoder-VL (Li et al., 2020a), ImageBERT (Qi et al., 2020), UNITER (Chen et al., 2020), VILLA (Gan et al., 2020), ERNIE-ViL (Yu et al., 2021), Oscar (Li et al., 2020b), VinVL (Zhang et al., 2021), ALBEF (Li et al., 2”
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision (2021, background): “…detection for image region feature regression or tagging as part of the pre-training objectives (Tan & Bansal, 2019; Su et al., 2020; Li et al., 2019; Chen et al., 2020b; Gan et al., 2020; Li et al., 2020; Yu et al., 2021; Li et al., 2021; Zhang et al., 2021; Hu et al., 2021; Cho et al., 2021).”
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (2021, background): “The first category focuses on modelling the interactions between image and text features with transformerbased multimodal encoders [10, 11, 12, 13, 1, 14, 15, 2, 3, 16, 8, 17, 18].”
MERLOT: Multimodal Neural Script Knowledge Models (2021, background): “Most recently, A variety of papers have proposed “Visual BERT” style models [95, 17, 7, 58, 63, 107, 30]; these approaches train on datasets of image captions such as MSCOCO [59] and Visual Genome [50].”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, methodology): “, 2020c), ERNIE-ViL (Yu et al., 2020), VILLA (Gan et al.”
Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solution (2020, methodology): “2) Lower lingual representation quality(VL-BERT has been pre-trained with text only MLM task, ERNIE-Vil have the independent text modality stream) let the entity tags that are represented in the text become less effective.”
RegionCLIP: Region-based Language-Image Pretraining (2021, background): “Beyond object labels, region representation learning benefits from from additional labels of object attributes [1, 29, 61], showing noticeable improvement on vision-language tasks [9, 31, 33, 51, 58, 63].”
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision (2021, methodology): “The pretrain-and-finetune scheme has been expanded to a joint domain of vision and language, giving birth to the category of Vision-and-Language Pretraining (VLP) models (Lu et al., 2019; Chen et al., 2019; Su et al., 2019; Li et al., 2019; Tan & Bansal, 2019; Li et al., 2020a; Lu et al., 2020; Qi e”
UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning (2020, positive): “We compare with most of the existed multi-modal pre-training models, including ViLBERT (Lu et al., 2019), VLP (Zhou et al., 2020), UNITER (Chen et al., 2020b), Os-car (Li et al., 2020), Villa (Gan et al., 2020) and ERNIE-ViL (Yu et al., 2020).”
M6: A Chinese Multimodal Pretrainer (2021, methodology): “ERNIE-ViL [50] enhances the model with the application of scene graph information.”
VinVL: Making Visual Representations Matter in Vision-Language Models (2021, methodology): “SoTAs: VQA is from ERNIE-VIL [43], GQA is from NSM [12], NoCaps is from VIVO [9], NLVR2 is from VILLA [7], the rest tasks are from OSCAR [21].”