Large-Scale Adversarial Training for Vision-and-Language Representation Learning

By Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, Jingjing Liu. 2020

559 citations recorded by the source corpus.

We present VILLA, the first known effort on large-scale adversarial training for vision-and-language (V+L) representation learning. VILLA consists of two training stages: (i) task-agnostic adversarial pre-training; followed by (ii) task-specific adversarial finetuning. Instead of adding adversarial perturbations on image pixels and textual tokens, we propose to perform adversarial training in the embedding space of each modality. To enable large-scale training, we adopt the "free" adversarial training strategy, and combine it with KL-divergence-based regularization to promote higher invariance in the embedding space. We apply VILLA to current best-performing V+L models, and achieve new state of the art on a wide range of tasks, including Visual Question Answering, Visual Commonsense Reasoning, Image-Text Retrieval, Referring Expression Comprehension, Visual Entailment, and NLVR2.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “[39] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu.”
Scaling Up Vision-Language Pretraining for Image Captioning (2021, background): “Along the journey of VLP, researchers have investigated different training strategies [11,31], robustness [25], compression [9, 10, 42], probing analysis [3, 27], and the extension to video-text modeling [21, 24, 38, 39, 50].”
Achieving Human Parity on Visual Question Answering (2021, background): “• VILLA [21]: proposes the idea of large-scale adversarial training for vision-language representation learning, which adds embedding-based adversarial training on both stages of task-agnostic pre-training and task-specific fine-tuning.”
FILIP: Fine-grained Interactive Language-Image Pre-Training (2021, methodology): “We compare our FILIP model against methods with complex attention layers including Unicoder-VL (Li et al., 2020a), ImageBERT (Qi et al., 2020), UNITER (Chen et al., 2020), VILLA (Gan et al., 2020), ERNIE-ViL (Yu et al., 2021), Oscar (Li et al., 2020b), VinVL (Zhang et al., 2021), ALBEF (Li et al., 2”
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts (2021, methodology): “The second category models the interaction of images and text using a deep fusion encoder with cross-modal attention [43, 30, 41, 24, 51, 3, 26, 25, 14, 49, 16, 17, 20, 23, 46].”
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision (2021, methodology): “…detection for image region feature regression or tagging as part of the pre-training objectives (Tan & Bansal, 2019; Su et al., 2020; Li et al., 2019; Chen et al., 2020b; Gan et al., 2020; Li et al., 2020; Yu et al., 2021; Li et al., 2021; Zhang et al., 2021; Hu et al., 2021; Cho et al., 2021).”
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (2021, methodology): “The first category focuses on modelling the interactions between image and text features with transformerbased multimodal encoders [10, 11, 12, 13, 1, 14, 15, 2, 3, 16, 8, 17, 18].”
MERLOT: Multimodal Neural Script Knowledge Models (2021, background): “Previous works [22, 36, 119] conduct what they call ‘second-stage pretraining’ with VCR training data.”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, methodology): “, 2020), VILLA (Gan et al., 2020) and Oscar (Li et al.”
Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solution (2020, methodology): “s not commonly see and accounted for in the general visual-linguistic model. 3 Approach The solution comprises four dierent VL transformer architecture ensemble, namely: VL-BERT[13], UNITER[3], VILLA[6], and ERNIE-Vil[15]. All the models besides ERNIE-Vil were modied for better performance of hatefu”