Data Efficient Masked Language Modeling for Vision and Language

By Yonatan Bitton, Gabriel Stanovsky, Michael Elhadad, Roy Schwartz. 2021

22 citations recorded by the source corpus.

Masked language modeling (MLM) is one of the key sub-tasks in vision-language pretraining. In the cross-modal setting, tokens in the sentence are masked at random, and the model predicts the masked tokens given the image and the text. In this paper, we observe several key disadvantages of MLM in this setting. First, as captions tend to be short, in a third of the sentences no token is sampled. Second, the majority of masked tokens are stop-words and punctuation, leading to under-utilization of the image. We investigate a range of alternative masking strategies specific to the cross-modal setting that address these shortcomings, aiming for better fusion of text and image in the learned representation. When pre-training the LXMERT model, our alternative masking strategies consistently improve over the original masking strategy on three downstream tasks, especially in low resource settings. Further, our pre-training approach substantially outperforms the baseline model on a prompt-based probing task designed to elicit image objects. These results and our analysis indicate that our method allows for better utilization of the training data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “For example, when using the aforementioned mask LM objective, models often ignore visual input completely in favor of texttext interactions [13]; this issue is magnified when training on videos with ASR, as spoken language often contains redundant words [126].”
Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing (2023, methodology): “Inspired by the ∆ image loss of [8], we define ∆ img as the change in loss by conditioning the estimation with a prior image for a given token w as follows:”
A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervision (2021, unknown): “[2] improves the masked-language-modeling component of pre-training.”
A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond (2022, background): “[100] also investigates the masking strategies of text data in language-vision tasks,”