Vokenization: Improving Language Understanding via Contextualized, Visually-Grounded Supervision

By Hao Hao Tan, Mohit Bansal. 2020

133 citations recorded by the source corpus.

Humans learn language by listening, speaking, writing, reading, and also, via interaction with the multimodal real world. Existing language pre-training frameworks show the effectiveness of text-only self-supervision while we explore the idea of a visually-supervised language model in this paper. We find that the main reason hindering this exploration is the large divergence in magnitude and distributions between the visually-grounded language datasets and pure-language corpora. Therefore, we develop a technique named "vokenization" that extrapolates multimodal alignments to language-only data by contextually mapping language tokens to their related images (which we call "vokens"). The "vokenizer" is trained on relatively small image captioning datasets and we then apply it to generate vokens for large language corpora. Trained with these contextually generated vokens, our visually-supervised language models show consistent improvements over self-supervised alternatives on multiple pure-language tasks such as GLUE, SQuAD, and SWAG. Code and pre-trained models publicly available at this https URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Universal Multimodal Representation for Language Understanding (2023, methodology): “We collect a list of tokens that have more than ten occurrences in the Multi30K training set after removing all stop words following [72].”
A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, unknown): “[183] proposed contextual mapping of language”
Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions (2021, background): “In general-domain corpora like Wikipedia, grounded words take up a decent portion ( > 25 % ) (Tan and Bansal, 2020).”
Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation (2021, methodology): “Speciﬁcally, we follow Tan and Bansal (2020) to collect a list of visually grounded tokens .”
UniT: Multimodal Multitask Learning with a Unified Transformer (2021, methodology): “Prior works such as VirTex [13], Voken [56] and VisualBERT [31] show that pretraining on multimodal data such as image captions helps downstream vision, language, or multimodal tasks, which is often accomplished by building specialized models through ﬁne-tuning on each downstream task.”
Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers (2021, background): “Transformers have also been used in a variety of vision image tasks, such as image representation learning [12, 82, 18, 59], image generation [51], object detection [48, 10], video question-answering [35], few-shot learning [19], and image–text [49, 63, 67, 44, 68], video-text [65, 64, 89, 25, 53, 2”
Probing Contextual Language Models for Common Ground with Visual Representations (2020, background): “The majority of examined models are contextual representation models based on the transformer architecture (Vaswani et al., 2017) trained on text-only data.”
Shared computational principles for language processing in humans and deep language models (2022, background): “Furthermore, as opposed to autoregressive DLMs, it is likely that the brain relies on additional simple objectives at different timescales to facilitate learning 20,67 .”
Text Data Augmentation for Deep Learning (2021, methodology): “For the sake of Data Augmentation, a notable example is Vokenization from Tan and Bansal [96].”
Leveraging Visual Knowledge in Language Tasks: An Empirical Study on Intermediate Pre-training for Cross-Modal Knowledge Transfer (2022, background): “, 2022), and they are found to be not effective in improving language tasks (Tan and Bansal, 2020).”