Supervised Multimodal Bitransformers for Classifying Images and Text

By Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Davide Testuggine. 2019

320 citations recorded by the source corpus.

Self-supervised bidirectional transformer models such as BERT have led to dramatic improvements in a wide variety of textual classification tasks. The modern digital world is increasingly multimodal, however, and textual information is often accompanied by other modalities such as images. We introduce a supervised multimodal bitransformer model that fuses information from text and image encoders, and obtain state-of-the-art performance on various multimodal classification benchmark tasks, outperforming strong baselines, including on hard test sets specifically designed to measure multimodal performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “MMBT (Kiela et al., 2019)”
A Multimodal Framework for the Detection of Hateful Memes (2020, methodology): “Early-fusion methods such as MMBT [14], VisualBERT [19], and ViLBERT [23], on the other hand, employ more complex approaches to process the two modalities jointly within the model architecture.”
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, neutral): “These can be compared to more sophisticated multimodal methods, such as supervised multimodal bitransformers [40] using either Image-Grid or Image-Region features (MMBT-Grid and MMBT-Region), and versions of ViLBERT [50] and Visual BERT [46] that were only unimodally pretrained and not pretrained on”
HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “MMBT (Kiela et al., 2019)”
Multimodal Few-Shot Learning with Frozen Language Models (2021, background): “Learning to embed image representations into the ‘word’ space of a large pretrained language model was done previously by [16].”
Trusted Multi-View Classification With Dynamic Evidential Fusion (2022, methodology): “Following existing work [49], there are 60,101 samples used as the training set, 5,000 samples used as the validation set, and the remaining 21,695 samples as the test set.”
Multi-Modal Open-Domain Dialogue (2020, methodology): “In a similar fashion to VisualBERT (Li et al., 2019a) and multi-modal Bi-transformers (Kiela et al., 2019), we concatenate the projected image encoding from the visual input with the token embeddings from the textual input, assign each a different segment embedding, and jointly encode the text and i”
Transformers: State-of-the-Art Natural Language Processing (2020, unknown): “Datasets Language Modeling x1:n−1 xn ∈ V Generation WikiText-103 Sequence Classification x1:N y ∈ C Classification, Sentiment Analysis GLUE, SST, MNLI Question Answering x1:M ,xM :N y span [1 : N ] QA, Reading Comprehension SQuAD, Natural Questions Token Classification x1:N y1:N ∈ CN NER, Tagging On”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, unknown): “This might involve techniques such as concatenation [65] and attention mechanisms [13] or more sophisticated methods such as cross-modal attention [66,67] and tensor fusion [68].”
What BERT Sees: Cross-Modal Transfer for Visual Question Generation (2020, methodology): “, 2017) – with the notable exception of Kiela et al. (2019) who used pooling layers from a CNN.”
Multimodal Conditionality for Natural Language Generation (2021, methodology): “We drew inspiration from Kiela et al. (2019) which shows that self-supervised unimodal transformer models are capable of learning context between different modalities through supervised learning for classiﬁcation tasks.”
Are we pretraining it right? Digging deeper into visio-linguistic pretraining (2020, background): “From the recent plethora of visio-linguistic pretraining approaches [2,12,20,25,26,30,47], two major categories of architectures have emerged.”
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis (2020, background): “Such cross-modality matching has been highly prominent in recent cross-modal learning approaches [22, 23, 27, 49, 57].”
Preserving integrity in online social networks (2020, methodology): “The MultiModal bitransformer [55] uses pre-trained unimodal representations (BERT and ResNet-152) and then fine tunes them together for the task at hand.”