Bilinear Attention Networks

By Jin-Hwa Kim, Jaehyun Jun, Byoung-Tak Zhang. 2018

1,059 citations recorded by the source corpus.

Attention networks in multimodal learning provide an efficient way to utilize given visual information selectively. However, the computational cost to learn attention distributions for every pair of multimodal input channels is prohibitively expensive. To solve this problem, co-attention builds two separate attention distributions for each modality neglecting the interaction between multimodal inputs. In this paper, we propose bilinear attention networks (BAN) that find bilinear attention distributions to utilize given vision-language information seamlessly. BAN considers bilinear interactions among two groups of input channels, while low-rank bilinear pooling extracts the joint representations for each pair of channels. Furthermore, we propose a variant of multimodal residual networks to exploit eight-attention maps of the BAN efficiently. We quantitatively and qualitatively evaluate our model on visual question answering (VQA 2.0) and Flickr30k Entities datasets, showing that BAN significantly outperforms previous methods and achieves new state-of-the-arts on both datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “, 2017) and BAN (Kim et al., 2018) proposed advanced multimodal fusion methods for Visual Question Answering (VQA) (Antol et al.”
VisualBERT: A Simple and Performant Baseline for Vision and Language (2019, methodology): “Since multiple boxes could be aligned to the same phrase, we use the same cross entropy loss used in our experiment on VQA, different from the binary cross entropy loss used in BAN.”
LXMERT: Learning Cross-Modality Encoder Representations from Transformers (2019, methodology): “VQA The SotA result is BAN+Counter in Kim et al. (2018), which achieves the best accuracy among other recent works: MFH (Yu et al., 2018), Pythia (Jiang et al., 2018), DFAF (Gao et al., 2019a), and Cycle-Consistency (Shah et al., 2019).”
Towards VQA Models That Can Read (2019, background): “3 [40] and BAN [25], which are stateof-the-art on VQA 2.”
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (2022, methodology): “During inference on VQA, we use the decoder to rank the 3,128 candidate answers (Li et al., 2021a; Kim et al., 2018).”
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (2021, methodology): “For a fair comparison with existing methods, we constrain the decoder to only generate from the 3,192 candidate answers [55] during inference.”
Unifying Vision-and-Language Tasks via Text Generation (2021, background): “, 2020) have also shown to perform better than previous nonpretraining approaches (Yu et al., 2018a; Anderson et al., 2018; Kim et al., 2018; Yu et al., 2018b) in a wide range of discriminative (Goyal et al.”
Unified Vision-Language Pre-Training for Image Captioning and VQA (2019, background): “…proposal (Antol et al. 2015), there has been a signiﬁ-cant amount of works proposing model architectures to fuse question and image representations (Kim, Jun, and Zhang 2018; Anderson et al. 2018; Gao et al. 2019), new datasets or models to reduce the dataset bias (Zhang et al. 2016; Goyal et al.…”
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, methodology): “BAN [20]: Bilinear Attention Networks for VQA.”
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “In Table 2, we compare LLaMA-Adapter with popular VQA methods [2, 9, 17, 18, 22, 23, 28, 53] and language models [5, 16, 59].”
Multimodal Chain-of-Thought Reasoning in Language Models (2023, neutral): “Speciﬁcally, among the 8 question classes, Mutimodal-CoT Large achieves a 21.37% (67.43% → 88.80”
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (2022, neutral): “Our experiments show that current multimodal methods [54, 1, 20, 8, 25, 34] fail to achieve satisfactory performance on S CIENCE QA and do not generate correct explanations.”
Grounded Language-Image Pre-training (2021, background): “Such a setting is often denoted as linear probing [23], prompt tuning [66], or efficient task adapters [12].”
Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers (2020, methodology): “…introduction of Visual Genome Dataset [16] and the proposal of Bottom-Up and Top-Down Attention model [2], most recent researches related to vision and language utilize region-based visual features extracted from object detection models (e.g., Faster R-CNN [24]) for better performance [6,15,22,33].”
Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, methodology): “This includes simple linear models using ranking loss [16, 36], bi-linear pooling meth-ods [45, 20, 55, 17, 37], attention-based methods [1, 52, 64] and reasoning based approaches [54, 27, 33, 38, 29] on CLEVER and VQA-v1.”
Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning (2021, methodology): “We follow [19] to treat is as a 3, 192-way classification problem, and optimize the model via binary cross-entropy loss.”
In Defense of Grid Features for Visual Question Answering (2020, background): “02s per image – 40+ times faster with slightly better accuracy! In fact, extracting region features is so time-consuming that most state-of-the-art models [19, 49] are directly trained and evaluated on cached visual features.”
REMIND Your Neural Network to Prevent Catastrophic Forgetting (2019, background): “, attention-based [6,46,78], compositional [7,28], bi-modal fusion [10,19,71]) and it can be applied to similar tasks like image captioning [12] and referring expression recognition [43,63,67].”