Attention Bottlenecks for Multimodal Fusion

By Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, Chen Sun. 2021

825 citations recorded by the source corpus.

Humans perceive the world by concurrently processing and fusing high-dimensional inputs from multiple modalities such as vision and audio. Machine perception models, in stark contrast, are typically modality-specific and optimised for unimodal benchmarks, and hence late-stage fusion of final representations or predictions from each modality (`late-fusion') is still a dominant paradigm for multimodal video classification. Instead, we introduce a novel transformer based architecture that uses `fusion bottlenecks' for modality fusion at multiple layers. Compared to traditional pairwise self-attention, our model forces information between different modalities to pass through a small number of bottleneck latents, requiring the model to collate and condense the most relevant information in each modality and only share what is necessary. We find that such a strategy improves fusion performance, at the same time reducing computational cost. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple audio-visual classification benchmarks including Audioset, Epic-Kitchens and VGGSound. All code and models will be released.

Subfields: Multimodal Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

In Model Architectures

Citations in the corpus

Multiview Transformers for Video Recognition (2022, methodology): “We also note that “bottleneck” tokens have also been used by [31, 49].”
CoTracker: It is Better to Track Together (2023, background): “However, this approximation is crude as points are often strongly correlated ( e.g ., when they belong to the same object).”
Self-supervised learning in medicine and healthcare (2022, methodology): “Notably, a transformer-based design was used to build models that can take arbitrary input data to learn representations of the data 37–39 .”
Masked Autoencoders that Listen (2022, methodology): “MBT [11] uses ImageNet-21K pre-trained ViT; AST [10] and PaSST [28] employ DeiT [14] as the Transformer backbone.”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, methodology): “U NIFIED -IO 2 also achieves better performance on Kinetics-Sounds [7] than MBT [137], which is trained Table 5.”
Multimodal Learning With Transformers: A Survey (2022, background): “[175] propose Fusion via Attention Bottlenecks (FSN, fusion bottle-”
SCENIC: A JAX Library for Computer Vision Research and Beyond (2021, methodology): “On top of that, S CENIC has been used for numerous Google projects and research papers such as ViViT [6], OmniNet [16], Token-Learner [14], MBT [12], studies on scaling behaviour of various models [5, 17], and others.”
Multimodal biomedical AI (2022, background): “…previous time points to predict future actions, only previous words to predict the next or only the image to predict text); the network then learns to handle multiple modalities and tasks. to share the most relevant information across modalities and hence improving computational performance 136 .”
Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer (2021, background): “It exploits the image-audio alignment inherent in videos and learns audio and image / video representations jointly (Korbar et al., 2018; Wang et al., 2021; Nagrani et al., 2021).”
Pre-Trained Language Models for Text Generation: A Survey (2022, background): “Nagrani et al. [170] proposed a multimodal bottleneck transformer (MBT) to fuse audio and vision in videos, in which a tight fusion bottlenecks is used to collect and condense the most relevant inputs in each modality.”
ImageBind One Embedding Space to Bind Them All (2023, methodology): “, AudioCLIP [27] uses (audio, text) supervision and AVFIC [52] uses automatically mined (audio, text) pairs.”
BEATs: Audio Pre-Training with Acoustic Tokenizers (2022, methodology): “, AST [Gong et al., 2021a], PaSST [Koutini et al., 2021], MBT [Nagrani et al., 2021] and HTS-AT [Chen et al., 2022a] employ Transformer-based architectures [Vaswani et al., 2017] as the backbone, such as ViT [Dosovitskiy et al., 2021] and Swin Transformer [Liu et al., 2021], and obtain superior…”
A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks (2023, background): “MBT has shown promising results in various multimodal tasks, such as audio-visual speech recognition and audio-visual event detection (Nagrani et al., 2021).”
MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022, methodology): “It can either involve training separate encoders or one unified architecture (e.g., a Transformer [85]) to operate on modalities such as images and text [4, 14, 18, 39, 42–45, 56, 57, 76, 79, 94], video and audio [5,41,60,62], video, text and audio [3], and depth, images and video [32].”
Omnivore: A Single Model for Many Visual Modalities (2022, methodology): “Recently, Transformers have been used to fuse multiple modalities: Transformers have been used to fuse features in vision-and-language tasks [2, 17, 37, 40, 49, 56, 83, 86] and video-and-audio tasks [64], video-and-image tasks [7], and even tasks that involve video, audio, and text [1].”