Parameter Efficient Multimodal Transformers for Video Representation Learning

By Sangho Lee, Youngjae Yu, Gunhee Kim, Thomas Breuel, Jan Kautz, Yale Song. 2020

91 citations recorded by the source corpus.

The recent success of Transformers in the language domain has motivated adapting it to a multimodal setting, where a new visual model is trained in tandem with an already pretrained language model. However, due to the excessive memory requirements from Transformers, existing work typically fixes the language model and train only the vision module, which limits its ability to learn cross-modal information in an end-to-end manner. In this work, we focus on reducing the parameters of multimodal Transformers in the context of audio-visual video representation learning. We alleviate the high memory requirement by sharing the weights of Transformers across layers and modalities; we decompose the Transformer into modality-specific and modality-shared parts so that the model learns the dynamics of each modality both individually and together, and propose a novel parameter sharing scheme based on low-rank approximation. We show that our approach reduces parameters up to 80$\%$, allowing us to train our model end-to-end from scratch. We also propose a negative sampling approach based on an instance similarity measured on the CNN embedding space that our model learns with the Transformers. To demonstrate our approach, we pretrain our model on 30-second clips from Kinetics-700 and transfer it to audio-visual classification tasks.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Transformers in Vision: A Survey (2021, methodology): “The Transformer designs in this category can be grouped into single-stream (UNITER [35], OSCAR [36], VideoBERT [17], Unicoder-VL [132], VisualBERT [55] and VL-BERT [22]) and dual-stream architectures (LXMERT [21], ViLBERT [133] and PEMT [134]).”
Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers (2021, background): “Transformers have also been used in a variety of vision image tasks, such as image representation learning [12, 82, 18, 59], image generation [51], object detection [48, 10], video question-answering [35], few-shot learning [19], and image–text [49, 63, 67, 44, 68], video-text [65, 64, 89, 25, 53, 2”
Attention Bottlenecks for Multimodal Fusion (2021, methodology): “For many works, the inputs to trans-formers are the output representations of single modality CNNs [39, 23] – unlike these works we use transformer blocks throughout, using only a single convolutional layer to rasterise 2D patches.”
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction (2022, unknown): “…focuses on learning self-supervised audio-visual representations of general videos to solve high-level semantic tasks, e.g., action recognition and audio event detection (Arandjelovi ´ c & Zisserman, 2017; Bruno et al., 2018; Morgado et al., 2021; Chen et al., 2020; Lee et al., 2021).”
Skeleton-based action recognition via spatial and temporal transformer networks (2020, background): “Other applications of Transformers in segmentation tasks (Huang et al. (2019)), multi-modal tasks (Lee et al. (2020)) and generative modeling (Oord et al. (2016); Parmar et al. (2018)) have been recently developed, showing the potential of Transformer models on a broad range of tasks.”