Contrastive Bidirectional Transformer for Temporal Representation Learning

By Chen Sun, Fabien Baradel, Kevin P. Murphy, Cordelia Schmid. 2019

137 citations recorded by the source corpus.

This paper aims at learning representations for long sequences of continuous signals. Recently, the BERT model has demonstrated the effectiveness of stacked transformers for representing sequences of discrete signals (i.e. word tokens). Inspired by its success, we adopt the stacked transformer architecture, but generalize its training objective to maximize the mutual information between the masked signals, and the bidirectional context, via contrastive loss. This enables the model to handle continuous signals, such as visual features. We further consider the case when there are multiple sequences that are semantically aligned at the sequence-level but not at the element-level (e.g. video and ASR), where we propose to use a Transformer to estimate the mutual information between the two sequences, which is again maximized via contrastive loss. We demonstrate the effectiveness of the learned representations on modeling long video sequences for action anticipation and video captioning. The results show that our method, referred to by Contrastive Bidirectional Transformer ({\bf CBT}), outperforms various baselines significantly. Furthermore, we improve over the state of the art.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Video Representations from Textual Web Supervision (2020, neutral): “*Reimplemented by [56].”
Self-Supervised MultiModal Versatile Networks (2020, background): “We build on top of [42, 68, 69] who learn good representations from such narrated material, but consider learning representations using audio as well.”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “tion [18, 43], as well as video-and-language representation learning [63, 62, 41, 31].”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, neutral): “[71] have used a contrastive bidirectional transformer (CBT) to learn long term contextual video representations from instructional videos.”
Contrastive Multiview Coding (2019, background): “The idea of contrastive learning has also started to spread over many other tasks in various other domains [74, 86, 61, 76, 48, 38, 75].”
Self-Supervised Learning of Pretext-Invariant Representations (2019, background): “Finally, PIRL is also related to approaches that use a contrastive loss [24] in predictive learning [25, 26, 28, 59, 70, 73].”
On Mutual Information Maximization for Representation Learning (2019, background): “Despite these fundamental challenges, several recent works have demonstrated promising empirical results in representation learning using MI maximization (van den Oord et al., 2018; Hénaff et al., 2019; Tian et al., 2019; Hjelm et al., 2019; Bachman et al., 2019; Sun et al., 2019).”
Scaling Up Vision-Language Pretraining for Image Captioning (2021, background): “Along the journey of VLP, researchers have investigated different training strategies [11,31], robustness [25], compression [9, 10, 42], probing analysis [3, 27], and the extension to video-text modeling [21, 24, 38, 39, 50].”
MERLOT: Multimodal Neural Script Knowledge Models (2021, background): “Of late, works have sought to learn multimodal representations transferable to many tasks from uncurated sets of (usually how-to) videos [68, 93, 94, 69, 115, 8, 6, 3]; generally these are applied to video understanding tasks like activity recognition.”
VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding (2021, methodology): “CBT (Sun et al., 2019a), HERO (Li et al., 2020b), VideoAsMT (Korbar et al., 2020) and UniVL (Luo et al., 2020) adopt multi-task learning (MTL) to learn retrieval tasks on video and text encoders.”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “Methods using such readily-available data have shown significant improvements on several tasks including video retrieval, action localization, action recognition and video captioning [25, 56, 57, 58, 71, 72, 100], sometimes outperforming fully-supervised baselines.”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, methodology): “, 2019b), CBT (Sun et al., 2019a) and UniViLM (Luo et al.”
UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation (2020, methodology): “han patches of different videos. Other works predict the feature vectors in latent space using auto-regressive models with the noise-contrastive estimation (NCE) (Lotter et al.,2016;Oord et al.,2018).Sun et al. (2019a) adopt NCE to predict corrupted (masked) latent space using the auto-encoder model”
Unified Vision-Language Pre-Training for Image Captioning and VQA (2019, background): “CBT (Sun et al. 2019a) Video Two-stream Transformer encoder+ Transformer decoder Action anticipation Video captioning”
VL-BERT: Pre-training of Generic Visual-Linguistic Representations (2019, background): “VideoBERT (Sun et al., 2019b) is the ﬁrst work seeking to conduct pre-training for visual-linguistic tasks.”
What makes for good views for contrastive learning (2020, background): “In practice, v1 and v2 are two views of the data x, such as different augmentations of the same image [58, 4, 23, 8, 7], different image channels [53], or video and text pairs [52, 36, 33].”
Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models (2020, background): “CBT [29] introduced contrastive learning to handle real-valued video frame features, and HERO [20] proposed hierarchical Transformer architectures to leverage both global and local temporal visual-textual alignments.”
Pre-trained models for natural language processing: A survey (2020, neutral): “VideoBERT [165] and CBT [164] are joint video and text models.”