Learning Video Representations using Contrastive Bidirectional Transformer

By Chen Sun, Fabien Baradel, Kevin P. Murphy, Cordelia Schmid. 2019

283 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “For example, VideoBERT (Sun et al., 2019) applied BERT to learn a bidrectional joint distribution over quantized video frame features and linguistic tokens from video-text pairs.”
Self-Supervised MultiModal Versatile Networks (2020, background): “We build on top of [52, 78, 79] who learn good representations from such narrated material, but consider learning representations using audio as well.”
On Mutual Information Maximization for Representation Learning (2019, background): “Despite these fundamental challenges, several recent works have demonstrated promising empirical results in representation learning using MI maximization (van den Oord et al., 2018; Hénaff et al., 2019; Tian et al., 2019; Hjelm et al., 2019; Bachman et al., 2019; Sun et al., 2019).”
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, methodology): “The most widely used objectives are image-text retrieval [2, 5, 43, 55, 63, 73, 74, 75], masked language modeling with image clues [2, 23, 43, 55, 72, 73, 74, 75], and masked region modeling [15, 55, 72, 73, 75].”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, methodology): “To remedy this, CBT (Sun et al., 2019a) uses a contrastive loss but still mainly for video representa- ∗ Equal contribution. tion learning alone, with text input only considered as side information.”
UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation (2020, methodology): “Sun et al. (2019a) adopt NCE to make prediction on corrupted (masked) latent space using auto-encoder model.”
Unified Vision-Language Pre-Training for Image Captioning and VQA (2019, background): “The most similar work to ours is VideoBERT (Sun et al. 2019b), which addresses generation-based tasks ( e.g. , video captioning) and understanding-based tasks ( e.g. , action classiﬁcation).”
VL-BERT: Pre-training of Generic Visual-Linguistic Representations (2019, background): “In the following work of CBT (Sun et al., 2019a), such clustering mechanism is removed.”
Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning (2021, background): “Contrastive learning has recently attracted a surge of interest and shown promising results in various areas including computer vision [18, 6], natu-ral language processing [11, 15], audio processing [49, 42], graph [20, 44] and multimodal data [4, 51].”
Transformers in Vision: A Survey (2021, neutral): “This stands in contrast to BERT inspired VLP models [120, 181, 183, 184] which use independent encoder and decoder networks.”
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, methodology): “A few attempts have been made to use Transformer for vision tasks like image super-resolution [105], object detection [12] and multimodal video understanding [89, 24, 63].”