MSR-VTT: A Large Video Description Dataset for Bridging Video and Language

By Jun Xu, Tao Mei, Ting Yao, Yong Rui. 2016

2,526 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Evaluation & Benchmarks

Citations in the corpus

Self-Supervised MultiModal Versatile Networks (2020, methodology): “We note a decrease in performance for the video-text benchmarks (MSRVTT and YC2), which can simply be explained by the fact that only a half of the training samples contain text vs. Table 1a (the other half comes from AudioSet which does not have text).”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, background): “Text-to-Video retrieval: YouCook2 [90], MSR-VTT [87].”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, methodology): “It is also subjective since video annotation can often be an ill-defined task with low annotator consistency [58].”
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “10: Evolution over time of state-of-the-art (SOTA) results for video-to-text retrieval, with R@1 metric, on the popular MSR-VTT (Xu et al. 2016) 1k-A (Yu, Kim, and Kim 2018) dataset.”
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, methodology): “6-9 on MSRVTT [1], DiDeMo [79], LSMDC [3], and YouCook2 [13], respectively.”
Florence: A New Foundation Model for Computer Vision (2021, neutral): “Then, we perform zero-shot text-to-video evaluation on the MSR-VTT (Xu et al., 2016) dataset.”
VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding (2021, methodology): “…(Luo et al., 2020) 42.0 Pairwise Matching ActBERT (Zhu and Yang, 2020) 41.4 VLM (task-agnostic, zero-shot) 28.5 VLM (supervised on 540 videos) 46.5 (Yu et al., 2018) 83.4 Pairwise Matching ActBERT(Zhu and Yang, 2020) 85.7 VLM 91.64 ActBERT, which leverages pairwise matching for each video/answer…”
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, background): “, “Why is the toddler crying?”, “How did the lady react after the toddler fell?”), which lies at the core of human intelligence [39], remains a great challenge for computational models and is also much less explored by existing video understanding tasks [22, 37, 49, 52, 56].”
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, methodology): “MSR-VTT [72] contains 10K YouTube videos with 200K descriptions.”
Unifying Vision-and-Language Tasks via Text Generation (2021, background): “, 2018b) in a wide range of discriminative (Goyal et al., 2019; Hudson & Manning, 2019; Lei et al., 2018; Mao et al., 2016; Xu et al., 2016; Zhou et al., 2018) and generative tasks (Chen et al.”
ActBERT: Learning Global-Local Video-Text Representations (2020, methodology): “We evaluate the ﬁve tasks on CrossTask [59], COIN [37], YouCook2 [53], MSR-VTT [46] and LSMDC [32].”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, methodology): “…captioning (Venugopalan et al., 2015; Pan et al., 2016; Gan et al., 2017; Zhou et al., 2018b, 2019), mostly benchmarking on Youtube2Text (Guadarrama et al., 2013), MSR-VTT (Xu et al., 2016), YouCook2 (Zhou et al., 2018a), ActivityNet Captions (Krishna et al., 2017) and VATEX (Wang et al., 2019).”
UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation (2020, positive): “MSR-VTT (Xu et al., 2016) is the Out-domain dataset for downstream task.”
Attention on Attention for Image Captioning (2019, methodology): “To show the general applicability of AoA, we perform experiments on a video captioning dataset, MSR-VTT [41]: we use ResNet-101 [18] to extract feature vectors from sampled 20 frames of each video and then pass them to a biLSTM and a decoder, “base” or “decoder with AoA”.”
Women also Snowboard: Overcoming Bias in Captioning Models (2018, methodology): “ence quality, without regard to captured biases. Though we pay special attention to gender in this work, all captioning models trained on visual description data (MSCOCO [35], Flickr30k [36], MSR-VTT [37] to name a few) implicitly learn to classify gender. However current captioning models do not di”
Towards Automatic Learning of Procedures From Web Instructional Videos (2017, methodology): “We use long short-term memory (LSTM) for sequential prediction due to its state-of-the-art performance in se-Mean quence modeling (Xu et al. 2016; Zhang et al. 2016).”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “Although our model is not dedicated for video tasks, our model achieve new SOTA on MSRVD, MSRVTT, and VATEX for captioning and on MSVD-QA and TGIF-Frame for QA.”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “[228] Jun Xu, Tao Mei, Ting Yao, and Yong Rui.”