VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research

By Xin Eric Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-fang Wang, William Yang Wang. 2019

718 citations recorded by the source corpus.

We present a new large-scale multilingual video description dataset, VATEX, which contains over 41,250 videos and 825,000 captions in both English and Chinese. Among the captions, there are over 206,000 English-Chinese parallel translation pairs. Compared to the widely-used MSR-VTT dataset, \vatex is multilingual, larger, linguistically complex, and more diverse in terms of both video and natural language descriptions. We also introduce two tasks for video-and-language research based on \vatex: (1) Multilingual Video Captioning, aimed at describing a video in various languages with a compact unified captioning model, and (2) Video-guided Machine Translation, to translate a source language description into the target language using the video information as additional spatiotemporal context. Extensive experiments on the \vatex dataset show that, first, the unified multilingual model can not only produce both English and Chinese descriptions for a video more efficiently, but also offer improved performance over the monolingual models. Furthermore, we demonstrate that the spatiotemporal video context can be effectively utilized to align source and target languages and thus assist machine translation. In the end, we discuss the potentials of using \vatex for other video-and-language research.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Evaluation & Benchmarks

Citations in the corpus

Experience Grounds Language (2020, background): “This move towards using large scale (whether mono-or multilingual) has led to substantial advances in performance on existing and novel community benchmarks (Wang et al., 2019a).”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “Manually annotated vision and language datasets are costly to obtain and thus relatively small (10k-100k) in scale [3, 15, 69, 122, 129, 139].”
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, methodology): “Popular VidL tasks were introduced, such as text-to-video retrieval [1–5], video question answering [6–9], text-based video moment retrieval [2, 4, 10, 11], and video captioning [1, 3, 12, 13].”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, background): “…captioning (Venugopalan et al., 2015; Pan et al., 2016; Gan et al., 2017; Zhou et al., 2018b, 2019), mostly benchmarking on Youtube2Text (Guadarrama et al., 2013), MSR-VTT (Xu et al., 2016), YouCook2 (Zhou et al., 2018a), ActivityNet Captions (Krishna et al., 2017) and VATEX (Wang et al., 2019).”
CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning (2020, methodology): “…including image cap-tioning datasets, such as Flickr30k (Young et al., 2014), MSCOCO (Lin et al., 2014), Conceptual Captions (Sharma et al., 2018), as well as video captioning datasets including LSMDC (Rohrbach et al., 2017), ActivityNet (Krishna et al., 2017), and VATEX (Wang et al., 2019b).”
TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval (2020, neutral): “8K 2-3 X - VATEX [33] Activity 41K 826K 20 X - LSMDC [27] Movie 128K 128K 1 X - MST-VTT [35] Open 10k 200k 20 X - -”
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, methodology): “Future work could consider multilingual VidL datasets [63, 54, 24, 30] as a complementary evaluation to further test systems’ ability on processing information in different languages.”
Birds Have Four Legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models (2020, methodology): “We collected these sentences by ﬁrst obtaining a list of frequent nouns from various caption corpora such as MSCOCO (Lin et al., 2014), VA-TEX (Wang et al., 2019), etc.”
Learning the Best Pooling Strategy for Visual Semantic Embedding (2020, methodology): “VATEX contains 25,991 videos for training, 6000 for testing and 3000 for validation, and the 10 English descriptions for each video are used in the experiments.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “…on combining low-level linguistic units such as words with images or videos for building visual-semantic embeddings (Frome et al., 2013; Kiros et al., 2014b; Liu et al., 2015; Cao et al., 2016; Tsai et al., 2017; Guo et al., 2018; Wang et al., 2019) useful for further down stream applications.”
Visual Grounding in Video for Unsupervised Word Translation (2020, background): “same video or images are associated with captions in multiple languages [46].”
SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning (2021, methodology): “Comparison with state-of-the-art methods on YouCook2, TVC, and VATEX.”
CLIP2Video: Mastering Video-Text Retrieval via Image CLIP (2021, methodology): “• Besides, VATEX [38] includes 34,991 videos with multilingual annotations.”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, neutral): “Method K i n e ti c s - 400 [ 90 ] VA TE X C a p ti on [ 190 ] M S R -”
Violin: A Large-Scale Dataset for Video-and-Language Inference (2020, positive): “This distinguishes our collected statements from the straightforward captions in video/image captioning dataset [39, 33, 59], which are dominated by explicit factual descriptions without deeper inference.”
Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers (2021, methodology): “VATEX contains around 40K short 10 seconds clip from the Kinetics-600 dataset [4] annotated with multiple descriptions.”