How2: A Large-scale Dataset for Multimodal Language Understanding

By Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loïc Barrault, Lucia Specia, Florian Metze. 2018

326 citations recorded by the source corpus.

In this paper, we introduce How2, a multimodal collection of instructional videos with English subtitles and crowdsourced Portuguese translations. We also present integrated sequence-to-sequence baselines for machine translation, automatic speech recognition, spoken language translation, and multimodal summarization. By making available data and code for several multimodal natural language tasks, we hope to stimulate more research on these and similar challenges, to obtain a deeper understanding of multimodality in language processing.

Subfields: Artificial Intelligence, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Multimodal Learning

In Evaluation & Benchmarks

Citations in the corpus

End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, positive): “The nature of this supervision has the potential to also provide semantic information [52, 67], which is often missing in works that only exploit pixel-wise cues.”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “Numerous datasets of web instructional videos were proposed over the past years [2, 30, 45, 47, 50, 67, 68].”
VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (2019, methodology): “[49] collect English→Portuguese subtitles for the automatic speech recognition (ASR) task, which however do not directly deSplit train validation public test secret test”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, methodology): “The recent inﬂux of instructional video datasets such as How2 [28], Inria Instructional Videos [60], COIN [61], CrossTask [31], YouCook2 [23], Mining YouTube [62], and HowTo100M [25] has inspired a variety of methods for semi-supervised text-video modelling.”
FINDINGS OF THE IWSLT 2020 EVALUATION CAMPAIGN (2020, unknown): “• How2 (Sanabria et al., 2018)8”
Investigating Self-Supervised Pre-Training for End-to-End Speech Translation (2020, methodology): “How2 corpus [24] is used for our main experiments.”
Toward Self-Supervised Object Detection in Unlabeled Videos (2019, methodology): “In order to ground a certain category of object we consider the large corpus of How2 [16] instructional videos covering a wide variety of topics across 13,000 clips (about 300 hours total duration), with word-level time-aligned subtitles.”
Multimodal Abstractive Summarization for How2 Videos (2019, background): “• Introducing Abstractive Summarization for OpenDomain Videos •Provide fluent textual summary from multimodal information •The How2 corpus [1] of instructional videos, transcripts and summaries is useful for this task •Propose Content-F1: a more informative measure of multimodal abstractive summarie”
Looking Enhances Listening: Recovering Missing Speech Using Images (2020, background): “The audio and text captions in this data are more structured and clean, compared to the ‘in-the-wild’ nature of How2 [17].”
ESPnet-ST: All-in-One Speech Translation Toolkit (2020, methodology): “How2 corpus contains English speech extracted from YouTube videos, the corresponding transcription, as well as the Portuguese translation (Sanabria et al., 2018).”
UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training (2021, background): “As more benchmarks [48, 40, 3] on multilingual video-text pairs become available, we are interested in enhancing the grounding between vision and language by leveraging the temporal information from videos.”
Speech Translation and the End-to-End Promise: Taking Stock of Where We Are (2020, background): “…of the latter approach, so follow-up work concentrates on compiling ST corpora from available web sources (Godard et al., 2018; Kocabiyikoglu et al., 2018; Sanabria et al., 2018; di Gangi et al., 2019a; Boito et al., 2020; Beilharz et al., 2020; Iranzo-Sánchez et al., 2020; Wang et al., 2020a).”
How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language (2020, methodology): “How2Sign extends the How2 [27] dataset, an existing multimodal dataset with a new sign language modality, and therefore enables connecting with research performed in the vision, speech and language communities.”
MaSS: A Large and Clean Multilingual Corpus of Sentence-aligned Spoken Utterances Extracted from the Bible (2019, result): “Similar in size, the English-Portuguese dataset How2 [24] was created by translating English short tutorials into Portuguese using a crowdsourcing platform.”
Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization (2021, methodology): “We evaluate our models on the How2 dataset (Sanabria et al., 2018).”
Grounding Object Detections With Transcriptions (2019, methodology): “In this work, we use the How2 dataset to train our weaklysupervised visual grounding model (Sanabria et al., 2018).”
Learning to Detect and Retrieve Objects From Unlabeled Videos (2019, methodology): “Large-scale video data sets such as YouTube-8M [1] and How2 [10] can be lever-aged for this purpose.”