Just Ask: Learning to Answer Questions from Millions of Narrated Videos
By Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, Cordelia Schmid. 2020
352 citations recorded by the source corpus.
Recent methods for visual question answering rely on large-scale annotated datasets. Manual annotation of questions and answers for videos, however, is tedious, expensive and prevents scalability. In this work, we propose to avoid manual annotation and generate a large-scale training dataset for video question answering making use of automatic cross-modal supervision. We leverage a question generation transformer trained on text data and use it to generate question-answer pairs from transcribed video narrations. Given narrated videos, we then automatically generate the HowToVQA69M dataset with 69M video-question-answer triplets. To handle the open vocabulary of diverse answers in this dataset, we propose a training procedure based on a contrastive loss between a video-question multi-modal transformer and an answer transformer. We introduce the zero-shot VideoQA task and show excellent results, in particular for rare answers. Furthermore, we demonstrate our method to significantly outperform the state of the art on MSRVTT-QA, MSVD-QA, ActivityNet-QA and How2QA. Finally, for a detailed evaluation we introduce iVQA, a new VideoQA dataset with reduced language biases and high-quality redundant manual annotations.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Cordelia Schmid
- Trevor Darrell
- Jitendra Malik
- Xiaogang Wang
- Ming-Hsuan Yang
- Andrew Zisserman
In Question Answering & Reading Comprehension
- Radu Soricut
- Soravit Changpinyo
- Zhengyuan Yang
- Shuicheng Yan
- Ali Farhadi
- Yu Qiao
- Ivan Laptev
- Tat-seng Chua
Citations in the corpus
- STAR: A Benchmark for Situated Reasoning in Real-World Videos (2024, background): “The existing visual/video question answering benchmarks [13, 50, 24, 40, 44] adopted images [1, 50, 11]/videos [40, 44, 19, 28, 18, 9, 45] and types of visual comprehension questions.”
- MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “Of note, our model through automatic question-conversion outperforms Just Ask [122], which performs an analogous (supervised-guided) question conversion on all its YouTube transcripts, before pretraining.”
- VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, background): “JustAsk automatically generates 69M video-question-answer triplets from narrated videos for training, which is hardly extendable to text-to-video retrieval tasks.”
- MERLOT: Multimodal Neural Script Knowledge Models (2021, neutral): “ActivityNetQA splits questions by type, and we report our per-type test set results in comparison to [118] in Table 9.”
- GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “For the scene-text related VQA tasks, existing approaches (Yang et al., 2021c; Hu et al., 2020) typically leverages the OCR engine to generate the scene text and use dynamic pointer network to decide the current output token should be OCR or the general text.”
- Align and Prompt: Video-and-Language Pre-training with Entity Prompts (2021, methodology): “We obtain on-par results with VQA-T [54], which exploits 69M QA-specific domain data for pre-training.”
- Multimodal Learning With Transformers: A Survey (2022, background): “Various recently released datasets are million-scale, e.g., Product1M [137], Conceptual 12M [138], RUC-CAS-WenLan [139] (30 M), HowToVQA69M [140], HowTo100M [141], ALT200M [142], LAION-400M [143].”
- Video Graph Transformer for Video Question Answering (2022, methodology): “For a better analysis, we further replace the S3D encoder in VQA-T with our DGT module.”
- End-to-end Generative Pretraining for Multimodal Video Captioning (2022, methodology): “Even though MV-GPT is not designed for this particular task, our model slightly outperforms the previous state-of-the-art VQA-T [57] (which is specifically designed for VideoQA) on both datasets.”
- Composing Ensembles of Pre-trained Models via Iterative Consensus (2022, methodology): “…segmentation of the real-world image use the relative pixel-wise We evaluate methods for solving VQA tasks on ActivityNet-QA (Yu et al., 2019). pixel-wise offsets Our method generates free-form language answers instead of selecting an a pre-defined answer set (Yang et al., 2021; Lei et al., 2022).”
- ScanQA: 3D Question Answering for Spatial Scene Understanding (2021, methodology): “The ScanQA dataset was created in multiple phrases: automatic QA generation [2, 8, 30, 46], question filtering, question editing, and answer collection.”
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning (2023, methodology): “As shown in Figure 2, the final collection covers 11 task categories and 28 datasets, including image captioning [23, 3, 50], image captioning with reading comprehension [37], visual reasoning [16, 24, 28], image question answering [11, 12], knowledge-grounded image question answering [29, 35, 27], ”
- MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning (2023, methodology): “Video Question Answering (VideoQA) includes the iVQA (Yang et al., 2021), MVSD (Chen & Dolan, 2011), and NextQA (Xiao et al., 2021) dateset.”
- VideoAgent: Long-form Video Understanding with Large Language Model as Agent (2024, neutral): “ATP-hard subset Methods Acc@C Acc@T Acc@D Acc@All Acc@C Acc@T Acc@All Supervised VFC [57] 8.4 frames per video — significantly fewer by 2x to 30x compared to existing approaches.”
- Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, neutral): “[113] automatically generate VideoQA training data using language models [79] pretrained on a manually annotated text-only question-answer corpus [80].”
- VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, neutral): “…of publicly available datasets, including Kinetics-710 (Kay et al., 2017), SthSthv2 (Goyal et al., 2017), NExtQA (Xiao et al., 2021), CLEVRER (Yi et al., 2019), EgoQA (Fan, 2019), Tgif (Li et al., 2016), WebVidQA(Yang et al., 2021), RealworldQA (X. AI, 2024), and Hm3d (Ramakrishnan et al., 2021).”