Zero-Shot Video Question Answering via Frozen Bidirectional Language Models

By Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, Cordelia Schmid. 2022

297 citations recorded by the source corpus.

Video question answering (VideoQA) is a complex task that requires diverse multi-modal data for training. Manual annotation of question and answers for videos, however, is tedious and prohibits scalability. To tackle this problem, recent methods consider zero-shot settings with no manual annotation of visual question-answer. In particular, a promising approach adapts frozen autoregressive language models pretrained on Web-scale text-only data to multi-modal inputs. In contrast, we here build on frozen bidirectional language models (BiLM) and show that such an approach provides a stronger and cheaper alternative for zero-shot VideoQA. In particular, (i) we combine visual inputs with the frozen BiLM using light trainable modules, (ii) we train such modules using Web-scraped multi-modal data, and finally (iii) we perform zero-shot VideoQA inference through masked language modeling, where the masked text is the answer to a given question. Our proposed approach, FrozenBiLM, outperforms the state of the art in zero-shot VideoQA by a significant margin on a variety of datasets, including LSMDC-FiB, iVQA, MSRVTT-QA, MSVD-QA, ActivityNet-QA, TGIF-FrameQA, How2QA and TVQA. It also demonstrates competitive performance in the few-shot and fully-supervised setting. Our code and models are publicly available at https://github.com/antoyang/FrozenBiLM.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Yu Qiao

Citations in the corpus

ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering (2024, background): “…proven to possess the capability to solve a wide array of tasks, shining across various domains such as text (Brown et al., 2020; Chowdhery et al., 2023; Rae et al., 2021; Yu et al., 2022), images (Alayrac et al., 2022; Tsimpoukelli et al., 2021), and videos (Yang et al., 2022; Wang et al., 2022).”
Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models (2023, methodology): “To benchmark Video-ChatGPT, we compared its performance with other significant models, such as FrozenBiLM [35] and the generative video model, Video Chat.”
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (2023, methodology): “Following we employ GPT-assisted evaluation to conduct a more comprehensive comparison of the text generation performance between MovieChat and previous methods [34, 40, 72] on processed ActivityNet-QA [75].”
MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (2023, background): “Frame EgoSchema Subset Fullset FrozenBILM [92] 90 - 26.9 VIOLET [18] 5 - 19.9 mPLUG-Owl [ [40].”
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models (2023, background): “In recent years, the trend of using autoregressive language models as decoders in vision-language tasks has gained significant traction [6, 15, 36, 31, 2, 16, 17, 12].”
Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, background): “Other alternatives to building video captioners include mPLUG-2 [81], GIT [82], FrozenBiLM [83], and more.”
Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (2023, unknown): “4, Chat-UniVi outperforms the recently proposed state-of-the-art methods, e.g ., FrozenBiLM [64] and Video-ChatGPT, across various datasets.”
EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding (2023, background): “FrozenBiLM [58] adapts frozen multi-modal encoders trained on web-scale data for the task of question answering and achieves state-of-the-art zero-shot QA accuracy across 8 video question-answering datasets.”