Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners

By Zhenhailong Wang, Manling Li, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem and 3 more. 2022

171 citations recorded by the source corpus.

The goal of this work is to build flexible video-language models that can generalize to various video-to-text tasks from few examples, such as domain-specific captioning, question answering, and future event prediction. Existing few-shot video-language learners focus exclusively on the encoder, resulting in the absence of a video-to-text decoder to handle generative tasks. Video captioners have been pretrained on large-scale video-language datasets, but they rely heavily on finetuning and lack the ability to generate text for unseen tasks in a few-shot setting. We propose VidIL, a few-shot Video-language Learner via Image and Language models, which demonstrates strong performance on few-shot video-to-text tasks without the necessity of pretraining or finetuning on any video datasets. We use the image-language models to translate the video content into frame captions, object, attribute, and event phrases, and compose them into a temporal structure template. We then instruct a language model, with a prompt containing a few in-context examples, to generate a target output from the composed content. The flexibility of prompting allows the model to capture any form of text input, such as automatic speech recognition (ASR) transcripts. Our experiments demonstrate the power of language models in understanding videos on a wide variety of video-language tasks, including video captioning, video question answering, video caption retrieval, and video future event prediction. Especially, on video future event prediction, our few-shot model significantly outperforms state-of-the-art supervised models trained on large-scale video datasets. Code and resources are publicly available for research purposes at https://github.com/MikeWangWZHL/VidIL .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Multimodal Learning

Citations in the corpus

ViperGPT: Visual Inference via Python Execution for Reasoning (2023, background): “Other recent works [63, 45, 55, 35, 37, 15] show that large pretrained models can be used together to great effect, but hand-specify the particular way models are combined.”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, neutral): “To reduce this discrepancy, recent approaches (Cho et al., 2021; Wang et al., 2021b; Yang et al., 2021b; Wang et al., 2022b) have attempted to design unified generative models for pre-training, as most VL tasks can be cast as generation problems.”
ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering (2024, background): “…proven to possess the capability to solve a wide array of tasks, shining across various domains such as text (Brown et al., 2020; Chowdhery et al., 2023; Rae et al., 2021; Yu et al., 2022), images (Alayrac et al., 2022; Tsimpoukelli et al., 2021), and videos (Yang et al., 2022; Wang et al., 2022).”
Modular Visual Question Answering via Code Generation (2023, methodology): “Following Wang et al. (2022), we use sentence embeddings2 to retrieve the most similar questions for each test question.”
See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning (2023, unknown): “This shows the potential of our model that performance could be further improved with a stronger LLM like GPT-3 175B [59] or bloomz [50].”
Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, unknown): “The research advancements in prompt compression [202] and long-context support [374] have partially mitigated these challenges, albeit with a slight trade-off in accuracy or costs.”
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, background): “Recently, a promising line of work builds on frozen large autoregressive language models [22, 75, 102, 107, 116, 123] for zero-shot visual question answering.”