VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs

By Xudong Lin, Gedas Bertasius, Jue Wang, Shih-Fu Chang, Devi Parikh, Lorenzo Torresani. 2021

78 citations recorded by the source corpus.

We present VX2TEXT, a framework for text generation from multimodal inputs consisting of video plus text, speech, or audio. In order to leverage transformer networks, which have been shown to be effective at modeling language, each modality is first converted into a set of language embeddings by a learnable tokenizer. This allows our approach to perform multimodal fusion in the language space, thus eliminating the need for ad-hoc cross-modal fusion modules. To address the non-differentiability of tokenization on continuous inputs (e.g., video or audio), we utilize a relaxation scheme that enables end-to-end training. Furthermore, unlike prior encoder-only models, our network includes an autoregressive decoder to generate open-ended text from the multimodal embeddings fused by the language encoder. This renders our approach fully generative and makes it directly applicable to different "video+x to text" problems without the need to design specialized network heads for each task. The proposed framework is not only conceptually simple but also remarkably effective: experiments demonstrate that our approach based on a single architecture outperforms the state-of-the-art on three videobased text-generation tasks—captioning, question answering and audio-visual scene-aware dialog.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Natural Language Generation

Citations in the corpus

MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “[77] Xudong Lin, Gedas Bertasius, Jue Wang, Shih-Fu Chang, Devi Parikh, and Lorenzo Torresani.”
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, background): “developed [57, 76, 45, 41, 26, 29, 36, 20, 50, 31, 58, 44, 39, 73] in this field are often evaluated on different tasks, datasets and experimental settings, making fair comparison difficult.”
Orca: A Distributed Serving System for Transformer-Based Generative Models (2022, methodology): “While we use a language model as a driving example throughout the paper and conduct experiments only on language models, generative models in other domains can benefit from our approach as long as the models are based on the Transformer architecture and use the autoregressive generation procedure [1”
Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following (2023, background): “Numerous studies have proved its efficacy, involving 2D images, videos, texts, and audio [15, 17, 48], and enhance the cross-modal performance for downstream tasks [5, 25, 37, 61], and videotext-audio integration for text generation [36].”
Multimodal Learning With Transformers: A Survey (2022, background): “Depth to Binaural Audio [14], music piece & seed 3D motion to long-range future 3D motions [146], X-raying image & question to answer [244], video & text & audio to text [245]), and”
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, unknown): “[67] Xudong Lin, Gedas Bertasius, Jue Wang, Shih-Fu Chang, Devi Parikh, and Lorenzo Torresani.”
FindIt: Generalized Localization with Natural Language Queries (2022, background): “Large amounts of work have been done to learn across image and language, including visualgrounding [16,81,49,96,91,62], image captioning [6,1,5], visual question and answering (VQA) [33,12], visual dialog [11,39], visual reasoning [75,93,85], text-toimage and image-to-text retrieval [32,66,74], and ”