Multi-Task Video Captioning with Video and Entailment Generation

By Ramakanth Pasunuru, Mohit Bansal. 2017

121 citations recorded by the source corpus.

Video captioning, the task of describing the content of a video, has seen some promising improvements in recent years with sequence-to-sequence models, but accurately learning the temporal and logical dynamics involved in the task still remains a challenge, especially given the lack of sufficient annotated data. We improve video captioning by sharing knowledge with two related directed-generation tasks: a temporally-directed unsupervised video prediction task to learn richer context-aware video encoder representations, and a logically-directed language entailment generation task to learn better video-entailing caption decoder representations. For this, we present a many-to-many multi-task learning model that shares parameters across the encoders and decoders of the three tasks. We achieve significant improvements and the new state-of-the-art on several standard video captioning datasets using diverse automatic and human evaluations. We also show mutual multi-task improvements on the entailment generation task.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, methodology): “, 2017); moreover, models trained on these datasets were used as components for performing better video captioning (Pasunuru and Bansal, 2017), summarization (Pasunuru and”
Reinforced Video Captioning with Entailment Rewards (2017, background): “…shown in Table 3 and Table 4 We list previous works’ results as reported by the MSR-VTT dataset paper itself, as well as their 3 leaderboard winners ( http://ms-multimedia-challenge. com/leaderboard ), plus the 10-ensemble video+entailment generation multi-task model of Pasunuru and Bansal (2017).”
Evaluation of Text Generation: A Survey (2020, methodology): “It has been used to evaluate several text generation tasks, including machine translation (Padó et al., 2009), document summarization (Long et al., 2018), language modeling (Liu et al., 2019b), and video captioning (Pasunuru & Bansal, 2017).”
Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generation (2018, methodology): “Our entailment generation task is based on the recent SNLI classiﬁcation dataset and task (Bowman et al., 2015), converted to a generation task (Pasunuru and Bansal, 2017).”
Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation (2018, methodology): “This changed as large NLI datasets have recently been used to train, or pre-train, models to perform NLI, or other tasks (Conneau et al., 2017; Pasunuru and Bansal, 2017).”
CKD: Cross-Task Knowledge Distillation for Text-to-Image Synthesis (2020, methodology): “[43] propose a multi-task video captioning method by sharing knowledge with a temporally-directed unsupervised video prediction task and a logically-directed language entailment generation task.”
Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning (2018, background): “…numerous improvements have been introduced, such as attention (Yao et al. 2015), hierarchical recurrent neural network (Yu et al. 2016; Pan et al. 2016), multi-modal fusion(Gan et al. 2017; Shen et al. 2017; Wang, Wang, and Wang 2018), multi-task learning (Pasunuru and Bansal 2017), etc.”
Video Captioning via Hierarchical Reinforcement Learning (2017, background): “Since then, numerous improvements were introduced, such as attention [41, 43], hierarchical recurrent neural network (RNN) [42, 18, 3, 32], C3D features [28], joint embedding space [23], language fusion [8], multi-task learning [20], etc.”
SibNet: Sibling Convolutional Encoder for Video Captioning (2018, methodology): “It can be seen from Table 4 that SibNet has much smaller number of parameters than previous stateof-the-art approaches (44% of [45], 57% of [9], 77% of [29] and 90% of [27]).”
Text Generation with Efficient (Soft) Q-Learning (2021, background): “To this end, we consider the task of entailment generation [43].”
Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-ray Reports (2019, background): “Additionally, other tasks related to visual captioning, (e.g., dense captioning (Johnson et al., 2016), multi-task learning (Pasunuru and Bansal, 2017)) also attracted a lot of research attention.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “It is further approached from the reinforcement learning by providing entailment rewards (Pasunuru & Bansal, 2017b) and to address the description generation for multiple fine-grained actions (Wang et al., 2018b).”
Adaptive Feature Abstraction for Translating Video to Text (2016, methodology): “The M-to-M method (Pasunuru and Bansal 2017) is the only model showing better BLEU and CIDEr on Youtube2Text.”