Video Fill in the Blank with Merging LSTMs

By Amir Mazaheri, Dong Zhang, Mubarak Shah. 2016

18 citations recorded by the source corpus.

Given a video and its incomplete textural description with missing words, the Video-Fill-in-the-Blank (ViFitB) task is to automatically find the missing word. The contextual information of the sentences are important to infer the missing words; the visual cues are even more crucial to get a more accurate inference. In this paper, we presents a new method which intuitively takes advantage of the structure of the sentences and employs merging LSTMs (to merge two LSTMs) to tackle the problem with embedded textural and visual cues. In the experiments, we have demonstrated the superior performance of the proposed method on the challenging "Movie Fill-in-the-Blank" dataset.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

ActBERT: Learning Global-Local Video-Text Representations (2020, neutral): “Text-only BLSTM [23] 32.0 Text-only Human [23] 30.2 GoogleNet-2D + C3D [23] 35.7 Merging-LSTM [24] 34.2 SNUVL [50] 38.0 CT-SAN [51] 41.9 LR/RL LSTMs [25] 40.9 JSFusion [49] 45.5 ActBERT 48.6”
A Joint Sequence Fusion Model for Video Question Answering and Retrieval (2018, methodology): “To encode the sentential query sentence on the video context, MergingLSTMs [35] and LR/RL LSTMs [36] are proposed.”
A Dataset and Exploration of Models for Understanding Video Data through Fill-in-the-Blank Question-Answering (2016, methodology): “Similar network structure for fill-in-the-blank question has also been explored in [23].”
Open-Ended Long-form Video Question Answering via Adaptive Hierarchical Reinforced Networks (2018, methodology): “, 2016] and videobased question answering ones [Mazaheri et al., 2016; Tapaswi et al., 2016; Zhu et al., 2015; Zeng et al., 2016; Zhao et al., 2017].”
Open-Ended Video Question Answering via Multi-Modal Conditional Adversarial Networks (2020, methodology): “approaches [45], [46] fill the missing words in the textual video description by ranking the candidate answers based on both visual content and contextual text description.”
Learning Fashion Compatibility with Bidirectional LSTMs (2017, background): “Recently, several fill-in-the-blank (FITB) datasets [13, 14, 29, 30] have been created and evaluated to bridge visual and semantic information.”
Video Question Answering via Hierarchical Spatio-Temporal Attention Networks (2017, methodology): “On the other hand, the proposed models [Zhu et al., 2015; Mazaheri et al., 2016; Tu et al., 2014] for fill-inthe-blank [Yu et al.”