End-to-End Concept Word Detection for Video Captioning, Retrieval, and Question Answering

By Youngjae Yu, Hyungjin Ko, Jongwook Choi, Gunhee Kim. 2016

245 citations recorded by the source corpus.

We propose a high-level concept word detector that can be integrated with any video-to-language models. It takes a video as input and generates a list of concept words as useful semantic priors for language generation models. The proposed word detector has two important properties. First, it does not require any external knowledge sources for training. Second, the proposed word detector is trainable in an end-to-end manner jointly with any video-to-language models. To effectively exploit the detected words, we also develop a semantic attention mechanism that selectively focuses on the detected concept words and fuse them with the word encoding and decoding in the language model. In order to demonstrate that the proposed approach indeed improves the performance of multiple video-to-language tasks, we participate in all the four tasks of LSMDC 2016 [18]. Our approach has won three of them, including fill-in-the-blank, multiple-choice test, and movie retrieval.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, methodology): “Our off-the-shelf HowTo100M model outperforms [22, 24, 53, 64, 65] that are directly trained on MSR-VTT.”
ActBERT: Learning Global-Local Video-Text Representations (2020, neutral): “Text-only BLSTM [23] 32.0 Text-only Human [23] 30.2 GoogleNet-2D + C3D [23] 35.7 Merging-LSTM [24] 34.2 SNUVL [50] 38.0 CT-SAN [51] 41.9 LR/RL LSTMs [25] 40.9 JSFusion [49] 45.5 ActBERT 48.6”
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering (2017, background): “ed to regions and to bind visual concepts associated with the same object. We are aware of a few models that seek to combine bottom-up and top-down attention processing. Semantic attention mechanisms [56,57] apply top-down attention to a list of visual attributes detected in an image. Attribute dete”
A Joint Sequence Fusion Model for Video Question Answering and Retrieval (2018, methodology): “Other recent successful methods benefit from incorporating concept words as semantic priors [9,29], or relying on strong representation of videos like RNN-FV [30].”
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, background): “Towards this goal recent works study representations of vision and language addressing tasks such as visual question answering [1], [2], action learning and discovery [3], [4], [5], text-based event localization [6] as well as video captioning, retrieval and summarization [7], [8], [9], [10].”
Multi-modal Transformer for Video Retrieval (2020, neutral): “0 CT-SAN [32] 16.”
Learning Joint Embedding with Multimodal Cues for Cross-Modal Video-Text Retrieval (2018, methodology): “We could not compare with method [39] which focuses on movie retrieval task and results are not available on our experimented datasets.”
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, neutral): “Accuracy ↑ S UPERVISED LSTM-fusion (Yu et al., 2018) 38.3 C+LSTM+SA-FC7 (Torabi et al., 2016) 60.2 SNUVL (Yu et al., 2016) 65.4 EITanque (Kaufman et al., 2017) 65.5 CT-SAN (Yu et al., 2017) 66.4 VSE-LSTM (Kiros et al., 2014) 67.3 MLB (Kim et al., 2016) 76.1 JSFusion(Yu et al., 2018) 83.4 ActBERT(Zhu”
Dual Encoding for Zero-Example Video Retrieval (2018, methodology): “Variants of recurrent neural networks are being exploited, see the usage of LSTM, bidirectional LSTM, and Gated Recurrent Unit (GRU) in [37], [36], and [24], respectively.”
Motion-Appearance Co-memory Networks for Video Question Answering (2018, background): “ing. Xiong et al. [33] improved the memory and input module of DMN so that it can be applied to image QA. Video question answering. Video QA is a relatively new task compared with image QA. Yu et al. [39] adopted a semantic attention mechanism, which combines the detected concepts in videos with tex”
Predicting Visual Features From Text for Image and Video Caption Retrieval (2017, methodology): “Similar to the image domain, the state-of-the-art methods for video caption retrieval also operate in a shared subspace [8], [43], [44].”
HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrieval (2021, unknown): “Methods R@1 R@5 R@10 MedR CT-SAN [67] 5.”
Video Question Answering with Spatio-Temporal Reasoning (2019, methodology): “Yu et al. (2017) proposed to use LSTMs to represent both videos and QA pairs.”