Video Captioning and Retrieval Models with Semantic Attention

By Youngjae Yu, Hyungjin Ko, Jongwook Choi, Gunhee Kim. 2016

46 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, methodology): “Our off-the-shelf HowTo100M model outperforms [22, 24, 53, 64, 65] that are directly trained on MSR-VTT.”
ActBERT: Learning Global-Local Video-Text Representations (2020, neutral): “…VideoQA, we ﬁne-tune the pre-trained ActBERT on the Method Accuracy LSTM-fusion [49] 38.3 C+LSTM+SA-FC7 [39] 60.2 VSE-LSTM [14] 67.3 SNUVL [50] 65.4 EITanque [11] 65.5 CT-SAN [51] 66.4 MLB [13] 76.1 JSFusion [49] 83.4 ActBERT 85.7 Table 5: Video question answering (multiple-choices)…”
A Joint Sequence Fusion Model for Video Question Answering and Retrieval (2018, methodology): “[9,29] attempt to detect semantic concept words from videos and integrate them with Bidirectional LSTM that encodes the language query.”
Movie Description (2016, methodology): “To model multi-sentence description, Yu et al. (2016) propose to use two stacked RNNs where the ﬁrst one models words within a sentence and the second one, sentences within a paragraph.”
A Dataset and Exploration of Models for Understanding Video Data through Fill-in-the-Blank Question-Answering (2016, methodology): “In [44], the authors use an LSTM on pretrained ImageNet features from layer conv5b of a ResNet to encode the video, using temporal attention on frames, and a bidirectional LSTM with semantic attention for encoding the question.”
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, methodology): “To train the model, we use the bi-directional max-margin ranking loss [21], [22], [50], [51] as we would like to learn an embedding that works for both text-to-video and video-to-text retrieval tasks.”
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, neutral): “Accuracy ↑ S UPERVISED LSTM-fusion (Yu et al., 2018) 38.3 C+LSTM+SA-FC7 (Torabi et al., 2016) 60.2 SNUVL (Yu et al., 2016) 65.4 EITanque (Kaufman et al., 2017) 65.5 CT-SAN (Yu et al., 2017) 66.4 VSE-LSTM (Kiros et al., 2014) 67.3 MLB (Kim et al., 2016) 76.1 JSFusion(Yu et al., 2018) 83.4…”
CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval (2021, methodology): “The pixel-level approach trains the model with raw video as input directly (Torabi et al., 2016; Kiros et al., 2014; Yu et al., 2016a; Kaufman et al., 2017; Yu et al., 2017, 2018).”
Temporal Tessellation: A Unified Approach for Video Analysis (2016, unknown): “108 Yu et al [71] .”
Unifying the Video and Question Attentions for Open-Ended Video Question Answering (2017, unknown): “[33] have used the TGIF dataset for movie retrieval.”
Video Captioning via Hierarchical Reinforcement Learning (2017, background): “Since then, numerous improvements were introduced, such as attention [41, 43], hierarchical recurrent neural network (RNN) [42, 18, 3, 32], C3D features [28], joint embedding space [23], language fusion [8], multi-task learning [20], etc.”
Video Fill In the Blank Using LR/RL LSTMs with Spatial-Temporal Attentions (2017, methodology): “SNUVL [35] is the best reported method on LSMDC FIB.”
TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment (2021, background): “Realistic application scenarios around videos have prompted emergence of various video-language tasks, such as text-video retrieval [30, 55, 53], video question answering [21, 27], video captioning [54, 59], etc .”
Learning from Video and Text via Large-Scale Discriminative Clustering (2017, unknown): “[42] 3.”