A Joint Sequence Fusion Model for Video Question Answering and Retrieval

By Youngjae Yu, Jongseok Kim, Gunhee Kim. 2018

405 citations recorded by the source corpus.

We present an approach named JSFusion (Joint Sequence Fusion) that can measure semantic similarity between any pairs of multimodal sequence data (e.g. a video clip and a language sentence). Our multimodal matching network consists of two key components. First, the Joint Semantic Tensor composes a dense pairwise representation of two sequence data into a 3D tensor. Then, the Convolutional Hierarchical Decoder computes their similarity score by discovering hidden hierarchical matches between the two sequence modalities. Both modules leverage hierarchical attention mechanisms that learn to promote well-matched representation patterns while prune out misaligned ones in a bottom-up manner. Although the JSFusion is a universal model to be applicable to any multimodal sequence data, this work focuses on video-language tasks including multimodal retrieval and video QA. We evaluate the JSFusion model in three retrieval and VQA tasks in LSMDC, for which our model achieves the best performance reported so far. We also perform multiple-choice and movie retrieval tasks for the MSR-VTT dataset, on which our approach outperforms many state-of-the-art methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Mohit Bansal
3. Zhe Gan
4. Cordelia Schmid
5. Yejin Choi
6. Lijuan Wang
7. Zicheng Liu
8. Linjie Li

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Cordelia Schmid
4. Ming-Hsuan Yang
5. Trevor Darrell
6. Andrew Zisserman
7. Yi Yang
8. Xiaogang Wang

In Multimodal Learning

1. Zhe Gan
2. Linjie Li
3. Lijuan Wang
4. Mohit Bansal
5. Jianfeng Gao
6. Cordelia Schmid
7. Zicheng Liu
8. Jianfeng Wang

Citations in the corpus

• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, methodology): “It shows that state-of-the-art performance [63] can be attained with only 20% of MSR-VTT samples.”
• Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “…and Internet image domains, (v) quantitative comparisons to state-of-the-art video understanding models on the popular MSR-VTT (Xu et al. 2016; Yu, Kim, and Kim 2018) dataset for video-to-text retrieval, and (vi) a framework for unsupervised quantitative model selection of Socratic Models…”
• VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, methodology): “MSRVTT-MC and MSRVTT-QA [7] are created based on videos and captions in MSRVTT [1].”
• Florence: A New Foundation Model for Computer Vision (2021, methodology): “We report results on the 1K-A test (Yu et al., 2018), which contains 1K video and caption pairs.”
• MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “MSRVTT Multichoice [121] The task objective for the MSRVTT Multichoice benchmark is identical to those of corresponding tasks in the LSMDC benchmark [110].”
• Unifying Vision-and-Language Tasks via Text Generation (2021, methodology): “…performance, these results suggest that referring expression comprehension can be effectively formulated as a text-generation task, rather than previously (Yu et al., 2018a; Chen et al., 2020) formulated classification task over a set of visual regions, allowing more flexible architecture design.”
• ActBERT: Learning Global-Local Video-Text Representations (2020, methodology): “Text-only BLSTM [23] 32.0 Text-only Human [23] 30.2 GoogleNet-2D + C3D [23] 35.7 Merging-LSTM [24] 34.2 SNUVL [50] 38.0 CT-SAN [51] 41.9 LR/RL LSTMs [25] 40.9 JSFusion [49] 45.5 ActBERT 48.6”
• Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, methodology): “(6) We use a combined hinge loss Lh (Yu et al., 2018) over positive and negative query-video pairs.”
• UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation (2020, methodology): “Yu et al. (2018) densely align each token with each frame.”
• Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, methodology): “Note that our approach significantly outperforms all other available results including JSFusion [57]6, which is the winning method of the LSMDC 2017 Text-to-Video and Video-to-Text retrieval challenge.”
• Multi-modal Transformer for Video Retrieval (2020, methodology): “1k-A and 1k-B denote test sets of 1000 randomly sampled caption-video pairs used in [31] and [16] resp.”
• VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, methodology): “We use the testing split and data in (Yu et al., 2018) on MSR-VTT to evaluate multiple-choice VideoQA.”
• Divide and Conquer: Question-Guided Spatio-Temporal Contextual Attention for Video Question Answering (2020, methodology): “Most existing Video QA methods (Jang et al. 2017; Xu et al. 2017; Yu, Kim, and Kim 2018; Gao et al. 2018; Li et al. 2019; Fan et al. 2019) first generate a question embedding and then fuse it with video features to generate a joint representation for question answering.”