Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos

By Shaoxiang Chen, Wenhao Jiang, Wei David Liu, Yu-Gang Jiang. 2020

113 citations recorded by the source corpus.

Automatically generating sentences to describe events and temporally localizing sentences in a video are two important tasks that bridge language and videos. Recent techniques leverage the multimodal nature of videos by using off-the-shelf features to represent videos, but interactions between modalities are rarely explored. Inspired by the fact that there exist cross-modal interactions in the human brain, we propose a novel method for learning pairwise modality interactions in order to better exploit complementary information for each pair of modalities in videos and thus improve performances on both tasks. We model modality interaction in both the sequence and channel levels in a pairwise fashion, and the pairwise interaction also provides some explainability for the predictions of target tasks. We demonstrate the effectiveness of our method and validate specific design choices through extensive ablation studies. Our method turns out to achieve state-of-the-art performances on four standard benchmark datasets: MSVD and MSR-VTT (event captioning task), and Charades-STA and ActivityNet Captions (temporal sentence localization task).

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Zhe Gan
  2. Lijuan Wang
  3. Zicheng Liu
  4. Jianfeng Wang
  5. Linjie Li
  6. Luc Van Gool
  7. Xiaowei Hu
  8. Cordelia Schmid

In Multimodal Learning

  1. Zhe Gan
  2. Lijuan Wang
  3. Zicheng Liu
  4. Jianfeng Wang
  5. Linjie Li
  6. Xiaowei Hu
  7. Zhengyuan Yang
  8. Ce Liu

Citations in the corpus