DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization

By Zineng Tang, Jie Lei, Mohit Bansal. 2021

54 citations recorded by the source corpus.

Leveraging large-scale unlabeled web videos such as instructional videos for pre-training followed by task-specific finetuning has become the de facto approach for many video-and-language tasks. However, these instructional videos are very noisy, the accompanying ASR narrations are often incomplete, and can be irrelevant to or temporally misaligned with the visual content, limiting the performance of the models trained on such data. To address these issues, we propose an improved video-and-language pre-training method that first adds automatically-extracted dense region captions from the video frames as auxiliary text input, to provide informative visual cues for learning better video and language associations. Second, to alleviate the temporal misalignment issue, our method incorporates an entropy minimization-based constrained attention loss, to encourage the model to automatically focus on the correct caption from a pool of candidate ASR captions. Our overall approach is named DeCEMBERT (Dense Captions and Entropy Minimization). Comprehensive experiments on three video-and-language tasks (text-to-video retrieval, video captioning, and video question answering) across five datasets demonstrate that our approach outperforms previous state-of-the-art methods. Ablation studies on pre-training and downstream tasks show that adding dense captions and constrained attention loss help improve the model performance. Lastly, we also provide attention visualization to show the effect of applying the proposed constrained attention loss.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Zhe Gan
  3. Jianfeng Gao
  4. Yejin Choi
  5. Linjie Li
  6. Jingjing Liu
  7. Lijuan Wang
  8. Zicheng Liu

In Computer Vision and Pattern Recognition

  1. Zhe Gan
  2. Cordelia Schmid
  3. Luc Van Gool
  4. Mohit Bansal
  5. Dacheng Tao
  6. Trevor Darrell
  7. Andrew Zisserman
  8. Xiaogang Wang

In Multimodal Learning

  1. Zhe Gan
  2. Linjie Li
  3. Lijuan Wang
  4. Zicheng Liu
  5. Mohit Bansal
  6. Jingjing Liu
  7. Cordelia Schmid
  8. Yu Cheng

Citations in the corpus