Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos

By Brian Chen, Andrew Rouditchenko, Kevin Duarte, Hilde Kuehne, Samuel Thomas, Angie Boggust, Rameswar Panda, Brian Kingsbury, Rogério Schmidt Feris, David F. Harwath and 3 more. 2021

100 citations recorded by the source corpus.

Multimodal self-supervised learning is getting more and more attention as it allows not only to train large networks without human supervision but also to search and retrieve data across various modalities. In this context, this paper proposes a framework that, starting from a pre-trained backbone, learns a common multimodal embedding space that, in addition to sharing representations across different modalities, enforces a grouping of semantically similar instances. To this end, we extend the concept of instance-level contrastive learning with a multimodal clustering step in the training pipeline to capture semantic similarities across modalities. The resulting embedding space enables retrieval of samples across all modalities, even from unseen datasets and different domains. To evaluate our approach, we train our model on the HowTo100M dataset and evaluate its zero-shot retrieval capabilities in two challenging domains, namely text-to-video retrieval, and temporal action localization, showing state-of-the-art results on four different datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Jianfeng Gao
  2. Jianwei Yang
  3. Lijuan Wang
  4. Zicheng Liu
  5. Lu Yuan
  6. Chunyuan Li
  7. Zhe Gan
  8. Jianfeng Wang

In Computer Vision and Pattern Recognition

  1. Jianfeng Gao
  2. Jianwei Yang
  3. Lu Yuan
  4. Luc Van Gool
  5. Dacheng Tao
  6. Zicheng Liu
  7. Cordelia Schmid
  8. Ming-Hsuan Yang

In Representation & Self-Supervised Learning

  1. Jianfeng Gao
  2. Jianwei Yang
  3. Chunyuan Li
  4. Lu Yuan
  5. Xuedong Huang
  6. Ce Liu
  7. Bin Xiao
  8. Jianfeng Wang

Citations in the corpus