Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning

By Elad Amrani, Rami Ben-Ari, Daniel Rotman, A. Bronstein. 2020

132 citations recorded by the source corpus.

One of the key factors of enabling machine learning models to comprehend and solve real-world tasks is to leverage multimodal data. Unfortunately, annotation of multimodal data is challenging and expensive. Recently, self-supervised multimodal methods that combine vision and language were proposed to learn multimodal representations without annotation. However, these methods often choose to ignore the presence of high levels of noise and thus yield sub-optimal results. In this work, we show that the problem of noise estimation for multimodal data can be reduced to a multimodal density estimation task. Using multimodal density estimation, we propose a noise estimation building block for multimodal representation learning that is based strictly on the inherent correlation between different modalities. We demonstrate how our noise estimation can be broadly integrated and achieves comparable results to state-of-the-art performance on five different benchmark datasets for two challenging multimodal tasks: Video Question Answering and Text-To-Video Retrieval. Furthermore, we provide a theoretical probabilistic error bound substantiating our empirical results and analyze failure cases. Code: https://github.com/elad-amrani/ssml.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

MERLOT: Multimodal Neural Script Knowledge Models (2021, background): “Of late, works have sought to learn multimodal representations transferable to many tasks from uncurated sets of (usually how-to) videos [68, 93, 94, 69, 115, 8, 6, 3]; generally these are applied to video understanding tasks like activity recognition.”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “[6] propose a text-video pretraining approach and finetune for VideoQA.”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, background): “The previous results [24, 25, 71] on these datasets mainly focus on text to video retrieval (denoted by T → V).”
CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval (2021, background): “, 2019), ActBERT (Zhu and Yang, 2020), NoiseE (Amrani et al., 2021), UniVL (Luo et al.”
Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos (2021, methodology): “Such visual caption pairs have been widely used in self-supervised models in vision and language tasks recently [3, 16, 30, 40, 45, 35, 27, 15].”
MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrieval (2022, unknown): “7 NoiseEst [4] 2021 ResNeXt-101 HowTo100M 110M 17.”
Video Graph Transformer for Video Question Answering (2022, background): “Furthermore, recent works [3,70] reveal that these models may suffer from performance lose on open-domain questions due to the heavy noise [1,39] and limited data scope of HowTo100M.”
Object-aware Video-language Pre-training for Retrieval (2021, methodology): “8 NE [1] AAAI’21 ImageNet, Kinetics [136M] HowTo100M 17.”
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, unknown): “[3] Elad Amrani, Rami Ben-Ari, Daniel Rotman, and Alex Bronstein.”