MoPro: Webly Supervised Learning with Momentum Prototypes

By Junnan Li, Caiming Xiong, Steven C. H. Hoi. 2020

115 citations recorded by the source corpus.

We propose a webly-supervised representation learning method that does not suffer from the annotation unscalability of supervised learning, nor the computation unscalability of self-supervised learning. Most existing works on webly-supervised representation learning adopt a vanilla supervised learning method without accounting for the prevalent noise in the training data, whereas most prior methods in learning with label noise are less effective for real-world large-scale noisy data. We propose momentum prototypes (MoPro), a simple contrastive learning method that achieves online label noise correction, out-of-distribution sample removal, and representation learning. MoPro achieves state-of-the-art performance on WebVision, a weakly-labeled noisy dataset. MoPro also shows superior performance when the pretrained model is transferred to down-stream image classification and detection tasks. It outperforms the ImageNet supervised pretrained model by +10.5 on 1-shot classification on VOC, and outperforms the best self-supervised pretrained model by +17.3 when finetuned on 1\% of ImageNet labeled samples. Furthermore, MoPro is more robust to distribution shifts. Code and pretrained models are available at this https URL.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (2021, methodology): “The recent CLIP [6] and ALIGN [7] perform pre-training on massive noisy web data using a contrastive loss, one of the most effective loss for representation learning [24, 25, 26, 27].”
Continual Learning on Noisy Data Streams via Self-Purified Replay (2021, neutral): “Although [56, 47] also use self-supervised learning with noisy labeled data, they focus on the loss or prediction from the model for selecting suspicious samples.”
Exploring Visual Engagement Signals for Representation Learning (2021, background): “This motivates a plethora of work investigating weakly-supervised [73, 55, 41, 42, 47], semisupervised [86, 87, 84] and self-supervised learning [3, 96, 14, 4, 22, 48, 90].”
Few-Shot Named Entity Recognition: A Comprehensive Study (2020, methodology): “To gain a stronger transfer learning ability, one may combine supervised and self-supervised methods (Li et al., 2020c,b).”
Concept Generalization in Visual Representation Learning (2020, background): “3.4 for the justification of our protocol and the choice of metrics.”
CoMatch: Semi-supervised Learning with Contrastive Graph Regularization (2020, background): “The widely adopted contrastive learning [35, 28, 5, 6, 14, 20] optimizes for the task of instance discrimination, and formulates the loss using the normalized low-dimensional embeddings z:”
Generalized Out-of-Distribution Detection: A Survey (2021, unknown): “OOD Detection & Open-Set Noisy Labels Existing methods of learning from open-set noisy labels focus on suppressing the negative effects of noise [84], [349].”