Learning Visual Features from Large Weakly Supervised Data

By Armand Joulin, L. V. D. van der Maaten, Allan Jabri, Nicolas Vasilache. 2015

431 citations recorded by the source corpus.

Convolutional networks trained on large supervised datasets produce visual features which form the basis for the state-of-the-art in many computer-vision problems. Further improvements of these visual features will likely require even larger manually labeled data sets, which severely limits the pace at which progress can be made. In this paper, we explore the potential of leveraging massive, weakly-labeled image collections for learning good visual features. We train convolutional networks on a dataset of 100 million Flickr photos and comments, and show that these networks produce features that perform well in a range of vision problems. We also show that the networks appropriately capture word similarity and learn correspondences between different languages.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “Joulin et al. (2016) modernized this line of work and demonstrated that CNNs trained to predict words in image captions learn useful image representations.”
Exploring the Limits of Weakly Supervised Pretraining (2018, methodology): “Other prior studies [16,38] trained convolutional networks to predict words or n-grams in comments on a collection of 100 million Flickr photos and corresponding comments [39].”
What makes ImageNet good for transfer learning? (2016, methodology): “ present in the 1000 way ImageNet challenge. A reasonable middle ground between the expensive, fully-supervised pre-training and free unsupervised pretraining is to use weak supervision. For example, [18] use the YFCC100M dataset of 100 million Flickr images labeled with noisy user tags as pre-train”
Growing a Brain: Fine-Tuning by Increasing Model Capacity (2017, methodology): “imized [4] 66.4 — — MsML+ [30] 89.5 DeCaf+DPD [7] 65.0 — — Places-CNN [59] 68.2 MPP [55] 91.3 MsML+ [30] 66.6 — — — — Deep Optimized [4] 91.3 MsML+* [30] 67.9 — — Data Augmented CNNs Combined-AlexNet [18] 58.8 Combined-AlexNet [18] 83.3 — — Combined-AlexNet [18] 56.4 Multi-Task CNNs Joint [22] 63.9 ”
Big Transfer (BiT): General Visual Representation Learning (2019, methodology): “In [23,30] the authors use a dataset containing 100M Flickr images [56].”
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era (2017, methodology): “However, unlike ours, [23] showed plateauing of detection performance when trained on 100M images.”
Learning Video Representations from Textual Web Supervision (2020, methodology): “Another approach is to use text captions, and label each image with key words present in the caption [50, 30, 39, 48].”
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, background): “as auxiliary source of information to train their models [11, 34, 48]. Although web images are noisy, utilizing such weakly-labeled images has been shown to be very effective in many multimedia tasks [13, 22, 34] Our work is motivated by these works on learning more powerful models by realizing the ”
Self-supervised Pretraining of Visual Features in the Wild (2021, background): “Prior attempts to train self-supervised models on uncurated data [6, 13, 19, 27] have only used a few millions of images for which using small model architectures is sufﬁcient.”
SLIP: Self-supervision meets Language-Image Pre-training (2021, background): “Initial attempts at leveraging the YFCC dataset for representation learning included predicting the bag-of-words representation [33] or n-gram occurrence [39] from images.”
Self-Supervised Learning of Pretext-Invariant Representations (2019, neutral): “Most representation learning methods are sensitive to the data distribution used during pre-training [21, 32, 46, 71].”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, methodology): “We note a similar trend for the Colorization problem wherein pre-training ImageNet, rather than YFCC, provides a greater benefit when transferring to VOC07 classification (also noted in [8, 12, 31]).”
Combined Scaling for Zero-shot Transfer Learning (2021, methodology): “Indeed, recent efforts have shown that the more data we train on, the better the models become (Joulin et al., 2016; Mahajan et al., 2018; Kolesnikov et al., 2020; Dosovitskiy et al., 2021; Sun et al., 2017).”
Learning to Prompt for Vision-Language Models (2021, background): “, 2013), multi-label classification (Joulin et al., 2016; Gomez et al., 2017), n-gram language learning (Li et al.”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, background): “These works are however limited to small datasets such as Flickr (Joulin et al., 2015; Li et al., 2017) and COCO Captions (Desai & Johnson, 2020; Sariyildiz et al., 2020), and the resulting models don’t produce a vision-language representation that is needed for tasks like cross-modal retrieval.”
RegionCLIP: Region-based Language-Image Pretraining (2021, background): “The most relevant work is learning visual representations from natural language, such as image tags [3, 8, 12, 25, 28] and text descriptions [11,23,43,53,62].”
Deep Clustering for Unsupervised Learning of Visual Features (2018, methodology): “For example, metadata, like hashtags, exhibits a Zipf distribution, with a few labels dominating the whole distribution [28].”
Efficient softmax approximation for GPUs (2016, methodology): “Sampling based approaches have been successfully applied to approximate the softmax function over large dictionaries in different domains, such as language modeling [22], machine translation [19] and computer vision [21].”