Learning Visual N-Grams from Web Data

By Ang Li, Allan Jabri, Armand Joulin, L. V. D. van der Maaten. 2016

152 citations recorded by the source corpus.

Real-world image recognition systems need to recognize tens of thousands of classes that constitute a plethora of visual concepts. The traditional approach of annotating thousands of images per class for training is infeasible in such a scenario, prompting the use of webly supervised data. This paper explores the training of image-recognition systems on large numbers of images and associated user comments, without using manually labeled images. In particular, we develop visual n-gram models that can predict arbitrary phrases that are relevant to the content of an image. Our visual n-gram models are feed-forward convolutional networks trained using new loss functions that are inspired by n-gram models commonly used in language modeling. We demonstrate the merits of our models in phrase prediction, phrase-based image retrieval, relating images and captions, and zero-shot transfer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “For example, Li et al. (2017) reach only 11.5% accuracy on ImageNet in a zero-shot setting.”
Exploring the Limits of Weakly Supervised Pretraining (2018, background): “Other prior studies [16,38] trained convolutional networks to predict words or n-grams in comments on a collection of 100 million Flickr photos and corresponding comments [39].”
Big Transfer (BiT): General Visual Representation Learning (2019, methodology): “rvised Learning of Representations A number of prior works use large supervised datasets for pre-training visual representaLarge Scale Learning of General Visual Representations for Transfer 15 tions [21,40,27,31]. In [21,27] the authors use a dataset containing 100M Flickr images [44]. This dataset”
Learning Visual Representations with Caption Annotations (2020, background): “Ground-truth concept vectors can be easily obtained by considering the most frequent bi-grams [32] or even n-grams [37] in captions.”
Learning Video Representations from Textual Web Supervision (2020, methodology): “Another approach is to use text captions, and label each image with key words present in the caption [50, 30, 39, 48].”
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, methodology): “Webly Supervised Learning: The method of manually annotating images for training does not scale well to the open-world setting as it is impracticable to collect and annotate images for all relevant concepts [33, 39].”
SLIP: Self-supervision meets Language-Image Pre-training (2021, background): “Initial attempts at leveraging the YFCC dataset for representation learning included predicting the bag-of-words representation [33] or n-gram occurrence [39] from images.”
Combined Scaling for Zero-shot Transfer Learning (2021, methodology): “Using images with accompanying text is related to the literature on image-captioning models, such as (Vinyals et al., 2015; Karpathy and Fei-Fei, 2015; Xu et al., 2015; Joulin et al., 2016; Li et al., 2017; Sariyildiz et al., 2020; Zhang et al., 2020; Desai and Johnson, 2021).”
Learning to Prompt for Vision-Language Models (2021, background): “Matching images and text features has been formulated as metric learning (Frome et al., 2013), multi-label classiﬁcation (Joulin et al., 2016; Gomez et al., 2017), n-gram language learning (Li et al., 2017), and the recently proposed captioning (Desai and Johnson, 2021).”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, background): “Image-text retrieval results on Flickr30K and MSCOCO datasets (zero-shot and ﬁne-tuned).”
LAION-5B: An open large-scale dataset for training next generation image-text models (2022, methodology): “For instance, OpenAI’s CLIP models [58] achieved large gains in zero-shot classification on ImageNet [65], improving from the prior top-1 accuracy of 11.5% [41] to 76.2%.”
ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models (2022, background): “The research topic has witnessed two types of distinctive notions of zero-shot: the traditional class-level zero-shot that usually refers to the study of generalizing to unseen object categories [30], and the recently popular task-level zero-shot that refers to the study of generalizing to unseen da”
Large-Scale Weakly-Supervised Pre-Training for Video Action Recognition (2019, background): “Dataset sources: Many prior approaches use Internet as a natural source of content [6, 12, 13, 18, 24, 25, 34, 37, 44, 46, 51, 58, 67] and the associated search queries, hashtags, or user-comments as labels to construct datasets.”
Conditional Prompt Learning for Vision-Language Models (2022, background): “In terms of cross-modality alignment, common approaches include metric learning [12], multi-label classiﬁcation [16, 26], and n-gram language learning [31].”
Large Scale Learning of General Visual Representations for Transfer (2019, methodology): “In [21,27] the authors use a dataset containing 100M Flickr images [44].”
A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-Language Model (2021, unknown): “For example, Visual N-Grams [29] and CLIP [41] explored the use of web-crawled data for image classification and [19] introduced the vision-language pre-training model for the open-vocabulary object detection and showed it could significantly improve the long-tile object detection [22].”
ClusterFit: Improving Generalization of Visual Representations (2019, methodology): “…Learning: Training ConvNets on very large, weakly supervised images by deﬁning the proxy tasks using the associated meta-data [11, 20, 24, 31, 36, 38, 48, 51–53 , 56] has shown By contrast, our approach does not adhere to a particular target task and leverages all the data, since, they may…”
Multimodal Contrastive Training for Visual Representation Learning (2021, background): “Recently, several works [46, 35, 18, 19, 10, 49] propose pre-training approaches that use semantically dense captions to learn visual representations from scratch.”