Separating Self-Expression and Visual Content in Hashtag Supervision

By Andreas Veit, Maximilian Nickel, Serge J. Belongie, L. V. D. van der Maaten. 2017

29 citations recorded by the source corpus.

The variety, abundance, and structured nature of hashtags make them an interesting data source for training vision models. For instance, hashtags have the potential to significantly reduce the problem of manual supervision and annotation when learning vision models for a large number of concepts. However, a key challenge when learning from hashtags is that they are inherently subjective because they are provided by users as a form of self-expression. As a consequence, hashtags may have synonyms (different hashtags referring to the same visual content) and may be polysemous (the same hashtag referring to different visual content). These challenges limit the effectiveness of approaches that simply treat hashtags as image-label pairs. This paper presents an approach that extends upon modeling simple image-label pairs with a joint model of images, hashtags, and users. We demonstrate the efficacy of such approaches in image tagging and retrieval experiments, and show how the joint model can be used to perform user-conditional retrieval and tagging.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Exploring the Limits of Weakly Supervised Pretraining (2018, background): “Other work [40] also trained networks to predict hashtags on the Flickr dataset but, unlike our study, does not investigate transfer of the resulting networks to other tasks.”
ClusterFit: Improving Generalization of Visual Representations (2019, methodology): “…ConvNets on very large, weakly supervised images by deﬁning the proxy tasks using the associated meta-data [11, 20, 24, 31, 36, 38, 48, 51–53 , 56] has shown By contrast, our approach does not adhere to a particular target task and leverages all the data, since, they may contain complementary…”
Probabilistic Embeddings for Cross-Modal Retrieval (2021, background): “[49] propose a conditional embedding approach to solve the multiplicity of hashtags, but it does not rely on a joint embedding space, hence cannot be directly applied to cross-modal retrieval.”