Semantic Jitter: Dense Supervision for Visual Comparisons via Synthetic Images

By Aron Yu, Kristen Grauman. 2016

163 citations recorded by the source corpus.

Distinguishing subtle differences in attributes is valuable, yet learning to make visual comparisons remains nontrivial. Not only is the number of possible comparisons quadratic in the number of training images, but also access to images adequately spanning the space of fine-grained visual differences is limited. We propose to overcome the sparsity of supervision problem via synthetically generated images. Building on a state-of-the-art image generation engine, we sample pairs of training images exhibiting slight modifications of individual attributes. Augmenting real training image pairs with these examples, we then train attribute ranking models to predict the relative strength of an attribute in novel pairs of real images. Our results on datasets of faces and fashion images show the great promise of bootstrapping imperfect image generators to counteract sample sparsity for learning to rank.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

GAN Inversion: A Survey (2021, background): “Besides, some GAN inversion studies also use other datasets [44], [45], [46], [47], [48], [49] in their experiments, such as DeepFashion [50], [51], [52], AnimeFaces [33], and StreetScapes [53].”
Interpreting the Latent Space of GANs via Correlation Analysis for Controllable Concept Manipulation (2020, methodology): “In this paper, we conducted designed experiments on two datasets: Fashion-MNIST[23] and UT Zappos50K[24].”
Identifying Ambiguous Similarity Conditions via Semantic Matching (2022, background): “UT-Zappos-50k Shoes contains 50,025 images of shoes with four similarity conditions collected online [45, 46].”
Task-Driven Modular Networks for Zero-Shot Compositional Learning (2019, background): “The second dataset is UT-Zappos50k [37, 36] which has”
Contextualizing Multiple Tasks via Learning to Decompose (2021, methodology): “We test L EAD N ET on UT Zappos50K [61, 23], where one image could be measured from six different attributes, namely, closure, heel height, insole, material, subcategory, toe style.”
A Modulation Module for Multi-task Learning with Applications in Image Retrieval (2018, methodology): “Dataset We use UT-Zappos50K dataset [2,3] for the product retrieval experiment.”
A Broader Study of Cross-Domain Few-Shot Learning (2019, methodology): “w-shot) task. In [44], a few-shot class density estimation is performed with an auto-regressive model, combined with an attention mechanism, where examples are synthesized by a sequential process. In [6,51,67] label and attribute semantics are used as additional information for training an example s”
Baby steps towards few-shot learning with multiple semantics (2019, methodology): “ional examples are synthesized via extracting, encoding, and transferring to the novel category instances, of the intra-class relations between pairs of instances of reference categories. Notably, in [5; 39] label and attribute semantics are used as additional information for training an example syn”
Jointly Optimize Data Augmentation and Network Training: Adversarial Data Augmentation in Human Pose Estimation (2018, background): “levant to ours in three categories. Adversarial learning. Generative Adversarial Networks (GANs) [12, 45, 47] are designed as playing minimax games between generator and discriminator. Yu and Grauman [44] use GANs to synthesize image pairs to overcome the sparsity of supervision when learning to com”
Delta-encoder: an effective sample synthesis method for few-shot object recognition (2018, unknown): “attribute vector) per-instance for example synthesis [4, 51].”
Lifelong Generative Modelling Using Dynamic Expansion Graph Model (2021, background): “…Chen, and Hsu 2014), 3D-Chair (Aubry et al. 2014), Ommiglot (Lake, Salakhutdinov, and Tenenbaum 2015), ImageNet* (Krizhevsky, Sutskever, and Hinton 2012), Car (Yang et al. 2015), Zappos (Yu and Grauman 2017), CUB (Wah et al. 2010) (detailed dataset setting is provided in Appendix-L.1 from SM1).”