MultiGrain: a unified image embedding for classes and instances

By Maxim Berman, Herv'e J'egou, Andrea Vedaldi, Iasonas Kokkinos, Matthijs Douze. 2019

127 citations recorded by the source corpus.

MultiGrain is a network architecture producing compact vector representations that are suited both for image classification and particular object retrieval. It builds on a standard classification trunk. The top of the network produces an embedding containing coarse and fine-grained information, so that images can be recognized based on the object class, particular object, or if they are distorted copies. Our joint training is simple: we minimize a cross-entropy loss for classification and a ranking loss that determines if two images are identical up to data augmentation, with no need for additional labels. A key component of MultiGrain is a pooling layer that takes advantage of high-resolution images with a network trained at a lower resolution. When fed to a linear classifier, the learned embeddings provide state-of-the-art classification accuracy. For instance, we obtain 79.4% top-1 accuracy with a ResNet-50 learned on Imagenet, which is a +1.8% absolute improvement over the AutoAugment method. When compared with the cosine similarity, the same embeddings perform on par with the state-of-the-art for image retrieval at moderate resolutions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Fixing the train-test resolution discrepancy (2019, background): “A recent approach [5] employs p-pooling instead of average pooling to adapt the network to test resolutions significantly higher than the training resolution.”
Early Convolutions Help Transformers See Better (2021, positive): “We compare this recipe to the one used for DeiT models in the Appendix, and observe that our setup provides substantially faster training convergence likely because we remove repeating augmentation [1, 20], which is known to slow training [1].”
Emerging Properties in Self-Supervised Vision Transformers (2021, methodology): “For reference, we also report the performance of the multigrain model [5], trained speciﬁcally for particular object retrieval.”
Grafit: Learning fine-grained image representations with coarse labels (2020, methodology): “proposed Multigrain [3], which simply adds to the classification objective a triplet loss that pulls together different data-augmentations of a same image.”
Meta-attention for ViT-backed Continual Learning (2022, methodology): “For data augmentation, extensive tricks are put to use, such as mixup [19], cutmix [18], Rand-Augment [2], repeated augmentation [1, 5], and random cropping.”
Going deeper with Image Transformers (2021, methodology): “The main parameters are as follows: we use a batch size of 1024 samples and train during 400 epochs with repeated augmentation [5, 29].”
LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference (2021, background): “In DeiT [3], the need for the large pre-training dataset is replaced with a student-teacher setup and stronger data augmentation and regularization, such as stochastic depth [27] or repeated augmentation [28, 29].”
Focal Self-attention for Local-Global Interactions in Vision Transformers (2021, background): “augmentation [4, 35] and exponential moving average (EMA) [49].”
DaViT: Dual Attention Vision Transformers (2022, methodology): “Following [37,35,58,75,15], we use the same set of data augmentation and regularization strategies used in [49] after excluding repeated augmentation [3,26] and exponential moving average (EMA) [41].”
Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error (2021, background): “, 2019], ‘repeated augmentation’ [Berman et al., 2019]).”
ResNet strikes back: An improved training procedure in timm (2021, methodology): “In addition to adapting the weight decay, Repeated-Augmentation [3, 18] (RA) and stochastic-Depth [22].”
Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samples (2021, background): “This idea of an adding a supervised cross-entropy loss to an unsupervised instance-based loss has also been exploited to learn representations suitable for both image classiﬁcation and instance recognition [17].”
Focal Attention for Long-Range Interactions in Vision Transformers (2021, methodology): “We use the same set of data augmentation and regularization strategies used in [55] after excluding random erasing [82], repeated augmentation [4, 34] and exponential moving average (EMA) [48].”
MetaFormer Baselines for Vision (2022, methodology): “We do not use repeated augmentation [2, 26] and LayerScale [69], but use ResScale [60] for the last two stages.”