Hard negative examples are hard, but useful

By Hong Xuan, Abby Stylianou, Xiaotong Liu, Xiaotong Liu, Robert Pless. 2020

157 citations recorded by the source corpus.

Triplet loss is an extremely common approach to distance metric learning. Representations of images from the same class are optimized to be mapped closer together in an embedding space than representations of images from different classes. Much work on triplet losses focuses on selecting the most useful triplets of images to consider, with strategies that select dissimilar examples from the same class or similar examples from different classes. The consensus of previous research is that optimizing with the \textit{hardest} negative examples leads to bad training behavior. That's a problem -- these hardest negatives are literally the cases where the distance metric fails to capture semantic similarity. In this paper, we characterize the space of triplets and derive why hard negatives make triplet loss training fail. We offer a simple fix to the loss function and show that, with this fix, optimizing with hard negative examples becomes feasible. This leads to more generalizable features, and image retrieval results that outperform state of the art for datasets with high intra-class variance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2021, neutral): “Best results correspond to FULL INSTRUCTION - NEG EXAMPLES, indicating all instruction elements are helpful except the NEGATIVE EXAMPLES, which remains a hurdle for NLP models as also discussed in previous work (Xuan et al., 2020; Lin et al., 2003).”
Hard Negative Mixing for Contrastive Learning (2020, background): “Hard negatives are critical for contrastive learning [1, 19, 23, 30, 37, 44, 48].”
Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (2024, unknown): “We train every base model using the MS-MARCO (Bajaj et al., 2016) dataset with mined hard negatives (Xuan et al., 2020) on 1.25M samples using sentence-transformers (Reimers and Gurevych, 2019).”