Heated-Up Softmax Embedding

By Xu Zhang, Felix X. Yu, Svebor Karaman, Shih-Fu Chang. 2018

48 citations recorded by the source corpus.

Metric learning aims at learning a distance which is consistent with the semantic meaning of the samples. The problem is generally solved by learning an embedding for each sample such that the embeddings of samples of the same category are compact while the embeddings of samples of different categories are spread-out in the feature space. We study the features extracted from the second last layer of a deep neural network based classifier trained with the cross entropy loss on top of the softmax layer. We show that training classifiers with different temperature values of softmax function leads to features with different levels of compactness. Leveraging these insights, we propose a "heating-up" strategy to train a classifier with increasing temperatures, leading the corresponding embeddings to achieve state-of-the-art performance on a variety of metric learning benchmarks.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

DER: Dynamically Expandable Representation for Class Incremental Learning (2021, methodology): “We train the classifier head only using the cross-entropy loss with a temperature δ in the Softmax [38].”
Circle Loss: A Unified Perspective of Pair Similarity Optimization (2020, background): “Re-scaling the cosine similarity under supervision is a common practice in modern classiﬁcation losses [21, 30, 29, 32, 39, 40].”
Dynamic Dual-Attentive Aggregation Learning for Visible-Infrared Person Re-Identification (2020, methodology): “To enhance the discriminability, an exponential function is added to magnify the relationship, which enlarges the part attention discrepancy [63].”
Deep Learning for Person Re-Identification: A Survey and Outlook (2020, background): “Generally, it is easy to train and automatically mine the hard samples during the training process, as demonstrated in [148].”
Learning Natural Language Generation with Truncated Reinforcement Learning (2022, methodology): “While temperature scaling (Bahdanau et al., 2015) is usually used at test time to control the smoothness of the language model distribution, temperature schedules during training of language models have been used in several settings (Jang et al., 2016; Zhang et al., 2018; Wang et al., 2020).”